Core file left on travis hardened rust builld

changed milestone to %Tor: unspecified

added 032-unreached-backport 033-unreached-backport 035-can ci component::core tor/tor milestone::Tor: unspecified priority::very high regression severity::normal status::needs-information tor-build travis type::defect labels

See #26788 (moved) as well. Duplicate?

Transferring keywords from duplicate ticket #26788 (moved).

I tried for a few minutes, but I don't know how to find the crash that creates the core file in either build: https://travis-ci.org/torproject/tor/jobs/403115782 https://travis-ci.org/torproject/tor/jobs/403730172

Trac:
Keywords: N/A deleted, tor-build, 035-must, regression added

What if this is coming from intentional crash in test-bt-cl?

If so we could fix it with "ulimit -c 0" in test_bt.sh.

I've made a PR at https://github.com/torproject/tor/pull/233 to test that theory. I have no idea if it will work.

The patch is simple enough, and it passes all our CI (except for a Windows failure due to #26076 (moved)).

Let's merge it as a precaution, and watch travis for a few days?

I have marked it for backport back to 0.2.9, because we backported distcheck CI back to 0.2.9.

Trac:
Status: new to merge_ready
Keywords: N/A deleted, 034-backport, 029-backport, 032-backport, 033-backport added

I'm not sure whether this will actually work -- I've merged it to master, but let's wait to see whether it fixes the bug before we backport.

Trac:
Status: merge_ready to needs_information

For what it's worth don't expect that the patch above will actually fix the bug for us: I think that if the backtrace test were the part dumping core, it would do it consistently, not just sometimes.

Another possibility here would be to add "core" to CLEANFILES, though I'm not sure whether it's a good idea.

Replying to nickm:

For what it's worth don't expect that the patch above will actually fix the bug for us: I think that if the backtrace test were the part dumping core, it would do it consistently, not just sometimes.

Another possibility here would be to add "core" to CLEANFILES, though I'm not sure whether it's a good idea.

It might be intermittently losing a race condition for cleaning up the core file.

It also seems that this failure turns up on maint-0.3.4 on Travis with DISTCHECK=yes and RUST_OPTIONS="--enable-rust --enable-cargo-online-mode". Maybe it's something having to do with the build.rs stuff.

Replying to catalyst:

It also seems that this failure turns up on maint-0.3.4 on Travis with DISTCHECK=yes and RUST_OPTIONS="--enable-rust --enable-cargo-online-mode". Maybe it's something having to do with the build.rs stuff. Doing an experiment on Travis to run the file ./core equivalent to see if that tells us anything useful.

The core file (at least one of them) indeed seems to be from test-bt-cl.

tor-0.3.4.4-rc-dev/_build/core: ELF 64-bit LSB  core file x86-64, version 1 (SYSV), SVR4-style, from './src/test/test-bt-cl crash'

I don't know why it happens mostly on the Rust-enabled distcheck builds and mostly (always?) not on the non-Rust distcheck builds.

We should backport to any release where we run distcheck in Travis, even though I've only seen 0.3.4 and master fail.

I've cherry-picked the fix to 0.2.9, added a changes file, and merged it forward.

Since this was merged to 0.2.9 and later, let's close the ticket, and re-open if we see the issue in CI again.

Trac:
Status: needs_information to closed
Resolution: N/A to fixed

This issue is not fixed in master: https://travis-ci.org/torproject/tor/jobs/435138514#L5514

Trac:
Status: closed to reopened
Milestone: Tor: 0.3.4.x-final to Tor: 0.3.5.x-final
Resolution: fixed to N/A

Trac:
Priority: Medium to Very High
Keywords: 034-backport deleted, 034-backport ci added

I wonder if the Rust builds somehow reactivate core dumps? Maybe an explicit setrlimit() in its runtime libraries? Or maybe it's a coredump from a different program?

I wonder if we should just do a top-level "ulimit -c 0" before we run "make distcheck". Not elegant, but it would make the issue go away.

Replying to nickm:

I wonder if we should just do a top-level "ulimit -c 0" before we run "make distcheck". Not elegant, but it would make the issue go away. I'm nervous about blanket ignoring of unexplained coredumps. That might hide more serious issues later on.

Trac:
Status: reopened to assigned
Owner: N/A to catalyst

Running make distcheck in a loop to try to figure out more details. So far in my test setup, MAKEFLAGS='-j4' gives some weird racy things involving cargo clean invocations colliding, but -j2 has been chugging for hours without failure.

Also repeatedly hitting "restart" on jobs in my travis-mystery-core branch in Travis, where I added some instrumentation.

I can't replicate it without commenting the ulimit -c 0 in test_bt.sh.

Maybe it's a different core file? Maybe we should merge a patch to run file on the core file?

Maybe we should merge a patch to run file on the core file?

This sounds great. We could even make it travis-only if we want.

Replying to nickm:

Maybe we should merge a patch to run file on the core file?

This sounds great. We could even make it travis-only if we want. Doing this in child ticket #28024 (moved).

Another mystery is why this happens more often in the rust build than the non-rust build. (I can't seem to get it to happen in the non-rust build.)

Trac:
Owner: catalyst to N/A

We need to know where the mystery core file comes from. #28024 (moved) should help with that.

Trac:
Status: assigned to needs_information

Trac:
Keywords: 035-must, 034-backport ci deleted, 034-backport, ci, 035-can added

0.3.2 is end of life, so 032-backport is now 032-unreached-backport.

Trac:
Keywords: 032-backport deleted, 032-unreached-backport added

This ticket doesn't have any code.

Trac:
Milestone: Tor: 0.3.5.x-final to Tor: unspecified

These open, non-merge_ready tickets can not get backported to 0.3.3, because 0.3.3 is now unsupported.

Trac:
Keywords: 033-backport deleted, N/A added

Hmm, I guess they should still get 033-backport-unreached

Trac:
Keywords: N/A deleted, 033-backport-unreached added

Removing 034-backport from all open tickets: 034 has reached EOL.

Trac:
Keywords: 034-backport deleted, N/A added

Fix 033-unreached-backport spelling.

Trac:
Keywords: 033-backport-unreached deleted, 033-unreached-backport added

0.2.9 is no longer maintained as of 1 January 2020. Therefore, we won't be backporting anything to it.

(None of these tickets are merge_ready, so we can just delete the backport tag.)

Trac:
Keywords: 029-backport deleted, N/A added

mentioned in issue #26788 (moved)

moved to tpo/core/tor#26787 (closed)

Core file left on travis hardened rust builld

Child items ...

Activity