#22701 closed enhancement (wontfix)

Missing makefile dependencies make parallel builds fail

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.3.4.x-final
Component: Core Tor/Tor Version: Tor: 0.3.0.8
Severity: Normal Keywords: 034-triage-20180328, 034-removed-20180328
Cc: Actual Points:
Parent ID: Points: 0.5
Reviewer: Sponsor:

Description

I think we're missing some dependencies in the src/ext/ed25519/ref10 makefile on Tor 0.3.0.8. I bet it still exists in 0.3.1.

I repeated the same make command, and it worked fine, because the files were there.

I'm hoping someone else knows how to fix it:

$ make -j8 all check
make  all-am
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_1.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_add.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_cmov.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_copy.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_frombytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_invert.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_isnegative.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_isnonzero.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_mul.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_neg.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_pow22523.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sq.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sq2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_tobytes.o
make[1]: Entering directory '/home/privcount/tor-privcount'
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_mul.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_add.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_double_scalarmult.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_frombytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_madd.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_msub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p1p1_to_p2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p1p1_to_p3.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_pow22523.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_double_scalarmult.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p1p1_to_p2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_to_cached.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_to_p2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_precomp_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_scalarmult_base.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_sub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_to_cached.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_scalarmult_base.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_sub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-open.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-open.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_muladd.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_muladd.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_reduce.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sign.o
mv: cannot stat 'src/ext/ed25519/ref10/.deps/src_ext_ed25519_ref10_libed25519_ref10_a-open.Tpo': No such file or directory
Makefile:5665: recipe for target 'src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-open.o' failed
make: *** [src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-open.o] Error 1
make: *** Waiting for unfinished jobs....
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_reduce.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sign.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keyconv.o
mv: mv: cannot stat 'src/ext/ed25519/ref10/.deps/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.Tpo': No such file or directorycannot stat 'src/ext/ed25519/ref10/.deps/src_ext_ed25519_ref10_libed25519_ref10_a-sign.Tpo'
: No such file or directory
Makefile:5651: recipe for target 'src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.o' failed
make: *** [src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.o] Error 1
Makefile:5707: recipe for target 'src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sign.o' failed
make: *** [src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sign.o] Error 1
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-blinding.o
  CC       src/ext/ed25519/donna/src_ext_ed25519_donna_libed25519_donna_a-ed25519_tor.o
  CC       src/ext/keccak-tiny/src_ext_keccak_tiny_libkeccak_tiny_a-keccak-tiny-unrolled.o
...

Child Tickets

Change History (15)

comment:1 Changed 22 months ago by nickm

I do parallel builds all the time, and I'm afraid I've never seen this happen. Is it reproducible? What automake version are you on?

comment:2 in reply to:  1 Changed 22 months ago by teor

Replying to nickm:

I do parallel builds all the time, and I'm afraid I've never seen this happen. Is it reproducible?

I've only seen it once.

What automake version are you on?

1.15.1

comment:3 Changed 22 months ago by nickm

Milestone: Tor: 0.3.1.x-finalTor: 0.3.2.x-final
Status: newneeds_information

I tried to reproduce this over and over, with no luck. I routinely build with "make -kj", so I'll probably run into it sooner or later. But for now, I have no idea how this could happen.

comment:4 Changed 19 months ago by nickm

Resolution: worksforme
Status: needs_informationclosed

comment:5 Changed 16 months ago by teor

Resolution: worksforme
Status: closedreopened

I can reproduce this using make clean all:

$ make clean all
...
test -z "src/ext/ed25519/ref10/libed25519_ref10.a src/ext/ed25519/donna/libed25519_donna.a src/ext/keccak-tiny/libkeccak-tiny.a src/trunnel/libor-trunnel.a src/trunnel/libor-trunnel-testing.a src/common/libor.a src/common/libor-ctime.a src/common/libor-crypto.a src/common/libor-event.a src/common/libor-testing.a src/common/libor-ctime-testing.a src/common/libor-crypto-testing.a src/common/libor-event-testing.a  src/common/libcurve25519_donna.a src/or/libtor.a src/or/libtor-testing.a " || rm -f src/ext/ed25519/ref10/libed25519_ref10.a src/ext/ed25519/donna/libed25519_donna.a src/ext/keccak-tiny/libkeccak-tiny.a src/trunnel/libor-trunnel.a src/trunnel/libor-trunnel-testing.a src/common/libor.a src/common/libor-ctime.a src/common/libor-crypto.a src/common/libor-event.a src/common/libor-testing.a src/common/libor-ctime-testing.a src/common/libor-crypto-testing.a src/common/libor-event-testing.a  src/common/libcurve25519_donna.a src/or/libtor.a src/or/libtor-testing.a
...
rm -f src/ext/ed25519/ref10/*.o
...
make[1]: Entering directory '/home/privcount/tor-privcount'
...
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_1.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_add.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_cmov.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_copy.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_frombytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_invert.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_isnegative.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_isnonzero.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_mul.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_neg.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_pow22523.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sq.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_sq2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_add.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_double_scalarmult.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_frombytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_madd.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_msub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p1p1_to_p2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p1p1_to_p3.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p2_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_to_cached.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_dbl.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_to_p2.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_p3_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_precomp_0.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_scalarmult_base.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keypair.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_muladd.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_tobytes.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-open.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-ge_sub.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sc_reduce.o
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-sign.o
...
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-keyconv.o
...
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-blinding.o
...
  AR       src/ext/ed25519/ref10/libed25519_ref10.a
ar: src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_0.o: No such file or directory
Makefile:3159: recipe for target 'src/ext/ed25519/ref10/libed25519_ref10.a' failed
make[1]: *** [src/ext/ed25519/ref10/libed25519_ref10.a] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/privcount/tor-privcount'
Makefile:2682: recipe for target 'all' failed
make: *** [all] Error 2
$ make all
make  all-am
make[1]: Entering directory '/home/privcount/tor-privcount'
  CC       src/ext/ed25519/ref10/src_ext_ed25519_ref10_libed25519_ref10_a-fe_0.o
...
  AR       src/ext/ed25519/ref10/libed25519_ref10.a
...
  CCLD     src/or/tor
...
make[1]: Leaving directory '/home/privcount/tor-privcount'

comment:6 Changed 16 months ago by nickm

I've tried running that a bunch of times, but it doesn't reproduce for me. Will try again on osx.

comment:7 Changed 16 months ago by teor

Weird. It may depend on the number of cores available. Or it might be because I run builds in a VM. Or it might be a macOS thing. (Linux returns directory entries in arbitrary (usually date) order, macOS returns them in sorted order.

Could you try make -jN clean all, where N is the number of cores you have available?
(I have 8.)

comment:8 Changed 16 months ago by nickm

I've been trying this in a loop, and it hasn't failed yet. I'll keep it running another hour or so

comment:9 Changed 16 months ago by teor

$ make -v
GNU Make 3.81
...
This program built for i386-apple-darwin11.3.0
$ automake --version
automake (GNU automake) 1.15.1
...
$ autoconf --version
autoconf (GNU Autoconf) 2.69
...
Last edited 16 months ago by teor (previous) (diff)

comment:10 Changed 16 months ago by teor

I just had a similar error on a completely separate macOS machine, not a VM, different number of cores:

$ make clean src/test/test-slow
...
rm -f src/or/*.o
...
  CC       src/or/src_or_libtor_testing_a-git_revision.o
...
  AR       src/or/libtor-testing.a
ar: src/or/src_or_libtor_testing_a-addressmap.o: No such file or directory
ar: src/or/src_or_libtor_testing_a-bridges.o: No such file or directory
ar: src/or/src_or_libtor_testing_a-channel.o: No such file or directory
(everything else is missing, too)
make: *** [src/or/libtor-testing.a] Error 1
make: *** Waiting for unfinished jobs....
Exit 2

comment:11 Changed 16 months ago by nickm

Milestone: Tor: 0.3.2.x-finalTor: 0.3.3.x-final

So, apparently this is a known problem with trying to run "clean" and "all" in parallel with one another: it's not generally easy to make sure that if "clean" is specified, it comes first.

Searching for "make parallel clean all" gives some ideas, but they tend to be stuff like this:

https://stackoverflow.com/questions/22645139/makefile-parallel-cleancompile-issue

The solutions tend to be kludgy ("define a clean_all target") or to require gmake, or both.

Not sure what to do here, but I don't think we can block 032 on it.

comment:12 Changed 15 months ago by nickm

Milestone: Tor: 0.3.3.x-finalTor: 0.3.4.x-final
Type: defectenhancement

Label a bunch of (arguable and definite) enhancements as enhancements for 0.3.4.

comment:13 Changed 13 months ago by nickm

Keywords: 034-triage-20180328 added

comment:14 Changed 13 months ago by nickm

Keywords: 034-removed-20180328 added

Per our triage process, these tickets are pending removal from 0.3.4.

comment:15 Changed 13 months ago by nickm

Resolution: wontfix
Status: reopenedclosed

Calling this wontfix -- parallel clean and all do not seem to work in automake.

Note: See TracTickets for help on using tickets.