Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#16953 closed enhancement (fixed)

Add make test-network-all to Tor's test suite

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.2.7.x-final
Component: Core Tor/Tor Version: Tor:
Severity: Keywords: CoreTorTeam201509 SponsorS testing Post027Freeze
Cc: Actual Points:
Parent ID: #16949 Points:
Reviewer: Sponsor:


make test-network-all should run comprehensive tests on:

  • clients, bridge clients, bridge clients over IPv6 (if configured),
  • exits,
  • hidden services, and
  • old tor version vs compiled tor version (if multiple versions present).

It should report results PASS / SKIP / FAIL like the existing make check (?) output.

Child Tickets

#16952defectclosednickmRetry configure/verify on transient chutney failures
#17012enhancementclosednickmPut BridgeClient first in bridges+hs and bridges+ipv6+hs

Change History (18)

comment:1 Changed 4 years ago by teor

I want to run the following chutney flavours / networks:


  • basic-min: authorities, exit relays, exit client
  • bridges-min: + bridge authority, bridge, bridge-exit client
  • hs-min: + non-exit relays, hidden service, hs client
  • bridges+hs: + bridge-hs client (with HS_MULTI_CLIENT and #17012), larger network (12)

IPv6: only if an IPv6 ping of localhost works

  • bridges+ipv6-min: + ipv6 bridge, ipv6 bridge client
  • ipv6-exit-min: + ipv6 exit, ipv6 exit client
  • we could also test bridge-ipv6 to hs or bridge ipv4/ipv6 to ipv4/ipv6 exit, but is that really necessary? (Split to #17013 to do with #16954)

Note that we'll need to fix #17011 for this to test actual IPv6 connectivity, and IPv6 DNS will be required for IPv6 exits until #16971 is fixed. But the chutney changes can happen after this branch is merged into tor.

Mixed: only if a stable version of tor exists (try harder to find one #17015)

  • mixed: + mixed authorities, mixed exit relays, mixed clients
  • think about this more during #16954

comment:2 Changed 4 years ago by teor

Status: newneeds_review

See my branch make-test-network-all on

Works on OS X (including IPv6 and mixed) and Linux (mixed failed as it was too old, no IPv6 due to AWS) with occasional failures due to --sleep being too short in bridges+hs. (These are resolved by #16952.)

This branch merges and works well with retry-test-network-on-failure from #16952.

comment:3 Changed 4 years ago by teor

Does the exit status of make test-network-all need to change if any of the tests fail?
(It doesn't change at the moment.)

comment:4 Changed 4 years ago by nickm

It probably should, so you can do it in scripts. Otherwise LGTM!

comment:5 Changed 4 years ago by teor

Updated the make-test-network-all branch to exit 1 if any test report (.trs) file contains "FAIL".

It's best if this branch is merged with retry-test-network-on-failure from #16952.

Since this script runs 7 test networks, it could run for a maximum of around 20 minutes.
Typical runtimes are around 20-30 seconds per script, or 3 minutes total for the 7 test networks. (Failures take longer due to retries.)

comment:6 Changed 4 years ago by nickm

Looks good to me. Holding off on question about #16952.

comment:7 Changed 4 years ago by teor

Pushed make-test-network-all with the default boostrap sleep time increased to 30 seconds.
This is ok to merge, we'll be smarter about chutney bootstrap once #16950 is completed.

comment:8 Changed 4 years ago by nickm

Resolution: implemented
Status: needs_reviewclosed

Great; merged that and added a changes file!

comment:9 Changed 4 years ago by teor

Oops, did the changes file not come through on that branch?
I thought I had included it as chages/make-test-network-all in 5feae32f46b3f0cc3da93ece80163c1a994c83b9

comment:10 Changed 4 years ago by nickm_mobile

Hmm. Could you investigate if i merged the right branch? I may have done something wrong

comment:11 Changed 4 years ago by teor

Oops, it looks like I left the changes file out. Yours is more succinct than mine, otherwise they're very similar.

comment:12 Changed 4 years ago by teor

Resolution: implemented
Status: closedreopened

The default bootstrap time of 30 seconds doesn't work well for i386 on OS X. (The combination is quite slow).

What do you think about increasing it to 35 seconds?

comment:13 Changed 4 years ago by nickm

No objection from me, though do they even still support 32-bit OSX?

comment:14 Changed 4 years ago by teor

Nope, all OS X instances are 64-bit and have been for several years now. But 32-bit programs still run unmodified on 64-bit OS X.

I can't see the point of anyone using a 32-bit tor on OS X, except to find obscure bugs.

However, 32-bit OS X performance gives an idea of speed on other 32-bit systems, or other slow/old systems.

comment:15 Changed 4 years ago by nickm

Resolution: fixed
Status: reopenedclosed

6f35fd0 changes a 30 to a 35. I hope it was the right 30! Please reopen if not.

comment:16 Changed 4 years ago by teor

Resolution: fixed
Status: closedreopened

I can't see this change in master, or see any likely branch.

I've made the change to the right line in a fixup on my make-test-network-all.

comment:17 Changed 4 years ago by nickm

Milestone: Tor: 0.2.7.x-final
Resolution: fixed
Status: reopenedclosed

Thanks for checking; merged that patch!

I must have been more distracted than usual yesterday. :(

comment:18 Changed 4 years ago by teor

You're not the only one...

Note: See TracTickets for help on using tickets.