Opened 3 years ago

Closed 3 years ago

#14175 closed enhancement (implemented)

chutney verify: report client, bridge, and HS performance

Reported by: teor Owned by: teor
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Keywords: tor-perf SponsorR SponsorS tor-hs tor-client tor-bridge TorCoreTeam201507
Cc: nickm, dgoulet Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

There is a need to load-test Tor relays when bringing them up, and to performance-test tor in general, whether relaying or hidden services.

See:
Tor BSD Underperformance https://lists.torproject.org/pipermail/tor-dev/2015-January/008048.html
Simulate high Tor load https://lists.torproject.org/pipermail/tor-relays/2015-January/006203.html

It would be nice to have a command for this, that would execute the same tests, and then report performance as a series of MB/s figures for clients->exits, bridge clients->exits, and clients->hidden services.

Nick, is this SponsorR, or SponsorS, or both?

Child Tickets

TicketTypeStatusOwnerSummary
#14067defectclosedteorchutney verify doesn't verify bridge clients or hidden services
#14173enhancementclosednickmchutney verify: verify in parallel, rather than one at a time (Tor load / performance testing)
#14174enhancementclosedteorchutney verify: make amount of data sent configurable (Tor load / performance testing)
#15860defectclosednickmchutney verify: make number of connections configurable
#15936defectclosedteorAdd additional HS networks to chutney

Change History (13)

comment:1 Changed 3 years ago by teor

Keywords: SponsorR SponsorS added

Tagged with appropriate sponsors per nickm in #14174

comment:2 Changed 3 years ago by teor

If we're going to test performance, we need to report:

  • time to contact server TCP SYN (if available)
  • time to first response first TCP ACK (if available)
  • time to connect second TCP ACK (if available)
  • ongoing latency
  • ongoing throughput

As connection time, latency, and throughput all impact perceived performance.

comment:3 Changed 3 years ago by teor

Alternately, we could leave this feature for Chutney 2.0, and just report performance each time we run chutney verify. This would work well with the configurable data volumes in #14174.

comment:4 Changed 3 years ago by teor

Owner: changed from nickm to teor
Status: newassigned
Summary: chutney perf: a new command that reports client, bridge, and HS performancechutney verify: report client, bridge, and HS performance

comment:5 Changed 3 years ago by teor

I have bundled up all these changes into an early draft in branch: bug14175-bandwidth
on: https://github.com/teor2345/chutney.git

It may need to be split into separate commits, but there are some dependencies I have to work out first.

comment:6 Changed 3 years ago by teor

Note: The bandwidth parameters can be changed by modifying constants in TorNet.py.
If we decide it's a good idea, I can do command-line arguments for these.

comment:7 Changed 3 years ago by teor

You'll likely need to change DATALEN in TorNet.py to 50MB or 100MB.
(I just hung python or tor on EC2 using 1GB.)

As a sanity check, the script doesn't calculate bandwidth until it takes > 1 second, and has sent > 5MB.

Expect to see something like this:

Verifying data transmission:
Connecting:
  Exit to 127.0.0.1:4747 via client localhost:9008
  Exit to 127.0.0.1:4747 via client localhost:9009
Transmitting Data:
........................................................................................................................
Single Stream Bandwidth: 2034.50 Mbps
Overall tor Bandwidth: 48828.06 Mbps
Transmission: Success

comment:8 Changed 3 years ago by teor

Status: assignedneeds_review

I've refactored the code, added environmental variables to configure the options, and split it into commits:

Branch: feature14175-chutney-performance-v2
Repository: https://github.com/teor2345/chutney.git

I've also added code to set the environmental variables from command-line arguments to tor/src/test/test-network.sh

Branch: feature14175-chutney-performance-v2
Repository: https://github.com/teor2345/tor.git

The available options are (env variable for chutney, argument for test-network.sh):

CHUTNEY_DATA_BYTES=n sends n bytes per test connection (10 KBytes)
--bytes n

CHUTNEY_CONNECTIONS=n makes n test connections per client (1)
--connections n

CHUTNEY_HS_MULTI_CLIENT=1 makes each client connect to each HS (0)
--hs-multi-client 1


When enough data is transmitted, chutney verify reports:

Single Stream Bandwidth: the speed of the slowest stream, end-to-end

Overall tor Bandwidth: the sum of the bandwidth across each tor instance
This approximates the CPU-bound tor performance on the current machine,
assuming everything is multithreaded and network performance is infinite.

Note: using --connections 7 or greater on a HS will trigger a verification failure due to #15937.

These branches cover this task and all child tasks.

comment:9 Changed 3 years ago by teor

I have tested these changes myself on Linux and OS X with the following commands:

src/test/test-network.sh --sleep 30 --bytes 102400000 --flavour bridges+ipv6+hs

Verifying data transmission:
Connecting:
  Exit to 127.0.0.1:4747 via client localhost:9010
  Exit to 127.0.0.1:4747 via client localhost:9011
  HS to jarz6vgn5xbbvnrk.onion:5858 (127.0.0.1:4747) via client localhost:9010
Transmitting Data:
.........................................................
Single Stream Bandwidth: 79.08 MBytes/s
Overall tor Bandwidth: 1265.23 MBytes/s
Transmission: Success

src/test/test-network.sh --sleep 30 --flavour bridges+ipv6+hs --connections 5

Verifying data transmission:
Connecting:
  Exit to 127.0.0.1:4747 via client localhost:9010
  Exit to 127.0.0.1:4747 via client localhost:9011
  HS to t4c3o37sgnc43flm.onion:5858 (127.0.0.1:4747) via client localhost:9010
Transmitting Data:
...............
Transmission: Success

src/test/test-network.sh --sleep 30 --flavour hs-025 --hs-multi-client 1

Verifying data transmission:
Connecting:
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9014
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9015
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9016
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9017
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9018
  HS to q6hmwsdh5z4kmtoj.onion:5858 (127.0.0.1:4747) via client localhost:9019
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9014
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9015
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9016
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9017
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9018
  HS to ubz5w5ncwbvvqk75.onion:5858 (127.0.0.1:4747) via client localhost:9019
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9014
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9015
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9016
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9017
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9018
  HS to kdgd6wb4w4gnvbrq.onion:5858 (127.0.0.1:4747) via client localhost:9019
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9014
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9015
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9016
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9017
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9018
  HS to gooa5v7yjkf6fbup.onion:5858 (127.0.0.1:4747) via client localhost:9019
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9014
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9015
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9016
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9017
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9018
  HS to ckn2xwnlouxasn52.onion:5858 (127.0.0.1:4747) via client localhost:9019
Transmitting Data:
..............................
Transmission: Success

comment:10 Changed 3 years ago by nickm

Sounds plausible, looks reasonable. Merging!

Please reparent and/or close subtickets as appropriate, then close this?

comment:11 Changed 3 years ago by teor

Resolution: fixed
Status: needs_reviewclosed

(Code runs on OS X and Linux, on at least both of my boxes, and Corey's from tor-dev.)

Child tickets all closed.

comment:12 Changed 3 years ago by teor

Resolution: fixed
Status: closedreopened

Hi nickm,

Looks like the tor branch for this didn't get merged.

Would you mind merging it?

The details are:

I've also added code to set the environmental variables from command-line arguments to tor/src/test/test-network.sh

Branch: feature14175-chutney-performance-v2
Repository:https://github.com/teor2345/tor.git

The available options are (env variable for chutney, argument for test-network.sh):

CHUTNEY_DATA_BYTES=n sends n bytes per test connection (10 KBytes)
--bytes n

CHUTNEY_CONNECTIONS=n makes n test connections per client (1)
--connections n

CHUTNEY_HS_MULTI_CLIENT=1 makes each client connect to each HS (0)
--hs-multi-client 1

comment:13 Changed 3 years ago by nickm

Keywords: TorCoreTeam201507 added
Resolution: implemented
Status: reopenedclosed

Merged, sorry for missing this!

Note: See TracTickets for help on using tickets.