prop289: add bidirectional data transfers to chutney

To test authenticated sendmes, we need to send data in both directions.

added actualpoints::2 component::core tor/chutney network-team-roadmap-2019-Q1Q2 owner::nickm points::5 priority::medium prop289 resolution::fixed severity::normal sponsor::19 status::closed type::enhancement labels

These chutney tickets are on the network team roadmap, or they are required for tickets that are on the network team roadmap.

Trac:
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added

Trac:
Sponsor: SponsorV to Sponsor19

Trac:
Owner: N/A to nickm
Actualpoints: N/A to 1
Status: new to accepted

See branch bidi with PR at https://github.com/torproject/chutney/pull/28

This was much easier than I'd expected, once I refactored Traffic.py to be a little more ergonomic.

Trac:
Status: accepted to needs_review

Travis is complaining; I suspect there are some per-version issues to solve.

Trac:
Status: needs_review to needs_revision

I've updated the branch, and the travis failure rate is lower, but still real. I don't know why; I've spent about a day searching for the problem in various ways.

For some reason, the issue seems to be that one of the Tor clients is sent a socks request, and never replies. I don't know why that would happen. It does not seem to happen with 0.4.0 or later quite so often as it does with 0.3.5.

I'm going to take a break from this and hope that Teor or I will have a big insight.

Trac:
Actualpoints: 1 to 2

Hm. I didn't see this before, but it shows up in my logs on the exit node:

May 13 09:45:20.498 [info] connection_handle_write_impl(): in-progress connect failed. Removing. (Connection refused)
May 13 09:45:20.498 [info] errno_to_stream_end_reason(): Didn't recognize errno 0 (Success); telling the client that we are ending a stream for 'misc' reason.
May 13 09:45:20.498 [info] connection_handle_write_impl(): in-progress connect failed. Removing. (Connection refused)
May 13 09:45:20.498 [info] errno_to_stream_end_reason(): Didn't recognize errno 0 (Success); telling the client that we are ending a stream for 'misc' reason.

If I am diagnosing this right, it might be because of our evdns code on 0.3.5.x and earlier not having a fix for #21900 (moved). I have NO IDEA why this is not affecting chutney without this patch though.

About a month ago, chutney CI was unstable. This failure pattern looks familiar: older releases and newer pythons.

Since then, I've fixed a bunch of bugs, and merged to master.

I closed and re-opened the pull request to get the latest merge, let's see how it goes.

I expect CI to fail on obsolete Tor versions due to #30826 (moved).

Seems to work fine now, except for 0.3.4, which is an expected fail.

Let's re-run CI after the #30876 (moved) merge, and then merge this PR?

We'll need to close and re-open the pull request so GitHub re-does the merge to master.

Trac:
Status: needs_revision to merge_ready

Merged to master as 49dceb9.

Trac:
Status: merge_ready to closed
Resolution: N/A to fixed

closed

changed time estimate to 40h

added 16h of time spent

mentioned in issue #30553 (moved)

moved to tpo/core/chutney#29263 (closed)

mentioned in issue tpo/core/chutney#30553