Opened 9 months ago

Closed 7 weeks ago

#21272 closed enhancement (fixed)

Onionperf deployment

Reported by: hiro Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics Version:
Severity: Normal Keywords:
Cc: karsten, iwakeh, teor, robgjansen, irl@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Hello,
I have finished deploying a test of OnionPerf on OTF Cloud.
It can be accessed at: https://199.119.112.144/ (certificate is self signed atm)

Now that we can look at the generated files, we can start thinking and planning about how we want to consume them.
I think this is a good moment to decide if we want to include an option to generate a different format within OnionPerf, or we want to write a script to parse the files in a format we can use.

Child Tickets

Attachments (14)

op-tp-comparison.png (160.7 KB) - added by karsten 7 months ago.
check_allowed_exits.py (1.3 KB) - added by robgjansen 7 months ago.
use stem to check percent of exit bandwidth allows exit to certain ips and ports
op-nl-51200.tpf (126.0 KB) - added by hiro 7 months ago.
op-nl-1048576.tpf (21.2 KB) - added by hiro 7 months ago.
op-nl-5242880.tpf (10.6 KB) - added by hiro 7 months ago.
op-nl-51200.2.tpf (188.2 KB) - added by hiro 7 months ago.
op-nl-1048576.2.tpf (33.3 KB) - added by hiro 7 months ago.
op-nl-5242880.2.tpf (14.6 KB) - added by hiro 7 months ago.
onionperf.analysis.json.xz (11.8 KB) - added by hiro 7 months ago.
onionperf.analysis.json.2.xz (17.4 KB) - added by hiro 6 months ago.
op-nl-51200.3.tpf (82.2 KB) - added by hiro 6 months ago.
op-nl-1048576.3.tpf (14.9 KB) - added by hiro 6 months ago.
op-nl-5242880.3.tpf (6.5 KB) - added by hiro 6 months ago.
op-tp-comparison-2017-04-12.png (151.3 KB) - added by karsten 6 months ago.

Download all attachments as: .zip

Change History (111)

comment:1 Changed 8 months ago by iwakeh

Type: defectenhancement

Adding a link from our mail conversation to the first analysis:
https://github.com/hiromipaw/onionperf-notebook

Changed ticket type to 'enhancement'.

comment:2 Changed 8 months ago by hiro

Hi,

So I have updated the wiki for onionperf:
https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure/onionperf
Now we have onionperf-one which is in DC, United States, and onionperf-eu which is in Amsterdam.

I am finishing installing also the tpo instance as I will need some libraries installed there first. I will let you know once this is completed by updating this ticket.

My notebook can be accessed on:
https://github.com/hiromipaw/onionperf-notebook
And the results can be displayed from https://github.com/hiromipaw/onionperf-notebook/blob/master/transfer_times.html which is basically the quick-n-dirty test I did to make sure measurements made some sense.

I could try to make onionperf measurements available for collector, or maybe you prefer me to do something else instead. I haven't honestly touched Java in a very long time ;)

Last edited 8 months ago by hiro (previous) (diff)

comment:3 Changed 8 months ago by karsten

hiro, I'm having a difficult time reading your results. Can you put them on GitHub Pages or export some images and attach them here? And can you add explanations what your results show and why you consider them close enough to Torperf data that we can switch over? That would be awesome!

Regarding the CollecTor integration, I started hacking on that yesterday evening. No need for you to touch Java just for this. :)

comment:4 Changed 8 months ago by hiro

Hi Karsten, sure I will generate some annotated images.

comment:5 Changed 8 months ago by hiro

I have started this board: https://github.com/hiromipaw/onionperf-notebook/blob/master/board.md
I will be collecting graphs for all the measurements provided by onionperf. At the moment I only have the 50KiB download time but I will be adding more measurements before our meeting tomorrow.

My idea up to this point has been that if I do not find something that it is completely unreasonable when comparing w/ Torperf I assume the measurements (and my analysis) can be considered as accurate.

Talk more tomorrow at the meeting.

comment:6 Changed 8 months ago by hiro

Hi,

I have discovered I was making an error in plotting the time differences. In fact I was just plotting the microseconds difference :(
Now I have fixed this and it looks more accurate: https://github.com/hiromipaw/onionperf-notebook/blob/master/board.md

comment:7 Changed 8 months ago by teor

Cc: teor added

comment:8 Changed 8 months ago by karsten

Looks good to me, thanks! I'd say let's go ahead and include these measurements on Metrics, and if we later find out that there are measurement issues, we can still take broken measurements out. But if nobody looks at the data, nobody will spot any issues.

So, seems like the missing piece is the CollecTor integration. I'll continue working on that.

comment:9 Changed 8 months ago by karsten

hiro, can you rename the sources to op-$cc with $cc being the country code where the instance is running in as discussed at the last team meeting?

And can you check why https://37.218.247.40/ is still empty?

And what's the URL for the third instance, or did you not set that up yet?

Thanks!

comment:10 Changed 8 months ago by karsten

Oh, and can you look into getting (Let's Encrypt) certificates, maybe using op-$cc.$something as domain names? Otherwise we'll first have to look into ways to tell CollecTor which (self-signed) certificate to expect on each OnionPerf instance, which might be possible but which is probably not trivial. Thanks!

comment:11 Changed 8 months ago by hiro

Hi Karsten,
I have renamed the op-us and op-nd ones already. Also renamed the files too. I will have left to rename the hostname fields in the already generated log. Doing that at the moment.
Regarding the logs I am checking why these are generated but not copied over. Maybe there is a mismatch in directories. I am checking that.
Regarding op-tpo I am working to have the webserver running.
Also installing let's encrypt right now :)

comment:12 Changed 8 months ago by hiro

Also do you think we could have these running on op-$cc.torproject.org ?

comment:13 Changed 8 months ago by hiro

Update: I have also replaced the REMOTEHOSTNAME in the tpf files.

comment:14 Changed 8 months ago by hiro

Let's encrypt is running. New domain names are:

  • op-us.onionperf.torproject.net
  • op-nl.onionperf.torproject.net

Working on tpo setup.

For OFT Cloud we have a new location available: Hong Kong. Are we interested in setting this up?

Regarding the tpf files not showing up in the directory. Logs are generated but files are not "packed" into tpf at midnight. I am trying to investigate why this is happening.

comment:15 Changed 8 months ago by karsten

Thanks for configuring domain names and Let's Encrypt! Downloading using Java works just fine now!

However, I just looked at one of the .tpf files and found a couple of onion addresses:

$ grep START op-us-1048576-2017-03-07.tpf | sed 's/.*ENDPOINTREMOTE=\([^ ]*\) .*/\1/' | sort | uniq -c
  19 199-119-112-144.i95.net:199.119.112.144:8080
  17 5rk7srcqqtesjxpo.onion:0.0.0.0:8080
   1 ggxgcnswjxwjcqoe.onion:0.0.0.0:8080
   7 ysfuzafp5bhhhhis.onion:0.0.0.0:8080

Any idea what's up with those? Could it be that this instance is configured to make more requests than it should? Can you maybe paste the configuration here?

comment:16 Changed 8 months ago by karsten

Status: newneeds_review

Please review my task-21272 branch with a commit that downloads .tpf files from OnionPerf instances. Still quite rough around the edges, but passed an initial test by successfully downloading .tpf files from https://op-us.onionperf.torproject.net/.

comment:17 Changed 8 months ago by iwakeh

From reading the git diffs, I have a few questions and suggestions:

Does this code reflect the transition period of processing both
torperf and onionperf files? Is that the reason for introducing
the empty property with the meaning of 'nothing to download'?

Surely tests will be added for the new functionality of Configuration's
getStringArray and getStringArrayArray methods?
I assume current code that uses these methods relies on receiving a non-empty
array or double array. It needs to be verified that we don't run
into trouble in the existing code using these methods.

All storing of files should be done by the persist-mechanism, i.e.,
a o.t.c.persist.TpfPersistence class needs to be introduced (this would
shorten TorperfDownloader quite a bit and prevent another
re-implementation of this functionality).
And, this is a prerequisite for also syncing these files with
CollecTor's sync-mechanism, which was left out because we knew
a Torperf replacement is coming.

It might be useful to put the Configuration's getUrlArray method
to work. The old Torperf code wasn't modernized, b/c we knew
it will be replaced. Now, it might be better to have
OnionPerfHosts as Url array property. This way the url-property
checking in the downloader class is not necessary anymore as Configuration
already provides it.
Why is the host-nick-name needed and could not be replaced by the hostname
itself? (For torperf it is used for further configuration.) Or, is this
one of the rough edges and note yet available?
If so, maybe have a different configuration approach instead of the hybrid-String-Url array?
It would be great to make the configuration simpler to read
and edit and keep allmost all property-checking code in the
Configuration class. For example, an url-array and a string-array-array
property, the latter for the configuration similar to torperf.
The only check left for the TorperfDownloader would be to match
length of these two array- properties, but there might be other
good approaches.

comment:18 in reply to:  17 Changed 8 months ago by karsten

Replying to iwakeh:

From reading the git diffs, I have a few questions and suggestions:

Wow, quick review there. Thanks!

Does this code reflect the transition period of processing both
torperf and onionperf files? Is that the reason for introducing
the empty property with the meaning of 'nothing to download'?

Huh, good point. What I had in mind that we could use this code to 0) run the same configuration as we do now, 1) start collecting OnionPerf files, and soon after 2) stop collecting Torperf files. But we could achieve the same by deploying this code when we do 1) and deploy another patch for 2). I can undo that change.

Surely tests will be added for the new functionality of Configuration's
getStringArray and getStringArrayArray methods?
I assume current code that uses these methods relies on receiving a non-empty
array or double array. It needs to be verified that we don't run
into trouble in the existing code using these methods.

Right. Let me undo this part of the change.

All storing of files should be done by the persist-mechanism, i.e.,
a o.t.c.persist.TpfPersistence class needs to be introduced (this would
shorten TorperfDownloader quite a bit and prevent another
re-implementation of this functionality).
And, this is a prerequisite for also syncing these files with
CollecTor's sync-mechanism, which was left out because we knew
a Torperf replacement is coming.

Hmm, is this something you might be able to do? I'd really want us to deploy this code before Amsterdam, so that we can discuss next steps there. Otherwise, how about we defer this part until after Amsterdam?

It might be useful to put the Configuration's getUrlArray method
to work. The old Torperf code wasn't modernized, b/c we knew
it will be replaced. Now, it might be better to have
OnionPerfHosts as Url array property. This way the url-property
checking in the downloader class is not necessary anymore as Configuration
already provides it.
Why is the host-nick-name needed and could not be replaced by the hostname
itself? (For torperf it is used for further configuration.) Or, is this
one of the rough edges and note yet available?
If so, maybe have a different configuration approach instead of the hybrid-String-Url array?
It would be great to make the configuration simpler to read
and edit and keep allmost all property-checking code in the
Configuration class. For example, an url-array and a string-array-array
property, the latter for the configuration similar to torperf.
The only check left for the TorperfDownloader would be to match
length of these two array- properties, but there might be other
good approaches.

Well, my idea was that we should use the configured source name to make sure that an OnionPerf host doesn't send us measurement results for other sources, either accidentally or on purpose. I'm not sure how we could use the hostname there if we want to stick to the current naming schema of .tpf files. But it could be that I overlooked something.

However, if we want to keep this, what we could do is add a new method Configuration.getStringUrlMap() that returns a (sorted) map of Strings to URLs. Again, would you want to write such a thing?

Thanks again!

comment:19 Changed 8 months ago by iwakeh

Only two short questions about naming scheme and host url:

Currently the beginning of the host url is reflected in the file name.
Why can't we just use the beginning of the host url, i.e., the op-$cc part
for checking the filenames?

Is the goal now to just download all available measurements regarding file size?

comment:20 in reply to:  19 ; Changed 8 months ago by karsten

Replying to iwakeh:

Only two short questions about naming scheme and host url:

Currently the beginning of the host url is reflected in the file name.
Why can't we just use the beginning of the host url, i.e., the op-$cc part
for checking the filenames?

Sure, we could do that. And we could extend that if we need to. Works for me.

Is the goal now to just download all available measurements regarding file size?

Hmm, I guess the main reason for specifying file sizes explicitly in TorperfFilesLines was that we didn't parse a directory listing, so we had to know which file sizes exist. But here we do know which file sizes are measured, so the only reason for specifying those would be to exclude sizes we don't recognize. Do we have to do that? I'd say as long as the file sizes stated in the .tpf file name matches the FILESIZE value contained in that file, we'll take what we get.

comment:21 in reply to:  20 Changed 8 months ago by iwakeh

Replying to karsten:

Replying to iwakeh:

Only two short questions about naming scheme and host url:

Currently the beginning of the host url is reflected in the file name.
Why can't we just use the beginning of the host url, i.e., the op-$cc part
for checking the filenames?

Sure, we could do that. And we could extend that if we need to. Works for me.

So, all paved for using getUrlArray, isn't it?

Is the goal now to just download all available measurements regarding file size?

Hmm, I guess the main reason for specifying file sizes explicitly in TorperfFilesLines was that we didn't parse a directory listing, so we had to know which file sizes exist. But here we do know which file sizes are measured, so the only reason for specifying those would be to exclude sizes we don't recognize. Do we have to do that? I'd say as long as the file sizes stated in the .tpf file name matches the FILESIZE value contained in that file, we'll take what we get.

I think it's fine, to reduce the configuration.
The verification of filename vs. FILESIZE value should be done by descriptor, if we feel the need to introduce it.

(more about your other comment:18 tomorrow)

comment:22 Changed 7 months ago by hiro

Hi Karsten,
I just wanted to point out that there is no configuration option in onion perf. The measurements are started by the onionperf process. Something we could do is investigating how to provide a small config. I could dig through the code and see what Rob thinks. What do you think?

comment:23 Changed 7 months ago by karsten

iwakeh, please find my updated task-21272 branch with some tweaks as discussed above.

hiro, I see. I'll take a closer look at the results and get back to you on this ticket.

comment:24 Changed 7 months ago by karsten

hiro, I just looked a bit more at the data and found a few differences that we can work with. For example, OnionPerf randomly picks a file size (with different weights, though) every five minutes whereas Torperf had three separate schedules for each file size, which is okay. Also, OnionPerf alternates between direct download and download via onion address, which is different from Torperf, but which is okay, too.

However, there's one difference that we should fix: the direct downloads go to port 8080, not to port 80. The issue is that the set of exits permitting port 8080 may be different from the set permitting port 80, so this might impact measurement results. As far as I can see there are two ways to do this: either run OnionPerf on port 80 (which may have security implications), or configure a firewall rule to forward port 80 to 8080. I assume we'll want to do the latter. In that case we'll have to use OnionPerf's options --tgen-listen-port and --tgen-connect-port to tell it where to listen (8080) and where to connect (80).

Would you want to try making that change?

comment:25 in reply to:  23 ; Changed 7 months ago by iwakeh

Replying to karsten and what's left from 18:

iwakeh, please find my updated task-21272 branch with some tweaks as discussed above.

Thanks, for these changes!

Couldn't downloadFromOnionPerfHost do some of the filename checking before
calling downloadAndParseOnionPerfTpfFile?

About the older code:
Could we just also make the change for #20514 now?
In addition, there are quite a few places where try-with resources
or some of Files' methods would prevent unclosed readers/writers.
Still this older code needs even more work, sigh. Maybe, after Amsterdam?

And, regarding the descriptor parsing don't the checks inside this loop belong into descriptor/metrics-lib itself?
(that might be a new ticket, though)

I can take a look at the Persistence topic at some point.

comment:26 in reply to:  25 ; Changed 7 months ago by karsten

Replying to iwakeh:

Replying to karsten and what's left from 18:

iwakeh, please find my updated task-21272 branch with some tweaks as discussed above.

Thanks, for these changes!

Couldn't downloadFromOnionPerfHost do some of the filename checking before
calling downloadAndParseOnionPerfTpfFile?

Well, that wouldn't change functionality but would be a simple refactoring, right? What's the goal there? Make methods more testable or easier to read or something else? In any case, would you want to suggest new methods, and I'll move around code? Or do you want to work on a patch?

About the older code:
Could we just also make the change for #20514 now?
In addition, there are quite a few places where try-with resources
or some of Files' methods would prevent unclosed readers/writers.
Still this older code needs even more work, sigh. Maybe, after Amsterdam?

No need to spend time on this. Let's just remove the Torperf code in a few weeks when we're certain that the OnionPerfs can take over.

And, regarding the descriptor parsing don't the checks inside this loop belong into descriptor/metrics-lib itself?
(that might be a new ticket, though)

Well, if we moved that code to metrics-lib, users wouldn't be able to read Torperf results from anything else than the originally named .tpf file. We usually avoid dependencies on file names if we can. This case is a bit different, because we're archiving .tpf files, and we should be certain that they contain what they say. I'd say that's specific to the CollecTor case though and cannot be generalized in metrics-lib.

Note that we could have picked a different approach by parsing descriptors from files and appending them to new files with file names taken from descriptor contents. The result might be the exact same output, or it could be a file with fewer descriptors or descriptors in a different order, etc. But I felt it's easier to verify .tpf file contents and either take them or leave them.

I can take a look at the Persistence topic at some point.

Sounds good! The last paragraph above might be relevant here. Hope this works with the persistence classes.

comment:27 in reply to:  26 ; Changed 7 months ago by iwakeh

Replying to karsten:

Replying to iwakeh:

Replying to karsten and what's left from 18:

iwakeh, please find my updated task-21272 branch with some tweaks as discussed above.

Thanks, for these changes!

Couldn't downloadFromOnionPerfHost do some of the filename checking before
calling downloadAndParseOnionPerfTpfFile?

Well, that wouldn't change functionality but would be a simple refactoring, right? What's the goal there? Make methods more testable or easier to read or something else? In any case, would you want to suggest new methods, and I'll move around code? Or do you want to work on a patch?

No, I intended to avoid the superfluous parsing of the URL for each file and avoid download when the filename doesn't make sense.
No refactoring.

About the older code:
Could we just also make the change for #20514 now?
In addition, there are quite a few places where try-with resources
or some of Files' methods would prevent unclosed readers/writers.
Still this older code needs even more work, sigh. Maybe, after Amsterdam?

No need to spend time on this. Let's just remove the Torperf code in a few weeks when we're certain that the OnionPerfs can take over.

That's fine, too.

And, regarding the descriptor parsing don't the checks inside this loop belong into descriptor/metrics-lib itself?
(that might be a new ticket, though)

Well, if we moved that code to metrics-lib, users wouldn't be able to read Torperf results from anything else than the originally named .tpf file. We usually avoid dependencies on file names if we can. This case is a bit different, because we're archiving .tpf files, and we should be certain that they contain what they say. I'd say that's specific to the CollecTor case though and cannot be generalized in metrics-lib.

There could be a boolean parameter for strict checking?
But, I didn't mean to hijack this ticket. I can just write that down on a list on my desk ;-)

Note that we could have picked a different approach by parsing descriptors from files and appending them to new files with file names taken from descriptor contents. The result might be the exact same output, or it could be a file with fewer descriptors or descriptors in a different order, etc. But I felt it's easier to verify .tpf file contents and either take them or leave them.

Hmm, ok, good to know.

I can take a look at the Persistence topic at some point.

Sounds good! The last paragraph above might be relevant here. Hope this works with the persistence classes.

Yes, I noticed this. We'll see.

comment:28 in reply to:  27 Changed 7 months ago by karsten

Replying to iwakeh:

Replying to karsten:

Replying to iwakeh:

Couldn't downloadFromOnionPerfHost do some of the filename checking before
calling downloadAndParseOnionPerfTpfFile?

Well, that wouldn't change functionality but would be a simple refactoring, right? What's the goal there? Make methods more testable or easier to read or something else? In any case, would you want to suggest new methods, and I'll move around code? Or do you want to work on a patch?

No, I intended to avoid the superfluous parsing of the URL for each file and avoid download when the filename doesn't make sense.
No refactoring.

Hmm, I'm a bit lost what you mean here. We only download if the filename makes sense. But this is discussion has become very theoretical. Want to provide a patch? :D

Well, if we moved that code to metrics-lib, users wouldn't be able to read Torperf results from anything else than the originally named .tpf file. We usually avoid dependencies on file names if we can. This case is a bit different, because we're archiving .tpf files, and we should be certain that they contain what they say. I'd say that's specific to the CollecTor case though and cannot be generalized in metrics-lib.

There could be a boolean parameter for strict checking?
But, I didn't mean to hijack this ticket. I can just write that down on a list on my desk ;-)

Sounds like a new ticket. But I'm not yet sold on the idea. If it's something that only CollecTor needs as provider of .tpf files and none of the consumers, then we shouldn't add make it part of metrics-lib. Of course, if there's a plausible use case for consumers, happy to reconsider. But, new (metrics-lib) ticket?

comment:29 Changed 7 months ago by iwakeh

Sure, I'll add a patch.

comment:30 in reply to:  24 Changed 7 months ago by hiro

Replying to karsten:

hiro, I just looked a bit more at the data and found a few differences that we can work with. For example, OnionPerf randomly picks a file size (with different weights, though) every five minutes whereas Torperf had three separate schedules for each file size, which is okay. Also, OnionPerf alternates between direct download and download via onion address, which is different from Torperf, but which is okay, too.

However, there's one difference that we should fix: the direct downloads go to port 8080, not to port 80. The issue is that the set of exits permitting port 8080 may be different from the set permitting port 80, so this might impact measurement results. As far as I can see there are two ways to do this: either run OnionPerf on port 80 (which may have security implications), or configure a firewall rule to forward port 80 to 8080. I assume we'll want to do the latter. In that case we'll have to use OnionPerf's options --tgen-listen-port and --tgen-connect-port to tell it where to listen (8080) and where to connect (80).

Would you want to try making that change?

Hi Karsten sure I can do that.
In the meantime I have fixed the logrotation issue on op-nl. Hopefully our tpo instance should also be ready. I will let you know if everything is working correctly on https://onionperf.torproject.org

More soon.

comment:31 Changed 7 months ago by hiro

Hi Karsten,
I have setup a proxy forward for 8080 to 80 for op-nl. I'll check if everything is ok and eventually will set it up also for op-us and op.tpo.
More soon.

comment:32 Changed 7 months ago by karsten

Thanks, hiro! Curious to see how that goes.

comment:33 Changed 7 months ago by hiro

Proxy forward up for op-nl and op-us

comment:34 Changed 7 months ago by hiro

Also measurements are up on tpo: https://onionperf.torproject.org/
Still some finishing touches, but looks good.

comment:35 in reply to:  29 Changed 7 months ago by iwakeh

Replying to iwakeh:

Sure, I'll add a patch.

This change is indeed shorter than its description.

(Just noticed, I accidentally, slipped in a filename length readability tweak, too.)

comment:36 Changed 7 months ago by karsten

iwakeh, patch looks good, pushed to my task-21272 branch. Thanks! Do you want to change anything else, or can I squash and merge?

hiro: now looking at new data...

comment:37 Changed 7 months ago by karsten

hiro, I looked at the data and have two questions:

  • It seems that the .tpf files produced by OnionPerf don't show whether connections were made to port 80 (and forwarded by firewall rules) or were made to port 8080. But we should only use data from measurements to port 80, or we'll compare two different measurements. Can you delete older measurements from before you changed that?
  • The OnionPerf source name (papillare) and DNS name (onionperf) of https://onionperf.torproject.org/papillare-5242880-2017-03-12.tpf don't match, which is something we implicitly require on CollecTor (which you couldn't know). Ideally, we'd stick with the naming scheme of the other instances and rename both to op-de. That's probably easiest to understand for users who will see these source names on Tor Metrics and would rightfully ask what a papillare is. Can you change the name of both OnionPerf instance and in the DNS entry?

Thanks!

comment:38 Changed 7 months ago by hiro

Hi Karsten,
Sure I will delete all the old measurements starting from the day before yesterday. Regarding papillare, the host and dns names where the bits that I was missing. Will have that changed. :)
-Silvia

comment:39 Changed 7 months ago by hiro

Update:
I have changed the onionperf source name on papillare as onionperf. I was chatting w the tpa team and they would prefer not having to change the dns to op-de. If we can stick with onionperf as service name and subdomain we are good, otherwise I'll bump them again.
Hopefully the ports are correct now.
Will finish deleting old files when we will check the new results tomorrow, to compare.

comment:40 in reply to:  36 Changed 7 months ago by iwakeh

Replying to karsten:

iwakeh, patch looks good, pushed to my task-21272 branch. Thanks! Do you want to change anything else, or can I squash and merge?

So far, this seems ready for merge. Tests pass and the download works.

The persistence topic from comment:18 and before, and also the later sync-addition has a separate ticket #21759.

Let's move any further things related to CollecTor java code to #21760, which already lists the open final move to onionperf and the tests etc.

The test run showed empty files
https://op-us.onionperf.torproject.net/op-us-5242880-2017-03-15.tpf (size 0)
on the us source.

comment:41 Changed 7 months ago by karsten

Thanks for looking again, and thanks for opening those new tickets. I just squashed and merged to master, together with a change log entry.

I noticed the empty file, too. It's yet unclear whether it will go away tonight. If not, we should handle that case less loudly.

I'll keep this ticket open until deployment is complete, with data being collected by CollecTor. But let's wait at least until tomorrow to see what the three op-xx instances produce.

comment:42 Changed 7 months ago by karsten

I took another look at the data and compared it to Torperf data. Please find the attached graph. I have two remaining questions before adding the new data to CollecTor:

  • It looks like op-nl is quite a bit faster than the other OnionPerf and Torperf instances. One measurement only took 80 milliseconds from making the request until receiving the last byte. Is this realistic? Or does OnionPerf take any shortcuts that the other instances don't take?
@type torperf 1.0
BUILDTIMES=0.0799999237061,0.0899999141693,0.109999895096 CIRC_ID=5284 CONNECT=1489499550.79 DATACOMPLETE=1489499550.87 DATAPERC0=1489499550.86 DATAPERC10=1489499550.86 DATAPERC100=1489499550.87 DATAPERC20=1489499550.86 DATAPERC30=1489499550.86 DATAPERC40=1489499550.86 DATAPERC50=1489499550.87 DATAPERC60=1489499550.87 DATAPERC70=1489499550.87 DATAPERC80=1489499550.87 DATAPERC90=1489499550.87 DATAREQUEST=1489499550.82 DATARESPONSE=1489499550.84 DIDTIMEOUT=0 ENDPOINTLOCAL=localhost:127.0.0.1:60802 ENDPOINTPROXY=localhost:127.0.0.1:29128 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:8080 FILESIZE=51200 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=1489499310.22 NEGOTIATE=1489499550.79 PATH=$A1EF24A8E85E440F215A5C296790636C62F43A2A,$150F2AD7D0FE7715ECAA642D4CD4BCBF95E8BD0D,$5CECC5C30ACC4B3DE462792323967087CC53D947 QUANTILE=0.8 READBYTES=51269 REQUEST=1489499550.80 RESPONSE=1489499550.82 SOCKET=1489499550.79 SOURCE=op-nl START=1489499550.79 TIMEOUT=1500 USED_AT=1489499550.87 USED_BY=10787 WRITEBYTES=54
  • The IP address of op-hk gets resolved to the Netherlands, though the measurements look very different from the op-nl host. Can we confirm that the host is really located in Hong Kong and that it's just the IP-to-country resolution that is behind?

If we have answers to these questions, I'd say let's make the data available.

Changed 7 months ago by karsten

Attachment: op-tp-comparison.png added

comment:43 in reply to:  42 Changed 7 months ago by robgjansen

Cc: rob.g.jansen@… added

Replying to karsten:

I took another look at the data and compared it to Torperf data. Please find the attached graph. I have two remaining questions before adding the new data to CollecTor:

  • It looks like op-nl is quite a bit faster than the other OnionPerf and Torperf instances. One measurement only took 80 milliseconds from making the request until receiving the last byte. Is this realistic? Or does OnionPerf take any shortcuts that the other instances don't take?

Looking at the relays chosen for that path, is looks like the full path is:
Netherlands(client)-->France(guard)-->Germany(middle)-->France(exit)-->Netherlands(server)

This means the latency on each of those links is about 10 milliseconds. That seems feasible to me, given how close those countries are and that the machines are probably well connected to the backbone. The server can send all ~100 cells at once and it will likely travel through Tor in one piece, so I don't think that would cause any or much delay.

Since most of the Tor relay bandwidth is in Europe, more of the circuits of a client in Europe would not leave the continent compared to clients in other regions. I'm not surprised if an OnionPerf client in Europe would trend faster than other countries on average. I wonder if either but not both of the two TorPerf nodes are in Europe, as we could potentially use the data they collect to test my hypothesis.

Also, there is more information about that circuit in particular and the download process in general in the json.xz files that are also dumped to the data directory. I added some notes to the elements there quite some time ago to help us understand what is available:

https://github.com/robgjansen/onionperf/blob/master/README_JSON.md

Finally, I have been running my OnionPerf instance since April 2016, and would like to contribute to Tor metrics. Is that possible? I think it would be great if you could import the data that I have been collecting if it's valid (someone should double check that it's valid first, and I'm happy to do so). If there is something about my setup that you would prefer that I change before accepting data from my instance, please let me know. Also, if you want all instances to be run by TPI, let me know that too so I can stop paying for the VPS.

http://onionperf.robgjansen.com:8081/

comment:44 Changed 7 months ago by karsten

Quick and only partial answer: I think it would be great to add your data to CollecTor and Tor Metrics. Here's one thing that we'd need to change though:

We currently require that the subdomain of the OnionPerf source ("onionperf" in your case) must match the OnionPerf source name ("phantomtrain" in your case). The reason is that we want to avoid a case where one instance produces data with the source name of another instance, which would mess up things quickly. You could probably work around this by adding yet another subdomain in front of your current domain. But let's first discuss which name you should pick.

Another requirement is that we pick source names in a way that Tor Metrics users are not confused by seeing them listed on https://metrics.torproject.org/torperf.html. It was not very smart to pick "torperf" as one name there, because all three are Torperf instances, and the names "moria" and "siv" don't have much meaning, either. That's why we picked "op-us" for "OnionPerf instance located in the U.S." etc. for the current instances. Obviously, that scheme does not scale either, with your instance being hosted in the U.S., too. But "onionperf" is just too generic. "phantomtrain" might work, because it's just a name. What do you think? Still time to do it right this time. :)

Thanks! (Will reply to the other parts later today.)

comment:45 in reply to:  42 Changed 7 months ago by hiro

Replying to karsten:

I took another look at the data and compared it to Torperf data. Please find the attached graph. I have two remaining questions before adding the new data to CollecTor:

  • It looks like op-nl is quite a bit faster than the other OnionPerf and Torperf instances. One measurement only took 80 milliseconds from making the request until receiving the last byte. Is this realistic? Or does OnionPerf take any shortcuts that the other instances don't take?
@type torperf 1.0
BUILDTIMES=0.0799999237061,0.0899999141693,0.109999895096 CIRC_ID=5284 CONNECT=1489499550.79 DATACOMPLETE=1489499550.87 DATAPERC0=1489499550.86 DATAPERC10=1489499550.86 DATAPERC100=1489499550.87 DATAPERC20=1489499550.86 DATAPERC30=1489499550.86 DATAPERC40=1489499550.86 DATAPERC50=1489499550.87 DATAPERC60=1489499550.87 DATAPERC70=1489499550.87 DATAPERC80=1489499550.87 DATAPERC90=1489499550.87 DATAREQUEST=1489499550.82 DATARESPONSE=1489499550.84 DIDTIMEOUT=0 ENDPOINTLOCAL=localhost:127.0.0.1:60802 ENDPOINTPROXY=localhost:127.0.0.1:29128 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:8080 FILESIZE=51200 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=1489499310.22 NEGOTIATE=1489499550.79 PATH=$A1EF24A8E85E440F215A5C296790636C62F43A2A,$150F2AD7D0FE7715ECAA642D4CD4BCBF95E8BD0D,$5CECC5C30ACC4B3DE462792323967087CC53D947 QUANTILE=0.8 READBYTES=51269 REQUEST=1489499550.80 RESPONSE=1489499550.82 SOCKET=1489499550.79 SOURCE=op-nl START=1489499550.79 TIMEOUT=1500 USED_AT=1489499550.87 USED_BY=10787 WRITEBYTES=54
  • The IP address of op-hk gets resolved to the Netherlands, though the measurements look very different from the op-nl host. Can we confirm that the host is really located in Hong Kong and that it's just the IP-to-country resolution that is behind?

If we have answers to these questions, I'd say let's make the data available.

Hi Karsten, I can have a chat w/ the people at Greenhost and find out about your questions. I was wondering that too actually.

comment:46 Changed 7 months ago by karsten

robgjansen, thanks for the response on those very fast measurements. Sounds plausible to me. I'll add add op-us and op-nl to CollecTor tomorrow afternoon. And I'll op-hk as soon as we're confident where it's located and your instance as soon as we have resolved the naming part. Thanks!

hiro, it would be great if you could chat with Greenhost people about the location. Just keep it running, and once we know better where it is we can add its data to CollecTor. Thanks!

comment:47 Changed 7 months ago by robgjansen

For naming, it seems like the current scheme works fine for now. Mine would be 'op-ca' since it appears that mine is located in Canada:
https://geoiptool.com/en/?ip=167.114.171.3

But yeah, you may run into collisions if you plan to run more instances. Some other schemes I can think of are:

  • IP address: 'op-ca-167.114.171.3'
  • AS number: 'op-ca-as1234'
  • Simple increment: 'op-ca1', 'op-ca2', etc.

I like tying it to an IP or AS number, because it is more precise. The name is longer though, and if you move the instance, you would need to rotate the name (which may be a good thing anyway since it moved network locations).

comment:48 Changed 7 months ago by robgjansen

I would have no problem updating the names in the data hosted by my instance before you import it.

I looked through my data a bit. I started out running my instance in a different network location (216.17.99.183) before moving it to a new VPS at 167.114.171.3. It appears that I was running the TGen server listening on port 216.17.99.183:6666 before, and now it is listening on 167.114.171.3:8080. So, connections from Tor will request these ports.

I think you are trying to get your instances to run a TGen server at port 80, right? Is that a hard requirement? If so, my existing data is useless. I wrote a quick script (attached in case it's useful) to help us understand how much exit bandwidth by consensus weight allows exit to various ports:

167.114.171.3:80 is allowed by 83.1011269711 percent of exit bandwidth
167.114.171.3:443 is allowed by 85.7649427029 percent of exit bandwidth
167.114.171.3:6666 is allowed by 78.7485340684 percent of exit bandwidth
167.114.171.3:8080 is allowed by 77.0591360016 percent of exit bandwidth
167.114.171.3:8443 is allowed by 76.3254015495 percent of exit bandwidth
216.17.99.183:80 is allowed by 83.1011269711 percent of exit bandwidth
216.17.99.183:443 is allowed by 85.7649427029 percent of exit bandwidth
216.17.99.183:6666 is allowed by 78.7485340684 percent of exit bandwidth
216.17.99.183:8080 is allowed by 77.0591360016 percent of exit bandwidth
216.17.99.183:8443 is allowed by 76.3254015495 percent of exit bandwidth

I think that the difference between port 6666 or 8080 and 80 is minor.

Let me know how to proceed. If you want to use my existing data, perhaps I should use a different name for each of the data sets that were gathered in different locations? It looks like 216.17.99.183 is in California, so that would be another 'op-us-XXX' name.

Last edited 7 months ago by robgjansen (previous) (diff)

Changed 7 months ago by robgjansen

Attachment: check_allowed_exits.py added

use stem to check percent of exit bandwidth allows exit to certain ips and ports

comment:49 Changed 7 months ago by karsten

robgjansen, thanks for the great suggestions above! I think we can make this work. Here are my suggestions:

  • Source names: Maybe we should just pick a naming scheme that works for the moment without expecting that it will scale in the future. Your example of an instance moving from the U.S. to Canada is a great example. Sure, you could split the data into two sets and pick different names. But if we open up contributions to other community members, which I think we should, we'll see similar changes happening, and we can't expect other operators to get the naming right.

My suggestion for the source name of your instance would be to just stick with 'phantomtrain'. The only change that you'd have to make though is make the data (also) available under http://phantomtrain.robgjansen.com:8081/. The reason is that we're currently parsing the first part of the domain name and comparing that to the beginning of file names and source names contained in the data. If this is not an easy change for you, we can look into making CollecTor a little more flexible. But maybe it's just a simple configuration change for you.

  • Local IP address: Even if we pick source names that don't indicate locations, we could resolve instance IP addresses to country codes and AS numbers when displaying results on Tor Metrics. But therefore we'll have to find out the public IP address and include it in the data. I'm yet unsure how we'd do that. I mean, "endpoint_remote" contains a host name and IP address, but only for direct downloads, but this information is also relevant for onion services.

Hmm, I don't really have a good suggestion for this. Do you have an idea?

  • Ports 80, 6666, and 8080: I agree that the difference is minimal, and we probably don't even have to mention this on Tor Metrics. But still I'd like to preserve this information somewhere in the data. Because who knows if somebody wants to run an OnionPerf instance with a rather unusual port.

I wonder if we can include both --tgen-listen-port and --tgen-connect-port in the output? Maybe we'll have to add a new field "endpoint_connect" with remote host name, remote address, and connect port? Or maybe there's a better way to include this information, possibly even related to the previous change where we include the local IP address?

comment:50 Changed 7 months ago by karsten

Quick update: the two OnionPerf instances op-nl and op-us are now available on CollecTor and Tor Metrics.

comment:51 Changed 7 months ago by robgjansen

Cc: robgjansen added; rob.g.jansen@… removed

I made the easy config update, so my data is now also available here: ​http://phantomtrain.robgjansen.com:8081/

comment:52 Changed 7 months ago by robgjansen

For IPs and ports, here is the related data for a direct download:

HOSTNAMELOCAL=onionperf
HOSTNAMEREMOTE=onionperf
ENDPOINTLOCAL=localhost:127.0.0.1:55998
ENDPOINTPROXY=localhost:127.0.0.1:45640
ENDPOINTREMOTE=onionperf.robgjansen.com:167.114.171.3:8080

And here is what I see for a .onion download:

HOSTNAMELOCAL=onionperf
HOSTNAMEREMOTE=onionperf
ENDPOINTLOCAL=localhost:127.0.0.1:45664
ENDPOINTPROXY=localhost:127.0.0.1:45640
ENDPOINTREMOTE=ih7hhuuppsy5wysu.onion:0.0.0.0:8080

Notice that the server listen port that was used for this download, 8080, is available in both cases. This will always be the same place that the client connects, otherwise the download won't happen.

If my OnionPerf instance runs a server on --tgen-listen-port=8080 and I instruct the client to connect to someone else's OnionPerf server at --tgen-connect-port=443, then 443 is the port that will show up in the download data. So I don't think we need to worry about the --tgen-*-port options.

For IPs and hostnames, we have:

I'm not sure that we should change any of this...

This is tricky, since OnionPerf could be run by a client that sits behind a firewall and only contributes .onion downloads, or that connects to someone else's OnionPerf server. These clients probably don't have FQDNs. They of course will have some kind of public IP address, but that won't be discoverable by the server end if the downloads are done over Tor. And the IP address assigned to it's local interface may not be the public-facing IP address.

comment:53 Changed 7 months ago by hiro

Hi,
I have updated measurements that were failing on https://op-us.onionperf.torproject.net and sent Rob a pull request that should fix the issue: https://github.com/robgjansen/onionperf/pull/27
Also I got in contact with the people from Greenhost so that we can find out if they route the traffic for that hong kong instance in a "funny way".
More soon.

comment:54 Changed 7 months ago by hiro

Also, answer from Greenhost regarding our hk onionperf instance:

Hi Silvia,

The IP's are from us, so registered on name of Greenhost, so whois will
say they are dutch, however, they are just routed/announced directly in HK.

(actually, using Whois/GeoIP would not give a clear determination where
the IP's actually are hosted, triangulating whould be a better approach)

Just use mtr/traceroute to determine the path it takes.

Regards,

Mart

comment:55 in reply to:  53 Changed 7 months ago by robgjansen

Replying to hiro:

I have updated measurements that were failing on https://op-us.onionperf.torproject.net and sent Rob a pull request that should fix the issue: https://github.com/robgjansen/onionperf/pull/27

Merged. Thanks!

comment:56 Changed 7 months ago by karsten

hiro, I just made op-hk measurements available on CollecTor and Tor Metrics. Thanks!

robgjansen, thanks for making your data available under the new URL. I can download the files just fine, but I ran into problems with some files containing measurements from other dates than what the file name indicates. I sent you details via email.

Regarding the local IP address I see how this might be problematic. We should figure out something there though, because measurement results highly depend on the location, and we need to include that somehow.

Regarding ports, are you certain that the --tgen-connect-port will be included and not the --tgen-listen-port? I believe hiro changed all op-* instances to download from port 80, but where would I find that port 80 in the .tpf files?

comment:57 in reply to:  56 ; Changed 7 months ago by robgjansen

Replying to karsten:

Regarding ports, are you certain that the --tgen-connect-port will be included and not the --tgen-listen-port? I believe hiro changed all op-* instances to download from port 80, but where would I find that port 80 in the .tpf files?

It looks like your recent torperf files are not including all of the keys that OnionPerf produces. Here is an example of the output produced by my node:
http://phantomtrain.robgjansen.com:8081/phantomtrain-5242880-2017-03-19.tpf

I think either your data is coming from a Torperf instance, or the output of OnionPerf is being filtered to remove some of the key=value entries.

comment:58 in reply to:  57 ; Changed 7 months ago by karsten

Ah, that directory contains .tpf files from Torperf instances (torperf-* and siv*) as well as OnionPerf instances (op-*). The latter should contain all keys that OnionPerf produces. Mind checking again?

comment:59 in reply to:  58 Changed 7 months ago by robgjansen

Replying to karsten:

Ah, that directory contains .tpf files from Torperf instances (torperf-* and siv*) as well as OnionPerf instances (op-*). The latter should contain all keys that OnionPerf produces. Mind checking again?

Ahh, ok. I'm now looking at op-us-5242880-2017-03-21.tpf. The data element I think you are looking for is:

ENDPOINTREMOTE=199-119-112-144.i95.net:199.119.112.144:8080

I think that 8080 is the port that the tgen client requests Tor to connect to.

I see the problem now. From the code, it appears that this port (8080 in this case) is the --tgen-connect-port, i.e., the port that the tgen client asks Tor to connect to. This not necessarily be the same as the --tgen-listen-port, since the remote server machine could have firewall rules set up to forward requests on various ports to the actual listening port.

So, I wonder which of the following is correct:

  • hiro is using --tgen-connect-port=8080 and --tgen-listen-port=8080
  • hiro is using --tgen-connect-port=8080 and --tgen-listen-port=80 and forwarding requests on 8080 over to the server running on 80
  • my conclusion after reading the code is incorrect.

comment:60 Changed 7 months ago by hiro

Please note that I am using the nginx as reverse proxy, so even though tgen is running on port 8080 this is being redirected to 80 externally. Is this correct?

comment:61 Changed 7 months ago by robgjansen

I just realized, the easiest way to find out where you are instructing your client to connect is by looking inside the onionperf-data/tgen-client/tgen.graphml.xml file in your installation. Here is mine:

 <graph edgedefault="directed">
    <node id="start">
      ...
      <data key="d2">ih7hhuuppsy5wysu.onion:8080,167.114.171.3:8080</data>
      ...
    </node>

Notice that my client is connecting to port 8080. What port is listed in your client tgen file, hiro?

comment:62 Changed 7 months ago by hiro

Hi,
I have updated the configuration. Now the tgen server would be on 8080 and the client on 80. Please let me know if this is correct. I will check the new measurements tomorrow morning.

More soon.

comment:63 Changed 7 months ago by robgjansen

According to a .tpf file from 2017-03-26, the client is now correctly requesting port 80 for non-onion downloads. (It is still requesting 8080 for onion downloads, but that should not affect affect path selection for onion service circuits.)

Unfortunately, it looks like all of the downloads are now timing out, since the data shows DIDTIMEOUT=1 for all entries in the .tpf file. More debugging is needed.

comment:64 Changed 7 months ago by hiro

Ok I think what is missing is a port forwarding so that 80 is forwarded to 8080.
More soon.

Last edited 7 months ago by hiro (previous) (diff)

comment:65 Changed 7 months ago by hiro

This is now implemented after changing the supported kernel in all 3 instances. I am seeing successful transfers. It is probably a good idea to clean up old logs with no transfers and wrong ports in the measurements. I'll do this starting from tomorrow. Please let me know if you prefer otherwise.

comment:66 Changed 7 months ago by karsten

Can you trigger generation of .tpf files without having to wait for the daily cronjob to do that? Then we could check if it's really using port 80 and telling us about it.

Removing old data once we're sure it's buggy is probably a good idea.

comment:67 Changed 7 months ago by hiro

Here is an example:

@type torperf 1.0
CONNECT=1490960319.10 DATACOMPLETE=1490960321.08 DATAPERC0=1490960319.56 DATAPERC10=1490960319.98 DATAPERC100=1490960321.08 DATAPERC20=1490960320.19 DATAPERC30=1490960320.36 DATAPERC40=1490960320.46 DATAPERC50=1490960320.56 DATAPERC60=1490960320.68 DATAPERC70=1490960320.80 DATAPERC80=1490960320.92 DATAPERC90=1490960321.04 DATAREQUEST=1490960319.31 DATARESPONSE=1490960319.52 DIDTIMEOUT=0 ENDPOINTLOCAL=localhost:127.0.0.1:46700 ENDPOINTPROXY=localhost:127.0.0.1:33806 ENDPOINTREMOTE=37.218.247.40:37.218.247.40:80 FILESIZE=1048576 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=op-nl LAUNCH=0.0 NEGOTIATE=1490960319.10 READBYTES=1048642 REQUEST=1490960319.10 RESPONSE=1490960319.31 SOCKET=1490960319.10 SOURCE=op-nl START=1490960319.10 WRITEBYTES=53

comment:68 Changed 7 months ago by hiro

List of files on all VMs now updated.

Last edited 7 months ago by hiro (previous) (diff)

comment:69 Changed 7 months ago by karsten

Hmmmm, it looks like the direct downloads are fine now, but all onion service requests time out. Ugh.

Maybe OnionPerf tries to connect to port 80 for onion service requests, too, and there's no onion service and no local port forwarding.

robgjansen, any ideas how to fix/work around this?

comment:70 in reply to:  69 Changed 7 months ago by robgjansen

Replying to karsten:

robgjansen, any ideas how to fix/work around this?

It looks like that was a bug:
https://github.com/robgjansen/onionperf/commit/e3ea7db4d8b1c3f2f410aa85326177495a7215b1

I pushed a fix to the OnionPerf repo on GitHub. @hiro Testing would be appreciated ;-)

comment:71 Changed 7 months ago by hiro

Attaching new tpf files. Please note that I pulled the updates and reinstalled around 12pm.

Changed 7 months ago by hiro

Attachment: op-nl-51200.tpf added

Changed 7 months ago by hiro

Attachment: op-nl-1048576.tpf added

Changed 7 months ago by hiro

Attachment: op-nl-5242880.tpf added

comment:72 Changed 7 months ago by robgjansen

That's great, it looks like both onion and regular downloads are working correctly. But...

SOURCEADDRESS=unknown

it looks like my new feature to add the IP address of the measurement machine needs more testing on my end.

comment:73 Changed 7 months ago by karsten

Uhm, is it just me, or do these files still contain lots of timed out requests to onion services?

@type torperf 1.0
CONNECT=1491478019.16 DATACOMPLETE=0.0 DATAPERC0=1491478139.78 DATAPERC10=0.0 DATAPERC20=0.0 DATAPERC30=0.0 DATAPERC40=0.0 DATAPERC50=0.0 DATAPERC60=0.0 DATAPERC70=0.0 DATAPERC80=0.0 DATAPERC90=0.0 DATAREQUEST=0.0 DATARESPONSE=0.0 DIDTIMEOUT=1 ENDPOINTLOCAL=localhost:127.0.0.1:52680 ENDPOINTPROXY=localhost:127.0.0.1:30850 ENDPOINTREMOTE=7va4ssp7wgszolfj.onion:0.0.0.0:8080 FILESIZE=51200 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=(null) LAUNCH=0.0 NEGOTIATE=1491478019.16 READBYTES=0 REQUEST=1491478019.16 RESPONSE=1491478139.78 SOCKET=1491478019.16 SOURCE=op-nl SOURCEADDRESS=unknown START=1491478019.16 WRITEBYTES=0

comment:74 Changed 7 months ago by hiro

Yes, I see the timeouts again in all the instances. Attaching results generated just now.

Changed 7 months ago by hiro

Attachment: op-nl-51200.2.tpf added

Changed 7 months ago by hiro

Attachment: op-nl-1048576.2.tpf added

Changed 7 months ago by hiro

Attachment: op-nl-5242880.2.tpf added

Changed 7 months ago by hiro

Attachment: onionperf.analysis.json.xz added

comment:75 Changed 7 months ago by hiro

I am noticing that every time onionperf tries to connect to a remote hostname on port 8080 it fails:
@type torperf 1.0
CONNECT=1491496019.19 DATACOMPLETE=0.0 DATAPERC0=1491496020.86 DATAPERC10=0.0 DATAPERC20=0.0 DATAPERC30=0.0 DATAPERC40=0.0 DATAPERC50=0.0 DATAPERC60=0.0 DATAPERC70=0.0 DATAPERC80=0.0 DATAPERC90=0.0 DATAREQUEST=0.0 DATARESPONSE=0.0 DIDTIMEOUT=1 ENDPOINTLOCAL=localhost:127.0.0.1:53300 ENDPOINTPROXY=localhost:127.0.0.1:30850 ENDPOINTREMOTE=7va4ssp7wgszolfj.onion:0.0.0.0:8080 FILESIZE=51200 HOSTNAMELOCAL=op-nl HOSTNAMEREMOTE=(null) LAUNCH=0.0 NEGOTIATE=1491496019.19 READBYTES=0 REQUEST=1491496019.19 RESPONSE=1491496020.86 SOCKET=1491496019.19 SOURCE=op-nl SOURCEADDRESS=unknown START=1491496019.19 WRITEBYTES=0

comment:76 Changed 6 months ago by robgjansen

  • issue 1 - onion service downloads failing

It should be connecting to .onion:80 instead of .onion:8080. I found the bug, fixed it, tested it, and push the fix to the onionperf master branch.

  • issue 2 - SOURCEADDRESS=unknown

hiro, I need some help from you to figure out why this is not working (it works for me in my tests). Can you run the following from one of your onionperf hosts and let me know the results:

source venv/bin/activate
$ python
>>> from onionperf import util
>>> util.get_ip_address()
Last edited 6 months ago by robgjansen (previous) (diff)

comment:77 Changed 6 months ago by hiro

Hi Rob,
Updated the VMs so issue 1 should be live now.

Regarding issue 2:

$ python

from onionperf import util
util.get_ip_address()

'37.218.247.40'

comment:78 Changed 6 months ago by robgjansen

Hmm, OK, so it seems like OnionPerf should be producing correct SOURCEADDRESS lines. Are you running the onionperf analyze mode by some external script to regenerate the .tpf files and ignore the ones that OnionPerf produces? If so, you need to use something like

onionperf analyze --address=37.218.247.40 --name=op-nl --date=2017-04-10 -t --tgen ... --torctl ...

This will ensure that the correct SOURCE and SOURCEADDRESS appear in the .tpf. output files, and that only downloads for the given date are included.

comment:79 Changed 6 months ago by hiro

I am letting onionperf generate the files each day together with the log rotation. But when I generate the tpf in the midlde of the day to test I am just running:
onionperf analyze --torperf --tgen tgen-client/onionperf.tgen.log --torctl tor-client/onionperf.tor.log -p /home/cloud/onionperf/onionperf-data

Which is why you might see the source address empty in this case. If you check a tpf generated at midnight though, the source address is present, e.g.: https://op-nl.onionperf.torproject.net/op-nl-1048576-2017-04-09.tpf

Attaching newly generated measurements just now to check if it's connecting to the right port (and it actually is).

Last edited 6 months ago by hiro (previous) (diff)

Changed 6 months ago by hiro

Changed 6 months ago by hiro

Attachment: op-nl-51200.3.tpf added

Changed 6 months ago by hiro

Attachment: op-nl-1048576.3.tpf added

Changed 6 months ago by hiro

Attachment: op-nl-5242880.3.tpf added

comment:80 Changed 6 months ago by hiro

I can confirm we do not have timeouts anymore. I have also deleted old corrupted measurements.

comment:81 Changed 6 months ago by karsten

hiro, I can confirm that .tpf files op-??-*-2017-04-11.tpf and later look good. I'm attaching a graph comparing measurements from 2017-04-12 with Torperf data. Can you please delete files from 2017-04-10? Those contain timeouts and connections to port 8080. Once that's done, I'll delete files from CollecTor.

Moving forward, I think the following tasks remain:

  • Include data from Rob's phantomtrain. I'm not sure where we are with this.
  • Add irl's instance op-ab, after changing its DNS to op-ab.something and possibly after switching that to port 80 using the same iptables trick that we use, but not necessarily.
  • Explain to ln5 how to set up an instance, maybe op-se, as replacement for the Torperf instance siv, ideally on port 80 with the same iptables trick that we use.
  • Turn off onionperf.torproject.org to reduce the complexity of our setup a little bit.
  • (Anything else?)

Changed 6 months ago by karsten

comment:82 Changed 6 months ago by hiro

Hi Karsten,

I have actually already have got to chat with ln5 and irl regarding old-torperf and op-ab.

  • Regarding old-torperf VM I have chatted with ln5 already and explained the setup last week. Haven't had opportunity to chat again.
  • Will ping irl again (last week he was a bit busy w something else) and see if the VM is ready so I can request a tpn subdomain for that VM.

Regarding our VM onionperf.tpo, I have requested the shut down.

Something else I was working on is a way to orchestrate deployments and managements of the VMs, but this will go into https://trac.torproject.org/projects/tor/wiki/org/operations/Infrastructure/onionperf and will not affect the measurements.

comment:83 Changed 6 months ago by irl

Cc: irl@… added

Hiro - the IP address for op-ab will remain stable, I have a few /24s there to play with and I try to never change an address while a system is running. Some machines have had the same address over 20 years now (though the hardware has changed). It is set to 139.133.232.234. It also has an IPv6 address (native connectivity) but I'm not sure if Onionperf makes any use of this.

I would like to add the port forwarding next week, if you could let me know what the firewall rules are that are required.

For orchestration, I would love it to be Salt based as I'm learning this anyway.

If you can ping me once there is a domain set up, I'll renew the lets encrypt cert to include the new domain.

Thanks,

comment:84 Changed 6 months ago by hiro

Hi irl,
subdomain requested. Yes I have looked at salt in the last few days and was thinking to do it since we are starting to have more machines.

The firewall rules I used are:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT --to-port 8080

Let me know if is there anything else :)

Also I'll ping you once the subdomain is setup.

comment:85 Changed 6 months ago by robgjansen

Replying to karsten:

Moving forward, I think the following tasks remain:

  • Include data from Rob's phantomtrain. I'm not sure where we are with this.

I added one last date filter to make sure that downloads that started on a day previous to the day that it finished are not included in the results; this is now pushed to the master branch on GitHub.

I regenerated my tpf files and I think the downloads are properly filtered now. The data I have been collecting is archived here, separated by measurement location (let me know if you want me to move the files for some reason):

I also finished migrating my onionperf instance as I previously mentioned I would, and the new data will be available here:

I am planning on replacing the twisted web server with caddy, which will automatically get me a let's encrypt cert and host the files over https. Once that is set up, the links above should automatically get redirected the the https versions. You may want to wait until I have that set up to fetch the data. I'll be collecting measurements in the meantime :)

comment:86 Changed 6 months ago by robgjansen

https://phantomtrain.robgjansen.com is now encrypted via a let's encrypt cert.

comment:87 in reply to:  84 ; Changed 6 months ago by irl

Replying to hiro:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT --to-port 8080

Why do we need UDP rules? Should I be allowing UDP traffic through the firewall? (Currently it's blocked).

Edit: The rules are in, managed with iptables-persist, and I've added the --tgen-*-port arguments.

Last edited 6 months ago by irl (previous) (diff)

comment:88 in reply to:  87 Changed 6 months ago by robgjansen

Replying to irl:

Replying to hiro:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 80 -j REDIRECT --to-port 8080

Why do we need UDP rules? Should I be allowing UDP traffic through the firewall? (Currently it's blocked).

OnionPerf only uses TCP; no need to allow UDP ports. (I am only forwarding TCP ports on my instance and it's working fine.)

comment:89 Changed 6 months ago by hiro

Hi,
While I was testing I was afraid I had overlooked something and I made sure all possible traffic on port 80 was being redirected. So yes you are right that udp rule shouldn't be there.

Regarding the latest updates. I have "refreshed" all three instances us, nl and hk.

Regarding op-ab.onionperf.torproject.net subdomain that's now active.

comment:90 in reply to:  89 Changed 6 months ago by irl

Replying to robgjansen:

OnionPerf only uses TCP; no need to allow UDP ports. (I am only forwarding TCP ports on my instance and it's working fine.)

Ok cool, as I suspected. Thanks for clarifying.

Replying to hiro:

Regarding op-ab.onionperf.torproject.net subdomain that's now active.

The cert has been refreshed to include this name.

comment:91 Changed 6 months ago by karsten

So, I deleted measurements from op-hk, op-nl, and op-us until 2017-04-10 from CollecTor and Tor Metrics. Looks good so far.

I also noticed that the last remaining Torperf instance had an issue that I cannot fix easily, so I decided to remove it, too, and add the OnionPerf instance running on that host in the near future. But as stated above somewhere, removing the last Torperf instance requires removing the code from CollecTor that downloads and merges Torperf files. Please find my branch task-21272-2 with a patch.

Adding other OnionPerf hosts is on my list for the next days, or more realistically next week.

comment:92 in reply to:  91 Changed 6 months ago by iwakeh

Replying to karsten:

So, I deleted measurements from op-hk, op-nl, and op-us until 2017-04-10 from CollecTor and Tor Metrics. Looks good so far.

I also noticed that the last remaining Torperf instance had an issue that I cannot fix easily, so I decided to remove it, too, and add the OnionPerf instance running on that host in the near future. But as stated above somewhere, removing the last Torperf instance requires removing the code from CollecTor that downloads and merges Torperf files. Please find my branch task-21272-2 with a patch.

Looks fine; all tests and checks pass.

I added three commits here; the latter two rename all 'torperf' occurrences and the package to 'onionperf'; the final one also renames the properties.

comment:93 Changed 6 months ago by karsten

Thanks for looking! And thanks for those three patches, all merged.

comment:94 Changed 6 months ago by irl

Looking at the .tpf output files from op-ab, it only seems to be testing Onion services and isn't doing any testing of the clearnet tgen instance, unless I'm misinterpreting the results.

Any ideas what might have gone wrong?

comment:95 Changed 6 months ago by robgjansen

The new onion data on the metrics performance page looks great! What's the status of adding the archived data and the current data from https://phantomtrain.robgjansen.com? Once that is done, we should have historical data going back to April 2016 :)

comment:96 Changed 3 months ago by iwakeh

When testing collector I noticed that the op-{hk,nl,us} hosts all provide still data since 2017/04/11 , which currently is no problem. But, at some point in future a deletion policy might be needed (CollecTor will supply historical data). Or, is there one in place?

comment:97 Changed 7 weeks ago by karsten

Resolution: fixed
Status: needs_reviewclosed

I'm closing this ticket now. We have three OnionPerf instances deployed and integrated into CollecTor and Tor Metrics.

Regarding irl's op-ab instance, we should discuss any remaining issues either at team meetings or on a new ticket, possibly even on GitHub where OnionPerf is developed.

Regarding robgjansen's phantomtrain instance, we discussed putting that data on Tor Metrics under Research, but that discussion happens via email.

Regarding iwakeh's question about a deletion policy, that discussion should probably happen on GitHub, too.

However, nothing important remains for deploying OnionPerf. Closing.

Note: See TracTickets for help on using tickets.