Onionoo documents should be the same accross all tp.o instances

Investigate current differences and define the prerequisites for achieving this goal.

I'll add examples for differences in comments, as this allows for different discussion threads.

added component::metrics/onionoo owner::metrics-team priority::medium resolution::implemented severity::normal status::closed type::enhancement labels

The details documents have geoip related differences (data retrieved on 2017-08-15), i.e., coordinates changes and spelling

h> "region_name":"Pennsylvania","city_name":"Lansdale","latitude":40.2262,"longitude":-75.2931,
m> "region_name":"Pennsylvania","city_name":"Lansdale","latitude":40.2415,"longitude":-75.2838,
                                                                      ^^^                  ^^^
h> "region_name":"Stockholm","city_name":"Norrtaelje",
m> "region_name":"Stockholm","city_name":"Norrtalje",
                                                ^^^

Other differences might be due to different update times, from details:

h> {"nickname":"mrkoolltor","fingerprint":"92808CA58D8F32CA34A34C547610869BF4E2A6EC","or_addresses":["77.120.94.233:9001"],"last_seen":"2017-08-15 08:00:00",...,"consensus_weight_fraction":3.3634333E-6,"guard_probability":0.0,"middle_probability":8.677337E-6,"exit_probability":0.0,"recommended_version":false,"measured":true}
m> {"nickname":"mrkoolltor","fingerprint":"92808CA58D8F32CA34A34C547610869BF4E2A6EC","or_addresses":["77.120.94.233:9001"],"last_seen":"2017-08-15 07:00:00",...,"consensus_weight_fraction":3.3669173E-6,"guard_probability":0.0,"middle_probability":8.666976E-6,"exit_probability":0.0,"recommended_version":false,"measured":true},
                                                                                                                                                   ^^                                            ^^^^^                                                    ^^^^^

clients

h> "fingerprint":"FCC1A739D1FCF2FCC15DEF3AE86D346548B0165D",..,"countries":{"by":0.042976454,"ca":0.020426456,"ch":0.09072858,"cn":0.02642937,"de":0.01072343,"es":0.021451455,"fr":0.01072343,"gb":0.034091588,"ir":0.122246176,"jp":0.07436535,"kz":0.06670313,"ma":0.010213228,"pa":0.016547084,"ro":0.013131949,"ru":0.122246176,"sa":0.01072343,"tr":0.122246176,"us":0.10079472,"vn":0.08319506},"transports":{"\u003cOR\u003e":0.31281802,"obfs4":0.687182},"versions":{"v4":1.0}},"1_month":{"first":"2017-07-13 12:00:00","last":"2017-08-13 12:00:00","interval":86400,"factor":0.0028379379379379383,"count":32,"values":[570,557,613,null,704,704,557,569,557,422,422,275,288,422,422,422,422,422,422,422,422,569,557,569,851,986,839,999,974,851,986,795],"countries":{"by":0.029526738,"ca":0.041479144,"ch":0.04099562,"cn":0.05778515,"de":0.038172565,"es":0.019129056,"fr":0.02451852,"gb":0.025931904,"ir":0.14300486,"it":0.015917324,"jp":0.0392698,"kz":0.08019847,"nl":0.01588199,"ru":0.12153813,"tr":0.115521945,"us":0.09202813,"vn":0.04192548},"transports":
m> "fingerprint":"FCC1A739D1FCF2FCC15DEF3AE86D346548B0165D",..,"countries":{"by":0.032763224,"ca":0.010213228,"ch":0.09200638,"cn":0.02642937,"de":0.01072343,"es":0.021451455,"fr":0.01072343,"gb":0.034091588,"ir":0.12352398,"jp":0.07564315,"kz":0.06798094,"ma":0.010213228,"pa":0.016547084,"ro":0.013131949,"ru":0.12352398,"sa":0.01072343,"tr":0.12352398,"us":0.10207252,"vn":0.09468609},"transports":{"\u003cOR\u003e":0.31281802,"obfs4":0.687182},"versions":{"v4":1.0}},"1_month":{"first":"2017-07-13 12:00:00","last":"2017-08-13 12:00:00","interval":86400,"factor":0.0028379379379379383,"count":32,"values":[570,557,613,null,704,704,557,569,557,422,422,275,288,422,422,422,422,422,422,422,422,569,557,569,851,986,839,999,974,851,986,795],"countries":{"by":0.025394445,"ca":0.03734685,"ch":0.04151262,"cn":0.05778515,"de":0.038172565,"es":0.019129056,"fr":0.02451852,"gb":0.025931904,"ir":0.14352186,"it":0.015917324,"jp":0.0397868,"kz":0.08071547,"nl":0.01588199,"ru":0.122055136,"tr":0.11603895,"us":0.09254514,"vn":0.04657477},"transports":
                                                                                    ^^^^^^^^         ^^^^^^^^          ^^^^^^^^                                                                                          ^^^^^^^

m> is from the main Onionoo instance.

Some entries appear in different order. If there is need I can also attach the entire file here.

bandwidth

Differences in first seen minutes and slight differences in counts, which seem to be due to different starting times of the updater.

h> "fingerprint":"D7EF14045DEDAE9E9E61D8CEF0F84F87703AD55D","write_history":{"3_days":{"first":"2017-08-12 04:37:30","last":"2017-08-15 03:07:30","interval":900,"factor":8.02902902902903,"count":283,
m> "fingerprint":"D7EF14045DEDAE9E9E61D8CEF0F84F87703AD55D","write_history":{"3_days":{"first":"2017-08-12 04:22:30","last":"2017-08-15 03:07:30","interval":900,"factor":8.02902902902903,"count":284,
                                                                                                              ^^                                                                                     ^

The following should have come from the copied data, but still differences?

h> {"fingerprint":"96CF829ABA63490D052396E717250FA84F60667E","write_history":{"1_month":{"first":"2017-07-15 02:00:00","last":"2017-08-14 18:00:00",
m> {"fingerprint":"96CF829ABA63490D052396E717250FA84F60667E","write_history":{"1_month":{"first":"2017-07-14 22:00:00","last":"2017-08-14 18:00:00",
                                                                                                          ^^ ^^

All samples of relays only showing up in the 20170815 diff for the main instance are disappeared in the current documents for both the main and the mirror instance.

Replying to iwakeh:

The details documents have geoip related differences (data retrieved on 2017-08-15), i.e., coordinates changes and spelling {{{ h> "region_name":"Pennsylvania","city_name":"Lansdale","latitude":40.2262,"longitude":-75.2931, m> "region_name":"Pennsylvania","city_name":"Lansdale","latitude":40.2415,"longitude":-75.2838, ^^^ ^^^ h> "region_name":"Stockholm","city_name":"Norrtaelje", m> "region_name":"Stockholm","city_name":"Norrtalje", ^^^ }}}

These differences shouldn't cause problems for clients. But it would be better to have consistent data, which is immediately achievable by using the same geoip edition.

In the long run ticket #21515 (moved) ought to provide a single source for geoip related data and thus solve the issue thoroughly.

Since both instances use the same geoip database the details doc differences reduced quite a bit.

In total, there is a small percentage when only looking at running nodes:

doc-type count diff. lines percentage 
summary	 9262	 25	   0.003
details	 9262	 649	   0.070
bandwd.  9262	 753	   0.081
weights	 6785	 361	   0.053
clients	 2484	 16	   0.006

These should be fine with clients when rotating.

Still some should be investigated like the diffs for details, weights, and bandwith. -> new ticket

Trac:
Status: new to needs_information

Without looking at the differences yet: We might resolve some of them by simply doing #22033 (moved). And #16513 (moved) might take a little more effort but move us forward in this direction, too.

Replying to karsten:

Without looking at the differences yet: We might resolve some of them by simply doing #22033 (moved). And #16513 (moved) might take a little more effort but move us forward in this direction, too.

Yes, these two tasks will make lots of diffs go away. #22033 (moved) removes most differences left in the clients docs. #16513 (moved) is important, because we will otherwise see an increase of differences in a few weeks for the monthly values, too. So, we already have the appropriate tickets, no need to open a new one as mentioned above.

I think this ticket could be closed as the work will be done in the two mentioned tickets.

Sounds good to me. Thanks for checking! Closing.

Trac:
Resolution: N/A to duplicate
Status: needs_information to closed

Only opening to close it with an appropriate solution.

Trac:
Status: closed to reopened
Resolution: duplicate to N/A

Resolved as 'implemented' b/c the ticket was about investigation of differences and defining measures to resolve these. One measure was to use identical geoip dbs, which is now deployed on both hosts. The other steps are the two tickets named. So, this is no duplicate, but done ;-)

Trac:
Resolution: N/A to implemented
Status: reopened to closed

closed