Find more reliable and universal way to get ASN and ASorg mappings

This ticket is intended to continue the discussion started at #19420 (moved) about GeoIP database reliability. To outline:

• We use only one source of ASN mappings (MaxMind's GeoIP)
• This source became unreliable recently (#19154 (moved), #19420 (moved))
• The data is maintained manualy and this process is pretty crafty. So it's good to move this process to vcs like Git.

added as asn bgp component::metrics/onionoo diversity geoip maxmind organisations priority::medium resolution::fixed severity::normal status::closed type::project labels

Also consider:

• related issue #19118 (moved) and
• a different approach for providing the necessary data: https://quaxio.com/bgp/

Trac:
Component: - Select a component to Metrics/Onionoo

Component changed from - Select a component to Metrics/Onionoo I guess it's more general and also relevant for at least little-t-tor, no?

Another option is to parse this ownership from the bgp tables that are maintained by routeviews

They have historic tables of RIP announcements from a few different gateway servers that can be parsed: archive.routeviews.org/oix-route-views/

There's also a current direct-ownership one that's a bit easier to deal with: archive.routeviews.org/dnszones/originas.bz2

Replying to twim:

Component changed from - Select a component to Metrics/Onionoo I guess it's more general and also relevant for at least little-t-tor, no?

tor has no AS awareness, it uses maxmind for country level lookups only (AFAIK)

As a stopgap solution for the currently broken MaxMind DB, I trying to repair at least nameless maxmind entries with data from http://thyme.apnic.net/current/data-used-autnums

quick'n dirty solution (this reduces nameless AS entries by 49k records):

#convert data-used-autnums into a csv file and prefix AS numbers with "AS"
sed 's/^[ \t]*//' data-used-autnums.orig | sed 's/ /,/'|sed 's/"//g'|awk '{ print "AS" $0 }'|sort -t, -k 1b,1 > data-used-autnums.csv

#get all maxmind entries without AS names
grep -v '"' GeoIPASNum2.csv.orig |sort -t, -k 3 > nameless_AS_entries 
join -o 1.1,1.2,1.3,2.2 -a 1  -t, -1 3 -2 1  nameless_AS_entries data-used-autnums.csv|awk -F, '{ print $1 "," $2 ",." $3 " " $4 "."}'|sed 's,\.,",g'  > repaired_entries
grep '"' GeoIPASNum2.csv.orig > entries_with_names
cat entries_with_names repaired_entries > GeoIPASNum2.csv.partially-repaired

Note: also records with AS names are broken (but I'm not touching them) https://lists.torproject.org/pipermail/tor-dev/2016-June/011095.html

would be great to have at least a workaround deployed soon - otherwise I'll have to postprocess onionoo data before using it.

Also note that records generated by the join/awk commands above are slightly different when they result in nameless AS entries:

maxmind nameless example:

1,2,AS123

my output (if data-used-autnums did not contain any record for AS123):

1,2,"AS123 "

changing the state to needs_review (even though that is not aimed to be the real fix)

Trac:
Status: new to needs_review

Let's not deploy a hack just yet. Nusenu contacted MaxMind's support team on June 19. Let's give them a few more days to resolve this issue in the next database, to be published on June 27. If that doesn't solve the issue, let's reconsider.

Okay, their June 27 database is still broken. But! I just found the May 23 database on my local hard drive by coincidence. That one is not broken and it's not that old. I'll put that in place as band-aid for the next two weeks in the hope that MaxMind fixes their stuff. It'll take an hour or so for both Onionoo instances to update.

for the record (got better, but I would still stick with the old version) https://lists.torproject.org/pipermail/tor-dev/2016-July/011192.html https://lists.torproject.org/pipermail/tor-dev/2016-July/011193.html

Trac:
Status: needs_review to new

Thanks for the update. I'll update once they have fixed the problem.

Since the situation with maxmind isn't to promising I'm planing to look into data used by https://github.com/hadiasghari/pyasn they get their data from http://archive.routeviews.org (https://github.com/hadiasghari/pyasn/blob/master/pyasn-utils/pyasn_util_download.py )

and http://www.cidr-report.org/as2.0/autnums.html ( https://github.com/hadiasghari/pyasn/blob/master/pyasn-utils/pyasn_util_asnames.py)

pyasn's ASnumber->name mapping source (​http://www.cidr-report.org/as2.0/autnums.html) isn't as good as maxmind or http://thyme.apnic.net/current/data-used-autnums currently

I filed a feature request to consider APNIC data as well: https://github.com/hadiasghari/pyasn/issues/30

https://iptoasn.com/ run by Frank Denis of OVH might be useful.

Replying to phoul:

https://iptoasn.com/ run by Frank Denis of OVH might be useful.

Thanks. Do you know is there a writeup about this service and the sources of the data there? I can't find any.

See the update on #19420 (moved). (I lost track where this discussion should happen.)

Here we should just discuss finding more reliable and universal way to get ASN and ASorg mappings.

After having resolved #19420 (moved) three months ago we're not looking for another source of ASN data for Onionoo anymore. Closing.

Trac:
Resolution: N/A to fixed
Status: new to closed

closed