Opened 3 years ago

Closed 3 years ago

#19437 closed project (fixed)

Find more reliable and universal way to get ASN and ASorg mappings

Reported by: twim Owned by:
Priority: Medium Milestone:
Component: Metrics/Onionoo Version:
Severity: Normal Keywords: asn, as, geoip, maxmind, bgp, organisations, diversity
Cc: iwakeh, karsten Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

This ticket is intended to continue the discussion started at #19420 about GeoIP database reliability.
To outline:

  • We use only one source of ASN mappings (MaxMind's GeoIP)
  • This source became unreliable recently (#19154, #19420)
  • The data is maintained manualy and this process is pretty crafty. So it's good to move this process to vcs like Git.

Child Tickets

Change History (18)

comment:1 Changed 3 years ago by iwakeh

Also consider:

comment:2 Changed 3 years ago by iwakeh

Component: - Select a componentMetrics/Onionoo

comment:3 Changed 3 years ago by twim

Component changed from - Select a component to Metrics/Onionoo

I guess it's more general and also relevant for at least little-t-tor, no?

comment:4 Changed 3 years ago by willscott

Another option is to parse this ownership from the bgp tables that are maintained by routeviews

They have historic tables of RIP announcements from a few different gateway servers that can be parsed:
archive.routeviews.org/oix-route-views/

There's also a current direct-ownership one that's a bit easier to deal with:
archive.routeviews.org/dnszones/originas.bz2

comment:5 in reply to:  3 Changed 3 years ago by cypherpunks

Replying to twim:

Component changed from - Select a component to Metrics/Onionoo

I guess it's more general and also relevant for at least little-t-tor, no?

tor has no AS awareness, it uses maxmind for country level lookups only (AFAIK)

comment:6 Changed 3 years ago by cypherpunks

As a stopgap solution for the currently broken MaxMind DB, I trying to repair at least nameless maxmind entries with data from
http://thyme.apnic.net/current/data-used-autnums

quick'n dirty solution (this reduces nameless AS entries by 49k records):

#convert data-used-autnums into a csv file and prefix AS numbers with "AS"
sed 's/^[ \t]*//' data-used-autnums.orig | sed 's/ /,/'|sed 's/"//g'|awk '{ print "AS" $0 }'|sort -t, -k 1b,1 > data-used-autnums.csv

#get all maxmind entries without AS names
grep -v '"' GeoIPASNum2.csv.orig |sort -t, -k 3 > nameless_AS_entries 
join -o 1.1,1.2,1.3,2.2 -a 1  -t, -1 3 -2 1  nameless_AS_entries data-used-autnums.csv|awk -F, '{ print $1 "," $2 ",." $3 " " $4 "."}'|sed 's,\.,",g'  > repaired_entries
grep '"' GeoIPASNum2.csv.orig > entries_with_names
cat entries_with_names repaired_entries > GeoIPASNum2.csv.partially-repaired

Note: also records with AS names are broken (but I'm not touching them)
https://lists.torproject.org/pipermail/tor-dev/2016-June/011095.html

would be great to have at least a workaround deployed soon - otherwise I'll have to postprocess onionoo data before using it.

comment:7 Changed 3 years ago by cypherpunks

Status: newneeds_review

Also note that records generated by the join/awk commands above are slightly different when they result in nameless AS entries:

maxmind nameless example:

1,2,AS123

my output (if data-used-autnums did not contain any record for AS123):

1,2,"AS123 "

changing the state to needs_review (even though that is not aimed to be the real fix)

comment:8 Changed 3 years ago by karsten

Let's not deploy a hack just yet. Nusenu contacted MaxMind's support team on June 19. Let's give them a few more days to resolve this issue in the next database, to be published on June 27. If that doesn't solve the issue, let's reconsider.

comment:9 Changed 3 years ago by karsten

Okay, their June 27 database is still broken. But! I just found the May 23 database on my local hard drive by coincidence. That one is not broken and it's not that old. I'll put that in place as band-aid for the next two weeks in the hope that MaxMind fixes their stuff. It'll take an hour or so for both Onionoo instances to update.

comment:10 Changed 3 years ago by cypherpunks

Status: needs_reviewnew

comment:11 Changed 3 years ago by karsten

Thanks for the update. I'll update once they have fixed the problem.

comment:13 Changed 3 years ago by cypherpunks

pyasn's ASnumber->name mapping source (​http://www.cidr-report.org/as2.0/autnums.html) isn't as good as maxmind or
http://thyme.apnic.net/current/data-used-autnums
currently

I filed a feature request to consider APNIC data as well:
https://github.com/hadiasghari/pyasn/issues/30

comment:14 Changed 3 years ago by phoul

https://iptoasn.com/ run by Frank Denis of OVH might be useful.

comment:15 in reply to:  14 Changed 3 years ago by twim

Replying to phoul:

https://iptoasn.com/ run by Frank Denis of OVH might be useful.

Thanks. Do you know is there a writeup about this service and the sources of the data there? I can't find any.

comment:16 Changed 3 years ago by karsten

See the update on #19420. (I lost track where this discussion should happen.)

comment:17 Changed 3 years ago by iwakeh

Here we should just discuss finding more reliable and universal way to get ASN and ASorg mappings.

comment:18 Changed 3 years ago by karsten

Resolution: fixed
Status: newclosed

After having resolved #19420 three months ago we're not looking for another source of ASN data for Onionoo anymore. Closing.

Note: See TracTickets for help on using tickets.