Find a working alternative to using MaxMind's GeoLite2 databases

MaxMind has recently changed access and use of their GeoLite2 databases: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

This affects Onionoo and tor. I started a thread on tor-dev@ about this topic last week with some more details.

Let's use this ticket to brainstorm and discuss working alternatives to the way we used their databases in the past.

Trac:
Child Ticket(s): #29281 (moved)

added component::metrics owner::metrics-team priority::medium severity::normal status::new type::defect labels

I think I found a possible alternative by using data from another provider. But before I name it here, I'd first want to find out how accurate it is.

I tried resolving relay IP addresses of relays that have been running in the past week and compared that to our existing lookups using MaxMind's October database. The result is that 7669 relays (93%) had the same country code and ASN. I put the remaining 7% on this wiki page and would like to hear from relay operators which data source is right and which is wrong (or if both are wrong). Relay operators can leave comments either here or in the "Comments" column on that wiki page. I'll post to the tor-relays@ mailing list and ask relay operators for participation.

Fingerprint: 41A3C162

---

According to Whois:

• Country, region, city: EU (Region, City is not part of the whois)
• ASN, ISP: AS205365, "Matthias Fetzer"

According to old db:

• Country, region, city: null, null, null
• ASN, ISP: AS205365, "Matthias Fetzer"

According to new db:

• Country, region, city: Ukraine, Odessa, Odessa
• ASN, ISP: AS205365, "FETZER"

---

The new database is correct, regarding the physical location. The server is indeed running in Odessa, Ukraine. The allocation (before it was transfered to me) has been registered to Odessa, Ukraine before. So I am not sure if the new database uses old data (instead of "no data", which the old db did), or if they actually process traceroutes in order to determine the location.

As the officially registration states "EU" as the Country, I would actually expect it to be determined as "EU".

Trac:
Username: fetzerms

Looking at a sample of deltas and corresponding BGP announcements indicates the new database is indeed more accurate; it seems to prefer "actual" location to what can be derived from solely from WHOIS.

Trac:
Username: minxi

please note previous discussions about IP-to-ASN data

#26585 (moved)

What is the new geoip source? I am dumping a much larger quantity of IP's through a differ and alt source check. Thanks.

Trac:
Keywords: N/A deleted, metrics-team-roadmap-2020Q1 added

We are looking into the license of the new source to be sure it will work.

Fingerprint: 951307BA

The new one is more correct.

The old one i think is incomplete because its at an OVH reseller. The new one at least got the location correct.

---

Fingerprint: F51A927E

Answer from my hosting company: PINDC-AS should be the correct one, AS34665 is the newer ASN.

So it looks the new database is more accurate here as well!

Trac:
Username: computer_freak

185.220.100.0/24: 2a0b:f4c0:16c::/48: as: AS205100 country: server: DE; operator: DE; hoster: =operator

not listed :-)

185.220.101.0/24: as: AS208294 country: server: NL; operator: DE; hoster: GB

maxmind: as: AS200052 (outdated since some weeks) maxmind: country: DE <--------------------- Ops, but not completely wrong new-db: as: AS208294 new-db: country: NL <--------------------- for me it looks better

---

185.220.102.0/24: 2a0b:f4c1::/64: as: AS60729 country: server: DE; operator: DE; hoster: DE

maxmind: as: AS60729 maxmind: country: DE new-db: as: AS60729 new-db: country: NL (wrong!) <--------------------- Ops

185.220.103.0/24: as: AS4224 country: server: USA?; operator: USA; hoster: USA?

maxmind: as: AS4224 maxmind: country: DE (wrong!) <--------------------- Ops new-db: as: AS4224 new-db: country: NL (wrong!) <--------------------- Ops

---

Sumup:

AS Wrong:

• Maxmind: 1 (okay, this is just outdated, maybe they need some more weeks (years?))
• New-DB: 0 Country Wrong:
• Maxmind: 1.5 (listened the operator as country)
• New-DB: 2

My personal experience with maxmind is, that they just don't response to contact about wrong entries. Maybe it would be a good idea to the that on the New-DB.

Trac:
Username: RedDog

New geoip source is wrong for all AS28753 relays. Maxmind has correctly located these in Germany; new source has Netherlands. AS description says it: "LEASEWEB-DE-FRA-10," where FRA is Frankfurt.

Move some of the tickets from last metrics roadmap to the roadmap in April.

Trac:
Keywords: metrics-team-roadmap-2020Q1 deleted, metrics-team-roadmap-2020April added

Trac:
Keywords: metrics-team-roadmap-2020April deleted, N/A added
Cc: metrics-team to metrics-team, nickm

Replying to gaba:

We are looking into the license of the new source to be sure it will work.

Gaba, let us know if you need help with the license side of it.

It sounds from the above that the new database is not much worse than the current database (and possibly as good or better), so it sounds like a winner to me, if we are indeed unable to continue with the maxmind one.

mentioned in issue #29281 (moved)