Opened 2 years ago

Closed 2 years ago

#26963 closed defect (fixed)

regression in host_name field introduced in v6.1

Reported by: nusenu Owned by: irl
Priority: High Milestone: Onionoo 1.16.0
Component: Metrics/Onionoo Version:
Severity: Normal Keywords:
Cc: metrics-team Actual Points:
Parent ID: Points:
Reviewer: karsten Sponsor:


I hoped this gets clarified or becomes obsolete due to other tickets (#26864) but since there is no progress I'll file it separately for onionoo.

Before v6.1 got deployed the host_name field for most relays contained an actual hostname (not just IP), now (with v6.1) all but less than 10 relays only show an IP address. The verified_host_names field does not appear to match the host_name field.

Child Tickets

Attachments (1)

onionoo-rdns-stats.png (124.6 KB) - added by karsten 2 years ago.

Download all attachments as: .zip

Change History (9)

Changed 2 years ago by karsten

Attachment: onionoo-rdns-stats.png added

comment:1 Changed 2 years ago by karsten

Cc: metrics-team added
Priority: MediumHigh

Let's take a look at Onionoo's internal statistics:

2018-07-29 08:40:37,001 Reverse domain name resolver
    19 lookups performed
    11 verified results found
    1 unverified results found
    00:00.058 minutes minimum lookup time
    00:01.039 minutes median lookup time
    00:03.370 minutes maximum lookup time

Here's a plot of those six numbers for July 2018 so far:

No doubt, there's a change on or around July 16 when we deployed the new lookup code. The total number of lookups performed per hour went down a lot. It didn't go down to zero, so it's still doing something, but it went down enough to be too low to be useful.

comment:2 Changed 2 years ago by irl

Owner: changed from metrics-team to irl
Status: newaccepted

Will look at this this week.

comment:3 Changed 2 years ago by irl

InitialDirContext is very not thread-safe. Oops.

[INFO] Reverse domain name resolver
    7,184 lookups performed
    4,207 verified results found
    334 unverified results found
    00:00.001 minutes minimum lookup time
    00:00.003 minutes median lookup time
    00:10.000 minutes maximum lookup time

This looks better. Patch incoming.

comment:4 Changed 2 years ago by irl

Please review commit 6e6a279 on my branch task/26963.

In the implementation of more accurate DNS resolution, the DirContext used by JNDI lookups was shared in a utility class. It turns out that InitialDirContext, the implementation we used, is very not thread-safe and so as a result lookups would often fail to complete.

This commit creates a new DirContext for every lookup. The overhead in doing so is minimal and ensures that we do not have thread safety issues.

This commit also increases RDNS_LOOKUP_WORKERS_NUM to 30 (from 5). The majority of time in these threads is spent waiting for replies and so there are gains to be had in increasing the number of requests in flight.

comment:5 Changed 2 years ago by irl

Status: acceptedneeds_review

comment:6 Changed 2 years ago by karsten

Reviewer: karsten

The changes look good. I'm now running a local test to see if it works here, too.

comment:7 Changed 2 years ago by karsten

Status: needs_reviewneeds_revision

Test looks good. Merged to master, after realizing that we should probably have a change log entry, too. Want to write one?

comment:8 Changed 2 years ago by karsten

Milestone: Onionoo 1.16.0
Resolution: fixed
Status: needs_revisionclosed

Never mind, I just wrote something and pushed it to master. Closing. Thanks!

Note: See TracTickets for help on using tickets.