Opened 7 years ago

Closed 3 years ago

#8799 closed enhancement (fixed)

collector's downloads: avoid httpurl-connection

Reported by: karsten Owned by: iwakeh
Priority: High Milestone: CollecTor 1.2.0
Component: Metrics/CollecTor Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: #20518 Points:
Reviewer: Sponsor:

Description (last modified by iwakeh)

In November 2012, tor26 was very slow in responding to metrics-db, which made metrics-db exceed its 30-minute schedule. This happened again on March 2, 2013. The workaround was to avoid tor26 and other slow directories and download descriptors only from the other directory authorities. metrics-db should use a timeout similar to how DocTor does.

This should avoid httpurl-connection as suggested in #20323 for metrics-lib.

This is the parent ticket for three smaller tasks.

Child Tickets

#20514enhancementclosedCollecTor' torperf module: replace HttpURLConnection
#20515enhancementclosediwakehCollecTor's relaydescs module should avoid httpurlconnection
#20516enhancementclosediwakehCollecTor's exitlists module should avoid httpurlconnection

Change History (11)

comment:1 Changed 4 years ago by iwakeh

Keywords: ctip added
Severity: Normal

added this ticket to the "Areas for Improvement" of CTIP

comment:2 Changed 4 years ago by iwakeh

Summary: Add timeouts to metrics-db's relay descriptor downloadsAdd timeouts to collector's relay descriptor downloads

changed metrics-db to collector.

comment:3 Changed 4 years ago by iwakeh

Milestone: CollecTor 1.1.0

Added to second release milestone.

comment:4 Changed 4 years ago by iwakeh

Owner: set to iwakeh
Status: newassigned

assigning these issues.

comment:5 Changed 4 years ago by iwakeh

Milestone: CollecTor 1.1.0CollecTor 1.2.0

comment:6 Changed 4 years ago by iwakeh

see #20323

comment:7 Changed 4 years ago by iwakeh

Description: modified (diff)
Priority: MediumHigh
Summary: Add timeouts to collector's relay descriptor downloadscollector's relay descriptor downloads: avoid httpurl-connection

Moved from #20323 (comment 3, karsten):

It might be that this issue killed the relaydescs module 1.5 days ago by blocking the downloader indefinitely:

2016-10-30 23:05:00,000 INFO o.t.c.c.CollecTorMain:54 Starting relaydescs module of CollecTor.
2016-10-30 23:05:20,823 INFO o.t.c.r.CachedRelayDescriptorReader:255 Finished importing relay descriptors from local Tor data directories:
cached-consensus: 2016-10-30 23:00:00
cached-descriptors: parsed 0, skipped 24471 server descriptors parsed 753, skipped 6024 server descriptors
cached-extrainfo: parsed 0, skipped 24508 extra-info descriptors parsed 745, skipped 1463 extra-info descriptors
v3-status-votes: parsed 8, skipped 0 votes
2016-10-30 23:06:00,001 INFO o.t.c.c.CollecTorMain:54 Starting updateindex module of CollecTor.

There should be a line like this (copied over from the previous run), but there wasn't:

2016-10-30 22:35:19,933 INFO o.t.c.r.RelayDescriptorDownloader:1067 Finished downloading relay descriptors from the directory authorities.

Raising priority to high.

comment:8 Changed 4 years ago by iwakeh

Description: modified (diff)
Keywords: metrics-help added; ctip removed
Summary: collector's relay descriptor downloads: avoid httpurl-connectioncollector's downloads: avoid httpurl-connection

comment:9 Changed 4 years ago by iwakeh

Keywords: metrics-help removed
Parent ID: #20518

Set parent ticket and removed metrics-help tag as this ticket only groups the three subtickets.

comment:10 Changed 3 years ago by iwakeh

Status: assignedneeds_information

The download in ExitlistDownloader and RelayDescriptorDownloader should be based on a common code source, which is not part of this task. New ticket?

comment:11 Changed 3 years ago by karsten

Resolution: fixed
Status: needs_informationclosed

I'd rather simplify the download code to the point where sharing it between modules is not necessary anymore. In any case, that'd be a discussion for a new ticket. Closing. Thanks!

Note: See TracTickets for help on using tickets.