Opened 4 years ago

Last modified 2 years ago

#18203 assigned enhancement

Base direct user estimates on responses to directory requests, rather than responses

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/Statistics Version:
Severity: Minor Keywords: metrics-2018
Cc: dcf Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Relays report two different statistics related to directory requests:

  • "dirreq-v3-reqs" contain the number of requests broken down by country and
  • "dirreq-v3-resp" contain the number of responses broken down by status code.

We're using the number of responses to estimate bridge users but the number of requests to estimate direct users. We should use the same input for both estimates. Using the number of responses (successful requests) seems more intuitive, because we're assuming that each client needs to fetch 10 fresh consensuses every day, and they might have to make more than 10 requests if one or more of them is not successful.

I'll attach an analysis shortly.

This minor enhancement came up when working on #18167.

Child Tickets

Attachments (3)

direct-clients-2016-02-01.png (62.6 KB) - added by karsten 4 years ago.
direct-clients-2017-01-20.png (52.8 KB) - added by karsten 3 years ago.
0001-tmp-msg-Base-direct-user-estimates-on-resp-not-reqs.patch (5.6 KB) - added by karsten 3 years ago.

Download all attachments as: .zip

Change History (14)

Changed 4 years ago by karsten

comment:1 Changed 4 years ago by karsten

The following graph shows the effect of switching from requests to responses for direct user number estimates:


comment:2 Changed 3 years ago by dcf

Summary: Base direct user estimates on responses to directory requestsBase direct user estimates on responses to directory requests, rather than responses

Counting responses rather than requests may fix a problem we observe in #18203, where a Tor blocking event in Turkey caused a paradoxical increase rather than decrease in the direct-user graph. dgoulet found that the increase in the graphs corresponded with an increase in the number of failed consensus downloads from the directory authorities.

comment:13:ticket:21014:

This indeed makes me think that the metrics graphs are overcounting users in this case where consensus downloads are being interrupted. That's why the direct-user graphs show more users when in reality there are probably fewer.

#18203 is a proposal to base direct-user counts on directory responses, rather than directory requests. Doing that might solve this overcounting issue.

comment:3 Changed 3 years ago by arma

I'm a fan!

It would be an interesting and hopefully useful exercise to go through and do another comparison graph, this time of those two numbers for relays counting users from Turkey lately.

I suggest that exercise because this seems like a good bug to fix, but it's not clear that fixing this bug will address the Turkey counting issue that dcf points to -- for example, if a censor lets you bootstrap and then cuts off your connection, and also you suffer from some bug like #20269, then you could end up fetching many more consensuses than you "should".

comment:4 Changed 3 years ago by karsten

Priority: MediumHigh

Raising priority, though this might not happen in 2016 anymore.

Changed 3 years ago by karsten

comment:5 Changed 3 years ago by karsten

Priority: HighMedium

I made another comparison graph for Q4/2016 until mid-January 2017 with all users, users from Turkey, and users from the United States:


The effect is the same: user number estimates based on responses (new approach) are slightly lower than those that are based on requests (old approach). But there is no visible difference for users from Turkey.

I think the reason is that we're counting a response as soon as we consider the request as valid and start sending a response document. Quoting from dir-spec.txt, where the "responses" line in the graph corresponds to the "ok" case below:

    "dirreq-v3-resp" status=nul,... NL
        [At most once.]

        List of mappings from response statuses to the number of requests
        for v2/v3 network statuses that were answered with that response
        status, rounded up to the nearest multiple of 4. Only response
        statuses with at least 1 response are reported. New response
        statuses can be added at any time. The current list of response
        statuses is as follows:

        "ok": a network status request is answered; this number
           corresponds to the sum of all requests as reported in
           "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before
           rounding up.
        "not-enough-sigs: a version 3 network status is not signed by a
           sufficient number of requested authorities.
        "unavailable": a requested network status object is unavailable.
        "not-found": a requested network status is not found.
        "not-modified": a network status has not been modified since the
           If-Modified-Since time that is included in the request.
        "busy": the directory is busy.

I'm also attaching the patch that I used to create this comparison graph. But I'd say let's not merge it yet, because we're in the middle of changing user number estimates. Let's just keep this minor improvement in mind when we do so.

Changing priority back to medium.

comment:6 Changed 3 years ago by iwakeh

Request in dirreq-v3-reqs are counted only for 'ok' responses,
as specified in torspec:

    "dirreq-v3-reqs" CC=N,CC=N,... NL
        [At most once.]

        List of mappings from two-letter country codes to the number of
        requests for v2/v3 network statuses from that country, rounded up
        to the nearest multiple of 8. Only those requests are counted that
        the directory can answer with a 200 OK status code.

and implemented here.

So, the two implementations should show relevant differences.
The differences above might be due to the binning for each country that
increases the sum for the 'all' counts.

comment:7 Changed 3 years ago by iwakeh

typo in comment:6

- So, the two implementations should show relevant differences.
+ So, the two implementations shouldn't show relevant differences.

comment:8 Changed 2 years ago by karsten

Component: Metrics/WebsiteMetrics/Statistics

Moving all tickets to Metrics/Statistics that are more related to the data-aggregating modules rather than the website parts of metric-web.

comment:9 Changed 2 years ago by dcf

Cc: dcf added

comment:10 Changed 2 years ago by karsten

Keywords: metrics-2018 added

comment:11 Changed 2 years ago by karsten

Owner: set to metrics-team
Status: newassigned
Note: See TracTickets for help on using tickets.