Opened 4 years ago

Closed 3 years ago

#18663 closed defect (fixed)

Onionoo doesn't send certain headers on even-numbered responses

Reported by: dcf Owned by: karsten
Priority: Medium Milestone:
Component: Metrics/Onionoo Version:
Severity: Major Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

When I load this URL, the first time I get meaningful output:

Screenshot of the 1st, 3rd, 5th, etc. time loading.

But if I hit Ctrl+R to refresh, I get this garbled (maybe compressed?) output instead:

Screenshot of the 2nd, 4th, 6th, etc. time loading.

If I refresh again, it goes back to the readable version, and if I refresh yet again, it switches back to the garbled version. I can keep switching back and forth.

The same happens if I click the refresh icon in the address bar. I tried it in Tor Browser 6.0a4 and Chromium 49.0, and it happens in both.

The garbled version additionally causes this to be printed to the console:

The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.

With another type of document, namely https://onionoo.torproject.org/details?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534, the text remains readable while repeatedly refreshing, but the "character encoding" message appears in the console in alternating refreshes.

Child Tickets

Attachments (2)

1.png (13.0 KB) - added by dcf 4 years ago.
Screenshot of the 1st, 3rd, 5th, etc. time loading.
2.png (26.1 KB) - added by dcf 4 years ago.
Screenshot of the 2nd, 4th, 6th, etc. time loading.

Download all attachments as: .zip

Change History (13)

Changed 4 years ago by dcf

Attachment: 1.png added

Screenshot of the 1st, 3rd, 5th, etc. time loading.

Changed 4 years ago by dcf

Attachment: 2.png added

Screenshot of the 2nd, 4th, 6th, etc. time loading.

comment:2 in reply to:  description Changed 4 years ago by dcf

Here are the different headers for alternating retrieval of the bandwidth URL. The garbled text appears to be because of a missing Content-Encoding: gzip, but it is also missing Last-Modified, Access-Control-Allow-Origin, Content-Type, and Cache-Control.

good:

Date: Mon, 28 Mar 2016 04:08:57 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Last-Modified: Mon, 28 Mar 2016 04:06:20 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json;charset=utf-8
Cache-Control: public, max-age=2400
Content-Encoding: gzip
Content-Length: 2545
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive

garbled:

Date: Mon, 28 Mar 2016 04:12:10 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Content-Length: 2545
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

The difference in the details URL headers is similar, but the good one does not have Content-Encoding: gzip to begin with, which explains why it doesn't get garbled when Content-Type is missing.

good:

Date: Mon, 28 Mar 2016 04:17:04 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Last-Modified: Mon, 28 Mar 2016 04:06:20 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json;charset=utf-8
Cache-Control: public, max-age=1800
Content-Length: 519
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

bad:

Date: Mon, 28 Mar 2016 04:17:24 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Content-Length: 519
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

comment:3 Changed 4 years ago by dcf

Summary: Onionoo bandwidth doc switches between readable and garbled responses when refreshingOnionoo doesn't send certain headers on even-numbered responses

comment:4 Changed 4 years ago by karsten

Owner: set to karsten
Status: newaccepted

Interesting bug! Here's a preliminary analysis:

  • This bug seems unrelated to even-numbered responses and only dependent on the headers included in the request.
  • Two headers that must be included in the request to reproduce this problem are "Accept-Encoding: gzip" and "Cache-Control: max-age=0". Note that the second header disables Apache caching and lets Jetty answer all requests.
  • Another header that must be included to reproduce this problem and that produces different results based on its value is "If-Modified-Since:":
    • older than "Last-Modified" time: 200 result, gzipped with "Content-Encoding" header in response. All is fine here.
    • same as "Last-Modified" time: 200 result, gzipped without "Content-Encoding" header in response. This is a bug.
    • newer than "Last-Modified" time: 304 result without body. All is fine.

I don't know yet what exactly leads to the bug case.

In order to reproduce this problem, fetch an Onionoo document and replace the timestamps below with values smaller/equal to/larger than the returned "Last-Modified" timestamp.

curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:53:59 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null
curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:54:00 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null
curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:54:01 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null

comment:5 Changed 4 years ago by karsten

So, I don't have a fix yet, but I have a theory: Apache's mod_cache module might not play well with Jetty's GzipFilter. We could try taking out that Jetty filter and enabling mod_deflate in Apache. If this still seems like a reasonable idea tomorrow, I'll try it out.

comment:6 Changed 4 years ago by karsten

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

comment:7 in reply to:  6 ; Changed 4 years ago by dcf

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

I tried again just now with Tor Browser 6.0a4 and Chromium 49.0 and I get the same alternating pattern.
https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534

comment:8 in reply to:  7 Changed 4 years ago by teor

Replying to dcf:

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

I tried again just now with Tor Browser 6.0a4 and Chromium 49.0 and I get the same alternating pattern.
https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534

I can confirm I see this same pattern using Tor Browser via several different exits.
The first request via a new exit is always plain text, the first refresh is garbled, the second refresh is plain text, the third refresh is garbled, ...

comment:9 in reply to:  6 Changed 4 years ago by cypherpunks

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

https://gitweb.torproject.org/onionoo.git/log/ says the last commit was in February and https://gitweb.torproject.org/onionoo.git/tree/etc/web.xml.template#n55 still contains the GzipFilter. Am i looking at the wrong repository?

comment:10 Changed 4 years ago by karsten

Severity: NormalMajor

Hmmmm. I don't have a solution yet, but here's a quick update: I can confirm that the problem still persists, and the disabled GzipFilter is not reflected in Git yet, because I only took it out locally on the server without committing. I'll have to take a closer look at this and will report back once I have found something.

comment:11 Changed 3 years ago by karsten

Resolution: fixed
Status: acceptedclosed

I believe this issue was resolved a few months ago by switching from Apache to Varnish. At least I cannot reproduce it anymore. Optimistically closing, please re-open if the issue still persists, ideally with instructions for reproducing it. Thanks everyone!

Note: See TracTickets for help on using tickets.