Onionoo doesn't send certain headers on even-numbered responses

When I load this URL, the first time I get meaningful output:

But if I hit Ctrl+R to refresh, I get this garbled (maybe compressed?) output instead: If I refresh again, it goes back to the readable version, and if I refresh yet again, it switches back to the garbled version. I can keep switching back and forth.

The same happens if I click the refresh icon in the address bar. I tried it in Tor Browser 6.0a4 and Chromium 49.0, and it happens in both.

The garbled version additionally causes this to be printed to the console: The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.

With another type of document, namely https://onionoo.torproject.org/details?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534, the text remains readable while repeatedly refreshing, but the "character encoding" message appears in the console in alternating refreshes.

added component::metrics/onionoo owner::karsten priority::medium resolution::fixed severity::major status::closed type::defect labels

Trac:

Screenshot of the 1st, 3rd, 5th, etc. time loading.

Trac:

Screenshot of the 2nd, 4th, 6th, etc. time loading.

Forgot to post the URL that causes the problem. It is https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534

Here are the different headers for alternating retrieval of the bandwidth URL. The garbled text appears to be because of a missing Content-Encoding: gzip, but it is also missing Last-Modified, Access-Control-Allow-Origin, Content-Type, and Cache-Control.

good:

Date: Mon, 28 Mar 2016 04:08:57 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Last-Modified: Mon, 28 Mar 2016 04:06:20 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json;charset=utf-8
Cache-Control: public, max-age=2400
Content-Encoding: gzip
Content-Length: 2545
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive

garbled:

Date: Mon, 28 Mar 2016 04:12:10 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Content-Length: 2545
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

The difference in the details URL headers is similar, but the good one does not have Content-Encoding: gzip to begin with, which explains why it doesn't get garbled when Content-Type is missing.

good:

Date: Mon, 28 Mar 2016 04:17:04 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Last-Modified: Mon, 28 Mar 2016 04:06:20 GMT
Access-Control-Allow-Origin: *
Content-Type: application/json;charset=utf-8
Cache-Control: public, max-age=1800
Content-Length: 519
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

bad:

Date: Mon, 28 Mar 2016 04:17:24 GMT
Server: Jetty(8.y.z-SNAPSHOT)
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15768000
Content-Length: 519
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

Trac:
Summary: Onionoo bandwidth doc switches between readable and garbled responses when refreshing to Onionoo doesn't send certain headers on even-numbered responses

Interesting bug! Here's a preliminary analysis:

• This bug seems unrelated to even-numbered responses and only dependent on the headers included in the request.
• Two headers that must be included in the request to reproduce this problem are "Accept-Encoding: gzip" and "Cache-Control: max-age=0". Note that the second header disables Apache caching and lets Jetty answer all requests.
• Another header that must be included to reproduce this problem and that produces different results based on its value is "If-Modified-Since:":
  • older than "Last-Modified" time: 200 result, gzipped with "Content-Encoding" header in response. All is fine here.
  • same as "Last-Modified" time: 200 result, gzipped without "Content-Encoding" header in response. This is a bug.
  • newer than "Last-Modified" time: 304 result without body. All is fine.

I don't know yet what exactly leads to the bug case.

In order to reproduce this problem, fetch an Onionoo document and replace the timestamps below with values smaller/equal to/larger than the returned "Last-Modified" timestamp.

curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:53:59 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null
curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:54:00 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null
curl -H "If-Modified-Since: Mon, 28 Mar 2016 12:54:01 GMT" -H "Accept-Encoding: gzip" -H "Cache-Control: max-age=0" -v https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534 > /dev/null

Trac:
Owner: N/A to karsten
Status: new to accepted

So, I don't have a fix yet, but I have a theory: Apache's mod_cache module might not play well with Jetty's GzipFilter. We could try taking out that Jetty filter and enabling mod_deflate in Apache. If this still seems like a reasonable idea tomorrow, I'll try it out.

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

I tried again just now with Tor Browser 6.0a4 and Chromium 49.0 and I get the same alternating pattern. https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534

Replying to dcf:

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again?

I tried again just now with Tor Browser 6.0a4 and Chromium 49.0 and I get the same alternating pattern. https://onionoo.torproject.org/bandwidth?lookup=88F745840F47CE0C6A4FE61D827950B06F9E4534

I can confirm I see this same pattern using Tor Browser via several different exits. The first request via a new exit is always plain text, the first refresh is garbled, the second refresh is plain text, the third refresh is garbled, ...

Replying to karsten:

So, I took out that Jetty filter a week ago and learned yesterday that Apache's mod_deflate module was enabled all the time. I think everything is working as expected now. Can you check again? https://gitweb.torproject.org/onionoo.git/log/ says the last commit was in February and https://gitweb.torproject.org/onionoo.git/tree/etc/web.xml.template#n55 still contains the GzipFilter. Am i looking at the wrong repository?

Hmmmm. I don't have a solution yet, but here's a quick update: I can confirm that the problem still persists, and the disabled GzipFilter is not reflected in Git yet, because I only took it out locally on the server without committing. I'll have to take a closer look at this and will report back once I have found something.

Trac:
Severity: Normal to Major

I believe this issue was resolved a few months ago by switching from Apache to Varnish. At least I cannot reproduce it anymore. Optimistically closing, please re-open if the issue still persists, ideally with instructions for reproducing it. Thanks everyone!

Trac:
Resolution: N/A to fixed
Status: accepted to closed

closed

mentioned in issue #18754 (moved)