Opened 7 months ago

Closed 7 months ago

#9713 closed defect (fixed)

HTTPS Everywhere 4.0development.11 causes google.com OCSP meltdown

Reported by: erinn Owned by: micahlee
Priority: normal Milestone:
Component: EFF-HTTPS Everywhere Version:
Keywords: Cc: mikeperry, arma, eroseman@…, agl@…, MB, schoen, dtauerbach
Actual Points: Parent ID:
Points:

Description

On our blog we have several users complaining that they are sending non-stop connection requests to clients1.google.com and they say that downgrading their version of HTTPS Everywhere alleviates the problem.

The same problem: at times, perhaps after some inactivity delay, there appear some non-stop connection requests to clients1.google.com:443.
Why??? I have no business connecting to Google.
Look for yourself - monitor the circuits/connections in the Vidalia's Network Map... If the requests are seen, they stop only after Tor Browser is closed.

(This one is interesting because the user implies there are no deliberate connections to Google being made.)

with the dev version of https everywhere in the latest tor build I get (like the poster above) constant connections to clients1.google.com, after downgrading https everywhere to the stable version these connections don't show up.

One user says that he or she can reliably reproduce it by visiting gmail. Just the front page is enough to trigger it, but logging in is even worse.

Another user says:

the earlier reported non-stop outgoing https connections to clients1.google.com seemed to happen with the latest TBB x64 on Linux, but only _after the browser Add-ons were updated manually_ - that loaded the latest dev version of Https-Everywhere.
After I manually replaced it with the previous "4.0development.9" version that was on hand, the weird connections are no more.

For my part, I have been unable to reproduce this bug on Linux, even when logging into (and interacting with) gmail. I haven't tried manually updating any of the extensions yet.

https://blog.torproject.org/blog/new-tor-02417-rc-packages

Child Tickets

Change History (13)

comment:1 Changed 7 months ago by cypherpunks

comment:2 Changed 7 months ago by cypherpunks

I have tried the following.

Ubuntu 10.04.4 and64: TBB 2.4.17-beta-1 32-bit and 64-bit
Ubuntu 12.04.3 amd64: TBB 2.4.17-beta-1 32-bit and 64-bit
Ubuntu 13.04 i386: TBB 2.4.17-beta-1 32-bit

In all cases I trigger the fault simply by trying to visit https://www.google.com/.

The problem appears to occur during the OCSP. Tor Browser submits a HTTP POST for http://clients1.google.com/ocsp. This is some how failing. TBB should then move up the certificate hierarchy to POST http://gtglobal-ocsp.geotrust.com/. This doesn’t happen. TBB repeatedly attempts to access clients1.google.com. This happens even after a New Identity. The only way I could stop it was to toggle Work Offline, to work offline then back online.

Disabling the HTTPS Everywhere ruleset "Google Services" stops this from happening. This ruleset looks particularly complicated, and I don't fancy wading my way through it. But, I'm going to have a look...

As I find it so easy to trigger and completely consistent, I find it weird that only some people are seeing this. As you can probably tell from above, I'm an Ubuntu person. I'm tempted to see if it occurs with other Linux distributions on my hardware. Is it a race condition?

comment:3 follow-up: Changed 7 months ago by erinn

I still haven't been able to trigger it at all. FWIW, I'm using Debian (unstable).

comment:4 Changed 7 months ago by cypherpunks

cd /tmp/
git clone --depth 1 git://git.torproject.org/https-everywhere.git
tar -zxf ~/tor-browser-gnu-linux-x86_64-2.3.25-12-dev-en-US.tar.gz
cd tor-browser_en-US/
rm -rf Data/profile/extensions/https-everywhere@…/
mkdir Data/profile/extensions/https-everywhere@…/
tar -cf - -C ../https-everywhere/src/ . | tar -xf - -C Data/profile/extensions/https-everywhere@…/
rm Data/profile/extensions/https-everywhere@…/chrome/content/rules/make-trivial-rule
./start-tor-browser

load https://news.google.com/news?ned=us
ta da!

comment:5 Changed 7 months ago by pde

  • Cc mikeperry arma added
  • Owner changed from pde to micahlee
  • Status changed from new to assigned

In case anyone was wondering, clients1.google.com is the domain that is used for google search suggestions while typing queries in a search box (including the one built into Firefox).

comment:6 in reply to: ↑ 3 Changed 7 months ago by cypherpunks

Replying to erinn:

I still haven't been able to trigger it at all. FWIW, I'm using Debian (unstable).


Further to comment 2, I've tried Debian Testing amd64 in VirtualBox. The problem is identical. I've even dug up an ancient, much slower Windows XP machine, and it's just the same there. So, it doesn't seem to be a timing issue.

Erinn, I've got to ask: Do you have Online Certificate Status Protocol disabled in your Tor Browser? That, of course, also stops the problem from occurring.

I've now had a look at the Google Services ruleset. It does re-write for the clients1.google.com domain.

	<rule from="^http://(apis|appengine|books|calendar|cbks0|checkout|chrome|clients[12]|code|[\w-]+\.corp|developers|dl|docs\d?|drive|encrypted|encrypted-tbn[123]|feedburner|fiber|gg|glass||health|helpouts|history|(?:hosted)?talkgadget|investor|lh\d|(?:chatenabled\.)?mail|pack|pki|play|plus(?:\.sandbox)?|plusone|productforums|profiles|safebrowsing-cache|cert-test\.sandbox|sb-ssl|script|security|servicessites|sites|spreadsheets\d?|support|talk|tools)\.google\.com/"
		to="https://$1.google.com/" />

I edited that rule to remove "clients[12]|", put an edited copy of the ruleset in the HTTPSEverywhereUserRules directory in Tor Browser's profile and disabled the copy built in to the extension. The problem went away.

I don't think HTTPS Everywhere should be touching any of the requests that are part of OCSP. All responses that are not error codes are signed anyway.

comment:7 Changed 7 months ago by pde

  • Cc eroseman@… agl@… added

CC'ing a couple of knowledgeable Googlers.

I'm on vacation, so I was hoping other people might be able to figure this out, but it seems like they aren't :/

It sounds like google has added an OCSP responder on a domain that was previously used for search suggestions. We can't remove coverage for clients1.google.com without leaving people's partially typed search queries exposed. So we need a targetted exclusion for something like http://clients[0-6].google.com/ocsp .

I'm going to try to reproduce and then test that possible fix.

comment:8 Changed 7 months ago by cypherpunks

Google's SSL certificates now have an OCSP responder on http://clients1.google.com/ocsp (previously they didn't have an OCSP pointer).

OCSP responses are signed and the OCSP responder serves on HTTPS as well:

curl https://clients1.google.com/ocsp/MG4wbDBFMEMwQTAJBgUrDgMCGgUABBTy4Gr5hYodjXCbSRkjeqm1Gih+ZAQUSt0GFhu89mi1dvWBtrtiGrpagS8CCAqfIRlqRC5FoiMwITAfBgkrBgEFBQcwAQIEEgQQBauuhzzvamPnjgfhvS7VVw== | hexdump -C

I don't know why things would go so wrong in the client when writing those requests.

comment:9 Changed 7 months ago by pde

  • Summary changed from Users report HTTPS Everywhere 0.development.11 in some sort of clients1.google.com loop? to HTTPS Everywhere 4.0development.11 causes google.com OCSP meltdown

I was able to reproduce this, and this fixes it (I also did this which isn't necessary but maybe I'll leave there because I'm paranoid).

This bug seems to be a combination of the fact that we rewrote an OCSP request and the resulting HTTPS url required an OCSP request to the same server. Probably Firefox is launching an infinite number of distinct OCSP requests. Mike, lmk if you think of a way to use nsIContentPolicy to ensure that we never touch OCSP!

The trigger: it seems as though MB added a whole-domain rewrite for clients[12].google.com to the Google Services ruleset; it was merged and shipped in 4.0development.11. As Cypherpunks noted above, that's causing the problem and is separate from the search-specific rewrites we've had on clients[0-9].google.com for a long time

This is only affects our development branch users, but it is REEEEALLY nasty. We need a development release as soon as humanly possible!

Last edited 7 months ago by pde (previous) (diff)

comment:10 Changed 7 months ago by pde

  • Cc MB added

comment:11 Changed 7 months ago by pde

  • Cc schoen dtauerbach added

Another way to avoid this in future would be to feed in a list of all OCSP responder urls from the SSL Observatory, and ensure that none of them are rewritten to HTTPS.

comment:12 Changed 7 months ago by micahlee

Thanks pde! I'm starting to do a new development release now.

comment:13 Changed 7 months ago by micahlee

  • Resolution set to fixed
  • Status changed from assigned to closed

4.0development.12 released with this bug fix.

Note: See TracTickets for help on using tickets.