Opened 6 years ago

Closed 6 years ago

#8670 closed defect (fixed)

SSL Observatory request flood when server unreachable (Firefox)

Reported by: karukoff Owned by: pde
Priority: Very High Milestone:
Component: HTTPS Everywhere/EFF-HTTPS Everywhere Version: HTTPS-E 3.1.4
Severity: Keywords:
Cc: g.koppen@… Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I am using HTTPS Everywhere 3.1.4 and Firefox 20.0 under Linux. While browsing, I noticed that my browsing suddenly got really slow. All websites would resolve, but data transfer was really slow or stalled altogether, so no pages would finish loading.

I checked tcpdump to see what was going on, and this is what I saw (snippet, goes on like this for as long as tcpdump was running):

`19:27:45.224467 IP 192.168.1.65.53664 > 64.147.188.18.443: Flags [S], seq 814724169, win 14600, options [mss 1460,sackOK,TS val 2193005 ecr 0,nop,wscale 7], length 0
19:27:45.307799 IP 192.168.1.65.53647 > 64.147.188.18.443: Flags [S], seq 298083676, win 14600, options [mss 1460,sackOK,TS val 2193030 ecr 0,nop,wscale 7], length 0
19:27:45.394473 IP 192.168.1.65.53595 > 64.147.188.18.443: Flags [S], seq 3914974079, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.394487 IP 192.168.1.65.53596 > 64.147.188.18.443: Flags [S], seq 1679001607, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.394492 IP 192.168.1.65.53597 > 64.147.188.18.443: Flags [S], seq 3842378412, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.394495 IP 192.168.1.65.53598 > 64.147.188.18.443: Flags [S], seq 3268035818, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.394499 IP 192.168.1.65.53599 > 64.147.188.18.443: Flags [S], seq 2288422783, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.394503 IP 192.168.1.65.53600 > 64.147.188.18.443: Flags [S], seq 1924775660, win 14600, options [mss 1460,sackOK,TS val 2193056 ecr 0,nop,wscale 7], length 0
19:27:45.437796 IP 192.168.1.65.53665 > 64.147.188.18.443: Flags [S], seq 298888272, win 14600, options [mss 1460,sackOK,TS val 2193069 ecr 0,nop,wscale 7], length 0
19:27:45.474461 IP 192.168.1.65.53633 > 64.147.188.18.443: Flags [S], seq 1715559767, win 14600, options [mss 1460,sackOK,TS val 2193080 ecr 0,nop,wscale 7], length 0
19:27:45.474469 IP 192.168.1.65.53634 > 64.147.188.18.443: Flags [S], seq 1860803928, win 14600, options [mss 1460,sackOK,TS val 2193080 ecr 0,nop,wscale 7], length 0
19:27:45.474472 IP 192.168.1.65.53635 > 64.147.188.18.443: Flags [S], seq 1134654807, win 14600, options [mss 1460,sackOK,TS val 2193080 ecr 0,nop,wscale 7], length 0
19:27:45.474475 IP 192.168.1.65.53636 > 64.147.188.18.443: Flags [S], seq 2496139043, win 14600, options [mss 1460,sackOK,TS val 2193080 ecr 0,nop,wscale 7], length 0
19:27:45.474478 IP 192.168.1.65.53637 > 64.147.188.18.443: Flags [S], seq 52809697, win 14600, options [mss 1460,sackOK,TS val 2193080 ecr 0,nop,wscale 7], length 0
19:27:45.487795 IP 192.168.1.65.53638 > 64.147.188.18.443: Flags [S], seq 1193905635, win 14600, options [mss 1460,sackOK,TS val 2193084 ecr 0,nop,wscale 7], length 0
19:27:45.521130 IP 192.168.1.65.53648 > 64.147.188.18.443: Flags [S], seq 2435494456, win 14600, options [mss 1460,sackOK,TS val 2193094 ecr 0,nop,wscale 7], length 0
19:27:45.521137 IP 192.168.1.65.53649 > 64.147.188.18.443: Flags [S], seq 1076454250, win 14600, options [mss 1460,sackOK,TS val 2193094 ecr 0,nop,wscale 7], length 0
19:27:45.521140 IP 192.168.1.65.53650 > 64.147.188.18.443: Flags [S], seq 4273166310, win 14600, options [mss 1460,sackOK,TS val 2193094 ecr 0,nop,wscale 7], length 0
19:27:45.521142 IP 192.168.1.65.53651 > 64.147.188.18.443: Flags [S], seq 2238779580, win 14600, options [mss 1460,sackOK,TS val 2193094 ecr 0,nop,wscale 7], length 0
19:27:45.821142 IP 192.168.1.65.53601 > 64.147.188.18.443: Flags [S], seq 1218150538, win 14600, options [mss 1460,sackOK,TS val 2193184 ecr 0,nop,wscale 7], length 0
19:27:45.821157 IP 192.168.1.65.53602 > 64.147.188.18.443: Flags [S], seq 1564399171, win 14600, options [mss 1460,sackOK,TS val 2193184 ecr 0,nop,wscale 7], length 0
19:27:45.874464 IP 192.168.1.65.53535 > 64.147.188.18.443: Flags [S], seq 3871568603, win 14600, options [mss 1460,sackOK,TS val 2193200 ecr 0,nop,wscale 7], length 0
19:27:45.874474 IP 192.168.1.65.53536 > 64.147.188.18.443: Flags [S], seq 1200317769, win 14600, options [mss 1460,sackOK,TS val 2193200 ecr 0,nop,wscale 7], length 0
19:27:45.874477 IP 192.168.1.65.53537 > 64.147.188.18.443: Flags [S], seq 1066099685, win 14600, options [mss 1460,sackOK,TS val 2193200 ecr 0,nop,wscale 7], length 0
19:27:45.874480 IP 192.168.1.65.53538 > 64.147.188.18.443: Flags [S], seq 103573693, win 14600, options [mss 1460,sackOK,TS val 2193200 ecr 0,nop,wscale 7], length 0
19:27:45.874484 IP 192.168.1.65.53539 > 64.147.188.18.443: Flags [S], seq 2863165172, win 14600, options [mss 1460,sackOK,TS val 2193200 ecr 0,nop,wscale 7], length 0
19:27:45.927809 IP 192.168.1.65.53378 > 64.147.188.18.443: Flags [S], seq 1443651518, win 14600, options [mss 1460,sackOK,TS val 2193216 ecr 0,nop,wscale 7], length 0
19:27:46.077795 IP 192.168.1.65.53666 > 64.147.188.18.443: Flags [S], seq 3852535012, win 14600, options [mss 1460,sackOK,TS val 2193261 ecr 0,nop,wscale 7], length 0
19:27:46.077804 IP 192.168.1.65.53668 > 64.147.188.18.443: Flags [S], seq 566989182, win 14600, options [mss 1460,sackOK,TS val 2193261 ecr 0,nop,wscale 7], length 0
19:27:46.077807 IP 192.168.1.65.53669 > 64.147.188.18.443: Flags [S], seq 3777578631, win 14600, options [mss 1460,sackOK,TS val 2193261 ecr 0,nop,wscale 7], length 0
19:27:46.114462 IP 192.168.1.65.53639 > 64.147.188.18.443: Flags [S], seq 2001028625, win 14600, options [mss 1460,sackOK,TS val 2193272 ecr 0,nop,wscale 7], length 0
19:27:46.161132 IP 192.168.1.65.53652 > 64.147.188.18.443: Flags [S], seq 2738092749, win 14600, options [mss 1460,sackOK,TS val 2193286 ecr 0,nop,wscale 7], length 0
19:27:46.161140 IP 192.168.1.65.53653 > 64.147.188.18.443: Flags [S], seq 3553154323, win 14600, options [mss 1460,sackOK,TS val 2193286 ecr 0,nop,wscale 7], length 0
19:27:46.327796 IP 192.168.1.65.53640 > 64.147.188.18.443: Flags [S], seq 3162972276, win 14600, options [mss 1460,sackOK,TS val 2193336 ecr 0,nop,wscale 7], length 0
19:27:46.354904 IP 192.168.1.65.53670 > 64.147.188.18.443: Flags [S], seq 2952496245, win 14600, options [mss 1460,sackOK,TS val 2193344 ecr 0,nop,wscale 7], length 0
19:27:46.374464 IP 192.168.1.65.53654 > 64.147.188.18.443: Flags [S], seq 3991905334, win 14600, options [mss 1460,sackOK,TS val 2193350 ecr 0,nop,wscale 7], length 0
19:27:46.374473 IP 192.168.1.65.53655 > 64.147.188.18.443: Flags [S], seq 634040360, win 14600, options [mss 1460,sackOK,TS val 2193350 ecr 0,nop,wscale 7], length 0
19:27:46.374477 IP 192.168.1.65.53656 > 64.147.188.18.443: Flags [S], seq 4200584575, win 14600, options [mss 1460,sackOK,TS val 2193350 ecr 0,nop,wscale 7], length 0
19:27:46.567805 IP 192.168.1.65.53379 > 64.147.188.18.443: Flags [S], seq 1734267859, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567819 IP 192.168.1.65.53380 > 64.147.188.18.443: Flags [S], seq 2166714112, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567823 IP 192.168.1.65.53381 > 64.147.188.18.443: Flags [S], seq 1752055028, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567827 IP 192.168.1.65.53382 > 64.147.188.18.443: Flags [S], seq 3208704690, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567831 IP 192.168.1.65.53383 > 64.147.188.18.443: Flags [S], seq 1871889640, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567834 IP 192.168.1.65.53384 > 64.147.188.18.443: Flags [S], seq 1176559303, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.567838 IP 192.168.1.65.53385 > 64.147.188.18.443: Flags [S], seq 542685111, win 14600, options [mss 1460,sackOK,TS val 2193408 ecr 0,nop,wscale 7], length 0
19:27:46.587798 IP 192.168.1.65.53657 > 64.147.188.18.443: Flags [S], seq 1126902578, win 14600, options [mss 1460,sackOK,TS val 2193414 ecr 0,nop,wscale 7], length 0
19:27:46.587808 IP 192.168.1.65.53658 > 64.147.188.18.443: Flags [S], seq 2926788370, win 14600, options [mss 1460,sackOK,TS val 2193414 ecr 0,nop,wscale 7], length 0`

What is basically happening is that my IP (192.168.1.65) is sending TCP SYN packets at a very high rate (~35 req/sec) to 64.147.188.18 (observatory6.eff.org) port 443 (HTTPS), probably depleting Firefox's resources and making browsing impossible. I suspect this is some sort of a software bug / infinite loop scenario within the SSL Obserbatory component.

I disabled HTTPS Everywhere and restarted Firefox, which stopped the flood and all websites started loading normally again. Then, I re-enabled HTTPS Everywhere and again restarted Firefox, and now it's again working fine without flooding or anything. Moreover, I can't reproduce the situation that lead to the flood even if I tried re-visiting the websites I think I was visiting before the flood happened.

Possible(?) problem pointer:

  • I am using another add-on called FoxyProxy to enable retrieving Tor Hidden Services (pattern: *.onion/*) through a Tor SOCKS proxy (I know this is not a fully secure setup). I am NOT using Torbutton or other Tor-related add-ons. Just before the flood happened, I was trying to browse a .onion service. This MIGHT have something to do with the flood, but I don't think the .onion service was using HTTPS, though I can not be absolutely sure. I have never seen a .onion service use HTTPS, because it is a redundant form of encryption for them AFAIK.

Above timestamps correlate to UTC 16:27:45/46 on 9 April, 2013. Public IP address available if needed.

---

My SSL Observatory settings:
[x] Use the Observatory?
[x] Check certificates even if Tor is not available (the other radio option is unselectable/disabled)
[x] When you see a new cert, tell the Observatory which ISP you are connected to
[ ] Submit and check self-signed certs
[x] Submit and check certs signed by non-standard root CAs
[ ] Submit and check certs for non-public DNS names

Child Tickets

Change History (6)

comment:1 Changed 6 years ago by karukoff

Oh okay, now it happened again, my machine started flooding observatory6.eff.org soon after resuming browsing and after submitting the above report (though not with quite as high rate). This time I wasn't using any *.onion services, so I think that can be ruled out.

What I noticed though, is that the Observatory server is currently down for me:

$ nc -v observatory6.eff.org 443
nc: connect to observatory6.eff.org port 443 (tcp) failed: Connection timed out

This is probably the reason why the code enters the flood loop. Some form of give-up logic should probably be added in case the Observatory database is unreachable.

comment:2 Changed 6 years ago by karukoff

Summary: SSL Observatory request flood (Firefox)SSL Observatory request flood when server unreachable (Firefox)

comment:3 Changed 6 years ago by schoen

Priority: normalcritical

I encountered this problem myself and have been discussing it with other developers. The performance impact was really quite severe. Upgrading priority to "critical".

comment:4 Changed 6 years ago by gk

Cc: g.koppen@… added

comment:5 Changed 6 years ago by pde

We believe this should be fixed in the 4.0development.7 and 3.2.1 releases that just went out. Our solution was to bound the number of outstanding XHRs per client, though there may also be serverside fixes we need for the mysterious TCP socket exhaustion that we're seeing.

comment:6 Changed 6 years ago by pde

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.