Opened 9 years ago

Closed 8 years ago

Last modified 20 months ago

#2148 closed defect (fixed)

1.3.x: RefSpoofer fails on 5 test cases out of 12.

Reported by: T(A)ILS developers Owned by: mikeperry
Priority: Immediate Milestone: Torbutton: 1.3
Component: Applications/Torbutton Version: Torbutton: 1.3.0-alpha
Severity: Normal Keywords: TorbuttonIteration20110305 MikePerryIteration20110305
Cc: Actual Points: 8
Parent ID: Points: 6
Reviewer: Sponsor:

Description

I conducted a bunch of test on the new refSpoofer feature from version 1.3.0alpha. Here are the result, in 4 situations for each of the 3 modes.

A - nospoof B - smartspoof C - spoofblank
1 one.domain.tld/a -> one.domain.tld/b OK - sent OK - sent OK - not sent
2 domain.tld -> one.domain.tld BAD! - not sent BAD? - sent one.domain.tld OK - not sent
3 domain.tld -> www.domain.tld BAD! - not sent BAD! - not sent OK - not sent
4 google.com -> one.domain.tld BAD! - not sent OK - not sent OK - not sent

As you can see :

  • it is not leaking HTTP Referers when it shouldn't, except in case (B2) but it was not clear from the comments in the source code whether it should send it or not. I would say it should not.
  • the smartspoof mode works in the two most obvious cases (1) and (4) but the two cases (2) and (3) have to be better specified.
  • the nospoof fails is a non-ambiguous case where the user configure it to send Referers between different domains.

Child Tickets

Attachments (6)

torbutton-1.3.2pre4-alpha.xpi (470.3 KB) - added by mikeperry 8 years ago.
Should make the referer behavior more uniform
test-torbutton-1.3.2pre4-alpha.txt (3.3 KB) - added by T(A)ILS developers 8 years ago.
Test cases for torbutton-1.3.2pre4-alpha.xpi.
torbutton-1.3.2pre6-alpha.xpi (470.6 KB) - added by mikeperry 8 years ago.
test-torbutton-1.3.2pre6-alpha.txt (1.4 KB) - added by mikeperry 8 years ago.
test-torbutton-1.3.2pre6-alpha-tails.txt (1.6 KB) - added by T(A)ILS developers 8 years ago.
test-torbutton-1.3.2pre6-alpha-tails-https.txt (739 bytes) - added by T(A)ILS developers 8 years ago.

Download all attachments as: .zip

Change History (39)

comment:1 Changed 9 years ago by bee

HI TAILS!!!!!!!!!!!!

Try BeeFREE!!!!!!!! my addon!!!!!!!!!
Bee FREE is able also to forge referrers ("softly" keeping the subdomains, or "strictly", keeping only the top domains!!!!), but in its default configuration it works removing the referrers!!!!!!!!!!! You've got to use this key to change the way it works!!!!!!!!!!!!!!
http://honeybeenet.altervista.org/beefree/?id=114000#extensions.beefree.websites.default.header.referer.action
You can change it from the GUI too!!!!!!!!!!!!!!!!!!!!
Ha!!!! A forged referrer being send is always the hostname of the target host you're sending the HTTP request to, so even this way of working won't let leak information!!!!!!!!!!!!!! Filters of beefree can be configured to remove the referrers even when you're surfing between the pages of the same filtered website!!!!!! you've to look at the keys named "*.header.referer.action.boost" to change this!!! this is also a per-filter based configuration and there is no GUI for it!!!!!!! for example you've to create this integer key "extensions.beefree.website.generic.header.referer.action.boost" and set it to "1" if you want to enable this feature everywhere!!!!!!! so all referrers will be removed (or "FORGED" if you want!!!!!!!) every time on all pages even if hosted under the same "domain.name"!!!!!!!!!!!
YEAH!!!!!!!!!!

bye!!!!!!!!!!!!
~bee!!!!!!

comment:2 Changed 9 years ago by mikeperry

Priority: normalmajor

comment:3 Changed 9 years ago by mikeperry

Summary: RefSpoofer fails on 5 test cases out of 12.1.3.x: RefSpoofer fails on 5 test cases out of 12.

comment:4 Changed 9 years ago by mikeperry

Owner: changed from mikeperry to koryk
Status: newassigned

comment:5 Changed 8 years ago by mikeperry

Priority: majorcritical

comment:6 Changed 8 years ago by mikeperry

Based on my reading of this and my informal opinion of how this feature should work, I'm going to declare that case B2 and B3 are bugs.

I think it was Kory's intention to implement a superset of the same-origin policy here, not the same-origin policy itself. I think for the goal of making sure sites decide to render properly, case B3 *should* send a referrer, as should B2.

However, I would also suspect that case B2 should send the proper referrer. If the TAILS developers report correctly, it is sending the *destination* site in the referrer, and not the origin. Is this correct?

If so, that is also a bug. B2 should send the *source* site as the referrer.

comment:7 Changed 8 years ago by mikeperry

Points: 6

Reproduce: 2
Review: 1
Fix: 3

comment:8 Changed 8 years ago by mikeperry

Kory also reports he's having trouble reproducing these cases. Is this chart correct? Is row 2 possibly a typo?

comment:10 Changed 8 years ago by mikeperry

I can't reproduce case B2 or B3. According to both reading the source and from testing, it sends the unaltered referer in those cases. It's my estimation that this is correct behavior. It also sends the unaltered referer in the inverse case..

comment:11 Changed 8 years ago by T(A)ILS developers

Hi Mike,

When I reported the bug I was probably using an older version of Firefox, now I just tried again with version shipped in Debian squeeze, 3.5.16-4 and torbutton 1.3.1-alpha and as far as B column is concerned I get the same results.

In case B3, going from http://domain.tld/something.html to http://www.domain.tld/ I get :

domain.tld:80 xx.xx.xx.xx - - [15/Feb/2011:12:19:33 +0100] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

So no referrer are sent when the spec says it should send it. And I can agree with that.

In case B2, going from http://domain.tld/something.html to httphost.domain.tld/index.html I get :

host.domain.tld:80 xx.xx.xx.xx - - [15/Feb/2011:12:22:32 +0100] "GET /index.html HTTP/1.1" 304 - "http://host.domain.tld" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"

So a referrer from the *destination* host.domain.tld is sent, which is pretty weird indeed.

The spec is unclear but in my opinion it should not send any referrer in this case. I understand the choice made in the spec for base B3 since we can usually assume that domain.tld and www.domain.tld are run by the same entity. But in the more general case of different subdomains we should assume that it is not the case: they can be run by different entities, the DNS can point to different IPs, the admins and owners of the CMSes can be different. Thinking about two different wordpress.com blogs for example one.wordpress.com and two.wordpress.com, I might not want the admin of two.wordpress know that I'm coming from one.wordpress.com to visit her blog.

I would be glad to help you more debugging this. What extra information can I send you ?

comment:12 Changed 8 years ago by mikeperry

Keywords: TorbuttonIteration20110305 added; refspoofer removed
Priority: criticalblocker

comment:13 Changed 8 years ago by mikeperry

Owner: changed from koryk to mikeperry

comment:14 Changed 8 years ago by mikeperry

Keywords: MikePerryIteration20110305 added

comment:15 Changed 8 years ago by mikeperry

For your first example here, 'www' is a special case in the code for some reason. I think it should have sent *some* kind of referrer for this case though.. That may be a bug due to the special case of 'www'. I think that we should not special case 'www'. If we have to, I think we're probably doing something wrong.

'Not sending the referrer' in the case of the smart referrer code means it spoofs to the origin domain of the *destination* URL. Hence that is why you see the destination url in the second case. This makes intuitive sense to me, but it should be consistent.

I think this means that there are three steps to the solution of this general problem:

  1. Make the behaviour consistent wrt subdomains, with no special cases.
  2. Document all this in the design doc.
  3. Verify for sanity, adjusting the code as needed.

I think that solving this bug definitely means doing #1 here. I am thinking that 2 and 3 might be separate tickets.

comment:16 Changed 8 years ago by T(A)ILS developers

  1. About subdomains :

Ok, first we could recall why in the first place we're spoofing the referrer. For me we're spoofing to hide to the admins of site foo.tld (on the Apache or the CMS) that I was coming from site bar.tld to visit them, right? Hiding that from people sniffing my traffic while going from a private HTTPS site to a public HTTP site would is important too.

So why trying to do smartspoof across subdomains we should first ask ourselves: can we expect the admins of some.thing.tld to be the same as the admins of some.thing.else.tld and if the answer is « no » then spoof it.

Since I fear there are no simple answer to this question we should base our reasoning on what's being done on the Internet, plus take some extra precautions.

And I think it is wrong to assume that the admins of one.domain.tld are the same as the admins of two.domain.tld. Because, as I said before :

  • those two sites can be hosted on a different machine, through DNS,
  • even if hosted on the same machine, the admin of the CMS or the people able to view the stats of the site usually differ.

So I'm against sending any referrer while moving between subdomains.

I'm ok to keep that reasoning for the case www.domain.tld / domain.tld but we can usually assume they are administered by the same entity. But that might not always be the case so we could also say we don't same the referrer. I would be fine with that.

  1. About « Not sending the referrer » :

I really thing « Not sending the referrer » should mean « we're not sending a referrer » (like in the example of case B3 in my last comment) and not « we're sending a fake referrer » (like in the example of case B2 in my last comment). The referrer should just not be sent, the browser should behave like if we entered the URL by hand in the location bar or clicked on the link from another app.

So I'm not in favor of sending the « the origin domain of the *destination* URL » because I don't see the point. Even though I don't see major differences in the privacy implications of both options.

comment:17 in reply to:  16 Changed 8 years ago by bee

Replying to T(A)ILS developers:

So I'm against sending any referrer while moving between subdomains.

Hi TAILS!!!!!!!!!!!!!!!!!!!!!!!!
You're very smart!!!!!!!! YEAH!!!!!!!!!!!!!!!!!
Indeed we think in the same way!!!!!!!!!! and this the reason why BeeFREE (my addon!!) works in the very same way you're thinking!!!!!!!!!!!! I even wrote everything you wrote, in my blog when i've been asked for the same questions!!!!!!!!!!!!
http://honeybeenet.phpbb3now.com/viewtopic.php?f=14&t=237&p=1241#p1241
I knew that copying the code from my addon was a much more simple and very well working solution!!!!!!!! but mikeperry is always jealous of good suggestions!!!!!!!!!!!!

bye!!!!!!!!!!!!!
~bee!!!!!!!

comment:18 Changed 8 years ago by mikeperry

bee - in general, outgoing link clicks aren't something we're interested in blocking right now. It is an arms race that really needs to be planned out if we're even going to attempt it. We would need a way of describing how outgoing link tracking works on sites and providing a rule language that allows people to write rules to undo this tracking, like we have done with HTTPS Everywhere, and like has been done with AdBlock Plus. There are too many sites doing this to expect to make a dent otherwise. Sure, Google. Yahoo, and Mozilla are the big ones, but the reality is many other sites have their own implementations, too.

In this bug, we are only discussing the actual Referer header. You're right, this is only one side of the problem. But it is one that we *might* be able to solve, or at least improve, with a single, general change to browser behaviour.

Changed 8 years ago by mikeperry

Should make the referer behavior more uniform

comment:19 Changed 8 years ago by mikeperry

Status: assignedneeds_review

TAILS guys: ok, now the referer behavior should be more uniform. The attached .xpi is from origin/master 2589477ba1034c394d9ef74c33bd1123316da214. We may still want to change that behavior, but at least it is now easier to describe.

The referer is left as the default behavior if either the source or the destination hostname are full substrings of one another. Otherwise, the referer is spoofed to be the prefix of the destination url (scheme+host).

We may want to loosen this to remove the TLD, and/or the prefix domain, if the hostnames are short enough, before performing the suffix test. Thoughts?

Also, does this git revision behave as described for you?

comment:20 Changed 8 years ago by T(A)ILS developers

Hi Mike,

I like this version better. The smartspoof now behaves as expected while doing:

one.domain.tld/something → domain.tld (blank referrer)
domain.tld/something → one.domain.tld (blank referrer)

I'm fine with removing the the special case for www.

I guess now we'll have to find an agreement on what « not sending the referrer » means, as I said before in comment 16. Because the « intuitive sense » you advocated in comment 15 doesn't seem clear to me. But again I don't consider that a major security issue. Maybe it's personal taste as I usually prefer not saying anything instead of lying ;)

By the way, in the nospoof configuration we're still not sending the referrer in most of the cases. It's a status quo from my first report:

domain.tld → one.domain.tld (blank referrer)
google.com → domain.tld (blank referrer)
www.domain.tld → domain.tld (blank referrer)

comment:21 Changed 8 years ago by mikeperry

Woah, hold on here. Are you actually saying that this:

one.domain.tld/something → domain.tld (blank referrer)
domain.tld/something → one.domain.tld (blank referrer)

Because these exact cases are the only ones that *should ever* send a real referer now. Are you saying they do not? Which cases do send a referer now for you? Any? Something seems broken. Are you sure the scheme is the same in your tests? Referers do *not* get sent by default between https and http.

Re what "blank referrer" actually means, we chose to send the destination url of the site because we feel that this is least likely to break things. Some sites actually check the referrer and refuse to serve content if it is not as expected. This is also why we might strip the prefix between one.domain.tld and two.domain.tld, and possibly even .tld in a future release, but only if people report breakage.

Changed 8 years ago by T(A)ILS developers

Test cases for torbutton-1.3.2pre4-alpha.xpi.

comment:22 Changed 8 years ago by T(A)ILS developers

I just performed those tests again and I confirm: they do not send any referrer. You'll find in attachment the full results of the tests I performed on Monday. But again, from me those two cases are fine now. Are you going to answer my concern about subdomains at some point? I guess we should agree and what it should be doing first.

Re what "blank referrer" actually means, this sounds like a good reason to lie ;)

Changed 8 years ago by mikeperry

Changed 8 years ago by mikeperry

comment:23 Changed 8 years ago by mikeperry

Ok, something is wrong here. Either there is a bug in the 1.3.2-pre4-alpha xpi attached here initially, or there is another addon you have installed that is messing with your results. See my attached log where your two blank cases do not happen with the attached 1.3.2-pre6-alpha. AFAICT, the 1.3.2-pre4-alpha has the exact same code for this codepath, so I suspect an addon conflict on your end.

Try starting with a fresh firefox profile? ALso, all of my schemes were https to/from https, but that should not matter.

comment:24 Changed 8 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

I am going to declare this fixed, unless we can determine some specific breakage on the part of Torbutton that is causing the TAILS people to get different results.

comment:25 Changed 8 years ago by mikeperry

Actual Points: 8

Changed 8 years ago by T(A)ILS developers

comment:26 Changed 8 years ago by T(A)ILS developers

Sorry for the week-end break,

I performed the tests again on a Debian Live under Squeeze running this 1.3.2-pre6-alpha and no other extension. The results are 100 % coherent with the one I sent you last Wednesday and still disagree with yours on:

domain.tld → www.domain.tld
where you get I referrer but I don't.

domain.tld -> one.domain.tld
where you get a referrer but I don't.

one.domain.tld → domain.tld
where you get a referrer but I don't.

I tried on another server, with different websites, URLs and tests for the same use cases and I got the same results again. All my schemes are HTTP to HTTP.

What else could I try?

comment:27 Changed 8 years ago by mikeperry

Are you able to test using https schemes? That is what I tested with.. I'm not sure what else could be causing this. If that is not easy for you, I can try to set up new vhosts to test with http...

comment:28 Changed 8 years ago by mikeperry

Also, are you using plain a href links? Or are you testing img src or some other content element?

Changed 8 years ago by T(A)ILS developers

comment:29 Changed 8 years ago by T(A)ILS developers

Hi Mike,

I just performed the 3 problematic tests with HTTPS and it's the same again… No, I'm kidding ;) This time I'm sending same referrers as you do !!! See the attachment.

So it seems like it's behaving differently with HTTP and HTTPS. And yes, I always tested with absolute href HTML links say <a href="http://domain.tld/">http://domain.tld/</a></li>, and from https://xxx.domain.tld/referrer.html to https://domain.tld/.

It's good to feel that we're getting somewhere at last…

comment:30 Changed 8 years ago by mikeperry

Ah.. this just in: The JonDos people discovered this may be due to Polipo: #2902.

comment:31 Changed 8 years ago by T(A)ILS developers

Indeed, Tails uses polipo. Seems like our censorReferer = maybe setting is interfering. The one of us who did all the testing work should do it again with censorReferer = false.

comment:32 Changed 8 years ago by T(A)ILS developers

Hi Mike,

I did the tests again under Tails after changing the Polipo config to
'censorReferer = false' and now my results with plain HTTP perfectly match
yours.

domain.tld → www.domain.tld
Now I get an unspoofed referer as you do, good.

domain.tld → one.domain.tld
Now I get an unspoofed referer as you do.

one.domain.tld → domain.tld
Now I get an unspoofed referer as you do.

So now that we get the same results I guess this first part of the whole issue
is solved and understood.

Now, I'm insisting but I still disagree on sending unspoofed referer when moving
between one.domain.tld and domain.tld. Shall I open a new specific bug about
that, now that you closed this one?

comment:33 Changed 20 months ago by teor

Severity: Normal

Set all tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.