Opened 23 months ago

Last modified 4 months ago

#23731 new defect

some websites block requests by HTTP User-Agent

Reported by: cypherpunks Owned by: mrphs, alison
Priority: Medium Milestone:
Component: Community/Outreach Version:
Severity: Normal Keywords: User-Agent, blocking
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Some websites will use the HTTP User-Agent field to determine whether the browser is allowed to visit. Apparently, this is done in the name of "security," with the assumption that "insecure" browsers should not be allowed to visit the site. (Probably, we should not assume that this has anything to do with security per se; perhaps it is really about correctness.)

The approach is neither necessary nor sufficient to achieve the objectives of the site operators. It is unnecessary because web standards define how browsers ought to behave, and any correctness should be determined by adherence to the standards, not by whether the name of the browser in question happens to be on some list. It is insufficient because circumventing the filter is trivial and can be done simply by changing the HTTP User-Agent, which users of Tor Browser can edit by editing general.useragent.override on the about:config page.

The default User-Agent that ships with Tor Browser appears to be:

Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0

This seems to work well if we want to appear to be using Firefox. However, sometimes Firefox is not on the approved list for websites such as those described above. (At least one website approves Safari and Chrome while rejecting IE and Firefox.)

Browser-Info provides a list of popular HTTP User-Agents, and choosing from this list we can configure Tor Browser to appear to be Safari by changing general.useragent.override to:

Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1

Web users who do not value privacy may indeed have the option, inconvenient as it may be, to switch to a browser that satisfies the requirements of the site. Tor users do not have such an option, because there is only one Tor Browser (it happens to be based on Firefox).

We need to make it easier for everyday Tor users to circumvent filtering of this variety. Some possible suggestions:

  1. Maintain a list of popular User-Agents and provide an option in the drop-down onion menu on Tor Browser to choose which one to be for this site.
  2. Establish a Wiki page that allows users to report websites that block specific browsers by User-Agent, along with examples of User-Agent strings, if any, that work.
  3. Where appropriate, liaise with the websites in question, particularly if they are popular ones, to make sure that Tor Browser is on the list of suitable browsers.

Child Tickets

Change History (2)

comment:1 Changed 23 months ago by gk

Component: Applications/Tor BrowserCommunity/Outreach
Owner: changed from tbb-team to mrphs, alison

I don't think we'll ship such a drop-down menu with different user agents to pick but am fine with 2. and 3. getting done. Moving this to the outreach component.

comment:2 Changed 4 months ago by cypherpunks

websites that reject Firefox is internet vandalism. wontfix

  1. no, makes you stand out.
  2. we have a list of sites blocking tor
  3. if they block Firefox, website is wrong anyway. however, tbb might look more equal to Firefox in detail by useragent
Note: See TracTickets for help on using tickets.