Opened 6 months ago

Closed 6 months ago

#33046 closed defect (fixed)

Test ETP enabled in Tor Browser

Reported by: sysrqb Owned by: sysrqb
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-performance
Cc: tbb-team Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Last year I ran some experiments comparing the loading of pages with Enhanced Tracking Protection (ETP) disabled and then with it enabled. The results from these experiments were not statistically significant, but the preliminary results showed that ETP noticeably reduces the best and average case timing, but the variance is still problematic.

On the bright side, this is a good starting point. From an email I sent:

I started measuring Tor Browser's
speed while loading various webpages. This was with the goal of
comparing Tor Browser with the measurements Mozilla made two years ago
[0] when they released Quantum (and compared Firefox and Chrome). It
uses benchmarks provided by Firefox's Javascript API (using the
Navigation Timing API).

It seems like Tracking Protection does has a noticable impact on
page-load time. I still have the feeling that investigating ad and
tracker blockers is a worthwhile initiative (especially if we want to
remain relevant). There are some interesting results that require
further analysis because I can't explain them right now.

In the future, it would be very helpful if we extended these
measurements by including tor controller events related to circuit
build and stream sentconnect/connected events. We should also run these
tests from different geographic areas (particularly locations with low
throughput connections).

My takeaway from this is if (ideally) pages should load with 6 seconds
(as was Mozilla's target [0]), then I believe this is achievable as the
other network improvements (correcting weighting, solving bottlenecks,
etc.) are rolled out. In addition, as usual, this will require some
advocacy for helping promote tor-friendly websites.

----

Mozilla's measurements used Selenium for automating the tests. Luckily,
Kushal already made a Tor Browser Selenium driver[1], so adapting
Mozilla's test[2] was relatively simple.

These measurements used a slightly non-standard Tor Browser
configuration. The Navigation Timing API is a fingerprinting vector, so
it is disabled by default. |privacy.resistFingerprinting| was disabled
for these tests. Similarly, NoScript's XSS protection required manual
input while pages loaded, so the XSS protection was disabled for these
tests.

Two sets of tests were run. The first test did not change any other Tor
Browser settings. The second test enabled Tracking Protection. Each test
loaded 20 webpages[3] (19 webpages in that file plus
http://newsweek.com/). The list of webpages is shuffled before each run,
and Tor Browser is restarted between each run.

[0]
https://hacks.mozilla.org/2017/11/comparing-browser-page-load-time-an-introduction-to-methodology/
[1] https://github.com/webfp/tor-browser-selenium
[2] https://github.com/onkeltom/browser_pageloadspeed
[3]
https://github.com/onkeltom/browser_pageloadspeed/blob/master/news.txt
[4] https://w3c.github.io/navigation-timing/#processing-model

#30939 is a consequence of this work.

I described in another email how the tests were run:

I pushed a branch page_load_timing on
https://github.com/sysrqb/tor-browser-selenium/

It requires the same setup configuration as described in the README. I
installed the dependencies with --user.
  - pip install --user tbselenium
  - pip install --user -r tor-browser-selenium/requirements-dev.txt
  - Downloaded geckodriver from the Github repo

I didn't use xvfb (simply for convenience), so I ran the page-load test
directly with:

$ NO_XVFB=1 TBB_PATH=~/tor-browser_en-US/ py.test tor-browser-selenium/tbselenium/test/test_pageload.py

Change TBB_PATH and path/to/test_pageload.py, as needed. Don't be
surprised if the browser doesn't launch immediately (it take 5-10
seconds on slower computers).

And, in case you're not aware, tor-browser-selenium currently only works
on Linux (the README says Debian/Ubuntu and I successfully used it on
Fedora). You'll need a system tor installed, too (or at least an
instance of tor running already listening on port 9050). I'd like to add
support for letting Tor Browser bootstrap and control its own tor in the
future.

See #32976 for a better way we get geckodriver.

Child Tickets

Attachments (1)

torbrowser_perf.py (25.2 KB) - added by sysrqb 6 months ago.

Download all attachments as: .zip

Change History (2)

Changed 6 months ago by sysrqb

Attachment: torbrowser_perf.py added

comment:1 Changed 6 months ago by sysrqb

Resolution: fixed
Status: assignedclosed

And attached script produced some graphs from the results.

Note: See TracTickets for help on using tickets.