Opened 3 years ago

Closed 2 years ago

Last modified 17 months ago

#13313 closed enhancement (fixed)

Enable bundled fonts in Tor Browser

Reported by: dcf Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: tbb-fingerprinting-fonts, tbb-5.0a4, TorBrowserTeam201507R
Cc: anonym, intrigeri, gacar@…, brade, mcs, gk Actual Points:
Parent ID: #18097 Points:
Reviewer: Sponsor:

Description

Add the ability to include fonts with the browser and use them instead of system fonts.

This Firefox ticket is somewhat related:

dynamically load fonts packaged with Firefox
https://bugzilla.mozilla.org/show_bug.cgi?id=998844

Child Tickets

Attachments (8)

4.5-alpha-2-fonts-1.patch (40.7 KB) - added by dcf 3 years ago.
tor-browser patch against tor-browser-31.3.0esr-4.5-1-build1.
droid-sans-georgian-4.5-alpha-2-fonts-1.png (54.5 KB) - added by dcf 3 years ago.
Screenshot of 4.5-alpha-2-fonts-1 showing www.wikipedia.org and a copy-paste of the Fonts panel.
fontlist.log (14.9 KB) - added by dcf 3 years ago.
NSPR_LOG_MODULES=fontlist:5 for 4.5-alpha-2-fonts-1.
fontinit.log (1.1 KB) - added by dcf 3 years ago.
NSPR_LOG_MODULES=fontinit:5 for 4.5-alpha-2-fonts-1.
textrun.log (5.5 KB) - added by dcf 3 years ago.
NSPR_LOG_MODULES=textrun:5 for 4.5-alpha-2-fonts-1.
textrunui.log (13.5 KB) - added by dcf 3 years ago.
NSPR_LOG_MODULES=textrunui:5 for 4.5-alpha-2-fonts-1.
cmapdata.log (4.0 KB) - added by dcf 3 years ago.
NSPR_LOG_MODULES=cmapdata:5 for 4.5-alpha-2-fonts-1.
linux32diff (4.2 KB) - added by gk 2 years ago.

Download all attachments as: .zip

Change History (47)

comment:1 Changed 3 years ago by dcf

I started a branch to work on this:

https://gitweb.torproject.org/user/dcf/tor-browser-bundle.git/log/?h=fonts

This comment is being made when HEAD is tbb-4.0-alpha-3-fonts-1. Here is the overall diff so far.

At this point the branch only handles the linux bundles. linux is easier because we can control font behavior with a fonts.conf file without patching the browser source code. I added two fonts to the bundle, Droid and Lohit. Droid covers all the languages of TBB, and Lohit additionally covers various Indic scripts.

This is the main important change.

+export FONTCONFIG_PATH="${HOME}/TorBrowser/Data/fontconfig"
+export FONTCONFIG_FILE="fonts.conf"

It overrides global Fontconfig settings by an included configuration file that loads only those fonts included with the browser.

I uploaded bundles built from the branch.

https://people.torproject.org/~dcf/pt-bundle/4.0-alpha-3-fonts-1/

mac and windows are there, but they only include the font files, and don't actually use them. Here are the sizes of the bundles with included fonts:

40970478 TorBrowser-4.0-alpha-3-fonts-1-osx32_en-US.dmg
33539120 torbrowser-install-4.0-alpha-3-fonts-1_en-US.exe
34123852 tor-browser-linux32-4.0-alpha-3-fonts-1_en-US.tar.xz
35833512 tor-browser-linux64-4.0-alpha-3-fonts-1_en-US.tar.xz

Compare that to the size of 4.0-alpha-3 without included fonts:

34742225 TorBrowser-4.0-alpha-3-osx32_en-US.dmg
29868034 torbrowser-install-4.0-alpha-3_en-US.exe
30430008 tor-browser-linux32-4.0-alpha-3_en-US.tar.xz
32140468 tor-browser-linux64-4.0-alpha-3_en-US.tar.xz

Here is the diff of bundle contents:

+80K    ./Browser/fonts/DroidKufi-Bold.ttf
+80K    ./Browser/fonts/DroidKufi-Regular.ttf
+91K    ./Browser/fonts/DroidNaskh-Bold.ttf
+88K    ./Browser/fonts/DroidNaskh-Regular.ttf
+109K   ./Browser/fonts/DroidNaskhUI-Regular.ttf
+36K    ./Browser/fonts/DroidSansArabic.ttf
+14K    ./Browser/fonts/DroidSansArmenian.ttf
+190K   ./Browser/fonts/DroidSans-Bold.ttf
+218K   ./Browser/fonts/DroidSansEthiopic-Bold.ttf
+223K   ./Browser/fonts/DroidSansEthiopic-Regular.ttf
+4.4M   ./Browser/fonts/DroidSansFallbackFull.ttf
+3.8M   ./Browser/fonts/DroidSansFallback.ttf
+21K    ./Browser/fonts/DroidSansGeorgian.ttf
+30K    ./Browser/fonts/DroidSansHebrew-Bold.ttf
+30K    ./Browser/fonts/DroidSansHebrew-Regular.ttf
+1.2M   ./Browser/fonts/DroidSansJapanese.ttf
+117K   ./Browser/fonts/DroidSansMono.ttf
+187K   ./Browser/fonts/DroidSans.ttf
+259K   ./Browser/fonts/DroidSerif-BoldItalic.ttf
+245K   ./Browser/fonts/DroidSerif-Bold.ttf
+247K   ./Browser/fonts/DroidSerif-Italic.ttf
+244K   ./Browser/fonts/DroidSerif-Regular.ttf
+137K   ./Browser/fonts/Lohit-Assamese.ttf
+137K   ./Browser/fonts/Lohit-Bengali.ttf
+71K    ./Browser/fonts/Lohit-Devanagari.ttf
+61K    ./Browser/fonts/Lohit-Gujarati.ttf
+194K   ./Browser/fonts/Lohit-Kannada.ttf
+49K    ./Browser/fonts/Lohit-Malayalam.ttf
+70K    ./Browser/fonts/Lohit-Marathi.ttf
+96K    ./Browser/fonts/Lohit-Oriya.ttf
+24K    ./Browser/fonts/Lohit-Punjabi.ttf
+65K    ./Browser/fonts/Lohit-Tamil-Classical.ttf
+61K    ./Browser/fonts/Lohit-Tamil.ttf
+167K   ./Browser/fonts/Lohit-Telugu.ttf
-125K   ./Browser/precomplete
+127K   ./Browser/precomplete
+639    ./Browser/TorBrowser/Data/fontconfig/conf.d/65-0-lohit-marathi.conf
+636    ./Browser/TorBrowser/Data/fontconfig/conf.d/65-0-lohit-nepali.conf
+492    ./Browser/TorBrowser/Data/fontconfig/conf.d/65-droid.conf
+1.7K   ./Browser/TorBrowser/Data/fontconfig/conf.d/65-lohit.conf
+626    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-assamese.conf
+639    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-bengali.conf
+3.1K   ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-devanagari.conf
+643    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-gujarati.conf
+620    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-kannada.conf
+634    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-oriya.conf
+621    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-punjabi.conf
+663    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-tamil-classical.conf
+633    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-tamil.conf
+633    ./Browser/TorBrowser/Data/fontconfig/conf.d/66-lohit-telugu.conf
+643    ./Browser/TorBrowser/Data/fontconfig/conf.d/67-lohit-malayalam.conf
+6.3K   ./Browser/TorBrowser/Data/fontconfig/fonts.conf
+572    ./Browser/TorBrowser/Docs/Licenses/Droid-Fonts.txt
+4.3K   ./Browser/TorBrowser/Docs/Licenses/Lohit-Fonts.txt
-5.8K   ./Browser/TorBrowser/Docs/sources/versions
+6.1K   ./Browser/TorBrowser/Docs/sources/versions
Last edited 3 years ago by dcf (previous) (diff)

comment:2 Changed 3 years ago by intrigeri

Cc: anonym intrigeri added

Changed 3 years ago by dcf

Attachment: 4.5-alpha-2-fonts-1.patch added

tor-browser patch against tor-browser-31.3.0esr-4.5-1-build1.

Changed 3 years ago by dcf

Screenshot of 4.5-alpha-2-fonts-1 showing www.wikipedia.org and a copy-paste of the Fonts panel.

Changed 3 years ago by dcf

Attachment: fontlist.log added

NSPR_LOG_MODULES=fontlist:5 for 4.5-alpha-2-fonts-1.

comment:3 Changed 3 years ago by dcf

I started looking at Firefox changes to disable system fonts and use only bundled fonts. comment:1 shows that it's easy to do on linux with fonts.conf. It remains to do it on other platforms.

I started by trying to disable system fonts for DirectWrite. DirectWrite is one of two font rendering APIs used on Windows (the other is GDI). It doesn't quite work yet. The system fonts are not loaded, and bundled fonts are loaded. But all text is rendered as squares, even in browser chrome—except, curiously, Georgian text. It seems that for whatever reason the only font displayed by default is Droid Sans Georgian, even though you can select others from the Content menu. More on that below.

This is the code I tried, my fonts branch of tor-browser-bundle.git, and a patch against tor-browser.git. The tor-browser.git patch is Mozilla #998844 (for --enable-bundled-fonts), plus a dummy loader for system fonts in the DirectWrite renderer.

I had to set

gfx.font_rendering.directwrite.enabled=true

and restart in order to enable DirectWrite. (Running in KVM, about:support says "Direct2D Enabled: Blocked for your graphics card because of unresolved driver issues." and "DirectWrite Enabled: false (6.2.9200.16581)".)

Here's what it looks like:
Screenshot of 4.5-alpha-2-fonts-1 showing [https://www.wikipedia.org/ www.wikipedia.org] and a copy-paste of the Fonts panel.
Almost everything is rendered as boxes, except for the Georgian text. I copy-pasted from the Fonts panel in the Inspector into a text editor, which shows that the "Droid Sans Georgian" font is being used. However, if I go into the Content menu, I can select Droid Sans (by looking for the right pattern of boxes, "▯▯▯▯▯ ▯▯▯▯"), and then the Latin text shows up properly (not on the Wikipedia page, but on other pages).

If I delete fonts/DroidSansGeorgian.ttf, then the font that gets loaded is instead Lohit Oriya, and is similarly broken. The shape of the boxes in Lohit Oriya have a noticeably different shape. Perhaps the fonts selection governed by ordering in an internal hash table or something.

I turn on fontlist logging with

set NSPR_LOG_MODULES=fontlist:5
cd Browser
firefox.exe -console

and I see this on the console:

0[1197208]: (fontlist-postscript) name: Droid Sans Georgian Regular, psname: Droid SansGeorgian
0[1197208]: (fontlist-fullname) name: Droid Sans Georgian Regular, fullname: Droid Sans Georgian
0[1197208]: (fontlist) added (Droid Sans Georgian Regular) to family (Droid Sans Georgian) with style: normal weight: 400 stretch: 0 psname: DroidSansGeorgian fullname: Droid Sans Georgian
0[1197208]: (fontlist) added (Droid Sans Georgian Bold) to family (Droid Sans Georgian) with style: normal weight: 700 stretch: 0 psname: DroidSansGeorgian fullname: Droid Sans Georgian
0[1197208]: (fontlist-cmap) name: Droid Sans Georgian Bold, size: 304 hash: 29410df0 new
0[1197208]: (fontlist-cmap) name: Droid Sans Georgian Regular, size: 1880 hash:54f67428 new

a few seconds later, while it's sitting at the about:tor screen, I see it load the rest of the fonts. The full log is in attachment:fontlist.log. (However, note that only Droid Sans Georgian has fontlist-cmap lines.)

Changed 3 years ago by dcf

Attachment: fontinit.log added

NSPR_LOG_MODULES=fontinit:5 for 4.5-alpha-2-fonts-1.

Changed 3 years ago by dcf

Attachment: textrun.log added

NSPR_LOG_MODULES=textrun:5 for 4.5-alpha-2-fonts-1.

Changed 3 years ago by dcf

Attachment: textrunui.log added

NSPR_LOG_MODULES=textrunui:5 for 4.5-alpha-2-fonts-1.

Changed 3 years ago by dcf

Attachment: cmapdata.log added

NSPR_LOG_MODULES=cmapdata:5 for 4.5-alpha-2-fonts-1.

comment:4 Changed 3 years ago by dcf

Here are other Firefox logs from 4.5-alpha-2-fonts-1. I got the names of log modules from https://hg.mozilla.org/mozilla-central/file/636498d041b5/gfx/thebes/gfxPlatform.cpp#l2058.

The most interesting one appears to be cmapdata.log:

0[c07268]: (cmapdata) name: Droid Sans Georgian Bold u+000000 [80040000 80000000 00000000 00000000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Bold u+001000 [00000000 00000000 00000000 00000000 00000000 ffffffff fc00ffff fffffff8]
0[c07268]: (cmapdata) name: Droid Sans Georgian Bold u+002d00 [ffffffff fc000000 00000000 00000000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000000 [00000000 ffffffff ffffffff fffffffe 00000000 ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000100 [ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000200 [ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000300 [ffffffff ffffffff ffffffff ffffff3e 0febffff dfffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000400 [ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000500 [ffffffff ff000000 00000000 00000000 00007fff ffffffff ff00ffff ffe0f800]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000600 [fbfffff3 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000700 [00000000 00000000 0000ffff ffffffff 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+000800 [00000000 00000000 00000000 00000000 00000000 bff80000 00000000 0ffffffe]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+001d00 [ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffe00000 00000003]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+001e00 [ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+001f00 [fffffcfc ffffffff fcfcff55 fffffffc ffffffff fffffbff fbfff3f7 ffff3bfe]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002000 [ffff3fff e23fb86e 08000002 003f8fc1 0000f800 ffffffe4 00000000 00008000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002100 [04001300 22020000 0006181e 00000000 0800fc00 00800000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002200 [22016463 00500000 00800000 cc000000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002300 [20008000 c0000000 00000000 00000000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002500 [a0088888 08080808 0000ffff fff80000 8888f000 c0382028 083900c0 02000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002600 [00000000 00000038 a0000000 96310000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002c00 [00000000 00000000 00000000 ffffffff 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+002e00 [00000100 00000000 00000000 00000000 00000000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00a700 [000001ff c0000000 00000000 00000000 00f80000 00000000 00000000 00000000]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00fb00 [60000007 fffffefa dbffffff ffffffff ffffffff ffffffff c0001fff ffffffff]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00fc00 [00000000 00000000 00000003 f0000000 00000000 00000000 00000000 00003800]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00fd00 [00000000 0000000f 00000000 00000000 00000000 00000000 00000000 0000283c]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00fe00 [00000000 f0000000 00000000 0000fbff ffffffff ffffffff ffffffff fffffff8]
0[c07268]: (cmapdata) name: Droid Sans Georgian Regular u+00ff00 [00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000008]

If I guess at the meaning right, it looks like "Droid Sans Georgian Regular" is claiming support for lots of code points, including most of ASCII. Maybe it doesn't actually support those, but Firefox believes it does, so it doesn't try loading any other fonts.

comment:5 Changed 3 years ago by dcf

If I set the prefs,

font.name.sans-serif.x-unicode="Droid Sans"
font.name.sans-serif.x-western="Droid Sans"
font.name.serif.x-unicode="Droid Serif"
font.name.serif.x-western="Droid Serif"

then it begins to work right, for Latin text. Presumably we could set the mappings for all our fonts in all.js. The font used for browser chrome is still messed up.

comment:7 Changed 3 years ago by mikeperry

Keywords: tbb-fingerprinting added

comment:8 Changed 3 years ago by mikeperry

Keywords: tbb-fingerprinting-fonts added; tbb-fingerprinting removed

comment:9 Changed 3 years ago by gacar

Cc: gacar@… added

See this Mozilla bug: 1121643 - Add an option to only expose whitelisted system fonts to avoid fontlist fingerprinting.

This could be very useful for disallowing system fonts.

comment:10 Changed 3 years ago by mcs

Cc: brade mcs added

comment:11 Changed 2 years ago by dcf

I got some advice from Firefox maintainers. One suggests that the best way to do the job is to put code in gfxPlatformFontList that filters the system font list (based on file name; we check that the names are within the whitelisted bundle directory). Another says that in order to disable system fallback, one should set the pref gfx.font_rendering.fallback.always_use_cmaps to true, which will cause the renderer to explicitly iterate through the list of known fonts.

comment:12 in reply to:  11 ; Changed 2 years ago by arthuredelstein

Replying to dcf:

I got some advice from Firefox maintainers. One suggests that the best way to do the job is to put code in gfxPlatformFontList that filters the system font list (based on file name; we check that the names are within the whitelisted bundle directory). Another says that in order to disable system fallback, one should set the pref gfx.font_rendering.fallback.always_use_cmaps to true, which will cause the renderer to explicitly iterate through the list of known fonts.

Hi David -- I'm having a look at your work. Is the branch here your latest version? ​https://gitweb.torproject.org/user/dcf/tor-browser-bundle.git/log/?h=fonts

comment:13 in reply to:  12 Changed 2 years ago by dcf

Replying to arthuredelstein:

Replying to dcf:

I got some advice from Firefox maintainers. One suggests that the best way to do the job is to put code in gfxPlatformFontList that filters the system font list (based on file name; we check that the names are within the whitelisted bundle directory). Another says that in order to disable system fallback, one should set the pref gfx.font_rendering.fallback.always_use_cmaps to true, which will cause the renderer to explicitly iterate through the list of known fonts.

Hi David -- I'm having a look at your work. Is the branch here your latest version? ​https://gitweb.torproject.org/user/dcf/tor-browser-bundle.git/log/?h=fonts

Yes, it's that branch plus this tor-browser.git patch:

The patch only covers DirectWrite for Windows, and resulted in the weird graphical effects in comment:3. In order to use it, you may have to set the pref

gfx.font_rendering.directwrite.enabled=true

comment:14 Changed 2 years ago by gk

Cc: gk added

comment:15 Changed 2 years ago by arthuredelstein

Here are my latest patches for this ticket (some based on David's patches):
https://github.com/arthuredelstein/tor-browser/commits/13313+2
https://github.com/arthuredelstein/tor-browser-bundle/commits/13313+1

I'll be posting signed binaries shortly for testing.

These patches use a subset of fonts from Google's Noto font family. I arbitrarily chose them to cover every living language for which there exists an edition of Wikipedia:
https://meta.wikimedia.org/wiki/List_of_Wikipedias#All_Wikipedias_ordered_by_number_of_articles

I chose Noto for a couple of reasons:

The fonts add ~15 MB to each package, most of it due to NotoSansCJKsc-Regular.otf.

Last edited 2 years ago by arthuredelstein (previous) (diff)

comment:16 Changed 2 years ago by dcf

I saw on IRC you were asking about the glyph testing program. It's not online anymore, but here is the source code.

git clone https://repo.eecs.berkeley.edu/git-anon/users/fifield/fontfp.git
cd fontfp/webapp
go build
./webapp --http :8000

Then browse to http://localhost:8000/fastfp to get a quick fingerprint. http://localhost:8000/fontfp is the all-codepoint test that takes 10 minutes that you should run after fastfp detects no differences.

There's also a demo at
https://www.bamsoftware.com/talks/fc15-fontfp/fontfp.html#demo
It doesn't print out a fingerprint, but it should be easy to crib from the source code.

comment:17 Changed 2 years ago by mikeperry

Keywords: tbb-5.0a4 added

comment:18 Changed 2 years ago by mikeperry

Keywords: TorBrowserTeam201507 added

comment:19 Changed 2 years ago by arthuredelstein

Keywords: TorBrowserTeam201507R added; TorBrowserTeam201507 removed
Status: newneeds_review

I made corrections to both tor-browser.git and tor-browser-bundle.git patches. So here are my two new branches, for review:

  • The tor-browser-bundle.git patches enable font bundling as implemented in Mozilla Bug 998844, download the Noto fonts and bundle them in Tor Browser:

https://github.com/arthuredelstein/tor-browser-bundle/commits/13313+4

  • The tor-browser.git patches add a new mechanism for whitelisting fonts according to a pref (font.system.whitelist), and use this pref to whitelist the bundled Noto fonts only:

https://github.com/arthuredelstein/tor-browser/commits/13313+4

For ease of testing the results, I built packages for various platforms and languages, downloadable here:
https://people.torproject.org/~arthuredelstein/downloads/13313-builds/

(My gpg fingerprint is 20B2 4CEF E6AF D615 0B6A 6F18 D752 F538 C0D3 8C3A.)

Here are a couple of pages that may be useful for testing:

The whitelisted fonts are given in the font.system.whitelist pref in about:config. They are:
Cousine, Noto Kufi Arabic, Noto Naskh Arabic, Noto Sans, Noto Sans Armenian, Noto Sans Bengali, Noto Sans Buginese, Noto Sans CJK SC Regular, Noto Sans Canadian Aboriginal, Noto Sans Cherokee, Noto Sans Devanagari, Noto Sans Ethiopic, Noto Sans Georgian, Noto Sans Gujarati, Noto Sans Gurmukhi, Noto Sans Hebrew, Noto Sans Kannada, Noto Sans Khmer, Noto Sans Lao, Noto Sans Malayalam, Noto Sans Mongolian, Noto Sans Myanmar, Noto Sans Oriya, Noto Sans Sinhala, Noto Sans Tamil, Noto Sans Telugu, Noto Sans Thaana, Noto Sans Thai, Noto Sans Tibetan, Noto Sans Yi, Noto Serif, Noto Serif Armenian, Noto Serif Khmer, Noto Serif Lao, Noto Serif Thai

Note that the extra MB added to Tor Browser are mostly from the file NotoSansCJKsc-Regular.otf, which covers Chinese (simplifed and traditional), Japanese, Korean. Also, Cousine is included as a monospace font similar in size and shape to the Noto fonts.

comment:20 Changed 2 years ago by arthuredelstein

(It might be useful to mention that Cousine, Noto Sans and Noto Serif cover the large majority of languages in the list from Wikipedia -- those that use Latin-, Greek-, and Cyrillic-derived alphabets.)

comment:21 Changed 2 years ago by gk

Some first comments:

It builds and I get proper Tor Browser bundles which is good :). Please move --enable-bundled-fonts into the respective .mozconfig files. It is a normal config option which landed on mozilla trunk a while ago. Apart from that the tor-browser-bundle changes are good.

I tested the bundles a bit and was kind of surprised. My naive understanding is that tests like they are done on http://ip-check.info should show the same amount and the same fonts regardless of the underlying OS/testing user or am I missing something here? Anyway, with 5.0a3 I get 54 fonts on a linux machine and 250 fonts on a windows machine. With the patches I get 21 fonts on the same linux machine and 250 fonts on the same windows machine. While I could understand the former the latter sounds like a bug (provided there are no issues with the test itself) to me.

comment:22 in reply to:  21 Changed 2 years ago by gk

Replying to gk:

Anyway, with 5.0a3 I get 54 fonts on a linux machine and 250 fonts on a windows machine. With the patches I get 21 fonts on the same linux machine and 250 fonts on the same windows machine. While I could understand the former the latter sounds like a bug (provided there are no issues with the test itself) to me.

Okay, this was a bug on my side: I forgot the commit that added the font whitelist. Sorry, for the noise. Interestingly, the test on ip-check.info is falling back to the CSS test. I forgot how the code worked but one guess would be that it is doing this because it could not find any known font at all with JS...

comment:23 in reply to:  21 ; Changed 2 years ago by arthuredelstein

Replying to gk:

Some first comments:

It builds and I get proper Tor Browser bundles which is good :). Please move --enable-bundled-fonts into the respective .mozconfig files. It is a normal config option which landed on mozilla trunk a while ago. Apart from that the tor-browser-bundle changes are good.

As I mentioned on IRC, the reason I put --enable-bundled-fonts in tor-browser-bundle.git instead of mozconfigs is that the fonts aren't available to tor-browser.git. If someone is just building tor-browser.git, I don't want to make them have to download the Noto fonts and put them in the right directory just to be able to read text.

Alternative approaches could be:

  1. Find a preprocessor flag that is only active when we build tor-browser-bundle.git, and use that to disable the whitelisting pref for tor-browser.git alone.
  2. Add the Noto fonts directly to the tor-browser.git repo, and add something in the Mozilla build scripts to install them in the directory where fonts are bundled. That would avoid modifying tor-browser-bundle.git altogether.

Opinions welcome!

Last edited 2 years ago by arthuredelstein (previous) (diff)

comment:24 in reply to:  16 ; Changed 2 years ago by gk

Replying to dcf:

There's also a demo at
https://www.bamsoftware.com/talks/fc15-fontfp/fontfp.html#demo

I just tested that on two 32bit Linux systems (one Ubuntu 12.04 and one Debian testing) and even there are differeces visible with bundled fonts (the diff is attached). I guess this means shipping the alpha with it is fine (it can't get worse wrt to the status quo :) ) but we might want to have an estimation about what the current solution really helps us for the stable series before we ship it there.

Changed 2 years ago by gk

Attachment: linux32diff added

comment:25 in reply to:  23 ; Changed 2 years ago by gk

Replying to arthuredelstein:

  1. Add the Noto fonts directly to the tor-browser.git repo, and add something in the Mozilla build scripts to install them in the directory where fonts are bundled. That would avoid modifying tor-browser-bundle.git altogether.

I think this makes sense. Another thing that bothers me with the currently proposed solution is that it makes bisecting quite error-prone. Although this is not documented yet the fastest approach is to just take an existing Tor Browser bundle and just bisect the tor-browser parts copying the result over the respective bundle parts with each iteration. This is not working anymore with having so many parts in tor-browser-bundle.git. Having everything in tor-browser could help us debug issues due to font updates easier as well.

comment:26 in reply to:  24 ; Changed 2 years ago by arthuredelstein

Replying to gk:

Replying to dcf:

There's also a demo at
https://www.bamsoftware.com/talks/fc15-fontfp/fontfp.html#demo

I just tested that on two 32bit Linux systems (one Ubuntu 12.04 and one Debian testing) and even there are differeces visible with bundled fonts (the diff is attached). I guess this means shipping the alpha with it is fine (it can't get worse wrt to the status quo :) ) but we might want to have an estimation about what the current solution really helps us for the stable series before we ship it there.

Whoa, interesting result. I think, though, that it's a form of OS fingerprinting, similar to #13018, or am I missing something? Whereas this ticket attempts to solve an orthogonal problem, which is that it is possible to enumerate the system fonts installed on a user's machine.

Also, I *think* measuring glyph sizes is only possible with JS activated, whereas enumerating fonts is possible using CSS alone.

(I've opened #16672 regarding differences in text rendering between operating systems.)

comment:27 in reply to:  25 Changed 2 years ago by arthuredelstein

Replying to gk:

Replying to arthuredelstein:

  1. Add the Noto fonts directly to the tor-browser.git repo, and add something in the Mozilla build scripts to install them in the directory where fonts are bundled. That would avoid modifying tor-browser-bundle.git altogether.

I think this makes sense. Another thing that bothers me with the currently proposed solution is that it makes bisecting quite error-prone. Although this is not documented yet the fastest approach is to just take an existing Tor Browser bundle and just bisect the tor-browser parts copying the result over the respective bundle parts with each iteration. This is not working anymore with having so many parts in tor-browser-bundle.git. Having everything in tor-browser could help us debug issues due to font updates easier as well.

Very good point. I'll give this approach a try.

comment:28 in reply to:  26 ; Changed 2 years ago by dcf

Replying to arthuredelstein:

Whoa, interesting result. I think, though, that it's a form of OS fingerprinting, similar to #13018, or am I missing something? Whereas this ticket attempts to solve an orthogonal problem, which is that it is possible to enumerate the system fonts installed on a user's machine.

Whitelisting font files is meant to solve both: enumeration of font names, and differences in glyph rendering. Differences in glyph rendering provide much more precision than just the OS--it can vary based on what fonts are installed, what antialiasing settings you use, and what graphics card you have, for example. Glyph rendering is in scope for this ticket--that's the idea behind enforcing a single list of exact font files, not just a single list of font names. By standardizing the list of font file and rendering settings you should be able to bring down the variability a lot. See figures 4 and 5 on page 13 of https://bamsoftware.com/papers/fontfp.pdf.

comment:29 in reply to:  28 Changed 2 years ago by arthuredelstein

Replying to dcf:

Replying to arthuredelstein:

Whoa, interesting result. I think, though, that it's a form of OS fingerprinting, similar to #13018, or am I missing something? Whereas this ticket attempts to solve an orthogonal problem, which is that it is possible to enumerate the system fonts installed on a user's machine.

Whitelisting font files is meant to solve both: enumeration of font names, and differences in glyph rendering. Differences in glyph rendering provide much more precision than just the OS--it can vary based on what fonts are installed, what antialiasing settings you use, and what graphics card you have, for example. Glyph rendering is in scope for this ticket--that's the idea behind enforcing a single list of exact font files, not just a single list of font names. By standardizing the list of font file and rendering settings you should be able to bring down the variability a lot. See figures 4 and 5 on page 13 of https://bamsoftware.com/papers/fontfp.pdf.

What I understand from those figures is that most of the entropy saved is in standardizing the exact font files (please correct me if I'm mistaken). In comment:19 we have patches that enforce a single list of fonts, and bundle exactly the same font files on all platforms. I think that moves us from the red line to the blue line. To get closer to the green line, we need to adjust rendering settings -- I'd suggest punting that work to #16672, because I think it's going to take substantial experimentation to optimize those settings across platforms. In the meantime I think it would be nice to get user feedback for the bundled fonts in the alpha if possible.

comment:30 Changed 2 years ago by mikeperry

Well, if Debian and Ubuntu differ, what about normalizing the fonts.conf in addition, as David suggested in from comment:1?

Or do we suspect its the underlying font renderer versions themselves?

comment:31 in reply to:  30 Changed 2 years ago by arthuredelstein

Replying to mikeperry:

Well, if Debian and Ubuntu differ, what about normalizing the fonts.conf in addition, as David suggested in from comment:1?

Or do we suspect its the underlying font renderer versions themselves?

Thanks for pointing that out. I had forgotten about that part of David's fonts.conf. I'll give it a try.

comment:32 Changed 2 years ago by arthuredelstein

Here's the tor-browser.git branch in comment:19 rebased on top of tor-browser-38.1.0esr-5.0-1 (8422662a5b8771b6e93a02e4cb6895ff1d8a3126):

https://github.com/arthuredelstein/tor-browser/commits/13313+5

comment:33 Changed 2 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Thanks, this is going to ship in 5.0a4. I am closing this as we enable bundled fonts in this ticket. It seems to me the idea with the fonts.conf file belongs to #16672, too.

comment:34 in reply to:  16 Changed 2 years ago by dcf

Replying to dcf:

There's also a demo at
https://www.bamsoftware.com/talks/fc15-fontfp/fontfp.html#demo
It doesn't print out a fingerprint, but it should be easy to crib from the source code.

I made a dedicated page that's more tailored to the kind of tests we're doing now. It displays a single checksum, lets you download a text file of all the dimensions, and includes a code point viewer.

https://people.torproject.org/~dcf/fonttest.html

I posted some of my results at https://lists.torproject.org/pipermail/tor-qa/2015-August/000648.html.
In short,

		5.0a3		5.0a4
Debian 8	1-2d5db8b8	1-83e97f7d
Debian 8	1-ee150545	1-83e97f7d
Windows 8	1-e77cd884	1-0fe1d60b

comment:35 Changed 2 years ago by dcf

For what it's worth, my thinking behind this ticket was not only to enable bundled fonts, but also to prohibit loading any font files other than the bundled files. I.e., whitelist certain font filenames. The patches https://github.com/arthuredelstein/tor-browser-bundle/commits/13313+4 and https://github.com/arthuredelstein/tor-browser/commits/13313+5 enable bundled fonts and then whitelist certain font names, which is not quite the same if the user happens to have fonts with the identical names already installed on their system.

But at any rate, I think these patches are a positive step and will probably remove the majority of fingerprintability related to font files.

comment:36 in reply to:  35 Changed 2 years ago by arthuredelstein

Replying to dcf:

For what it's worth, my thinking behind this ticket was not only to enable bundled fonts, but also to prohibit loading any font files other than the bundled files. I.e., whitelist certain font filenames. The patches https://github.com/arthuredelstein/tor-browser-bundle/commits/13313+4 and https://github.com/arthuredelstein/tor-browser/commits/13313+5 enable bundled fonts and then whitelist certain font names, which is not quite the same if the user happens to have fonts with the identical names already installed on their system.

That's a good point. I considered the filename option somewhat but my impression was it was going to be pretty messy to implement on some platforms. But it may be worth another look. So I'm opening a new ticket: #16739.

comment:37 Changed 2 years ago by dcf

Moritz and I found a case where bundling fonts was not sufficient to determine Tor Browser's font selection. It was choosing different fonts (serif versus sans-serif) for the "cursive" and "fantasy" styles of one particular code point.

I speculated:

That's very interesting. In just that one code point, your computer chooses a sans-serif font for the "fantasy" and "cursive" styles, while mine chooses a serif font. Probably we would find more examples if we tested more of Unicode.
I would not have expected a difference like this. It does not come down to small rendering differences; the browser is simply choosing a different font. Perhaps there is something non-deterministic about how Firefox (or Fontconfig) loads its font files. Like, maybe it iterates a directory in inode order and takes the first match when there is a tie.

comment:38 Changed 21 months ago by arthuredelstein

Parent ID: 18097

comment:39 Changed 21 months ago by arthuredelstein

Parent ID: 18097#18097
Note: See TracTickets for help on using tickets.