Opened 3 years ago

Last modified 9 months ago

#18172 needs_revision defect

Emoji support is broken in Tor Browser 5.5

Reported by: gk Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Normal Keywords: tbb-fingerprinting-fonts, tbb-usability-website, tbb-5.5-regression, TorBrowserTeam201603
Cc: arthuredelstein, joel2017 Actual Points:
Parent ID: #18097 Points:
Reviewer: Sponsor:

Description

It seems our font fingerprinting defense breaks emoji support: https://blog.torproject.org/blog/tor-browser-55-released#comment-153807. See comment:10:ticket:17270 for one example.

Child Tickets

TicketStatusOwnerSummaryComponent
#18364newtbb-teamTor Browser in Gnu+Linux doesn't support Dingbats properlyApplications/Tor Browser

Attachments (3)

Screen Shot 2016-01-28 at 3.52.30 PM.png (15.3 KB) - added by arthuredelstein 3 years ago.
Screen Shot 2016-01-28 at 3.58.16 PM.png (7.0 KB) - added by arthuredelstein 3 years ago.
Screen Shot 2016-01-28 at 4.24.10 PM.png (6.0 KB) - added by arthuredelstein 3 years ago.

Download all attachments as: .zip

Change History (39)

comment:1 Changed 3 years ago by cypherpunks

I see no problem with the face characters in that comment. Gnu/linux, 64 bits.

Changed 3 years ago by arthuredelstein

Changed 3 years ago by arthuredelstein

Changed 3 years ago by arthuredelstein

comment:2 Changed 3 years ago by arthuredelstein

There's a unicode emoji table here:
http://apps.timwhitlock.info/emoji/tables/unicode

Here's what I get on different platforms:

LinuxMacWindows

So it looks like we need to fix Mac and Windows, but Linux is OK.

comment:3 Changed 3 years ago by arthuredelstein

Keywords: TorBrowserTeam201601R added
Status: newneeds_review

The last commit in this branch contains a fix for Windows and Mac. It has been applied on top of my fix for #18169:

https://github.com/arthuredelstein/tor-browser/commits/18169

Last edited 3 years ago by arthuredelstein (previous) (diff)

comment:4 in reply to:  2 ; Changed 3 years ago by gk

Status: needs_reviewneeds_revision

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

Last edited 3 years ago by gk (previous) (diff)

comment:5 in reply to:  4 Changed 3 years ago by cypherpunks

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken.

Me too (uncommon Linux distribution). If the native emojis work, that's a leak of installed fonts, right (so a bug)?

comment:6 in reply to:  4 ; Changed 3 years ago by arthuredelstein

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

comment:7 Changed 3 years ago by arthuredelstein

(I should mention, I decided against bundling Noto Color Emoji, because it is 2.3 MB. Instead I am bundling Noto Emoji, which is only 280kB.)

comment:8 in reply to:  6 ; Changed 3 years ago by gk

Replying to arthuredelstein:

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

I think we can start with this font as a fix for this bug. Two things regarding the patch:

1) You don't use NOTOEMOJIFONT_VER anywhere, so we don't need it. Which reminds me that the same seems to hold for NOTOCJKFONT_VER.
2) Please remove the superflous newline below +- "NotoEmoji-Regular.ttf"

comment:9 Changed 3 years ago by gk

Keywords: TorBrowserTeam201602R added; TorBrowserTeam201601R removed

Carrying over review tickets.

comment:11 in reply to:  8 Changed 3 years ago by arthuredelstein

Status: needs_revisionneeds_review

Replying to gk:

Replying to arthuredelstein:

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

I think we can start with this font as a fix for this bug. Two things regarding the patch:

1) You don't use NOTOEMOJIFONT_VER anywhere, so we don't need it. Which reminds me that the same seems to hold for NOTOCJKFONT_VER.
2) Please remove the superflous newline below +- "NotoEmoji-Regular.ttf"

Here's a patch with these things fixed:
https://github.com/arthuredelstein/tor-browser-bundle/commit/18172+1

comment:12 Changed 3 years ago by gk

Resolution: fixed
Status: needs_reviewclosed

Thanks, this is commit c885551e69ca5043d0af0de6940e250968e6f7e5, 7aa3abf90371fb82518edf5d54e862e91744c97b and ee6fd02653fc0e396bee5daa034bd00be890ccb5 on master/maint-5.5/hardened-builds.

comment:13 Changed 3 years ago by gk

Keywords: tbb-5.5-regression added

comment:14 Changed 3 years ago by bugzilla

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

comment:15 in reply to:  14 ; Changed 3 years ago by arthuredelstein

Resolution: fixed
Status: closedreopened

Replying to bugzilla:

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

  1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
  2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
  3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Last edited 3 years ago by arthuredelstein (previous) (diff)

comment:16 in reply to:  15 ; Changed 3 years ago by gk

Replying to arthuredelstein:

Replying to bugzilla:

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

Windows 8 seems to be affected as well.

  1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
  2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
  3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 and #18168 are more important and I would like to see some testing with this approach.

comment:17 in reply to:  16 Changed 3 years ago by gk

Replying to gk:

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 and #18168 are more important and I would like to see some testing with this approach.

Which means I think we back out the Windows change until we have a proper solution and ship 5.5.1 only with OS X and Linux emoji support fixing the Windows related bug in the next stable release.

comment:18 in reply to:  15 Changed 3 years ago by cypherpunks

Replying to arthuredelstein:

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

  1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
  2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
  3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Isn't a long term goal that the platform can't be detected? That points to 3.

comment:19 in reply to:  16 Changed 3 years ago by arthuredelstein

Keywords: tbb-fingerprinting-fonts, tbb-usability-website, TorBrowserTeam201602R tbb-5.5-regressiontbb-fingerprinting-fonts, tbb-usability-website, TorBrowserTeam201602R, tbb-5.5-regression
Status: reopenedneeds_review

Replying to gk:

[...]
Windows 8 seems to be affected as well.

That's unfortunate. It was available on the Windows 8 machine I used for testing, but it sounds like this cannot be relied on.

[...]

  1. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 and #18168 are more important and I would like to see some testing with this approach.

Here are my patches to bundle and whitelist Noto Emoji on Windows:
https://github.com/arthuredelstein/tor-browser-bundle/commit/18172+2
https://github.com/arthuredelstein/tor-browser/commit/18172

And here is the resulting build (I confirmed that the Noto Emoji font is used for emoji characters on a Windows machine. I don't expect there to be a difference on different Windows versions, as I am bundling the font.):
https://people.torproject.org/~arthuredelstein/downloads/18172-builds

Last edited 3 years ago by arthuredelstein (previous) (diff)

comment:20 Changed 3 years ago by gk

What about https://blog.torproject.org/blog/tor-browser-551-released#comment-155650

The emoji in Linux works again after this update, but they're terrible. Most of the faces at http://apps.timwhitlock.info/emoji/tables/unicode#emoji-modal look like PacMan ghosts and their facial expressions aren't identifiable at typical font sizes on the web.

and

https://blog.torproject.org/blog/tor-browser-551-released#comment-155569

Is it possible to just bundle Symbola, to support emojis?

?

Last edited 3 years ago by gk (previous) (diff)

comment:21 Changed 3 years ago by gk

Keywords: TorBrowserTeam201602 added; TorBrowserTeam201602R removed
Status: needs_reviewneeds_revision

I guess we want to revise that one given the user feedback?

comment:22 in reply to:  21 Changed 3 years ago by arthuredelstein

Replying to gk:

I guess we want to revise that one given the user feedback?

Yes, we definitely do. Unfortunately I haven't found a great solution yet. I'll be posting the results of my advanced emoji research soon.

comment:23 Changed 3 years ago by gk

Keywords: TorBrowserTeam201603 added; TorBrowserTeam201602 removed

comment:24 Changed 2 years ago by vegansalad

@arthuredelstein did you finish your advanced emoji research? i just found out about this Tor Browser on Linux emoji related issue and wrote some notes up over here: https://github.com/fedwiki/wiki/issues/97

from what I can discern, the decisions that Tor Browser makes related to what it whitelists might be pulled upstream into Mozilla (currently only manually) since they are adding font.system.whitelist in Firefox 52. here's some talk about it: https://bugzilla.mozilla.org/show_bug.cgi?id=1121643

comment:25 Changed 2 years ago by vegansalad

My issue has to do with dingbats, not emojis. Should I create a new ticket about this or does it make sense to tackle it here?

As an example:

Simple things like the pencil doesn't work on TBB on Linux: https://www.fileformat.info/info/unicode/char/270E/browsertest.htm

In TBB on Windows, the pencil works via Micro$oft's copyrighted MS PGothic (which Linux can't use) but it doesn't look very good.

Oh, and a simple X works in TBB Windows but not TBB Linux: https://www.fileformat.info/info/unicode/char/2715/browsertest.htm

Also, black flag dingbats don't seem to work on TBB on either Windows or Linux: https://www.fileformat.info/info/unicode/char/2691/browsertest.htm

comment:26 Changed 2 years ago by vegansalad

I should state, not every dingbat is broken in the Linux TBB. The Up Down Arrow works: https://www.fileformat.info/info/unicode/char/2195/browsertest.htm

Also, a regular multiplication sign works https://www.fileformat.info/info/unicode/char/00D7/browsertest.htm but a "MULTIPLICATION X" does not https://www.fileformat.info/info/unicode/char/2715/browsertest.htm

I'm not sure which whitelisted font allows some of these to work.

It might be good to find a good font candidate to whitelist in the Linux and Windows TBB to allow it to render more unicode characters than the current fonts allow. I haven't done any tests on the Mac TBB.

comment:27 Changed 2 years ago by vegansalad

Maybe we should make specific font recommendations at this ticket in order to try to get support for emojis and dingbats: https://trac.torproject.org/projects/tor/ticket/20842

comment:28 Changed 2 years ago by vegansalad

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=849042
RFP: fonts-noto-emoji -- "No Tofu" emoji font

Noto color emoji seems like it is being packages in Debian.

Might be a good TBB whitelisted font when it is released.

Noto emoji (black and white) seems to not have its source code online unfortunately.

comment:29 Changed 2 years ago by yawning

Noto emoji (black and white) seems to not have its source code online unfortunately.

NotoEmoji-Regular.ttf is bundled with Tor Browser.

comment:30 Changed 2 years ago by vegansalad

@yawning Thanks for the clarification. I was using https://arthuredelstein.github.io/tordemos/enumerate-fonts.html to get a master list of fonts that TBB ships and didn't see NotoEmoji-Regular.ttf on the list, but I don't think that page checks for it.

I saw that someone else filed a ticket https://trac.torproject.org/projects/tor/ticket/18364 about dingbats not working. You can see a good example of the problem here: http://danshort.com/HTMLentities/index.php?w=dingb

Actually, an even better example is happening right on this page. I'm not able to view the "bold" "italic" "link" or any other icons above the box that I'm typing in. Also, another person pointed out that the "reply to comment" icon to the right of every comment on this page is blank. https://trac.torproject.org/projects/tor/ticket/18860

I've also written about this error in two other trackers, but this has been determined to be TBB specific so both of those tickets were closed: https://github.com/fedwiki/wiki/issues/97 https://labs.riseup.net/code/issues/12154#change-65349

Dingbats are broken in TBB on Gnu+Linux and it isn't a fringe issue because it is happening on our very own Tracker.

Last edited 2 years ago by vegansalad (previous) (diff)

comment:31 Changed 2 years ago by vegansalad

uBlock is also having similar problems: https://trac.torproject.org/projects/tor/ticket/21320#ticket

comment:32 Changed 22 months ago by gk

Cc: joel2017 added

#22087 is a duplicate.

comment:34 Changed 18 months ago by joel2017

the native emoji are still not displayed in tor browser 7.0.4 in windows 10.

was it fixed for windows or is it still an open issue ?

Last edited 18 months ago by joel2017 (previous) (diff)

comment:35 Changed 9 months ago by vegansalad

This seems to affect Linux users of TBB the most, but joel2017 said in the last comment that it is still causing problems for windows users.

I posted a proposed solution to the emoji support problem for GNU+Linux users over here: https://trac.torproject.org/projects/tor/ticket/18364#comment:12

Since there is already a ticket for emoji stuff for GNU+Linux, would it make sense to change the title of this ticket to a Windows specific problem, since the solution is going to be different than it would be for GNU+Linux? If so, I can remove my GNU+Linux related posts in this ticket and move them to 18364.

comment:36 Changed 9 months ago by gk

I think we can have #18364 as a child ticket to this one. Let me do that now.

Note: See TracTickets for help on using tickets.