Emoji support is broken in Tor Browser 5.5

It seems our font fingerprinting defense breaks emoji support: https://blog.torproject.org/blog/tor-browser-55-released#comment-153807. See comment:10:ticket:17270 for one example.

added TorBrowserTeam201603 component::applications/tor browser owner::tbb-team parent::18097 priority::medium severity::normal status::needs-revision tbb-5.5-regression tbb-fingerprinting-fonts tbb-usability-website type::defect labels

I see no problem with the face characters in that comment. Gnu/linux, 64 bits.

Trac:

Trac:

Trac:

There's a unicode emoji table here: http://apps.timwhitlock.info/emoji/tables/unicode

Here's what I get on different platforms:

=Linux=	=Mac=	=Windows=

So it looks like we need to fix Mac and Windows, but Linux is OK.

The last commit in this branch contains a fix for Windows and Mac. It has been applied on top of my fix for #18169 (moved):

https://github.com/arthuredelstein/tor-browser/commits/18169

Trac:
Status: new to needs_review
Keywords: N/A deleted, TorBrowserTeam201601R added

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

Trac:
Status: needs_review to needs_revision

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken.

Me too (uncommon Linux distribution). If the native emojis work, that's a leak of installed fonts, right (so a bug)?

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

(I should mention, I decided against bundling Noto Color Emoji, because it is 2.3 MB. Instead I am bundling Noto Emoji, which is only 280kB.)

Replying to arthuredelstein:

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

I think we can start with this font as a fix for this bug. Two things regarding the patch:

1. You don't use NOTOEMOJIFONT_VER anywhere, so we don't need it. Which reminds me that the same seems to hold for NOTOCJKFONT_VER.
2. Please remove the superflous newline below +- "NotoEmoji-Regular.ttf"

Carrying over review tickets.

Trac:
Keywords: TorBrowserTeam201601R deleted, TorBrowserTeam201602R added

Will this fix resolve these 9 widespread "icons" https://blogs.windows.com/windowsexperience/2016/01/27/announcing-windows-10-insider-preview-build-14251/?

Replying to gk:

Replying to arthuredelstein:

Replying to gk:

Replying to arthuredelstein:

So it looks like we need to fix Mac and Windows, but Linux is OK.

That's actually not true. Surprisingly I have at least one Linux box where emoticons are broken. Is is running Debian testing. That said I realized that it can be a bit hard to see this if one is upgrading from 5.0.7 as it seems to work then but after shutting the browser down and restarting it no emoticons are shown. (This reminded me at one of the failure modes of our start script. :/)

You're right, this happened to me. Once I shutdown the browser and started it again, the Emojis were no longer available.

So here's a patch for Linux:

https://github.com/arthuredelstein/tor-browser-bundle/commit/18172

I built this version and confirmed that the Emojis are displayed, using the "Noto Emoji" font.

I think we can start with this font as a fix for this bug. Two things regarding the patch:

1. You don't use NOTOEMOJIFONT_VER anywhere, so we don't need it. Which reminds me that the same seems to hold for NOTOCJKFONT_VER.
2. Please remove the superflous newline below +- "NotoEmoji-Regular.ttf"

Here's a patch with these things fixed: https://github.com/arthuredelstein/tor-browser-bundle/commit/18172+1

Trac:
Status: needs_revision to needs_review

Thanks, this is commit c885551e69ca5043d0af0de6940e250968e6f7e5, 7aa3abf90371fb82518edf5d54e862e91744c97b and ee6fd02653fc0e396bee5daa034bd00be890ccb5 on master/maint-5.5/hardened-builds.

Trac:
Status: needs_review to closed
Resolution: N/A to fixed

Trac:
Keywords: TorBrowserTeam201602R deleted, TorBrowserTeam201602R tbb-5.5-regression added

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

Replying to bugzilla:

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Trac:
Status: closed to reopened
Resolution: fixed to N/A

Replying to arthuredelstein:

Replying to bugzilla:

No NotoEmoji-Regular.ttf in TBB 5.5.1 for Win, no emoji.

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

Windows 8 seems to be affected as well.

1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 (moved) and #18168 (moved) are more important and I would like to see some testing with this approach.

Replying to gk:

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 (moved) and #18168 (moved) are more important and I would like to see some testing with this approach.

Which means I think we back out the Windows change until we have a proper solution and ship 5.5.1 only with OS X and Linux emoji support fixing the Windows related bug in the next stable release.

Replying to arthuredelstein:

It appears that the Emoji font I add to the whitelist for Windows, Segoe UI Emoji, is not always available on Windows 7. So there are a couple of alternatives:

1. Whitelist Segoe UI Symbol instead. This covers a lot of Emoji characters, although they are monochrome.
2. Whitelist both. This of course allows Windows 7 and earlier to be distinguished from later versions, but there are other ways to distinguish these fonts.
3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion? Isn't a long term goal that the platform can't be detected? That points to 3.

Replying to gk:

[...] Windows 8 seems to be affected as well.

That's unfortunate. It was available on the Windows 8 machine I used for testing, but it sounds like this cannot be relied on.

[...] 3. Bundle NotoEmoji-Regular.ttf with Windows and remove Segoe UI Emoji from the whitelist. This makes the Emoji fonts identical on all versions of Windows. The Noto Emojis don't look the quite the same as the native Windows fonts.

I'm inclined to go for the third option. Does anyone else have an opinion?

Hm, this is pretty unfortunate especially as we probably want to test whether the bundling mechanism is working on Windows as well properly. But, yes, I think we could try that route. I am not really inclined to wait for that for 5.5.1, though, as #18169 (moved) and #18168 (moved) are more important and I would like to see some testing with this approach.

Here are my patches to bundle and whitelist Noto Emoji on Windows: https://github.com/arthuredelstein/tor-browser-bundle/commit/18172+2 https://github.com/arthuredelstein/tor-browser/commit/18172

And here is the resulting build (I confirmed that the Noto Emoji font is used for emoji characters on a Windows machine. I don't expect there to be a difference on different Windows versions, as I am bundling the font.): https://people.torproject.org/~arthuredelstein/downloads/18172-builds

Trac:
Status: reopened to needs_review
Keywords: TorBrowserTeam201602R tbb-5.5-regression deleted, TorBrowserTeam201602R, tbb-5.5-regression added

What about https://blog.torproject.org/blog/tor-browser-551-released#comment-155650

The emoji in Linux works again after this update, but they're terrible. Most of the faces at http://apps.timwhitlock.info/emoji/tables/unicode#emoji-modal look like PacMan ghosts and their facial expressions aren't identifiable at typical font sizes on the web.

and

https://blog.torproject.org/blog/tor-browser-551-released#comment-155569

Is it possible to just bundle Symbola, to support emojis?

?

I guess we want to revise that one given the user feedback?

Trac:
Status: needs_review to needs_revision
Keywords: TorBrowserTeam201602R deleted, TorBrowserTeam201602 added

Replying to gk:

I guess we want to revise that one given the user feedback?

Yes, we definitely do. Unfortunately I haven't found a great solution yet. I'll be posting the results of my advanced emoji research soon.

Trac:
Keywords: TorBrowserTeam201602 deleted, TorBrowserTeam201603 added

@arthuredelstein did you finish your advanced emoji research? i just found out about this Tor Browser on Linux emoji related issue and wrote some notes up over here: https://github.com/fedwiki/wiki/issues/97

from what I can discern, the decisions that Tor Browser makes related to what it whitelists might be pulled upstream into Mozilla (currently only manually) since they are adding font.system.whitelist in Firefox 52. here's some talk about it: https://bugzilla.mozilla.org/show_bug.cgi?id=1121643

Trac:
Username: vegansalad
Reviewer: N/A to N/A

My issue has to do with dingbats, not emojis. Should I create a new ticket about this or does it make sense to tackle it here?

As an example:

Simple things like the pencil doesn't work on TBB on Linux: https://www.fileformat.info/info/unicode/char/270E/browsertest.htm

In TBB on Windows, the pencil works via Micro$oft's copyrighted MS PGothic (which Linux can't use) but it doesn't look very good.

Oh, and a simple X works in TBB Windows but not TBB Linux: https://www.fileformat.info/info/unicode/char/2715/browsertest.htm

Also, black flag dingbats don't seem to work on TBB on either Windows or Linux: https://www.fileformat.info/info/unicode/char/2691/browsertest.htm

Trac:
Username: vegansalad

I should state, not every dingbat is broken in the Linux TBB. The Up Down Arrow works: https://www.fileformat.info/info/unicode/char/2195/browsertest.htm

Also, a regular multiplication sign works https://www.fileformat.info/info/unicode/char/00D7/browsertest.htm but a "MULTIPLICATION X" does not https://www.fileformat.info/info/unicode/char/2715/browsertest.htm

I'm not sure which whitelisted font allows some of these to work.

It might be good to find a good font candidate to whitelist in the Linux and Windows TBB to allow it to render more unicode characters than the current fonts allow. I haven't done any tests on the Mac TBB.

Trac:
Username: vegansalad

Maybe we should make specific font recommendations at this ticket in order to try to get support for emojis and dingbats: https://trac.torproject.org/projects/tor/ticket/20842

Trac:
Username: vegansalad

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=849042 RFP: fonts-noto-emoji -- "No Tofu" emoji font

Noto color emoji seems like it is being packages in Debian.

Might be a good TBB whitelisted font when it is released.

Noto emoji (black and white) seems to not have its source code online unfortunately.

Trac:
Username: vegansalad

Noto emoji (black and white) seems to not have its source code online unfortunately.

NotoEmoji-Regular.ttf is bundled with Tor Browser.

@yawning Thanks for the clarification. I was using https://arthuredelstein.github.io/tordemos/enumerate-fonts.html to get a master list of fonts that TBB ships and didn't see NotoEmoji-Regular.ttf on the list, but I don't think that page checks for it.

I saw that someone else filed a ticket https://trac.torproject.org/projects/tor/ticket/18364 about dingbats not working. You can see a good example of the problem here: http://danshort.com/HTMLentities/index.php?w=dingb

Actually, an even better example is happening right on this page. I'm not able to view the "bold" "italic" "link" or any other icons above the box that I'm typing in. Also, another person pointed out that the "reply to comment" icon to the right of every comment on this page is blank. https://trac.torproject.org/projects/tor/ticket/18860

I've also written about this error in two other trackers, but this has been determined to be TBB specific so both of those tickets were closed: https://github.com/fedwiki/wiki/issues/97 https://labs.riseup.net/code/issues/12154#change-65349

Dingbats are broken in TBB on Gnu+Linux and it isn't a fringe issue because it is happening on our very own Tracker.

Trac:
Username: vegansalad

uBlock is also having similar problems: https://trac.torproject.org/projects/tor/ticket/21320#ticket

Trac:
Username: vegansalad

#22087 (moved) is a duplicate.

Trac:
Cc: arthuredelstein to arthuredelstein, joel2017

https://bugzilla.mozilla.org/show_bug.cgi?id=1305010 https://emojipedia.org/unicode-9.0/

the native emoji are still not displayed in tor browser 7.0.4 in windows 10.

was it fixed for windows or is it still an open issue ?

Trac:
Username: joel2017

This seems to affect Linux users of TBB the most, but joel2017 said in the last comment that it is still causing problems for windows users.

I posted a proposed solution to the emoji support problem for GNU+Linux users over here: https://trac.torproject.org/projects/tor/ticket/18364#comment:12

Since there is already a ticket for emoji stuff for GNU+Linux, would it make sense to change the title of this ticket to a Windows specific problem, since the solution is going to be different than it would be for GNU+Linux? If so, I can remove my GNU+Linux related posts in this ticket and move them to 18364.

Trac:
Username: vegansalad

I think we can have #18364 (moved) as a child ticket to this one. Let me do that now.

It is still broken.

Trac:
Username: vegansalad

#33979 (moved) is a duplicate.

Can this ticket please get some love from someone?

I feel like it would be better to have the TBB for linux bundled with basic needed fonts, even if that makes the ROM slightly bigger, instead of leaving them broken. This is a basic feature that a browser should provide, the ability to render characters properly, especially characters that have been standardized for decades, like the multiplication sign: https://www.fileformat.info/info/unicode/char/2715/browsertest.htm

What are the blockers to allowing multiplication signs and other common characters render properly? I'm not trying to be rude, but it has been over three years since I've contributed to this issue.

Trac:
Username: vegansalad

If someone else is willing to write the patch and build a TBB for linux version, I'd be willing to test it. https://trac.torproject.org/projects/tor/ticket/18364#comment:13

Trac:
Username: vegansalad

mentioned in issue #18364 (moved)

mentioned in issue #22087 (moved)

mentioned in issue #31141 (moved)

mentioned in issue #31666 (moved)

mentioned in issue #33979 (moved)

moved to tpo/applications/tor-browser#18172 (closed)