Opened 3 years ago

Closed 3 years ago

#13019 closed defect (fixed)

New locale fingerprinting capabilities in FF31ESR

Reported by: mikeperry Owned by: arthuredelstein
Priority: High Milestone:
Component: Applications/Tor Browser Version:
Severity: Keywords: ff31-esr, tbb-fingerprinting, TorBrowserTeam201503, tbb-4.5-alpha
Cc: mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Various element properties (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-type) as well as the String, Date, and Number objects now expose locale-specific functions.

Also, the Intl object exposes locale information:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl

If extensions.torbutton.spoof_english is set, these functions should all behave as if the locale was en_US.

Child Tickets

Change History (38)

comment:1 Changed 3 years ago by mikeperry

See also #5926 for older, pre-31esr instances of this.

comment:2 in reply to:  description Changed 3 years ago by arthuredelstein

Replying to mikeperry:

Various element properties (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-type) ...

I looked through the different input element types but I didn't find any that appear to leak locale information. It seems these input types may display (at least in theory) some sort of locale-specific information, but when JS accesses the .value property, it receives non-localized data. Please correct me if I'm missing something!

Changed 3 years ago by arthuredelstein

comment:3 Changed 3 years ago by arthuredelstein

Status: newneeds_review

Posted a patch which covers String, Date, and Number and Intl objects. Not sure if I'm missing something on Element properties.

comment:4 Changed 3 years ago by arthuredelstein

Keywords: MikePerry201409R added

comment:5 Changed 3 years ago by mikeperry

Status: needs_reviewneeds_revision

There are a couple issues with this patch. You shouldn't need to store the current locale just to have something to do in DefaultJSLocaleSetter::Run() when the pref is empty. If the pref is empty, just do nothing. This eliminates the need to export JS_GetDefaultLocale() as well.

But beyond this, there's actually two bugs in the storage of this locale information. In the case of DefaultJSLocaleSetter::jsLocale, you leak it on XPCOM shutdown. In the case of DefaultJSLocaleSetting::systemLocale, you are keeping a pointer to a static buffer, so that subsequent calls to setlocale may cause this memory to get replaced with something else. It probably will always contain the actual current locale, but this seems a bit sloppy to rely on.

comment:6 Changed 3 years ago by mikeperry

Keywords: TorBrowserTeam201410 added; TorBrowserTeam201409 removed

comment:7 in reply to:  5 Changed 3 years ago by arthuredelstein

Replying to mikeperry:

There are a couple issues with this patch. You shouldn't need to store the current locale just to have something to do in DefaultJSLocaleSetter::Run() when the pref is empty. If the pref is empty, just do nothing. This eliminates the need to export JS_GetDefaultLocale() as well.

My thinking is that it is nice to be able to set this pref at runtime. So if the user sets the pref to empty at some point, then the original locale will be restored. That's why we need to store the original locale on startup. Otherwise we would have to require restarting the browser whenever the pref is changed.

But beyond this, there's actually two bugs in the storage of this locale information. In the case of DefaultJSLocaleSetter::jsLocale, you leak it on XPCOM shutdown. In the case of DefaultJSLocaleSetting::systemLocale, you are keeping a pointer to a static buffer, so that subsequent calls to setlocale may cause this memory to get replaced with something else. It probably will always contain the actual current locale, but this seems a bit sloppy to rely on.

Thanks for catching these two bad mistakes. I believe I've fixed them in the new patch version, added above. I also changed the implementation to use Preferences::RegisterCallback/UnRegisterCallback, which is more concise than the nsIObserver approach.

comment:8 Changed 3 years ago by arthuredelstein

Status: needs_revisionneeds_review

comment:9 Changed 3 years ago by mikeperry

Ok. Is there a reason why there is a 5926+ branch in your repo, but not this patch? I was looking at that branch for a while, and it seemed to fix these problems but did not use the RegisterCallback/UnregisterCallback, as the attachment does.

I'm a little nervous about the use of free() to free stuff allocated with the JS-runtime allocators. They appear to all boil down to the same js_free right now, but that may change? Not sure if we should put this as an XXX, or what. The JS_Free() calls do not seem to match JS_Strdup used by JS_GetDefaultLocale, in that they don't take a JSRuntime as an argument...

We also still need to set this new pref in the Torbutton code that sets extensions.torbutton.spoof_*, too.

I am thinking for both of these reasons this might not make it into the FF31 release on Tuesday, but we should perhaps aim for the next release after that for this.

comment:10 in reply to:  9 Changed 3 years ago by arthuredelstein

Replying to mikeperry:

Ok. Is there a reason why there is a 5926+ branch in your repo, but not this patch? I was looking at that branch for a while, and it seemed to fix these problems but did not use the RegisterCallback/UnregisterCallback, as the attachment does.

Sorry for the confusion -- this was an old branch.

I'm a little nervous about the use of free() to free stuff allocated with the JS-runtime allocators. They appear to all boil down to the same js_free right now, but that may change? Not sure if we should put this as an XXX, or what. The JS_Free() calls do not seem to match JS_Strdup used by JS_GetDefaultLocale, in that they don't take a JSRuntime as an argument...

I've updated the patch to use the JS_free() call instead. You're right that it doesn't match, but I think it's OK to use nullptr for the JSContext here.

We also still need to set this new pref in the Torbutton code that sets extensions.torbutton.spoof_*, too.

I've added a patch for torbutton that does this.

(Please see two patches attached today, 2014-10-10.)

Last edited 3 years ago by arthuredelstein (previous) (diff)

comment:11 Changed 3 years ago by mikeperry

Keywords: tbb-4.5-alpha added

comment:12 Changed 3 years ago by mikeperry

Resolution: fixed
Status: needs_reviewclosed

It looks like there's a new memory leak in that patch. You were strdupping an already strdupped pointer from JS_GetDefaultLocale().

I fixed this in a fixup commit and merged this and the torbutton fix.

comment:13 Changed 3 years ago by arthuredelstein

You're right. Thanks for catching it.

comment:14 in reply to:  12 ; Changed 3 years ago by gk

Resolution: fixed
Status: closedreopened

Replying to mikeperry:

It looks like there's a new memory leak in that patch. You were strdupping an already strdupped pointer from JS_GetDefaultLocale().

I fixed this in a fixup commit.

This seems to break the build:

/home/ubuntu/build/tor-browser/xpcom/build/nsXPComInit.cpp: In function 'void {anonymous}::StartWatchingDefaultLocalePref()':
/home/ubuntu/build/tor-browser/xpcom/build/nsXPComInit.cpp:496:39: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]
     sJSLocale = JS_GetDefaultLocale(rt);
                                       ^
make[5]: *** [nsXPComInit.o] Error 1
make[5]: Leaving directory `/home/ubuntu/build/tor-browser/obj-i686-pc-linux-gnu/xpcom/build'
make[4]: *** [xpcom/build/compile] Error 2
make[4]: *** Waiting for unfinished jobs....

comment:15 in reply to:  14 Changed 3 years ago by arthuredelstein

Replying to gk:

Replying to mikeperry:

It looks like there's a new memory leak in that patch. You were strdupping an already strdupped pointer from JS_GetDefaultLocale().

I fixed this in a fixup commit.

This seems to break the build:

I looked at it again, and I think the original patch is correct. JS_GetDefaultLocale is a minimal wrapper for JSRuntime::getDefaultLocale(), which returns a pointer to an internal JSRuntime member (char* JSRuntime::defaultLocale). The JS_strdup that allocates defaultLocale is matched internally in the JSRuntime class by a js_free in JSRuntime::resetDefaultLocale():

void                                                                                                                                                                                                        
JSRuntime::resetDefaultLocale()                                                                                                                                                                             
{                                                                                                                                                                                                           
    js_free(defaultLocale);                                                                                                                                                                                 
    defaultLocale = nullptr;                                                                                                                                                                                
}                                                                                                                                                                                                           
                                                                                                                                                                                                            
const char *                                                                                                                                                                                                
JSRuntime::getDefaultLocale()                                                                                                                                                                               
{                                                                                                                                                                                                           
    if (defaultLocale)                                                                                                                                                                                      
        return defaultLocale;                                                                                                                                                                               
                                                                                                                                                                                                            
    char *locale, *lang, *p;                                                                                                                                                                                
#ifdef HAVE_SETLOCALE                                                                                                                                                                                       
    locale = setlocale(LC_ALL, nullptr);                                                                                                                                                                    
#else                                                                                                                                                                                                       
    locale = getenv("LANG");                                                                                                                                                                                
#endif                                                                                                                                                                                                      
    // convert to a well-formed BCP 47 language tag                                                                                                                                                         
    if (!locale || !strcmp(locale, "C"))                                                                                                                                                                    
        locale = const_cast<char*>("und");                                                                                                                                                                  
    lang = JS_strdup(this, locale);                                                                                                                                                                         
    if (!lang)                                                                                                                                                                                              
        return nullptr;                                                                                                                                                                                     
    if ((p = strchr(lang, '.')))                                                                                                                                                                            
        *p = '\0';                                                                                                                                                                                          
    while ((p = strchr(lang, '_')))                                                                                                                                                                         
        *p = '-';                                                                                                                                                                                           
                                                                                                                                                                                                            
    defaultLocale = lang;                                                                                                                                                                                   
    return defaultLocale;                                                                                                                                                                                   
} 

So we need an extra strdup to get an independent copy into nsXPComInit.cpp:static char* sJSLocale;

comment:16 Changed 3 years ago by mikeperry

Resolution: fixed
Status: reopenedclosed

You are right, I missed that this string was also being stored and managed by the JS interpreter. Sorry, this one was my mistake. I reverted my fixup.

Calling this fixed, as these are both merged in torbutton and tor-browser for 4.5-alpha.

comment:17 Changed 3 years ago by gk

Resolution: fixed
Status: closedreopened

While investigating #15268 it turned out that even in 4.5-alpha-1 the OS locale is still leaking. The example URL http://www.fixtheworldproject.net/ shows a german date format on a german Windows 7 machine although I tested with an en-US Tor Browser.
We missed something in the patch, thus re-opening.

comment:18 Changed 3 years ago by mcs

Cc: mcs added

comment:19 Changed 3 years ago by gk

Keywords: TorBrowserTeam201503 added; MikePerry201409R TorBrowserTeam201410 removed
Owner: changed from tbb-team to arthuredelstein
Status: reopenedassigned

comment:20 in reply to:  17 Changed 3 years ago by arthuredelstein

Replying to gk:

While investigating #15268 it turned out that even in 4.5-alpha-1 the OS locale is still leaking. The example URL http://www.fixtheworldproject.net/ shows a german date format on a german Windows 7 machine although I tested with an en-US Tor Browser.
We missed something in the patch, thus re-opening.

Looks like I missed that some parts of the code require en-US and other parts en_US. :( Here is my fixup for the patch, that also simplifies the pref.
https://github.com/arthuredelstein/tor-browser/commit/c2338da39ec3cde6189d4467dd80b7aeebd1b462
Currently trying to get my gitian Windows build to work so I can test this on Windows.

comment:21 Changed 3 years ago by mikeperry

Last minute pref changes like this are not a good idea, especially since we've not consolidated the rest of our changes around this new resistFingerprinting pref, having it only apply to locale at this point seems weird. Not to mention we need to update Torbutton now just for this change, or it will stop working properly.

I'd much rather just fix the test and the underscore issue in this patch. It will make the delta smaller and easier to review and deal with at this late stage.

comment:22 Changed 3 years ago by mikeperry

Status: assignedneeds_revision

comment:23 Changed 3 years ago by arthuredelstein

After further discussion with Mike, I have kept the boolean pref, but renamed it to javascript.use_us_english_locale

Here's the revised patch for Tor Browser:
https://github.com/arthuredelstein/tor-browser/commit/917f5dfb669585ceb4b42f68ed93988ada61e158

And the accompanying patch for torbutton:
https://github.com/arthuredelstein/torbutton/commit/dcbd97834f991c6b4ef5f86154e6823251511075

comment:24 in reply to:  23 ; Changed 3 years ago by gk

Replying to arthuredelstein:

After further discussion with Mike, I have kept the boolean pref, but renamed it to javascript.use_us_english_locale

Here's the revised patch for Tor Browser:
https://github.com/arthuredelstein/tor-browser/commit/917f5dfb669585ceb4b42f68ed93988ada61e158

And the accompanying patch for torbutton:
https://github.com/arthuredelstein/torbutton/commit/dcbd97834f991c6b4ef5f86154e6823251511075

These patches do not solve the problem. I made sure I really built Tor Browser with the commit above and had the proper Torbutton version running. Still, the locale of the OS was not spoofed.

comment:25 Changed 3 years ago by mikeperry

Ugh, I did that thing again where I was in a rush and merged a branch from my build that I tested that contained this fix. But since there is at least a test now, maybe we can still call this an improvement. Definitely not calling this closed, and I think it may have just missed the train for 4.5a5 (unless something else delays the build).

comment:26 in reply to:  24 ; Changed 3 years ago by arthuredelstein

Replying to gk:

Replying to arthuredelstein:

After further discussion with Mike, I have kept the boolean pref, but renamed it to javascript.use_us_english_locale

Here's the revised patch for Tor Browser:
https://github.com/arthuredelstein/tor-browser/commit/917f5dfb669585ceb4b42f68ed93988ada61e158

And the accompanying patch for torbutton:
https://github.com/arthuredelstein/torbutton/commit/dcbd97834f991c6b4ef5f86154e6823251511075

These patches do not solve the problem. I made sure I really built Tor Browser with the commit above and had the proper Torbutton version running. Still, the locale of the OS was not spoofed.

I'm still having difficulty getting my Windows cross-compile to work. However, I think I understand better now what is happening on Windows. It turns out that Windows does not accept the "en_US" locale in setlocale(...). Instead you have to use "English_United States.1252". I used the latest alpha version of Tor Browser and set "javascript.default_locale" to "English_United States.1252". In that case, http://www.fixtheworldproject.net/ shows US English-formatted dates on Windows 7 when the system locale is set to German. As does Date().toLocaleString().

So I'm posting an extra fixup patch to my 13019+3 branch:
https://github.com/arthuredelstein/tor-browser/commits/13019%2B3

I would be grateful if someone could build this and test on Windows.

comment:27 in reply to:  26 Changed 3 years ago by arthuredelstein

Mike pointed out that "English_United States.1252" might conceivably be unavailable on some Windows machines. So I tried the "C" locale for Windows, OSX, and Linux using the latest alpha, and this works for Date().toLocaleFormat().

So here's yet another version, using the "C" locale for setlocale(...). Note that the JS_SetDefaultLocale call uses "en-US" as before, and applies not to Date() but to the Intl API calls, such as Intl.Collator().resolvedOptions().locale.

https://github.com/arthuredelstein/tor-browser/commits/13019+5

comment:28 Changed 3 years ago by gk

Works for me although (now?) I can see different date formats used based on OS.

Mac: Wed Mar 25 ...
Win: Wednesday, March 25 ...
Linux: 3/25 ...

I have not looked why this is happening.

comment:29 Changed 3 years ago by gk

Setting javascript.use_us_english_locale to false does not change anything wrt to the differing date formats (at least not on OS X nor on Linux; I currently don't have an en-US Windows available to test).

comment:30 in reply to:  29 ; Changed 3 years ago by arthuredelstein

Replying to gk:

Setting javascript.use_us_english_locale to false does not change anything wrt to the differing date formats (at least not on OS X nor on Linux; I currently don't have an en-US Windows available to test).

I just tested it on OS X, and it is working for me. However, if you are running Tor Browser from the command line, it is necessary to make sure that the shell locale is set to something other than English.

I did the following:

bash-3.2$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
bash-3.2$ export LC_ALL="de_DE"
bash-3.2$ locale
LANG="en_US.UTF-8"
LC_COLLATE="de_DE"
LC_CTYPE="de_DE"
LC_MESSAGES="de_DE"
LC_MONETARY="de_DE"
LC_NUMERIC="de_DE"
LC_TIME="de_DE"
LC_ALL="de_DE"
bash-3.2$ ./mach run
 0:00.42 /projects/torproject/tor-browser31/obj-x86_64-apple-darwin13.4.0/dist/Nightly.app/Contents/MacOS/firefox -no-remote -foreground

Then in the JavaScript console:

// set pref to true
new Date().toLocaleFormat()
--> "Thu Mar 26 10:31:21 2015"
// set pref to false
new Date().toLocaleFormat()
--> "Do 26 Mär 10:31:38 2015"

comment:31 in reply to:  30 ; Changed 3 years ago by gk

Replying to arthuredelstein:

Replying to gk:

Setting javascript.use_us_english_locale to false does not change anything wrt to the differing date formats (at least not on OS X nor on Linux; I currently don't have an en-US Windows available to test).

I just tested it on OS X, and it is working for me. However, if you are running Tor Browser from the command line, it is necessary to make sure that the shell locale is set to something other than English.

I think you misunderstood me. While testing I realized that you can infer the underlying OS if you just get an english date format back as the particular english date format used differs across OSes. This was somewhat surprising to me. Setting the pref above to false did not change that.

comment:32 in reply to:  31 Changed 3 years ago by arthuredelstein

Replying to gk:

I think you misunderstood me. While testing I realized that you can infer the underlying OS if you just get an english date format back as the particular english date format used differs across OSes. This was somewhat surprising to me. Setting the pref above to false did not change that.

Sorry, I did misunderstand. I've posted a ticket to keep us reminded of this issue: #15473

comment:33 Changed 3 years ago by mikeperry

Resolution: fixed
Status: needs_revisionclosed

Ok, I merged this for 4.5a5.

Note: See TracTickets for help on using tickets.