Opened 2 years ago

Last modified 8 months ago

#22867 reopened defect

Some URLs are saved in the Tor Browser places.sqlite database as part of the browsing history

Reported by: cypherpunks Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor Browser Version:
Severity: Major Keywords: tbb-disk-leak
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The situation that I am describing may or may not be due to an error in the Tor Browser software ("defect" may not be the best category), but it seems somewhat anomalous and may be worth noting and may be of interest to others.

In summary, from what I can tell, an instance of the URL https://us-u.openx.net/w/1.0/pd?plm=10&ph=e26121be-304d-460c-92c5-0b3d1d4c9b7a (which may well have been embedded in a different page) ended up in Tor Browser's places.sqlite database as if it was saved as part of the user's browsing history. This on-disk database is used to store such information as bookmarks, browsing history, favicons, and annotations, among other things. At the same time, from what I understand, Tor Browser is preconfigured to not save any browsing history to the disk, and I do not remember reconfiguring Tor Browser with the purpose of changing this aspect.

The software configuration in question was a recent version of the Tor Browser Bundle (most likely 7.0.2) 32-bit running under Xubuntu Linux 16.04. I do not remember installing any additional extensions into Tor Browser.

When entering text in the address bar, Tor Browser displayed a list of suggestions underneath the address bar. This would be expected. In addition to pages that had been bookmarked, however, the list of suggestions also included the URL https://us-u.openx.net/w/1.0/pd?plm=10&ph=e26121be-304d-460c-92c5-0b3d1d4c9b7a. This URL was not among the pages that I had bookmarked in the browser. In addition, from what I remember, the inclusion of the openx.net URL in the suggestions list happened when text was entered into the address bar immediately after Tor Browser had been launched.

From information elsewhere, there is an SQLite database in a file, places.sqlite, that is used by Tor Browser (and Mozilla Firefox) for storing information relating to bookmarked sites and browsing history. In the case of the Tor Browser Bundle under Xubuntu Linux, the location of this file is tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default/places.sqlite, assuming the default root directory tor-browser_en-US.

With the command-line sqlite3 software, I examined the contents of the places.sqlite file. For the moz_places table, there were a number of entries, including sites that had been bookmarked but also an entry for the openx.net URL that had shown up in the suggestions list.

sqlite> select id,url,title,rev_host,visit_count,hidden,typed,frecency,datetime((moz_places.last_visit_date/1000000), 'unixepoch') from moz_places where url like "%openx.net%";
8|https://us-u.openx.net/w/1.0/pd?plm=10&ph=e26121be-304d-460c-92c5-0b3d1d4c9b7a||ten.xnepo.u-su.|1|1|0|-1|2017-06-27 21:59:47

According to the schema for the moz_places table and the results of the above query, the primary key index (id) for the openx.net URL is 8. The hidden field has a value of 1, indicating that the URL was one that was not navigated to directly by the user (i.e. the URL was for content that was embedded in a page) and the typed field has a value of 0, indicating that the URL was not typed directly into the location bar.

In the moz_bookmarks table, which holds information about bookmarked pages, there is (as expected) no entry for the openx.net URL:

sqlite> select * from moz_bookmarks where fk is 8;
sqlite>

According to this page, the fk column in the moz_bookmarks table holds the primary key index for the moz_places entry that was bookmarked.

The moz_inputhistory table does not appear to contain any entries:

sqlite> select * from moz_inputhistory;
sqlite>

Notably, the moz_historyvists table does contain a single entry. This entry corresponds to the openx.net URL.

sqlite> select id, from_visit, place_id, datetime((moz_historyvisits.visit_date/1000000), 'unixepoch'), visit_type from moz_historyvisits;
1|0|8|2017-06-27 21:59:47|5

In the results of the query, the place_id column has the value 8, which corresponds to the primary key index for the openx.net URL. The visit_type field has a value of 5, which indicates a permanent redirect, according to this page.

As far as whether the openx.net URL has appeared elsewhere, a Web search led to this "Cookie and Security Scan Report" for kurl8.com. In the report, there is mention of a similar URL, http://us-u.openx.net/w/1.0/pd?plm=10&ph=e26121be-304d-460c-92c5-0b3d1d4c9b7a&bi=1e76f9dc-b164-49ba-aef9-3f1a93d491e5.

Child Tickets

Attachments (1)

xxx.png (130.4 KB) - added by AxelF 20 months ago.

Download all attachments as: .zip

Change History (27)

comment:1 Changed 2 years ago by gk

Keywords: tbb-disk-leak added

comment:2 Changed 2 years ago by cypherpunks

Severity: NormalMajor

This happened to me multiple times (TB alpha), and it usually seems to be urls of trackers such as in my case https://ib.adnxs.com/.........

comment:3 Changed 2 years ago by cypherpunks

Summary: us-u.openx.net URL saved in the Tor Browser places.sqlite database as part of the browsing historySome URLs are saved in the Tor Browser places.sqlite database as part of the browsing history

Happened for me again with this url saved https://accounts.google.com/ServiceLogin?continue=https://www.blogger.com/comment-iframe.g?blogID........

comment:5 Changed 2 years ago by cypherpunks

It happened to me again (!!) this time with https://subscribe.washingtonpost.com/acq?promo=....

comment:6 Changed 2 years ago by cypherpunks

Resolution: not a bug
Status: newclosed

comment:7 Changed 2 years ago by Dbryrtfbcbhgf

Resolution: not a bug
Status: closedreopened

Please provide a explanation on why the "Resolution = Not a bug", before closing the ticket.

Last edited 2 years ago by Dbryrtfbcbhgf (previous) (diff)

comment:8 Changed 2 years ago by cypherpunks

Got one again with https://secure.adnxs.com/bounce?%2Fgetuid%3Fhttps%253A%252F%252Fusermatch.targeting.unrulymedia.com%252Fusermatch%252Fappnexus....

comment:9 Changed 2 years ago by dcf

#24866 is a duplicate. Extra information from that ticket:

I experienced it with Tor Browser 7.0.11.

The two URLs being remembered are similar to the ones previously reported:

  • https://accounts.google.com/ServiceLogin?continue=https://www.blogger.com/comment-iframe.g[...]
  • https://id.rlcdn.com/463496.gif?credir=https%3A%2F%2Fsimage4.pubmatic.com%2FAdServer[...]

The remembered URLs appear also in the ctrl+H history panel. When I open it, there are two entries corresponding to the months of the timestamps in the moz_historyvisits table. However it doesn't list any URLs: if I click the expander arrows, the arrows just disappear without expanding anything. But if I search for a string like "http", I can make the URLs appear.


Ticket #23704 is potentially related; it's about the browser remembering tabs after upgrading. comment:3:ticket:23704 says "TBB 7.0.5/6 has places.history.enabled set to true by default. And now TBB 7.5a5 has it set to false as a non-default value for unknown reason!" For what it's worth, my about:config looks like this:

Preference Name Status Type Value
places.history.enabled default boolean true
places.history.expiration.transient_current_max_pages user set integer 122334

Changed 20 months ago by AxelF

Attachment: xxx.png added

comment:10 Changed 20 months ago by AxelF

Can confirm the issue still persists. Those are some tricky tracking URLs in places.sqlite, they persist even after TBB is restarted.

https://trac.torproject.org/projects/tor/attachment/ticket/22867/xxx.png

comment:11 Changed 20 months ago by cypherpunks

Maybe this can help some folks reproduce it: I'm not 100% sure but it's been my observation that this always happened to me with fresh clean installs of the Tor Browser and never happened afterwards.

comment:12 in reply to:  11 Changed 20 months ago by cypherpunks

Replying to cypherpunks:

Maybe this can help some folks reproduce it: I'm not 100% sure but it's been my observation that this always happened to me with fresh clean installs of the Tor Browser and never happened afterwards.

That's wrong, got one today (but reproducing it with fresh installs is easier):

https://ap.lijit.com/pixel?redir=https:%2F%2Fads.servebom.com%2Fpartner%3Fcb%3D2579%26svc%3Dus%26id%3D24%26uid%3D$UID&sovrn_retry=true

comment:13 Changed 20 months ago by cypherpunks

Got one today on aps.org, what's interesting is I always use HTTPS Everywhere with "Block all unencrypted traffic" and this url was saved in its http form, what I did was to click on the download link for some presentation then cancel it,

http://absuploads.aps.org/presentation/upload/***********.pptx

comment:14 in reply to:  13 ; Changed 20 months ago by cypherpunks

Replying to cypherpunks:

I always use HTTPS Everywhere with "Block all unencrypted traffic"

Could you reproduce that with HTTPSE disabled?

comment:15 in reply to:  14 Changed 20 months ago by cypherpunks

Replying to cypherpunks:

Replying to cypherpunks:

I always use HTTPS Everywhere with "Block all unencrypted traffic"

Could you reproduce that with HTTPSE disabled?

Well you know this bug is difficult to reproduce, I just tried again with that same url with the same procedure and I couldn't reproduce it with HTTPSE enabled.

comment:16 Changed 19 months ago by cypherpunks

Got three ones in the same browsing session:

http://xml.ezmob.com/redirect?feed=<redacted>&auth=<redacted>&subid=<redacted>&url=juicyads.com

http://redir.juicyads.com/pu_uu.php?cb=<redacted>&uu=<redacted>

http://redir.juicyads.com/pu_uu.php?cb=<redacted-different-one-than-above>&uu=<redacted-different-one-than-above>

On the two first ones appeared in the about:newtab page.

Edit: There was even a fourth one that didn't show up in the history but was nevertheless there when you start writing in the address bar: https://xapi.juicyads.com/<redacted>.php?juicy_code=<redacted>&u=http://www.juicyads.rocks

Last edited 19 months ago by cypherpunks (previous) (diff)

comment:17 Changed 19 months ago by cypherpunks

More example on an URL that doesn't appear in the history but appears when typing in the address bar:
https://ssum-sec.casalemedia.com/usermatch?s=<redacted>&cb=https%3A%2F%2Fcms-xch.33across.com%2Fmatch%3Fbidder_id%3D2%26external_user_id%3D

comment:18 Changed 19 months ago by cypherpunks

More examples of URLs that don't appear in the history but appear when typing in the address bar:

https://ipw.metadsp.co.uk/ul_cb/sync?ssp=common&caller=unrulyx
https://bsw.digitru.st/syncx?ssp=unrulyx
https://x.bidswitch.net/ul_cb/sync?ssp=unrulyx
https://gce-sc.bidswitch.net/sync?ssp=unrulyx&bsw_own_uuid=
https://secure.adnxs.com/bounce?%2Fgetuid%3Fhttps%253A%252F%252Fusermatch.targeting.unrulymedia.com%252Fusermatch%252Fappnexus%252F%2524UID

Last edited 19 months ago by cypherpunks (previous) (diff)

comment:19 in reply to:  18 ; Changed 19 months ago by Dbryrtfbcbhgf

Replying to cypherpunks:

More examples of URLs that don't appear in the history but appear when typing in the address bar:

https://ipw.metadsp.co.uk/ul_cb/sync?ssp=common&caller=unrulyx
https://bsw.digitru.st/syncx?ssp=unrulyx
https://x.bidswitch.net/ul_cb/sync?ssp=unrulyx
https://gce-sc.bidswitch.net/sync?ssp=unrulyx&bsw_own_uuid=
https://secure.adnxs.com/bounce?%2Fgetuid%3Fhttps%253A%252F%252Fusermatch.targeting.unrulymedia.com%252Fusermatch%252Fappnexus%252F%2524UID

Can you download compile the latest esr 60 TorBrowser nightly and see if the bug still occurs?

Linux compile instructions
https://github.com/arthuredelstein/tor-browser-build/blob/22563/README

And here is The latest TorBrowser source code.
https://gitweb.torproject.org/builders/tor-browser-build.git/

comment:20 in reply to:  19 Changed 19 months ago by cypherpunks

Replying to Dbryrtfbcbhgf:

Can you download compile the latest esr 60 TorBrowser nightly and see if the bug still occurs?

I'll wait till the alpha to test esr 60 so don't worry, I'll keep testing the hell out of it as long as I'm alive ;)

comment:21 Changed 18 months ago by cypherpunks

#26315 is a duplicate.

comment:22 Changed 18 months ago by cypherpunks

Is there some bugzilla bug for this? I still encounter this bug and I'd be happy to see it marked as resolved on Mozilla's favorite bug tracking status piece of code.

comment:23 Changed 18 months ago by cypherpunks

Resolution: worksforme
Status: reopenedclosed

More than two weeks of 8.0a9 usage and not a single url saved this time, so seems it has been fixed/no longer reproducible with the switch to ff60-esr. Please reopen if it happens again, thanks!

comment:24 Changed 17 months ago by ProTipGuyFWIWWeLoveARMA

By the way I think I found the reason. About:newtab had on TB based ff52-esr "Display top sites" enabled, maybe when a newtab fetch is done some kind of bug happens due to that.

comment:25 in reply to:  23 ; Changed 8 months ago by cypherpunks

Resolution: worksforme
Status: closedreopened

Replying to cypherpunks:

More than two weeks of 8.0a9 usage and not a single url saved this time, so seems it has been fixed/no longer reproducible with the switch to ff60-esr. Please reopen if it happens again, thanks!

Seems like I was wrong, it happened with TB 8.5a10: https://accounts.google.com/ServiceLogin?continue=https://www.blogger.com/comment-iframe.g?blogID......

comment:26 in reply to:  25 Changed 8 months ago by gk

Replying to cypherpunks:

Replying to cypherpunks:

More than two weeks of 8.0a9 usage and not a single url saved this time, so seems it has been fixed/no longer reproducible with the switch to ff60-esr. Please reopen if it happens again, thanks!

Seems like I was wrong, it happened with TB 8.5a10: https://accounts.google.com/ServiceLogin?continue=https://www.blogger.com/comment-iframe.g?blogID......

Hm. Is that with a clean, unmodified Tor Browser as we ship it? We could really use some steps to reproduce this behavior... :(

Note: See TracTickets for help on using tickets.