Opened 2 months ago

Last modified 6 weeks ago

#33702 assigned defect

RSA_get0_d could not be located in the dynamic link library tor.exe

Reported by: ner0 Owned by:
Priority: Medium Milestone:
Component: Core Tor/Tor Version:
Severity: Normal Keywords:
Cc: nickm, teor, mcs Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

OS: Win10 (win64)

Tor Browser will not run, when attempting to establish a connection the following error appeats:

The procedure entry point RSA_get0_d could not be located in the dynamic link library C:\Users\MyUser\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe

Child Tickets

Attachments (6)

procmon.png (251.7 KB) - added by ner0 2 months ago.
procmon_logs.7z (98 bytes) - added by ner0 2 months ago.
EnableExternalManifest.reg (126 bytes) - added by cypherpunks 7 weeks ago.
tor.exe.Manifest (417 bytes) - added by cypherpunks 7 weeks ago.
openssl.dll.Manifest (327 bytes) - added by cypherpunks 7 weeks ago.
internalmanifest.txt (59 bytes) - added by cypherpunks 7 weeks ago.

Download all attachments as: .zip

Change History (49)

comment:1 Changed 2 months ago by ner0

Could not specify before, but this starts with latest browser update v9.0.7 (tor 0.4.2.7)

Last edited 2 months ago by ner0 (previous) (diff)

comment:2 Changed 2 months ago by boklm

A user reported the same issue on the blog:
https://blog.torproject.org/comment/287165#comment-287165

comment:3 Changed 2 months ago by boklm

I think this is probably related with this change:
https://gitweb.torproject.org/tor.git/commit/?h=maint-0.4.2&id=8abdb394893a1704f885278f5f5d7913cdf516c9

We are including ./Browser/TorBrowser/Tor/libssl-1_1.dll in the bundle. I am wondering if tor.exe gets linked to wrong dll for some reason.

comment:4 Changed 2 months ago by boklm

Cc: nickm teor added

nickm, teor: do you have any idea about what could be causing this error?

comment:5 Changed 2 months ago by mcs

One additional data point: I am unable to reproduce this problem on a Win10 64-bit laptop that has minimal software installed (no anti-malware software except the built-in Defender).

comment:6 in reply to:  2 Changed 2 months ago by ner0

Replying to boklm:

A user reported the same issue on the blog:
https://blog.torproject.org/comment/287165#comment-287165

That was also me.

Replying to mcs:

One additional data point: I am unable to reproduce this problem on a Win10 64-bit laptop that has minimal software installed (no anti-malware software except the built-in Defender).

Can confirm. In the system showing the issue I have Comodo AV, I hav esince cloned the system to a VM, removed tons of software including AV and it doesn't work all the same.

Last edited 2 months ago by ner0 (previous) (diff)

comment:7 Changed 2 months ago by ner0

I have uninstalled virtually everything on the system and the issue persists. Should be noted that the issue does not happen when using the 32bit version of Tor Browser, only 64bit.

comment:8 Changed 2 months ago by cypherpunks

update

uninstalled virtually everything

reinstall torbrowser?

comment:9 in reply to:  8 Changed 2 months ago by ner0

Replying to cypherpunks:

update

uninstalled virtually everything

reinstall torbrowser?

Yes, reinstalled it multiple times. Also purged the whole folder, as well as Program Files and ProgramData leftovers. Tested with new Win user to exclude user profile issues. As mentioned by mcs, I also have been able to use the new win64 release on a another machine with no issue. On my computer (and VM) only the 32bit version is working, 64bit throws the error mentioned in the title.

I could try to be of further assistance if I knew a good way to debug or gather logging information that could be useful.

comment:10 in reply to:  4 ; Changed 2 months ago by nickm

Replying to boklm:

nickm, teor: do you have any idea about what could be causing this error?

Most frequently this kind of error happens when Tor is compiled with one version of openssl, but linked with another substantially different. I don't know so much about Windows linking behavior, however.

comment:11 Changed 2 months ago by nickm

Is there some kind of tool that can be used to tell which specific libraries Windows is trying to link Tor against here? If so, that would probably help figure this out.

Changed 2 months ago by ner0

Attachment: procmon.png added

comment:12 Changed 2 months ago by ner0

I don't know if procmon logs can be useful on their own.
I've attached logs and side by side comparison of v9.0.6 and v9.0.7, the issue arises very early on, not sue what is the trigger though.

Changed 2 months ago by ner0

Attachment: procmon_logs.7z added

comment:13 Changed 2 months ago by cypherpunks

12:01:42.7784859 PM	tor.exe	25560	CreateFile	C:\Windows\System32\libcrypto-1_1-x64.dll	SUCCESS	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

libcrypto-1_1-x64.dll in system32 dir (why?)

comment:14 Changed 2 months ago by boklm

When looking at Logfile_v9.0.7_win64_fail_hl.PML I see that it contains the string C:\Windows\System32\libssl-1_1-x64.dll.

So it looks like libssl is loaded from the Windows system directory instead of the Tor Browser directory.

When looking at:
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order
The directory from which the application loaded is first, so I don't know why it is not loaded from Tor Browser's directory.

However in Factors That Affect Searching they also say:

If the DLL is on the list of known DLLs for the version of Windows on which the application is running, the system uses its copy of the known DLL (and the known DLL's dependent DLLs, if any) instead of searching for the DLL. For a list of known DLLs on the current system, see the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs.

Maybe libssl is part of the known DLLs?

comment:15 in reply to:  13 Changed 2 months ago by ner0

Replying to cypherpunks:

12:01:42.7784859 PM	tor.exe	25560	CreateFile	C:\Windows\System32\libcrypto-1_1-x64.dll	SUCCESS	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

libcrypto-1_1-x64.dll in system32 dir (why?)

The OpenSSL Toolkit... that's where it came from.

that might have been a contributing factor, but weirder is how it was "fixed":

  1. Ran Tor Browser once as administrator;
  2. Done.

I don't have a clue as to what/why.
I can now run it without elevated privileges just fine, no more issues.

Last edited 2 months ago by ner0 (previous) (diff)

comment:16 in reply to:  14 Changed 2 months ago by ner0

Replying to boklm:

When looking at Logfile_v9.0.7_win64_fail_hl.PML I see that it contains the string C:\Windows\System32\libssl-1_1-x64.dll.

So it looks like libssl is loaded from the Windows system directory instead of the Tor Browser directory.

When looking at:
https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order
The directory from which the application loaded is first, so I don't know why it is not loaded from Tor Browser's directory.

However in Factors That Affect Searching they also say:

If the DLL is on the list of known DLLs for the version of Windows on which the application is running, the system uses its copy of the known DLL (and the known DLL's dependent DLLs, if any) instead of searching for the DLL. For a list of known DLLs on the current system, see the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs.

Maybe libssl is part of the known DLLs?

None of the DLLs on that registry key matches the ones in Tor's folder, can't speak to their dependency though, it's possible it could be called by any one of them.

Last edited 2 months ago by ner0 (previous) (diff)

comment:17 Changed 2 months ago by ner0

Just want to confirm that the presence of a different version of libcrypto-1_1-x64.dll in System32 (v1.1.0.10) was the actual issue (thanks for noticing it, cypherpunks). Not sure why running as admin would "bypass" that permanently and also very curious why, as boklm mentioned, Windows would ignore the DLL present in Tor Browser's folder which should have had priority. I have removed all OpenSSL DLLs from System32, I am not sure if it was me who put them there on my own, but its best to remove them to avoid future issues.

Another weird thing is that my system has not changed since the last release up to current, so maybe partially was due to:
boklm:

I think this is probably related with this change:
https://gitweb.torproject.org/tor.git/commit/?h=maint-0.4.2&id=8abdb394893a1704f885278f5f5d7913cdf516c9

Last edited 2 months ago by ner0 (previous) (diff)

comment:18 Changed 2 months ago by cypherpunks

comment:19 Changed 2 months ago by cypherpunks

new topic: Windows10 ignore the DLL present in the directory from which the application loaded.

comment:20 in reply to:  18 Changed 2 months ago by ner0

I didn't get the d3dcompiler_47.dll reference.
If anything, I get the idea that it is possible to prevent DLL pre-loading attacks by implementing some measures and not that it is so by default (even though my experience in this topic goes against that). Firefox, for example: https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29

Do it's a bit conflicting what I'm interpreting from Microsoft docs. Which excerpts specifically say that Windows (10) excludes the working dir by default from known DLL search?

Indeed, DLL loading has become the important topic.

comment:21 Changed 2 months ago by cypherpunks

Firefox, for example: ​https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29

"SetDefaultDllDirectories affect the implicit loading in child processes" huh?

comment:22 Changed 2 months ago by pili

Severity: BlockerNormal
Version: Tor: unspecified

comment:23 Changed 2 months ago by MadCabbit

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away.

comment:24 in reply to:  23 ; Changed 2 months ago by ner0

Replying to MadCabbit:

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away.

Have you tried removing possible overriding DLLs or just running Tor Browser once as administrator? For me both approaches solved the issue. In my case I shouldn't have had those DLLs in system32 to begin with, but I also think Tor Browser bears responsibility for not loading the correct (workingdir) DLL.

Last edited 2 months ago by ner0 (previous) (diff)

comment:25 in reply to:  24 Changed 8 weeks ago by MadCabbit

I'm not sure what DLLs would be overriding, but I tried running Tor as administrator, and that fixed the problem.
Replying to ner0:

Replying to MadCabbit:

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away.

Have you tried removing possible overriding DLLs or just running Tor Browser once as administrator? For me both approaches solved the issue. In my case I shouldn't have had those DLLs in system32 to begin with, but I also think Tor Browser bears responsibility for not loading the correct (workingdir) DLL.

comment:26 in reply to:  10 ; Changed 8 weeks ago by teor

Replying to nickm:

Replying to boklm:

nickm, teor: do you have any idea about what could be causing this error?

Most frequently this kind of error happens when Tor is compiled with one version of openssl, but linked with another substantially different. I don't know so much about Windows linking behavior, however.

We have seen similar issues in our Windows CI.

We tried copying the correct DLLs into the same directory as the exe, but that didn't help. We ended up disabling the failing unit test.

But if you work out how to modify the DLL search path, or find the correct directories to copy the DLLs into, please let us know.

I also don't know why running Tor Browser once as an administrator fixes the issue.

comment:27 Changed 8 weeks ago by cypherpunks

But if you work out how to modify the DLL search path, or find the correct directories to copy the DLLs into, please let us know.

manifest?
https://docs.microsoft.com/en-us/windows/win32/sbscs/assembly-manifests

comment:28 in reply to:  26 Changed 8 weeks ago by ner0

Replying to teor:

I also don't know why running Tor Browser once as an administrator fixes the issue.


At first glance it seems to be a Microsoft "feature", but lets not lose focus that Tor is the actual issue. To the extent that Windows 10 is concerned, my testing had the following results in terms of DLL search order.

  • When running as current user, '\Windows\System32\' is searched first, then \Windows\, and so on.
  • When current user invokes "run as administrator", current folder is searched first.

I can't really wrap my head around Microsoft's intention here, if the idea was to protect the user from potential system hijacking I would assume the OS to take the opposite approach since non-system files and folders would be easier to compromise, thus it would make much more sense for an elevated process to first search the system folder instead of an easily manipulable user folder. That aside, why does using "run as administrator" once fixes it for subsequent runs as a standard user? No clue.

Anyway, one thing is sure, Firefox process itself handles DLL search order perfectly regardless of the elevation while Tor process does not. The earliest example is firefox.exe searching for the DLL version.dll inside Tor Browser's folder first before searching in \Windows\System32\, meanwhile tor.exe searches for all DLLs first in \Windows\System32\, then '\Windows\', etc. I think Tor should implement the approach mentioned before: https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29

comment:29 Changed 7 weeks ago by teor

That Firefox code is for system DLLs, not application-specific DLLs.

We probably want something like this:

      LoadLibraryExW(fullModulePath, nullptr, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);

Or maybe there are flags we can set when linking the DLL?

Edit:

Here is the reference doc:
https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadlibraryexa

Last edited 7 weeks ago by teor (previous) (diff)

comment:30 Changed 7 weeks ago by gk

Component: Applications/Tor BrowserCore Tor/Tor

Sounds like a tor ticket now, no?

comment:31 in reply to:  30 Changed 7 weeks ago by teor

Replying to gk:

Sounds like a tor ticket now, no?

I'm not sure if it is an issue with:

  1. tor's code
  2. tor's build process
  3. the way Tor Browser builds or packages tor
  4. the local system that Tor Browser is installed on

But everything works fine in our Windows CI. So even if it is something we can fix (1 or 2), it's not a universal issue. So it's not going to be easy to test. (We might be able to test is using the OpenSSL version mismatch in #33643, but that's also unreliable.)

If we find out it's something Tor Browser can fix, we'll hand it back to you.

comment:32 Changed 7 weeks ago by teor

Owner: tbb-team deleted
Status: newassigned

comment:33 Changed 7 weeks ago by cypherpunks

Replying to gk:

Sounds like a tor ticket now, no?

FWIW, tor runs in Tor Browser with Virtualized UAC Disabled when it's on Windows 10.
According to #22450, they won't fix Windows issues anytime soon.

comment:34 Changed 7 weeks ago by mcs

Cc: mcs added

comment:35 in reply to:  33 Changed 7 weeks ago by teor

Replying to cypherpunks:

Replying to gk:

Sounds like a tor ticket now, no?

FWIW, tor runs in Tor Browser with Virtualized UAC Disabled when it's on Windows 10.

Can you help me understand how Virtualised UAC relates to this issue?

According to #22450, they won't fix Windows issues anytime soon.

We don't have a lot of Windows developers working on tor.
So we prioritise issues based on their impact on users.

I'm not sure that using a compatibility mode is actually a bug.
Does it cause any user-visible issues?

We'd love more people to help us patch and test Windows issues.

comment:36 Changed 7 weeks ago by cypherpunks

@ner0
"an app folder" != "a current folder"

and if "the app folder" == "the current folder" then windows_10 exclude "an app folder" ("a current folder" is last one)

Changed 7 weeks ago by cypherpunks

Attachment: EnableExternalManifest.reg added

Changed 7 weeks ago by cypherpunks

Attachment: tor.exe.Manifest added

Changed 7 weeks ago by cypherpunks

Attachment: openssl.dll.Manifest added

Changed 7 weeks ago by cypherpunks

Attachment: internalmanifest.txt added

comment:37 Changed 7 weeks ago by teor

Hi cypherpunks, can you please explain what these attachments do?

It's hard for us to guess what you're thinking.

comment:39 Changed 7 weeks ago by teor

I know about Windows manifests.

It would be more helpful if we had instructions, or git pull requests. Otherwise we just have to guess what you want done with the files.

Here's my review of what I think you want done:

  • we can't modify users' registries. Tor doesn't build packages for Windows. And Tor Browser doesn't modify the registry: it doesn't have an installer.
  • if you want to script to create the tor.exe manifest, that belongs in tor's Makefile. And we should check that mt.exe exists first.
  • we'll also need a patch for Tor Browser's build scripts, to get the manifest in the right place.
  • Tor Browser builds and packages OpenSSL, tor does not. So you'll need to add the OpenSSL manifest to Tor Browser's build scripts.

It looks like we need to split this ticket into two tickets, one for tor, and another for Tor Browser.

We can make that happen, but I need you to confirm my guesses first.

comment:41 in reply to:  39 Changed 7 weeks ago by teor

Replying to teor:.

It would be more helpful if we had instructions, or git pull requests. Otherwise we just have to guess what you want done with the files.

Here's my review of what I think you want done:

  • we can't modify users' registries. Tor doesn't build packages for Windows. And Tor Browser doesn't modify the registry: it doesn't have an installer.
  • if you want to script to create the tor.exe manifest, that belongs in tor's Makefile. And we should check that mt.exe exists first.
  • we'll also need a patch for Tor Browser's build scripts, to get the manifest in the right place.
  • Tor Browser builds and packages OpenSSL, tor does not. So you'll need to add the OpenSSL manifest to Tor Browser's build scripts.

It looks like we need to split this ticket into two tickets, one for tor, and another for Tor Browser.

We can make that happen, but I need you to confirm my guesses first.

I can't make progress on this ticket, until we have this information.

comment:42 in reply to:  36 Changed 6 weeks ago by ner0

Replying to cypherpunks:

@ner0
"an app folder" != "a current folder"

and if "the app folder" == "the current folder" then windows_10 exclude "an app folder" ("a current folder" is last one)

I don't really see the distinction here; by "a current folder" I meant the working directory, which I assume is the directory from which the process was run and also assume it is "an app folder". As for the second part of your comment I am not completely sure where you derive that information from. Although that exclusion is precisely the problem, why would running as Administrator changes that? Is it because as administrator the current folder/workingdir is System32 and thus relegated to last?

Note: See TracTickets for help on using tickets.