RSA_get0_d could not be located in the dynamic link library tor.exe

OS: Win10 (win64)

Tor Browser will not run, when attempting to establish a connection the following error appeats:

The procedure entry point RSA_get0_d could not be located in the dynamic link library C:\Users\MyUser\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe

Trac:
Username: ner0

added component::core tor/tor priority::medium reporter::ner0 severity::normal status::assigned type::defect labels

Could not specify before, but this starts with latest browser update v9.0.7 (tor 0.4.2.7)

Trac:
Username: ner0

A user reported the same issue on the blog: https://blog.torproject.org/comment/287165#comment-287165

I think this is probably related with this change: https://gitweb.torproject.org/tor.git/commit/?h=maint-0.4.2&id=8abdb394893a1704f885278f5f5d7913cdf516c9

We are including ./Browser/TorBrowser/Tor/libssl-1_1.dll in the bundle. I am wondering if tor.exe gets linked to wrong dll for some reason.

nickm, teor: do you have any idea about what could be causing this error?

Trac:
Cc: N/A to nickm, teor

One additional data point: I am unable to reproduce this problem on a Win10 64-bit laptop that has minimal software installed (no anti-malware software except the built-in Defender).

Replying to boklm:

A user reported the same issue on the blog: https://blog.torproject.org/comment/287165#comment-287165 That was also me.

Replying to mcs:

One additional data point: I am unable to reproduce this problem on a Win10 64-bit laptop that has minimal software installed (no anti-malware software except the built-in Defender). Can confirm. In the system showing the issue I have Comodo AV, I hav esince cloned the system to a VM, removed tons of software including AV and it doesn't work all the same.

Trac:
Username: ner0

I have uninstalled virtually everything on the system and the issue persists. Should be noted that the issue does not happen when using the 32bit version of Tor Browser, only 64bit.

Trac:
Username: ner0

update

uninstalled virtually everything

reinstall torbrowser?

Replying to cypherpunks:

update

uninstalled virtually everything

reinstall torbrowser? Yes, reinstalled it multiple times. Also purged the whole folder, as well as Program Files and ProgramData leftovers. Tested with new Win user to exclude user profile issues. As mentioned by mcs, I also have been able to use the new win64 release on a another machine with no issue. On my computer (and VM) only the 32bit version is working, 64bit throws the error mentioned in the title.

I could try to be of further assistance if I knew a good way to debug or gather logging information that could be useful.

Trac:
Username: ner0

Replying to boklm:

nickm, teor: do you have any idea about what could be causing this error?

Most frequently this kind of error happens when Tor is compiled with one version of openssl, but linked with another substantially different. I don't know so much about Windows linking behavior, however.

Is there some kind of tool that can be used to tell which specific libraries Windows is trying to link Tor against here? If so, that would probably help figure this out.

Trac:
Username: ner0

Trac:
Username: ner0

I don't know if procmon logs can be useful on their own. I've attached logs and side by side comparison of v9.0.6 and v9.0.7, the issue arises very early on, not sue what is the trigger though.

Trac:
Username: ner0

Trac:
Username: ner0

procmon_logs.7z

Trac:
Username: ner0

12:01:42.7784859 PM	tor.exe	25560	CreateFile	C:\Windows\System32\libcrypto-1_1-x64.dll	SUCCESS	Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

libcrypto-1_1-x64.dll in system32 dir (why?)

When looking at Logfile_v9.0.7_win64_fail_hl.PML I see that it contains the string C:\Windows\System32\libssl-1_1-x64.dll.

So it looks like libssl is loaded from the Windows system directory instead of the Tor Browser directory.

When looking at: https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order The directory from which the application loaded is first, so I don't know why it is not loaded from Tor Browser's directory.

However in Factors That Affect Searching they also say: If the DLL is on the list of known DLLs for the version of Windows on which the application is running, the system uses its copy of the known DLL (and the known DLL's dependent DLLs, if any) instead of searching for the DLL. For a list of known DLLs on the current system, see the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs.

Maybe libssl is part of the known DLLs?

Replying to cypherpunks:

{{{ 12:01:42.7784859 PM tor.exe 25560 CreateFile C:\Windows\System32\libcrypto-1_1-x64.dll SUCCESS Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened }}} libcrypto-1_1-x64.dll in system32 dir (why?) The OpenSSL Toolkit... that's where it came from.

that might have been a contributing factor, but weirder is how it was "fixed":

1. Ran Tor Browser once as administrator;
2. Done.

I don't have a clue as to what/why. I can now run it without elevated privileges just fine, no more issues.

Trac:
Username: ner0

Replying to boklm:

When looking at Logfile_v9.0.7_win64_fail_hl.PML I see that it contains the string C:\Windows\System32\libssl-1_1-x64.dll.

So it looks like libssl is loaded from the Windows system directory instead of the Tor Browser directory.

When looking at: https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order The directory from which the application loaded is first, so I don't know why it is not loaded from Tor Browser's directory.

However in Factors That Affect Searching they also say: If the DLL is on the list of known DLLs for the version of Windows on which the application is running, the system uses its copy of the known DLL (and the known DLL's dependent DLLs, if any) instead of searching for the DLL. For a list of known DLLs on the current system, see the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs.

Maybe libssl is part of the known DLLs? None of the DLLs on that registry key matches the ones in Tor's folder, can't speak to their dependency though, it's possible it could be called by any one of them.

Trac:
Username: ner0

Just want to confirm that the presence of a different version of libcrypto-1_1-x64.dll in System32 (v1.1.0.10) was the actual issue (thanks for noticing it, cypherpunks). Not sure why running as admin would "bypass" that permanently and also very curious why, as boklm mentioned, Windows would ignore the DLL present in Tor Browser's folder which should have had priority. I have removed all OpenSSL DLLs from System32, I am not sure if it was me who put them there on my own, but its best to remove them to avoid future issues.

Another weird thing is that my system has not changed since the last release up to current, so maybe partially was due to: [comment:3 boklm]:

I think this is probably related with this change: https://gitweb.torproject.org/tor.git/commit/?h=maint-0.4.2&id=8abdb394893a1704f885278f5f5d7913cdf516c9

Trac:
Username: ner0

See how d3dcompiler_47.dll is loaded ;) https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-security

new topic: Windows10 ignore the DLL present in the directory from which the application loaded.

I didn't get the d3dcompiler_47.dll reference. If anything, I get the idea that it is possible to prevent DLL pre-loading attacks by implementing some measures and not that it is so by default (even though my experience in this topic goes against that). Firefox, for example: https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29

Do it's a bit conflicting what I'm interpreting from Microsoft docs. Which excerpts specifically say that Windows (10) excludes the working dir by default from known DLL search?

Indeed, DLL loading has become the important topic.

Trac:
Username: ner0

Firefox, for example: ​https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29

"SetDefaultDllDirectories affect the implicit loading in child processes" huh?

Trac:
Severity: Blocker to Normal
Version: Tor: unspecified to N/A

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away.

Trac:
Username: MadCabbit

Replying to MadCabbit:

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away. Have you tried removing possible overriding DLLs or just running Tor Browser once as administrator? For me both approaches solved the issue. In my case I shouldn't have had those DLLs in system32 to begin with, but I also think Tor Browser bears responsibility for not loading the correct (workingdir) DLL.

Trac:
Username: ner0

I'm not sure what DLLs would be overriding, but I tried running Tor as administrator, and that fixed the problem. Replying to ner0:

Replying to MadCabbit:

This issue also popped up for me today when Tor tried upgrading to 9.0.7, Win10 x64. Tried downloading the installer and that also gives the error. Reinstalled 9.0.6 and the error went away. Have you tried removing possible overriding DLLs or just running Tor Browser once as administrator? For me both approaches solved the issue. In my case I shouldn't have had those DLLs in system32 to begin with, but I also think Tor Browser bears responsibility for not loading the correct (workingdir) DLL.

Trac:
Username: MadCabbit

Replying to nickm:

Replying to boklm:

nickm, teor: do you have any idea about what could be causing this error?

Most frequently this kind of error happens when Tor is compiled with one version of openssl, but linked with another substantially different. I don't know so much about Windows linking behavior, however.

We have seen similar issues in our Windows CI.

We tried copying the correct DLLs into the same directory as the exe, but that didn't help. We ended up disabling the failing unit test.

But if you work out how to modify the DLL search path, or find the correct directories to copy the DLLs into, please let us know.

I also don't know why running Tor Browser once as an administrator fixes the issue.

But if you work out how to modify the DLL search path, or find the correct directories to copy the DLLs into, please let us know.

manifest? https://docs.microsoft.com/en-us/windows/win32/sbscs/assembly-manifests

Replying to teor:

I also don't know why running Tor Browser once as an administrator fixes the issue.

---

At first glance it seems to be a Microsoft "feature", but lets not lose focus that Tor is the actual issue. To the extent that Windows 10 is concerned, my testing had the following results in terms of DLL search order.

• When running as current user, '\Windows\System32' is searched first, then \Windows, and so on.
• When current user invokes "run as administrator", current folder is searched first.

I can't really wrap my head around Microsoft's intention here, if the idea was to protect the user from potential system hijacking I would assume the OS to take the opposite approach since non-system files and folders would be easier to compromise, thus it would make much more sense for an elevated process to first search the system folder instead of an easily manipulable user folder. That aside, why does using "run as administrator" once fixes it for subsequent runs as a standard user? No clue.

Anyway, one thing is sure, Firefox process itself handles DLL search order perfectly regardless of the elevation while Tor process does not. The earliest example is firefox.exe searching for the DLL version.dll inside Tor Browser's folder first before searching in \Windows\System32, meanwhile tor.exe searches for all DLLs first in \Windows\System32, then '\Windows', etc. I think Tor should implement the approach mentioned before: [https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/loaddlls.cpp#8-29]

Trac:
Username: ner0

That Firefox code is for system DLLs, not application-specific DLLs.

We probably want something like this:

      LoadLibraryExW(fullModulePath, nullptr, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);

Or maybe there are flags we can set when linking the DLL?

Edit:

Here is the reference doc: https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadlibraryexa

Sounds like a tor ticket now, no?

Trac:
Component: Applications/Tor Browser to Core Tor/Tor

Replying to gk:

Sounds like a tor ticket now, no?

I'm not sure if it is an issue with:

1. tor's code
2. tor's build process
3. the way Tor Browser builds or packages tor
4. the local system that Tor Browser is installed on

But everything works fine in our Windows CI. So even if it is something we can fix (1 or 2), it's not a universal issue. So it's not going to be easy to test. (We might be able to test is using the OpenSSL version mismatch in #33643 (moved), but that's also unreliable.)

If we find out it's something Tor Browser can fix, we'll hand it back to you.

Trac:
Status: new to assigned
Owner: tbb-team to N/A

Replying to gk:

Sounds like a tor ticket now, no? FWIW, tor runs in Tor Browser with Virtualized UAC Disabled when it's on Windows 10. According to #22450 (moved), they won't fix Windows issues anytime soon.

Trac:
Cc: nickm, teor to nickm, teor, mcs

Replying to cypherpunks:

Replying to gk:

Sounds like a tor ticket now, no? FWIW, tor runs in Tor Browser with Virtualized UAC Disabled when it's on Windows 10.

Can you help me understand how Virtualised UAC relates to this issue?

According to #22450 (moved), they won't fix Windows issues anytime soon.

We don't have a lot of Windows developers working on tor. So we prioritise issues based on their impact on users.

I'm not sure that using a compatibility mode is actually a bug. Does it cause any user-visible issues?

We'd love more people to help us patch and test Windows issues.

@ner0 "an app folder" != "a current folder"

and if "the app folder" == "the current folder" then windows_10 exclude "an app folder" ("a current folder" is last one)

Trac:
EnableExternalManifest.reg

Trac:
tor.exe.Manifest

Trac:
openssl.dll.Manifest

Trac:
internalmanifest.txt

Hi cypherpunks, can you please explain what these attachments do?

It's hard for us to guess what you're thinking.

@teor https://en.wikipedia.org/wiki/Side-by-side_assembly

I know about Windows manifests.

It would be more helpful if we had instructions, or git pull requests. Otherwise we just have to guess what you want done with the files.

Here's my review of what I think you want done:

• we can't modify users' registries. Tor doesn't build packages for Windows. And Tor Browser doesn't modify the registry: it doesn't have an installer.
• if you want to script to create the tor.exe manifest, that belongs in tor's Makefile. And we should check that mt.exe exists first.
• we'll also need a patch for Tor Browser's build scripts, to get the manifest in the right place.
• Tor Browser builds and packages OpenSSL, tor does not. So you'll need to add the OpenSSL manifest to Tor Browser's build scripts.

It looks like we need to split this ticket into two tickets, one for tor, and another for Tor Browser.

We can make that happen, but I need you to confirm my guesses first.

https://searchfox.org/mozilla-central/source/browser/app/firefox.exe.manifest

Replying to teor:.

It would be more helpful if we had instructions, or git pull requests. Otherwise we just have to guess what you want done with the files.

Here's my review of what I think you want done:

• we can't modify users' registries. Tor doesn't build packages for Windows. And Tor Browser doesn't modify the registry: it doesn't have an installer.
• if you want to script to create the tor.exe manifest, that belongs in tor's Makefile. And we should check that mt.exe exists first.
• we'll also need a patch for Tor Browser's build scripts, to get the manifest in the right place.
• Tor Browser builds and packages OpenSSL, tor does not. So you'll need to add the OpenSSL manifest to Tor Browser's build scripts.

It looks like we need to split this ticket into two tickets, one for tor, and another for Tor Browser.

We can make that happen, but I need you to confirm my guesses first.

I can't make progress on this ticket, until we have this information.