Opened 8 years ago

Closed 4 years ago

#3197 closed defect (worksforme)

m:/tor-mingw/tbb-erinn.git/build-scripts/build-stable-windows/built/share\tor\fallback-consensus

Reported by: arma Owned by: tbb-team
Priority: Medium Milestone:
Component: Applications/Tor bundles/installation Version:
Severity: Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description (last modified by arma)

A user sent us logs of his failed (for some other reason I think) bootstrap. He's using Tor 0.2.1.30 -- I'm not sure which bundle.

But this line was part of his logs:

May 16 22:17:14.078 [Warning] Could not open
"m:/tor-mingw/tbb-erinn.git/build-scripts/build-stable-windows/built/share\tor\fallback-consensus": Permission denied

It looks like the bundle had some of Erinn's paths still in it.

What steps can we take in our build process (ideally automated) to reduce the chance that this recurs?

Child Tickets

Change History (9)

comment:1 Changed 8 years ago by arma

Description: modified (diff)
Summary: m:/tor-mingw/tbb-erinn.git/build-scripts/build-stable-windows/built/share\tor\fallback-consensus": Permission deniedm:/tor-mingw/tbb-erinn.git/build-scripts/build-stable-windows/built/share\tor\fallback-consensus

comment:2 Changed 7 years ago by arma

Is this (probably) still an issue?

comment:3 in reply to:  2 Changed 7 years ago by cypherpunks

Replying to arma:

Is this (probably) still an issue?

Yes, this is still an issue.

And this issue is not specific to TBB. It impacts all Windows Tor binaries, expert bundle, probably the half-dozen Vidalia bundles, as well.

For any tor.exe you build, try this:

strings tor.exe | grep y - erinn

you can see this invalid hard-coded path, and some other artifacts of Erinn's Mac OSX build system.

What steps can we take in our build process (ideally automated)
to reduce the chance that this recurs?

1) Document the Windows Tor build process. I mean update the old win32 build docs, which refer to MinGW instalers that no longer exist. Plus, update the instructions to discuss how Erinn is cross-compiling from a Mac OSX box. So contributors can help you fix it, building precisely like you do.

2) Ship a build log along with the resulting binary artifacts. It'll help with trust, and it'll help contributors who are trying to diagnose this problem.

3) Perhaps also automate the above grep and fail when erinn's name is located, except for the GCC flags line, I presume.

Thanks,
Lee

comment:4 Changed 7 years ago by Lee Fisher

Here's some more background on when Tor tries to open the invalid erinn path, showing the order of %APPDATA%\Tor\* files that Tor is trying to open, before and after this invalid path.

Tor tries to open these non-existant files, in this order:

C:\Users\user\AppData\Roaming\tor\router-stability
C:\Users\user\AppData\Roaming\tor\geoip6
C:\Users\user\AppData\Roaming\tor\cached-certs
C:\Users\user\AppData\Roaming\tor\cached-status
C:\Users\user\AppData\Roaming\tor\cached-consensus
C:\Users\user\AppData\Roaming\tor\unverified-consensus
C:\Users\user\AppData\Roaming\tor\cached-microdesc-consensus
C:\Users\user\AppData\Roaming\tor\unverified-microdesc-consensus
C:\Users\erinn\build-scripts.git\osx-bundles\build-alpha\share\tor\fallback-consensus
C:\Users\user\AppData\Roaming\tor\cached-microdescs
C:\Users\user\AppData\Roaming\tor\cached-microdescs.new
C:\Users\user\AppData\Roaming\tor\cached-descriptors
C:\Users\user\AppData\Roaming\tor\cached-extrainfo

All of these FILE_OPEN IRPs return STATUS_OBJECT_NAME_NOT_FOUND.

This is on a box with no active network, so, Tor's sockets are failing. No previous successful online connection has ever occurred, so there are no cached files, except what the Tor installer may've installed.

The only valid files in the initial Tor's user data directory are:

C:\Users\user\AppData\Roaming\tor\fingerprint
C:\Users\user\AppData\Roaming\tor\geoip
C:\Users\user\AppData\Roaming\tor\lock
C:\Users\user\AppData\Roaming\tor\state
C:\Users\user\AppData\Roaming\tor\torrc
C:\Users\user\AppData\Roaming\tor\keys\secret_id_key
C:\Users\user\AppData\Roaming\tor\keys\secret_onion_key

Perhaps the code that that makes the invalid erinn fallback-consensus open is this block in or/networkstatus.c's networkstatus_set_current_consensus()? It is at the end of a series of if loops that check for known values of flavor, and this is the tail end, with an unknown flavor value:


} else {

cached_dir_t *cur;
char buf[128];
tor_snprintf(buf, sizeof(buf), "cached-%s-consensus", flavor);
consensus_fname = get_datadir_fname(buf);
tor_snprintf(buf, sizeof(buf), "unverified-%s-consensus", flavor);
unverified_fname = get_datadir_fname(buf);
cur = dirserv_get_consensus(flavor);
if (cur) {

current_digests = &cur->digests;
current_valid_after = cur->published;

}


Given the code, and its previous 2 if blocks, and the previous 2 pairs of open requests:

C:\Users\user\AppData\Roaming\tor\cached-consensus
C:\Users\user\AppData\Roaming\tor\unverified-consensus

C:\Users\user\AppData\Roaming\tor\cached-microdesc-consensus
C:\Users\user\AppData\Roaming\tor\unverified-microdesc-consensus

I would have expected the errin I/O to be this:

C:\Users\user\AppData\Roaming\tor\cached-fallback-consensus
C:\Users\user\AppData\Roaming\tor\cached-fallback-consensus

instead of:

C:\Users\erinn\build-scripts.git\osx-bundles\build-alpha\share\tor\fallback-consensus

What is the value of 'flavor' when this happens? It is some GNU/Autotools environment variable, that is much longer when you cross-compile on MacOSX? The buffer here is hard-coded to 128 chars. Recall that MinGW still uses the Microsoft C Runtime Library, not GLibC.

Maybe you can do some strlen/sizeof tests with the input const char *flavor parameter, to see if this uninitialized buffer is big enough for this flavor?

Maybe you should give some debug log info about this dangling else case for flavors, since this is the only case where you don't explicitly check for the flavor string.

Again, this bug is not specific to TBB, it applies to all Windows builds of Tor.

Below is the Tor debug log of those files, from start-of-log until after file I/O is done:


Jan 20 20:53:57.000 [notice] Tor 0.2.4.6-alpha (git-b13c6becc892d971) opening log file.
Jan 20 20:53:57.000 [debug] tor_disable_debugger_attach(): Attemping to disable debugger attachment to Tor for unprivileged users.
Jan 20 20:53:57.000 [info] tor_lockfile_lock(): Locking "C:\Users\user\AppData\Roaming\tor\lock"
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 128.31.0.39:9131 (9695)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 111 dirserver at 86.59.21.38:80 (847B)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 102 dirserver at 194.109.206.212:80 (7EA6)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 16 dirserver at 82.94.251.203:80 (4A0C)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 76.73.17.194:9030 (F397)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 212.112.245.170:80 (F204)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 193.23.244.244:80 (7BE6)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 208.83.223.34:443 (0AD3)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 171.25.193.9:443 (BD6A)
Jan 20 20:53:57.000 [debug] parse_dir_server_line(): Trusted 100 dirserver at 154.35.32.5:80 (CF6D)
Jan 20 20:53:57.000 [info] or_state_load(): Loaded state from "C:\Users\user\AppData\Roaming\tor\state"
Jan 20 20:53:57.000 [debug] circuit_build_times_disabled(): CircuitBuildTime learning is not disabled. Consensus=0, Config=0, AuthDir=0, StateFile=0
Jan 20 20:53:57.000 [debug] circuit_build_times_disabled(): CircuitBuildTime learning is not disabled. Consensus=0, Config=0, AuthDir=0, StateFile=0
Jan 20 20:53:57.000 [info] circuit_build_times_parse_state(): Adding 0 timeouts.
Jan 20 20:53:57.000 [info] circuit_build_times_parse_state(): Loaded 0/0 values from 0 lines in circuit time histogram
Jan 20 20:53:57.000 [debug] circuit_build_times_disabled(): CircuitBuildTime learning is not disabled. Consensus=0, Config=0, AuthDir=0, StateFile=0
Jan 20 20:53:57.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\router-stability": No such file or directory
Jan 20 20:53:57.000 [info] cell_ewma_set_scale_factor(): Disabled cell_ewma algorithm because of value in Default value
Jan 20 20:53:57.000 [notice] Parsing GEOIP IPv4 file C:\Users\user\AppData\Roaming\tor\geoip.
Jan 20 20:53:57.000 [info] geoip_load_file(): Failed to open GEOIP file C:\Users\user\AppData\Roaming\tor\geoip6.
Jan 20 20:53:57.000 [info] crypto_global_init(): OpenSSL version matches version from headers (1000103f: OpenSSL 1.0.1c 10 May 2012).
Jan 20 20:53:57.000 [info] crypto_global_init(): NOT using OpenSSL engine support.
Jan 20 20:53:57.000 [info] evaluate_evp_for_aes(): This version of OpenSSL has a known-good EVP counter-mode implementation. Using it.
Jan 20 20:53:59.000 [info] tor_tls_init(): OpenSSL OpenSSL 1.0.1c 10 May 2012 looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Jan 20 20:53:59.000 [debug] mp_pool_new(): Capacity is 251, item size is 520, alloc size is 130520
Jan 20 20:53:59.000 [info] Bootstrapped 0%: Starting.
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\cached-certs": No such file or directory
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\cached-consensus": No such file or directory
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\unverified-consensus": No such file or directory
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\cached-microdesc-consensus": No such file or directory
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\unverified-microdesc-consensus": No such file or directory
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "c:/Users/erinn/build-scripts.git/osx-bundles/build-alpha/share\tor\fallback-consensus": No such file or directory
Jan 20 20:53:59.000 [info] tor_mmap_file(): Couldn't mmap file "C:\Users\user\AppData\Roaming\tor\cached-microdescs": The system cannot find the file specified.
Jan 20 20:53:59.000 [info] read_file_to_str(): Could not open "C:\Users\user\AppData\Roaming\tor\cached-microdescs.new": No such file or directory
Jan 20 20:53:59.000 [info] microdesc_cache_reload(): Reloaded microdescriptor cache. Found 0 descriptors.
Jan 20 20:53:59.000 [info] tor_mmap_file(): Couldn't mmap file "C:\Users\user\AppData\Roaming\tor\cached-descriptors": The system cannot find the file specified.
Jan 20 20:53:59.000 [info] tor_mmap_file(): Couldn't mmap file "C:\Users\user\AppData\Roaming\tor\cached-extrainfo": The system cannot find the file specified.
Jan 20 20:53:59.000 [info] I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Jan 20 20:54:01.000 [info] update_consensus_networkstatus_downloads(): Launching microdesc networkstatus consensus download.
Jan 20 20:54:01.000 [info] router_pick_directory_server(): No reachable router entries for dirservers. Trying them all again.
Jan 20 20:54:01.000 [info] directory_get_from_dirserver(): No router found for consensus network-status fetch; falling back to dirserver list.
Jan 20 20:54:01.000 [debug] directory_initiate_command_rend(): anonymized 0, use_begindir 1.
Jan 20 20:54:01.000 [debug] directory_initiate_command_rend(): Initiating consensus network-status fetch
Jan 20 20:54:01.000 [info] connection_ap_make_link(): Making internal direct tunnel to [scrubbed]:80 ...
Jan 20 20:54:01.000 [debug] connection_add_impl(): new conn type Socks, socket -1, address (Tor_internal), n_conns 2.
Jan 20 20:54:01.000 [debug] circuit_get_open_circ_or_launch(): considering 1, $BD6A829255CB08E66FBE7D3748363586E46B3810
Jan 20 20:54:01.000 [debug] onion_pick_cpath_exit(): Launching a one-hop circuit for dir tunnel.
Jan 20 20:54:01.000 [info] onion_pick_cpath_exit(): Using requested exit node '$BD6A829255CB08E66FBE7D3748363586E46B3810~BD6A829255CB08E66FB at 171.25.193.9'
Jan 20 20:54:01.000 [debug] onion_extend_cpath(): Path is 0 long; we want 1
<Rest of log omitted; it'll go on forever like this without working sockets...>


Thanks,
Lee

comment:5 Changed 7 years ago by Lee Fisher

I would have expected the errin I/O to be this:

C:\Users\user\AppData\Roaming\tor\cached-fallback-consensus
C:\Users\user\AppData\Roaming\tor\cached-fallback-consensus

Typo from last bug entry, the above second line should read unverified, not cached. As in a pair of (cached,unverified)-%s-consensus, where flavor %s is "fallback".

However, fallback-consensus is marked OBSOLETE in Tor, so why is Tor even using it? In src/or/config.c it is defined as:

OBSOLETE("FallbackNetworkstatusFile"),

Of the various other entries marked OBSOLETE, this appears to be the only init-time filename marked in this way. Tor gets the path correct for all other files except fallback-consensus.

Also, the output of "strings tor.exe | grep -y erinn":


c:/Users/erinn/build-scripts.git/osx-bundles/build-alpha/share\tor\fallback-consensus
OPENSSLDIR: "/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha"
gcc -D_WINDLL -DOPENSSL_USE_APPLINK -DOPENSSL_PIC -DOPENSSL_THREADS -D_MT -DDSO_WIN32 -I/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha/include -DL_ENDIAN -DWIN32_LEAN_AND_MEAN -fomit-frame-pointer -O3 -march=i486 -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha/lib/engines
/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha/private
/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha
/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha/certs
/c/Users/erinn/build-scripts.git/osx-bundles/build-alpha/cert.pem


shows that the this full path is hardcoded, not built at runtime in that %s. This is a built-time problem.

And the grep output shows that it is the only entry hard-coded path like this.

[There are some other similarly-invalid paths for OpenSSL dirs/files in the grep output, which might be another bug, or part of this bug, especially if those hard-coded invalid errinn OpenSSL dirs/paths are supposed to be usable dirs/paths, I'm not clear. And does Tor really need OpenSSL compiled with WHILRPOOL enabled? If resolving this bug doesn't resolve these other OpenSSL entries, this should become a separate bug, I suppose.]

So, where does Tor get this path:

"$PREFIX/share/tor/"

"C:\Users\erinn\build-scripts.git\osx-bundles\build-alpha\share\tor"?

When it should be using:

"%APPDATA\tor\"
aka:
"C:\Users\user\AppData\Roaming\tor\"

How does GNU/Autotools get/set "$PREFIX/share/tor" using MinGW toolchain?

Windows, of course, has no "/share" subdirectory anywhere to refer to, that is a Unix thing. Unless Tor's Windows installer creates the directory, and I don't recall seeing any of that in the NSIS/WIX installer scripts.

In Makefile.in this is interesting:

-----snip-----
FORCE:
# fallback_consensus

# If we don't have it, fake it.
src_config_fallback-consensus:

touch src/config/fallback-consensus

-----snip-----

And in src/config/include.am, this is interesting:

-----snip-----
EXTRA_DIST+= src/config/geoip src/config/geoip6
# fallback-consensus

conf_DATA = src/config/torrc.sample

tordata_DATA = src/config/geoip src/config/geoip6
# fallback_consensus

# If we don't have it, fake it.
src_config_fallback-consensus:

touch src/config/fallback-consensus

-----snip-----

Why does the makefile touch an obsolete file like this, at all?

Note that "src/config" doesn't exist on a Windows installation, AFAIK.

Perhaps this touch is failing on the Windows build, with that invalid path? Does your toolchain even have a touch.exe, or is that also failing? I'm afraid I can't figure out how to get further without Erinn's build log, and system and toolchain info. Someone with more GNU/Autotools skills is needed.

It appears this feature was added in 2008:


  • When we have no consensus, check FallbackNetworkstatusFile (defaults

to $PREFIX/share/tor/fallback-consensus) for a consensus. This way
we can start out knowing some directory caches. We don't ship with
a fallback consensus by default though, because it was making
bootstrapping take too long while we tried many down relays.
---snip----

Thanks,
Lee

comment:6 Changed 5 years ago by erinn

Keywords: needs-triage added

comment:7 Changed 4 years ago by cypherpunks

Status: newneeds_information

Is this still an issue?

comment:8 Changed 4 years ago by arma

Owner: changed from erinn to tbb-team
Status: needs_informationassigned

Reassigning to tbb-team in hopes they can tell us the bug is obsolete.

comment:9 Changed 4 years ago by gk

Keywords: needs-triage removed
Resolution: worksforme
Status: assignedclosed

The new Tor Browser does not contain any information of particular builders anymore. Thus WORKSFORME.

Note: See TracTickets for help on using tickets.