Opened 7 months ago

Closed 6 months ago

#20530 closed defect (fixed)

undefined reference to 'munmap' and 'mmap' building tor on Windows

Reported by: ice Owned by:
Priority: Medium Milestone: Tor: 0.3.0.x-final
Component: Core Tor/Tor Version: Tor: unspecified
Severity: Normal Keywords: windows, mingw, CoreTorTeam201611, review-group-12
Cc: Actual Points: 0.3
Parent ID: Points: 0.3
Reviewer: Sponsor:

Description

Using MinGW64 version 4.7.3-- thread model win32-- running naively (no cross-compile) inside msys2, I tried to build the tor-0.2.8.9 source downloaded from torproject.org.

While configuring the build, I noticed I had to build and install libevent, openssl, and zlib, which I did. After that Configure didn't fail anymore, and I proceeded to build tor by issuing "make". It seemed to go pretty well until the linker wanted to build tor.exe. It failed and complained about undefined references. Here is the output after issuing the make command again.

$ make
make all-am
make[1]: Entering directory `/e/users/v/Desktop/nfolder/tor-0.2.8.9'

CC src/common/compat.o

src/common/compat.c: In function 'tor_mmap_file':
src/common/compat.c:273:3: warning: implicit declaration of function 'getpagesize' [-Wimplicit-function-declaration]

AR src/common/libor.a
CC src/common/src_common_libor_testing_a-compat.o

src/common/compat.c: In function 'tor_mmap_file':
src/common/compat.c:273:3: warning: implicit declaration of function 'getpagesize' [-Wimplicit-function-declaration]

AR src/common/libor-testing.a
CCLD src/or/tor.exe

src/common/libor.a(compat.o): In function `tor_munmap_file':
e:\users\v\Desktop\nfolder\tor-0.2.8.9/src/common/compat.c:312: undefined reference to `munmap'
src/common/libor.a(compat.o): In function `tor_mmap_file':
e:\users\v\Desktop\nfolder\tor-0.2.8.9/src/common/compat.c:285: undefined reference to `mmap'
collect2.exe: error: ld returned 1 exit status
make[1]: * [src/or/tor.exe] Error 1
make[1]: Leaving directory `/e/users/v/Desktop/nfolder/tor-0.2.8.9'
make:
* [all] Error 2

I have googled around and have not found an answer to solve this problem.

Child Tickets

Attachments (5)

configure-output.txt (15.2 KB) - added by ice 7 months ago.
test-output.txt (28.7 KB) - added by ice 7 months ago.
test-output-no-mman-headers.txt (28.3 KB) - added by ice 7 months ago.
configureOutput-fix-mingw-pagesize.txt (29.4 KB) - added by ice 7 months ago.
test-output-fmp.txt (1.3 KB) - added by ice 7 months ago.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 7 months ago by arma

  • Component changed from - Select a component to Core Tor/Tor

comment:2 Changed 7 months ago by teor

  • Keywords windows mingw added

configure might be setting HAVE_SYS_MMAN_H incorrectly.

When you run configure, do you have any configure lines that mention mman.h, mmap, munmap, or getpagesize?

Clearly you have some header than mentions mmap and munmap.
Do you need to link to a special library to get access to mmap and munmap?

Do any of your headers mention getpagesize?

What happens when you disable HAVE_SYS_MMAN_H in orconfig.h?

Changed 7 months ago by ice

comment:3 Changed 7 months ago by ice

Sorry I am not that familiar with trac. I attached the config-output.txt file to the comment above? Anyway. I meant to attach it here.

So after reading the configure output, there is no mention of the items you listed above.

I have also disabled HAVE_SYS_MMAN_H in orconfig.h and there was no change; same error as above-- after cleaning and rebuilding.

The header files that mention getpagesize on my system are as follows:

E:\dev\MinGW64\mingw\include\commctrl.h 142 KB C/C++ Header 2012-09-24 2:56:07 PM 2014-05-26 8:40:29 PM 2014-05-26 8:40:29 PM 1
1808 #define TBM_GETPAGESIZE (WM_USER+22)

E:\dev\MinGW64\mingw\include\dbgeng.h 110 KB C/C++ Header 2012-09-24 2:56:07 PM 2014-05-26 8:40:35 PM 2014-05-26 8:40:35 PM 3
744 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE;
846 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE;
967 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE;

E:\dev\MinGW64\mingw\include\msrdc.h 16 KB C/C++ Header 2012-09-24 2:56:09 PM 2014-05-26 8:41:03 PM 2014-05-26 8:41:03 PM 3
261 STDMETHOD(GetPageSize)(THIS_ DWORD *pageSize) PURE;
275 #define ISimilarityTraitsMapping_GetPageSize(This,pageSize) (This)->lpVtbl->GetPageSize(This,pageSize)

I don't think they are related to this, or are they?

More importantly, I am not so sure about mmap and friends on Windows. I think these are linux/unix specific. On the gcc-patches mailing list[1] one can read the following.

"When compiling for windows, there shouldn't be any mmap/munmap references at all. While cygwin may supply some version of these, dlmalloc defines w32mmap/w32unmap which directly calls the appropriate routines for windows when WIN32 is defined (which I believe is always the case for the mingw compiler)."

On MSDN[2] there is mention of CreateFileMapping, and I think this and friends are what are used on Windows.

If you take a look at the mmapmodule[3] of Python, you'll notice how on Windows that function and its associated types are the ones used in place of mmap, etc.

[1] https://gcc.gnu.org/ml/gcc-patches/2009-06/msg02240.html
[2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx
[3] https://github.com/python-git/python/blob/master/Modules/mmapmodule.c

comment:4 Changed 7 months ago by teor

Ok, your configure output says:

checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes

https://trac.torproject.org/projects/tor/attachment/ticket/20530/configure-output.txt#L264

This is why HAVE_SYS_MMAN_H gets redefined in orconfig.h every time you run configure.
(If the header did not exist, tor would not try to use the functions in it.)

If you want to try rebuilding without HAVE_SYS_MMAN_H:
Disable it in orconfig.h and then do not run configure, just run make.
If this works, we can fix the order of the alternatives in compat.c so that we check for WIN32 first, then HAVE_SYS_MMAN_H.

If you want to try rebuilding with HAVE_SYS_MMAN_H:
What are the contents of sys/mman.h on your machine?
Does it contain mmap, munmap, or getpagesize?

If you want to try building with dlmalloc, that could be a bit harder.
Do you have dlmalloc installed? dlmalloc.c or dlmalloc.h?

comment:5 Changed 7 months ago by dgoulet

  • Keywords backport added
  • Milestone set to Tor: 0.3.0.x-final

comment:6 Changed 7 months ago by ice

My bad. I don't know why I was looking for mmap.h in the configure output. Also HAVE_SYS_MMAN_H was not (re)defined in orconfig.h; when you suggested I disable it, I understood it to mean undefine it as it was not defined there.

At any rate that mman.h is indeed on my system, but I think it is an artifact from the person (or build system) that build the compiler. There is no libmman.a on my system and that file must have come from this repository: https://github.com/witwall/mman-win32.

I downloaded it and built the libmman.a and installed it. Edited Makefile to link with that libary and then issued "make". And success: building tor went without problems.

Issuing make test however, resulted in 3 fails and 6 skipped tests. I am attaching the make test output for reference. If the fails are expected then maybe we document all this somewhere. If the fails are a sign of a more serious problem, I am open for suggestions.

Changed 7 months ago by ice

comment:7 Changed 7 months ago by ice

Okay. After more looking into this, it seems that I did have the libmman.a in one lib directory, which I suspect is the one picked up by the configure script. However, due to the nature of MinGW64 there are now several library directories that GCC can access depending on the type of build x64, X86, etc. So even though libmman.a was picked up by configure, the build system looked for it in MinGW64/x86_64-w64-mingw32/lib directory instead of MinGW64/mingw/lib where it is found.

Alas, building and installing the mman-win32 and installing it in the MinGW64/x86_64-w64-mingw32/lib and adding a link to it in the Makefile, results in the build succeeding albeit with failed tests related to mmap functions. As an aside the tests in the mman-win32 library all pass.

At any rate, seeing that my vanilla GCC 5.3.0 (still referred to as MinGW64) does not contain the mman.h or the libmman.a, I tried hiding the mman.h by renaming it in GCC 4.7.3 (which is the one I used here) and issuing configure, make, and make test. Sure enough the configure script cannot find the header file, the build succeeds and all the tests pass. Except those referring to mmap of course are never tested, since they are now not available. I enclose the test output for reference. The question now is whether the removal of said functionality renders tor unusable, and whether or not that functionality exists in the tor distributed by the tor project whether in the Expert package or in the torbrowser.

Thanks

Changed 7 months ago by ice

comment:8 Changed 7 months ago by teor

  • Actual Points set to 0.3
  • Keywords 029-proposed CoreTorTeam201611 added
  • Milestone changed from Tor: 0.3.0.x-final to Tor: 0.2.9.x-final
  • Points set to 0.3
  • Status changed from new to needs_review
  • Version set to Tor: unspecified

Ok, so we need to prefer sys/mman.h when it's available, even on Windows, so Cygwin can handle mmap() and fork() correctly. (Although this is not so important now, because Tor uses threads, rather than forking.)

I think tor is missing a check for getpagesize, and that's causing the problem.

We could also check for linkage to the mmap, munmap, and getpagesize functions, but I'm going to leave that for now, because other environments replace those functions using macros. (There would be nothing to link to, even though the code is correct.)

Can you please test this branch?
https://github.com/teor2345/tor/tree/fix-mingw-pagesize

comment:9 follow-up: Changed 7 months ago by ice

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system.

Also, kindly inform me if the tor I built without the mmap functionality is safe to use. I attempted to build tor only because the latest tor I have was complaining that I have an outdated version; and until I started my attempt, the torproject did not post an updated build in the form of the Expert package (the one included in the torbrowser is the "outdated" one I use). So can I use the one I built safely?

comment:10 Changed 7 months ago by ice

Okay. Enclosing configure output for branch fix-mingw-pagesize. Will try to build it and report back shortly.

comment:11 in reply to: ↑ 9 ; follow-up: Changed 7 months ago by cypherpunks

Replying to ice:

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system.

You can run the autogen.sh script situated in the root directory.

comment:12 in reply to: ↑ 11 Changed 7 months ago by ice

Replying to cypherpunks:

Replying to ice:

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system.

You can run the autogen.sh script situated in the root directory.

Yes I've already done that and the output of the resulting configure script is attached above.

I have built it now but unfortunately the test.exe crashes (twice). Apart from enclosing the few lines that the test printed out (attached file test-output-fmp.txt), I am not sure how else to help.

Before I forget: would you say it is safe to use the tor I built without the mman functionality?

Changed 7 months ago by ice

comment:13 follow-up: Changed 7 months ago by nickm

teor: fwiw, the patch looks good, but IIUC it doesn't resolve the problem above?

would you say it is safe to use the tor I built without the mman functionality?

ice: it should be just fine. The windows memory mapping code (which Tor uses when HAVE_SYS_MMAN_H is not defined) is probably a much better choice for Windows users.

comment:14 in reply to: ↑ 13 Changed 7 months ago by teor

Replying to nickm:

teor: fwiw, the patch looks good, but IIUC it doesn't resolve the problem above?

The test.exe error is a different bug to do with asprintf, not mman.
It's likely that some logging code called by the circuit_timeout and geoip tests use a format not understood by _vscprintf or _vsnprintf.
This is probably happening due to #19999.
I've split this off into #20560.

Otherwise, I am unable to tell if it resolves the mman problem above, because the build environment changed between:
https://trac.torproject.org/projects/tor/attachment/ticket/20530/configure-output.txt
and:
https://trac.torproject.org/projects/tor/attachment/ticket/20530/configureOutput-fix-mingw-pagesize.txt
likely due to:
https://trac.torproject.org/projects/tor/ticket/20530?replyto=13#comment:7

The first build had sys/mman.h available, the second did not.

However, the patch does prevent a call to getpagesize() when sys/mman.h is present, but getpagesize is not. So it's a net win.

would you say it is safe to use the tor I built without the mman functionality?

ice: it should be just fine. The windows memory mapping code (which Tor uses when HAVE_SYS_MMAN_H is not defined) is probably a much better choice for Windows users.

It is, as long as you are not using Cygwin.

If you are using Cygwin, you need to use Cygwin's mmap() so that Cygwin can handle fork() semantics correctly (and possibly other system calls).

comment:15 Changed 7 months ago by arma

Nick asked me to look through tickets with the 0.2.9.x milestone. A) This one is a bug report about 0.2.8.9, right? B) I would not object to a fix for this going into 0.2.9.x, but if turns into the remaining ticket in the way of a stable, back to 0.3.0 it goes! C) That said, if the fix in 0.2.9 might mess things up for a tiny fraction of people who will take a while to notice, to me that's a danger sign for a late-in-the-release-cycle merge.

comment:16 Changed 7 months ago by teor

  • Keywords backport 029-proposed removed
  • Milestone changed from Tor: 0.2.9.x-final to Tor: 0.3.0.x-final

I can't tell if this issue happened due to a broken build environment, or a genuine tor bug building on MinGW.

And since we're late in the alpha series, and no-one else has reported the same issue in either 0.2.8 or 0.2.9, it really does belong in 0.3.0.

comment:17 Changed 6 months ago by nickm

  • Keywords review-group-12 added

comment:18 Changed 6 months ago by nickm

  • Resolution set to fixed
  • Status changed from needs_review to closed

Taking your patch in master; closing this ticket. Please reopen if the problem still exists on git master.

Note: See TracTickets for help on using tickets.