undefined reference to 'munmap' and 'mmap' building tor on Windows

changed milestone to %Tor: 0.3.0.x-final

added CoreTorTeam201611 actualpoints::0.3 component::core tor/tor milestone::Tor: 0.3.0.x-final mingw points::0.3 priority::medium reporter::ice resolution::fixed review-group-12 severity::normal status::closed type::defect version::tor unspecified windows labels

Trac:
Component: - Select a component to Core Tor/Tor

configure might be setting HAVE_SYS_MMAN_H incorrectly.

When you run configure, do you have any configure lines that mention mman.h, mmap, munmap, or getpagesize?

Clearly you have some header than mentions mmap and munmap. Do you need to link to a special library to get access to mmap and munmap?

Do any of your headers mention getpagesize?

What happens when you disable HAVE_SYS_MMAN_H in orconfig.h?

Trac:
Keywords: N/A deleted, mingw, windows added

Trac:
Username: ice

configure-output.txt

Trac:
Username: ice

Sorry I am not that familiar with trac. I attached the config-output.txt file to the comment above? Anyway. I meant to attach it here.

So after reading the configure output, there is no mention of the items you listed above.

I have also disabled HAVE_SYS_MMAN_H in orconfig.h and there was no change; same error as above-- after cleaning and rebuilding.

The header files that mention getpagesize on my system are as follows:

E:\dev\MinGW64\mingw\include\commctrl.h 142 KB C/C++ Header 2012-09-24 2:56:07 PM 2014-05-26 8:40:29 PM 2014-05-26 8:40:29 PM 1 1808 #define TBM_GETPAGESIZE (WM_USER+22)

E:\dev\MinGW64\mingw\include\dbgeng.h 110 KB C/C++ Header 2012-09-24 2:56:07 PM 2014-05-26 8:40:35 PM 2014-05-26 8:40:35 PM 3 744 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE; 846 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE; 967 STDMETHOD(GetPageSize)(THIS_ PULONG Size) PURE;

E:\dev\MinGW64\mingw\include\msrdc.h 16 KB C/C++ Header 2012-09-24 2:56:09 PM 2014-05-26 8:41:03 PM 2014-05-26 8:41:03 PM 3 261 STDMETHOD(GetPageSize)(THIS_ DWORD *pageSize) PURE; 275 #define ISimilarityTraitsMapping_GetPageSize(This,pageSize) (This)->lpVtbl->GetPageSize(This,pageSize)

I don't think they are related to this, or are they?

More importantly, I am not so sure about mmap and friends on Windows. I think these are linux/unix specific. On the gcc-patches mailing list[1] one can read the following.

"When compiling for windows, there shouldn't be any mmap/munmap references at all. While cygwin may supply some version of these, dlmalloc defines w32mmap/w32unmap which directly calls the appropriate routines for windows when WIN32 is defined (which I believe is always the case for the mingw compiler)."

On MSDN[2] there is mention of CreateFileMapping, and I think this and friends are what are used on Windows.

If you take a look at the mmapmodule[3] of Python, you'll notice how on Windows that function and its associated types are the ones used in place of mmap, etc.

[1] https://gcc.gnu.org/ml/gcc-patches/2009-06/msg02240.html [2] https://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx [3] https://github.com/python-git/python/blob/master/Modules/mmapmodule.c

Trac:
Username: ice

Ok, your configure output says:

checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes

https://trac.torproject.org/projects/tor/attachment/ticket/20530/configure-output.txt#L264

This is why HAVE_SYS_MMAN_H gets redefined in orconfig.h every time you run configure. (If the header did not exist, tor would not try to use the functions in it.)

If you want to try rebuilding without HAVE_SYS_MMAN_H: Disable it in orconfig.h and then do not run configure, just run make. If this works, we can fix the order of the alternatives in compat.c so that we check for WIN32 first, then HAVE_SYS_MMAN_H.

If you want to try rebuilding with HAVE_SYS_MMAN_H: What are the contents of sys/mman.h on your machine? Does it contain mmap, munmap, or getpagesize?

If you want to try building with dlmalloc, that could be a bit harder. Do you have dlmalloc installed? dlmalloc.c or dlmalloc.h?

Trac:
Milestone: N/A to Tor: 0.3.0.x-final
Keywords: N/A deleted, backport added

My bad. I don't know why I was looking for mmap.h in the configure output. Also HAVE_SYS_MMAN_H was not (re)defined in orconfig.h; when you suggested I disable it, I understood it to mean undefine it as it was not defined there.

At any rate that mman.h is indeed on my system, but I think it is an artifact from the person (or build system) that build the compiler. There is no libmman.a on my system and that file must have come from this repository: https://github.com/witwall/mman-win32.

I downloaded it and built the libmman.a and installed it. Edited Makefile to link with that libary and then issued "make". And success: building tor went without problems.

Issuing make test however, resulted in 3 fails and 6 skipped tests. I am attaching the make test output for reference. If the fails are expected then maybe we document all this somewhere. If the fails are a sign of a more serious problem, I am open for suggestions.

Trac:
Username: ice

test-output.txt

Trac:
Username: ice

Okay. After more looking into this, it seems that I did have the libmman.a in one lib directory, which I suspect is the one picked up by the configure script. However, due to the nature of MinGW64 there are now several library directories that GCC can access depending on the type of build x64, X86, etc. So even though libmman.a was picked up by configure, the build system looked for it in MinGW64/x86_64-w64-mingw32/lib directory instead of MinGW64/mingw/lib where it is found.

Alas, building and installing the mman-win32 and installing it in the MinGW64/x86_64-w64-mingw32/lib and adding a link to it in the Makefile, results in the build succeeding albeit with failed tests related to mmap functions. As an aside the tests in the mman-win32 library all pass.

At any rate, seeing that my vanilla GCC 5.3.0 (still referred to as MinGW64) does not contain the mman.h or the libmman.a, I tried hiding the mman.h by renaming it in GCC 4.7.3 (which is the one I used here) and issuing configure, make, and make test. Sure enough the configure script cannot find the header file, the build succeeds and all the tests pass. Except those referring to mmap of course are never tested, since they are now not available. I enclose the test output for reference. The question now is whether the removal of said functionality renders tor unusable, and whether or not that functionality exists in the tor distributed by the tor project whether in the Expert package or in the torbrowser.

Thanks

Trac:
Username: ice

test-output-no-mman-headers.txt

Trac:
Username: ice

Ok, so we need to prefer sys/mman.h when it's available, even on Windows, so Cygwin can handle mmap() and fork() correctly. (Although this is not so important now, because Tor uses threads, rather than forking.)

I think tor is missing a check for getpagesize, and that's causing the problem.

We could also check for linkage to the mmap, munmap, and getpagesize functions, but I'm going to leave that for now, because other environments replace those functions using macros. (There would be nothing to link to, even though the code is correct.)

Can you please test this branch? https://github.com/teor2345/tor/tree/fix-mingw-pagesize

Trac:
Actualpoints: N/A to 0.3
Status: new to needs_review
Keywords: N/A deleted, CoreTorTeam201611, 029-proposed added
Milestone: Tor: 0.3.0.x-final to Tor: 0.2.9.x-final
Version: N/A to Tor: unspecified
Points: N/A to 0.3

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system.

Also, kindly inform me if the tor I built without the mmap functionality is safe to use. I attempted to build tor only because the latest tor I have was complaining that I have an outdated version; and until I started my attempt, the torproject did not post an updated build in the form of the Expert package (the one included in the torbrowser is the "outdated" one I use). So can I use the one I built safely?

Trac:
Username: ice

Okay. Enclosing configure output for branch fix-mingw-pagesize. Will try to build it and report back shortly.

Trac:
Username: ice

configureOutput-fix-mingw-pagesize.txt

Trac:
Username: ice

Replying to ice:

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system. You can run the autogen.sh script situated in the root directory.

Replying to cypherpunks:

Replying to ice:

The branch and master are missing the configure script. If I am expected to generate it, please advise as to how as I am not that familiar with the automake build system. You can run the autogen.sh script situated in the root directory.

Yes I've already done that and the output of the resulting configure script is attached above.

I have built it now but unfortunately the test.exe crashes (twice). Apart from enclosing the few lines that the test printed out (attached file test-output-fmp.txt), I am not sure how else to help.

Before I forget: would you say it is safe to use the tor I built without the mman functionality?

Trac:
Username: ice

test-output-fmp.txt

Trac:
Username: ice

teor: fwiw, the patch looks good, but IIUC it doesn't resolve the problem above?

would you say it is safe to use the tor I built without the mman functionality?

ice: it should be just fine. The windows memory mapping code (which Tor uses when HAVE_SYS_MMAN_H is not defined) is probably a much better choice for Windows users.

Replying to nickm:

teor: fwiw, the patch looks good, but IIUC it doesn't resolve the problem above?

The test.exe error is a different bug to do with asprintf, not mman. It's likely that some logging code called by the circuit_timeout and geoip tests use a format not understood by _vscprintf or _vsnprintf. This is probably happening due to #19999 (moved). I've split this off into #20560 (moved).

Otherwise, I am unable to tell if it resolves the mman problem above, because the build environment changed between: https://trac.torproject.org/projects/tor/attachment/ticket/20530/configure-output.txt and: https://trac.torproject.org/projects/tor/attachment/ticket/20530/configureOutput-fix-mingw-pagesize.txt likely due to: https://trac.torproject.org/projects/tor/ticket/20530?replyto=13#comment:7

The first build had sys/mman.h available, the second did not.

However, the patch does prevent a call to getpagesize() when sys/mman.h is present, but getpagesize is not. So it's a net win.

would you say it is safe to use the tor I built without the mman functionality?

ice: it should be just fine. The windows memory mapping code (which Tor uses when HAVE_SYS_MMAN_H is not defined) is probably a much better choice for Windows users.

It is, as long as you are not using Cygwin.

If you are using Cygwin, you need to use Cygwin's mmap() so that Cygwin can handle fork() semantics correctly (and possibly other system calls).

Nick asked me to look through tickets with the 0.2.9.x milestone. A) This one is a bug report about 0.2.8.9, right? B) I would not object to a fix for this going into 0.2.9.x, but if turns into the remaining ticket in the way of a stable, back to 0.3.0 it goes! C) That said, if the fix in 0.2.9 might mess things up for a tiny fraction of people who will take a while to notice, to me that's a danger sign for a late-in-the-release-cycle merge.

I can't tell if this issue happened due to a broken build environment, or a genuine tor bug building on MinGW.

And since we're late in the alpha series, and no-one else has reported the same issue in either 0.2.8 or 0.2.9, it really does belong in 0.3.0.

Trac:
Milestone: Tor: 0.2.9.x-final to Tor: 0.3.0.x-final
Keywords: backport, 029-proposed deleted, N/A added

Trac:
Keywords: N/A deleted, review-group-12 added

Taking your patch in master; closing this ticket. Please reopen if the problem still exists on git master.

Trac:
Resolution: N/A to fixed
Status: needs_review to closed

closed

changed time estimate to 2h 24m

added 2h 24m of time spent

mentioned in issue #20560 (moved)

mentioned in issue #25398 (moved)

moved to tpo/core/tor#20530 (closed)

mentioned in issue tpo/core/tor#20560 (closed)

undefined reference to 'munmap' and 'mmap' building tor on Windows

Child items ...

Activity