Opened 4 years ago

Closed 4 years ago

#16651 closed defect (implemented)

Tor fails to build on OpenBSD 5.8 due to libevent config options

Reported by: teor Owned by:
Priority: Medium Milestone: Tor: 0.2.7.x-final
Component: Core Tor/Tor Version:
Severity: Normal Keywords: bsd, 027-backport, libevent, PostFreeze027, pre028-patch, TorCoreTeam201512, 201511-deferred
Cc: seth+bugs@…, cb@…, tyseom Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Child Tickets

Attachments (5)

tor-libevent2-4.patch (1.0 KB) - added by attila 4 years ago.
patch: make --enable-bufferevents (w/libevent2) work under OpenBSD (at least)
event2.patch (1.1 KB) - added by rubiate 4 years ago.
event2-a.patch (1.2 KB) - added by rubiate 4 years ago.
pass save_LIBS when searching libs
event2-b.patch (1.3 KB) - added by rubiate 4 years ago.
don't clear LIBS, manually add results to TOR_LIBEVENT_LIBS
0001-Quote-variables-in-case-they-contain-spaces.patch (1.2 KB) - added by cypherpunks 4 years ago.

Download all attachments as: .zip

Change History (46)

comment:1 Changed 4 years ago by nickm

Keywords: 026-backport libevent added
Status: newneeds_review

Hm. The "evthread_use_pthreads" is wrong; Tor only uses libevent from the main thread, except when windows bufferevents are in use. (That could be a comment instead.)

Has anybody tested this with older (pre-2.0) versions of Libevent, to make sure it doesn't break them?

comment:2 Changed 4 years ago by sysfu

Cc: seth+bugs@… added

comment:3 Changed 4 years ago by rubiate

Cc: cb@… added

comment:4 Changed 4 years ago by tyseom

Cc: tyseom added

Changed 4 years ago by attila

Attachment: tor-libevent2-4.patch added

patch: make --enable-bufferevents (w/libevent2) work under OpenBSD (at least)

comment:5 in reply to:  1 Changed 4 years ago by attila

Replying to nickm:

Hm. The "evthread_use_pthreads" is wrong; Tor only uses libevent from the main thread, except when windows bufferevents are in use. (That could be a comment instead.)

Has anybody tested this with older (pre-2.0) versions of Libevent, to make sure it doesn't break them?

You are correct, the original patch and several mods that were tossed around
on the tor-bsd mailing list were all subtly wrong. I have attached a patch
that I think is correct: only compiled in w/libevent2 and --enable-bufferevents
turned on. Works for me under OpenBSD/i386:

Jul 28 09:55:44.277 [notice] Tor v0.2.7.1-alpha-dev (git-aeeb6376aab186fd) (with
 bufferevents) running on OpenBSD with Libevent 2.0.22-stable, OpenSSL LibreSSL 
2.2 and Zlib 1.2.3.

Successfully builds circuits and otherwise seems to work. Most of the test suite
passes but there are four tests that have never passed for me under OpenBSD,
patch notwithstanding... that can be a separate ticket.

comment:6 Changed 4 years ago by sysfu

Used this patch on the Tor 0.2.6.10 source on an OpenBSD 5.7 system with libevent 2.0.22 installed from ports. Command './configure --enable-bufferevents' fails as per below. Same error when trying to configure the 0.2.7.2-alpha.

configure: error: You've asked for bufferevent support, but you're using a version of Libevent without SSL support. This won't work. We need Libevent 2.0.8-rc or later, and you don't seem to even have Libevent 2.0.3-alpha.

$ pkg_info |grep libevent
libevent-2.0.22 event notification library

Last edited 4 years ago by sysfu (previous) (diff)

comment:7 Changed 4 years ago by rubiate

You need to tell it what paths to look in.

The command from the original mail works for me:

    $ env CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib \
    ./configure --with-ssl-dir=/usr --disable-gcc-hardening \
        --enable-bufferevents --disable-asciidoc

comment:8 Changed 4 years ago by sysfu

Using freshly extracted tor 0.2.7.2-alpha archive and the commands below it got bit further this time, before failing as per below. System is updated with LibreSSL 2.2.1 and platform is amd64.

Commands used:

$ patch < tor-libevent2-4.patch
$ env CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure --with-ssl-dir=/usr --disable-gcc-hardening --enable-bufferevents --disable-asciidoc
$ AUTOCONF_VERSION=2.69 AUTOMAKE_VERSION=1.15 make check

<snipped make command output>
src/common/libor-event.a(procmon.o)(.text+0x256): In function `tor_process_monitor_new':
: undefined reference to `event_new'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_get_write_max'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_init_common'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_generic_adj_timeouts'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_add_event'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_eventcb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_get_read_max'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_unsuspend_read'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_incref_and_lock'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_incref'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_mm_calloc_'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_readcb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `evutil_closesocket'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decref_and_unlock'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_suspend_read'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_evthread_lock_fns'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decrement_write_buckets'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_enable_locking'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_assign'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_get_fd'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_writecb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decrement_read_buckets'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_del_generic_timeout_cbs'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_init_generic_timeout_cbs'
collect2: ld returned 1 exit status

  • Error 1 in /usr/local/src/tor-0.2.7.2-alpha (Makefile:2656 'src/or/tor': @echo " CCLD " src/or/tor;gcc -std=gnu99 -I/usr/local/incl...)
Last edited 4 years ago by sysfu (previous) (diff)

comment:9 in reply to:  8 Changed 4 years ago by attila

Replying to sysfu:

Using freshly extracted tor 0.2.7.2-alpha archive and the commands below it got bit further this time, before failing as per below. System is updated with LibreSSL 2.2.1 and platform is amd64.

Commands used:

$ patch < tor-libevent2-4.patch
$ env CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure --with-ssl-dir=/usr --disable-gcc-hardening --enable-bufferevents --disable-asciidoc
$ AUTOCONF_VERSION=2.69 AUTOMAKE_VERSION=1.15 make check

<snipped make command output>
src/common/libor-event.a(procmon.o)(.text+0x256): In function `tor_process_monitor_new':
: undefined reference to `event_new'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_get_write_max'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_init_common'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_generic_adj_timeouts'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_add_event'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_eventcb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_get_read_max'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_unsuspend_read'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_incref_and_lock'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_incref'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_mm_calloc_'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_readcb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `evutil_closesocket'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decref_and_unlock'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_suspend_read'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_evthread_lock_fns'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decrement_write_buckets'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `bufferevent_enable_locking'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_assign'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `event_get_fd'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_run_writecb'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_decrement_read_buckets'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_del_generic_timeout_cbs'
/usr/local/lib/libevent_openssl.so.0.0: undefined reference to `_bufferevent_init_generic_timeout_cbs'
collect2: ld returned 1 exit status

  • Error 1 in /usr/local/src/tor-0.2.7.2-alpha (Makefile:2656 'src/or/tor': @echo " CCLD " src/or/tor;gcc -std=gnu99 -I/usr/local/incl...)

The patch has two parts, one to configure.ac and one to compat_libevent.c. It looks like
perhaps you didn't regenerate configure after patching configure.ac.

The full sequence (after applying the patch):

$ env AUTOMAKE_VERSION=1.11 AUTOCONF_VERSION=2.69 ./autogen.sh
$ env CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib \
  ./configure --with-ssl-dir=/usr --disable-gcc-hardening \
              --disable-asciidoc --enable-bufferevents
$ gmake

Also, to be sure: you're running -current, right? What does the first line of your
dmesg output look like? Here's the one from the machine where this worked:

$ dmesg | head -1
OpenBSD 5.8-beta (GENERIC) #987: Sat Jul 11 00:10:38 MDT 2015

That's actually not that recent a snap... I'll try it with the latest today once my
new machine is up to snuff. This might all work on -stable as well but I don't have
anything but -current to test with at the moment.

comment:10 Changed 4 years ago by nickm

FWIW, --enable-bufferevents is not recommended; it has known bugs, and may make relays and clients unstable.

comment:11 Changed 4 years ago by sysfu

@attila, I was trying to do this on a 5.7-release system. Will re-install as -current and try again using the steps detailed above.

Any chance of getting the patch backported to 5.7 BTW?

comment:12 Changed 4 years ago by sysfu

I tried this again on an OpenBSD 5.8 amd64 current snapshot against the latest Tor source from the git repository. Build crapped again, commands and build error output below.

$ git clone https://git.torproject.org/git/tor
$ cd tor
$ ftp https://trac.torproject.org/projects/tor/raw-attachment/ticket/16651/tor-libevent2-4.patch
$ patch < tor-libevent2-4.patch 
$ env AUTOMAKE_VERSION=1.15 AUTOCONF_VERSION=2.69 ./autogen.sh
$ env CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure --with-ssl-dir=/usr --disable-gcc-hardening --disable-asciidoc
$ env AUTOMAKE_VERSION=1.15 AUTOCONF_VERSION=2.69 gmake

compat_libevent.c:(.text+0x88e): undefined reference to `event_new'
src/common/libor-event.a(procmon.o): In function `tor_process_monitor_new':
procmon.c:(.text+0x256): undefined reference to `event_new'
collect2: ld returned 1 exit status
Makefile:2655: recipe for target 'src/or/tor' failed
gmake[1]: *** [src/or/tor] Error 1
gmake[1]: Leaving directory '/usr/local/src/tor'
Makefile:1864: recipe for target 'all' failed
gmake: *** [all] Error 2
Last edited 4 years ago by sysfu (previous) (diff)

comment:13 in reply to:  12 Changed 4 years ago by rubiate

Replying to sysfu:

I tried this again on an OpenBSD 5.8 amd64 current snapshot against the latest Tor source from the git repository. Build crapped again, commands and build error output below.

I get the same when building without bufferevents. With bufferevents it builds fine on both 5.7-release and a -current snapshot.

comment:14 Changed 4 years ago by cypherpunks

This patch makes tor 0.2.7.2-alpha compile with libevent2 without bufferevents:

http://lists.nycbug.org/pipermail/tor-bsd/2015-August/000350.html

comment:15 Changed 4 years ago by nickm

Priority: normalcritical

comment:16 Changed 4 years ago by nickm

Keywords: PostFreeze027 added

If we wind up with a nice patch for any of these in the appropriate window, we should sure merge it.

comment:17 Changed 4 years ago by nickm

cypherpunks at comment:15 : that patch will make us fail for Libevent 1.4 and earlier, where we didn't have libevent_extra or libevent_core.

comment:18 Changed 4 years ago by nickm

Status: needs_reviewneeds_revision

comment:19 Changed 4 years ago by nickm

Milestone: Tor: 0.2.7.x-finalTor: 0.2.???
Severity: Normal
Summary: Tor fails to build on OpenBSD due to libevent config optionsTor fails to build on OpenBSD with --enable-bufferevents due to libevent config options

comment:20 Changed 4 years ago by nickm

Milestone: Tor: 0.2.???Tor: 0.2.8.x-final
Priority: Very HighMedium
Summary: Tor fails to build on OpenBSD with --enable-bufferevents due to libevent config optionsTor fails to build on OpenBSD 5.8 due to libevent config options

comment:21 Changed 4 years ago by nickm

Keywords: pre028-patch added

comment:22 Changed 4 years ago by rubiate

Here's a patch I've been playing with. On OpenBSD without libevent2 it properly picks libevent 1.x from base ("-levent"). With libevent2 installed, it picks "-levent_extra -levent_core -levent".

For other platforms, I've tested it on FreeBSD (with libevent2 installed) and it correctly picks just "-levent" as that's all that is needed there.

Changed 4 years ago by rubiate

Attachment: event2.patch added

comment:23 Changed 4 years ago by nickm

Status: needs_revisionneeds_review

comment:24 Changed 4 years ago by nickm

Thanks, rubiate! I have a few questions, to make sure I understand.

  • Is it really true that on OpenBSD with libevent2, -levent is necessary? Or would -levent_extra -levent_core be sufficient? (I hope that the latter is true, or else the libevent installation is really mangled!)
  • I'm a little worried by the LIBS="-levent" line: on some platforms (like Linux or Windows), Libevent simply won't link with only -levent, since you also need -lrt or -lws2_32 or something. I realize this is only a theoretical issue, since those platforms don't seem to have the same problem as OpenBSD does here. But do you think LIBS="$saved_LIBS -levent" could work instead for this test?

comment:25 in reply to:  24 Changed 4 years ago by rubiate

You're right, it doesn't need -levent, so instead I'll add a conditional to set -levent for the libevent1 case.

At present, -lrt is only added to TOR_LIBEVENT_LIBS if you're building with --enable-static-libevent and it seems adding $save_LIBS would also mix in libraries that aren't needed for libevent (-lpthread, -lexecinfo).

To avoid cluttering TOR_LIBEVENT_LIBS and keep what would have been in LIBS available, how about just passing $save_LIBS?

Changed 4 years ago by rubiate

Attachment: event2-a.patch added

pass save_LIBS when searching libs

comment:26 Changed 4 years ago by rubiate

Or, the alternative approach; no messing with LIBS and manually adding the results to TOR_LIBEVENT_LIBS.

Changed 4 years ago by rubiate

Attachment: event2-b.patch added

don't clear LIBS, manually add results to TOR_LIBEVENT_LIBS

comment:27 Changed 4 years ago by nickm

Keywords: 027-backport added; 026-backport removed
Milestone: Tor: 0.2.8.x-finalTor: 0.2.7.x-final

Applied to master as 5dff4ae0ad5e2a1fb93ceae3446f0a315fbd8a3a. If it works well for everybody, it might be a backport candidate for 0.2.7

comment:28 Changed 4 years ago by cypherpunks

The applied patch misses quotes around the variables. This causes configure to report test errors such as test: too many arguments (and curiously does not terminate the configuration process). The attached patch adds the quotes.

Regarding the applied patch, the autoconf manual states

The result of this test is cached in the ac_cv_search_function variable as ‘none required’ if function is already available, as ‘no’ if no library containing function was found, otherwise as the -llibrary option that needs to be prepended to LIBS.

There is no test whether the variables contain 'no'.

comment:29 Changed 4 years ago by teor

Keywords: TorCoreTeam201511 added

I think we could get this finished in November, if not, let's push it to December at the latest.

We also need to work out what to do if the variable contains "no": terminate configure and tell the user to install libevent?

comment:30 in reply to:  29 ; Changed 4 years ago by rubiate

Replying to teor:

We also need to work out what to do if the variable contains "no": terminate configure and tell the user to install libevent?

configure terminates before this if libevent isn't installed. Also, these checks are only run if the libevent2 header was found (which also turns on the event2 ifdefs) so a result of "no" at this point means something is rather wrong.

comment:31 Changed 4 years ago by teor

Milestone: Tor: 0.2.7.x-finalTor: 0.2.8.x-final

Thanks, let's get the attached patch 0001-Quote-variables-in-case-they-contain-spaces.patch merged to master - it's a good idea to fix quoting.

comment:32 Changed 4 years ago by nickm

Milestone: Tor: 0.2.8.x-finalTor: 0.2.7.x-final

Applied. If we do a backport, we should backport these:

  • 5dff4ae0ad5e2a1fb93ceae3446f0a315fbd8a3a
  • 4e3e5264938e8d2ea2f7c27005baf8e074e098e9

comment:33 in reply to:  30 ; Changed 4 years ago by cypherpunks

Replying to rubiate:

Replying to teor:

We also need to work out what to do if the variable contains "no": terminate configure and tell the user to install libevent?

configure terminates before this if libevent isn't installed. Also, these checks are only run if the libevent2 header was found (which also turns on the event2 ifdefs) so a result of "no" at this point means something is rather wrong.

The Windows compilation in Jenkins seems to fail on this exact problem. When looking at the latest log you can find the problem by searching for "no no". This is the result of the two AC_SEARCH_LIB queries finding no suitable libraries and their result being appended to the TOR_LIBEVENT_LIBS variable.

comment:34 Changed 4 years ago by nickm

Keywords: TorCoreTeam201512 201511-deferred added; TorCoreTeam201511 removed

Bulk-move uncompleted items to december. :/

comment:35 in reply to:  33 Changed 4 years ago by rubiate

Replying to cypherpunks:

The Windows compilation in Jenkins seems to fail on this exact problem. When looking at the latest log you can find the problem by searching for "no no". This is the result of the two AC_SEARCH_LIB queries finding no suitable libraries and their result being appended to the TOR_LIBEVENT_LIBS variable.

Doh! I think I see the problem but I can't test that this patch fixes it, so here's hoping:

diff --git a/configure.ac b/configure.ac
index 0530d77..a1bfded 100644
--- a/configure.ac
+++ b/configure.ac
@@ -500,7 +500,7 @@ AC_CHECK_MEMBERS([struct event.min_heap_idx], , ,
 
 AC_CHECK_HEADERS(event2/event.h event2/dns.h event2/bufferevent_ssl.h)
 
-LIBS="$save_LIBS"
+LIBS="$STATIC_LIBEVENT_FLAGS $TOR_LIB_WS32 $save_LIBS"
 
 AM_CONDITIONAL(USE_EXTERNAL_EVDNS, test x$ac_cv_header_event2_dns_h = xyes)

comment:36 Changed 4 years ago by rubiate

For handling the "no" case:

diff --git a/configure.ac b/configure.ac
index 0530d77..e133e93 100644
--- a/configure.ac
+++ b/configure.ac
@@ -512,8 +512,8 @@ if test "$enable_static_libevent" = "yes"; then
    fi
 else
      if test "x$ac_cv_header_event2_event_h" = "xyes"; then
-       AC_SEARCH_LIBS(event_new, [event event_core])
-       AC_SEARCH_LIBS(evdns_base_new, [event event_extra])
+       AC_SEARCH_LIBS(event_new, [event event_core], , AC_MSG_ERROR("libevent2 is installed but linking it failed while searching for event_new"))
+       AC_SEARCH_LIBS(evdns_base_new, [event event_extra], , AC_MSG_ERROR("libevent2 is installed but linking it failed while searching for evdns_base_new"))
 
        if test "$ac_cv_search_event_new" != "none required"; then
          TOR_LIBEVENT_LIBS="$ac_cv_search_event_new"

Not sure if this is the best error message for this though.

comment:37 Changed 4 years ago by cypherpunks

Patch looks like something that could work but i also have no way of testing it.
For the error messages i would mention these are functions it was looking for to communicate the issue better.

FWIW i submitted #17744 because the configure script still contains instances where strings are compared without quotes but these are out of the scope of this ticket.

comment:38 Changed 4 years ago by teor

This issue is causing build failures on our Windows buildservers (jenkins).

Can someone with Windows please check the patches in comment 35 & 36 resolve this issue?

comment:39 Changed 4 years ago by nickm

Hm. I can't reproduce the bug on my local mingw setup. The second patch looks "obviously correct", so applying it. Will see if it makes jenkins happy as I look hard at the first.

comment:40 Changed 4 years ago by nickm

The second patch, though correct, did not make jenkins happy. The first looks good, so applying that too.

comment:41 Changed 4 years ago by nickm

Resolution: implemented
Status: needs_reviewclosed

I think it's building on windows now? Jenkins has been acting a little funny though, so please reopen if I'm wrong and this code is why.

Note: See TracTickets for help on using tickets.