Opened 4 years ago

Last modified 20 months ago

#16323 new defect

Compatibility fixes for BWAuths on FreeBSD 10

Reported by: ioerror Owned by:
Priority: Medium Milestone:
Component: Core Tor/Torflow Version:
Severity: Normal Keywords: freebsd
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I'm attempting to run BWAuth scanner code in a FreeBSD 10 jail as a test to see if it will work at all on FreeBSD 10. This assumes you've used ezjail to make a jailed machine - I used a jail behind a NAT for ease of use. The ezjail steps are out of scope but should be around three commands if NAT or a public IP is already assigned to your box.

First - we need to install some packages:

pkg install bash python27 py27-virtualenv py27-sqlite3 py27-sqlite3 sqlite3 py27-sqlite3dbm sqlite
pkg install automake libevent2 gmake openssl

I suggest the following patch:

diff --git a/NetworkScanners/BwAuthority/setup.sh b/NetworkScanners/BwAuthority/setup.sh
index 15ad931..d184060 100755
--- a/NetworkScanners/BwAuthority/setup.sh
+++ b/NetworkScanners/BwAuthority/setup.sh
@@ -1,4 +1,6 @@
-#!/bin/bash -e
+#!/usr/bin/env bash
+set -e
+set -x
 
 SCANNER_DIR=$(dirname "$0")
 SCANNER_DIR=$(readlink -f "$SCANNER_DIR")
@@ -35,7 +37,7 @@ then
   git checkout release-0.2.6
   ./autogen.sh
   ./configure --disable-asciidoc
-  make -j4
+  make
   popd
 fi
 
@@ -53,8 +55,8 @@ peep install -r $SCANNER_DIR/requirements.txt
 
 # 6. Prepare cron script
 cp cron.sh cron-mine.sh
-echo -e "45 0-23 * * * $SCANNER_DIR/cron-mine.sh" | crontab
-echo -e "@reboot $SCANNER_DIR/run_scan.sh\n`crontab -l`" | crontab
+echo -e "45 0-23 * * * $SCANNER_DIR/cron-mine.sh" | crontab -
+echo -e "@reboot $SCANNER_DIR/run_scan.sh\n`crontab -l`" | crontab -
 echo "Prepared crontab. Current crontab: "
 crontab -l
 

Then run everything as an unpriv user:

 git clone https://git.torproject.org/torflow.git
 cd torflow/NetworkScanners/BwAuthority
 ./setup.sh

If Tor fails to build, we'll see the following:

$ ./setup.sh 
/usr/home/builder/torflow /usr/home/builder/torflow/NetworkScanners/BwAuthority
/usr/home/builder/torflow/NetworkScanners/BwAuthority
/usr/home/builder /usr/home/builder/torflow/NetworkScanners/BwAuthority
fatal: destination path 'tor' already exists and is not an empty directory.

Just remove Tor and ensure you have proper build deps:

rm -rf ../../../tor

If Tor fails to build:

--- src/or/src_or_libtor_testing_a-onion_ntor.o ---
  CC       src/or/src_or_libtor_testing_a-onion_ntor.o
--- src/or/src_or_libtor_testing_a-config_codedigest.o ---
  CC       src/or/src_or_libtor_testing_a-config_codedigest.o
src/or/config_codedigest.c:10:10: fatal error: 'or_sha1.i' file not found
#include "or_sha1.i"
         ^
1 error generated.
*** [src/or/src_or_libtor_testing_a-config_codedigest.o] Error code 1

make[1]: stopped in /usr/home/builder/tor
1 error

make[1]: stopped in /usr/home/builder/tor
*** [all] Error code 2

make: stopped in /usr/home/builder/tor
1 error

make: stopped in /usr/home/builder/tor

The above error means that make was running with -j4 - please apply the above patch and try again.

    In file included from src/module.c:24:
    src/connection.h:33:10: fatal error: 'sqlite3.h' file not found
    #include "sqlite3.h"
             ^
    1 error generated.
    error: command 'cc' failed with exit status 1  

This means you need to tell cc where to find sqlite3.h

export CPPFLAGS=-I/usr/local/include

Run setup.sh again - it may error out like so:

crontab: usage error: file name must be specified for replace
usage: crontab [-u user] file
       crontab [-u user] { -e | -l | -r }

This means you didn't apply the above patch for crontab (on BSD it expects {{{-}} or a file name but not an empty string, I guess).

  cd torflow/NetworkScanners/BwAuthority
  export CPPFLAGS=-I/usr/local/include
  ./setup.sh

The patch adds -e and -x which makes debugging easier - -x could be removed.

It looks like it may even work:

$ ./run_scan.sh
Waiting for 60 seconds to refresh tors...
Jun 09 20:28:31.505 [notice] Tor v0.2.6.8 (git-41db4bffd69f7de9) running on FreeBSD with Libevent 2.0.22-stable, OpenSSL 1.0.2a and Zlib 1.2.8.
Jun 09 20:28:31.505 [notice] Tor v0.2.6.8 (git-41db4bffd69f7de9) running on FreeBSD with Libevent 2.0.22-stable, OpenSSL 1.0.2a and Zlib 1.2.8.
Jun 09 20:28:31.505 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 09 20:28:31.505 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Jun 09 20:28:31.505 [notice] Read configuration file "/usr/home/builder/torflow/NetworkScanners/BwAuthority/./data/tor.1/torrc".
Jun 09 20:28:31.505 [notice] Read configuration file "/usr/home/builder/torflow/NetworkScanners/BwAuthority/./data/tor.2/torrc".
Jun 09 20:28:31.513 [notice] Opening Socks listener on 127.0.0.1:9120
Jun 09 20:28:31.513 [notice] Opening Socks listener on 127.0.0.1:9110
Jun 09 20:28:31.513 [notice] Opening Control listener on 127.0.0.1:9121
Jun 09 20:28:31.513 [notice] Opening Control listener on 127.0.0.1:9111
Jun 09 20:28:31.513 [warn] Fixing permissions on directory ./data/tor.2
Jun 09 20:28:31.513 [warn] Fixing permissions on directory ./data/tor.1

Child Tickets

Change History (8)

comment:1 Changed 4 years ago by ioerror

Eventually it detaches:

+ [ -f bwauthenv/bin/activate ]
+ echo 'Using virtualenv in bwauthenv...'
Using virtualenv in bwauthenv...
+ . bwauthenv/bin/activate
+ deactivate nondestructive
+ unset pydoc
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' -o -n '' ]
+ [ -n '' ]
+ unset VIRTUAL_ENV
+ [ ! nondestructive = nondestructive ]
+ VIRTUAL_ENV=/usr/home/builder/torflow/NetworkScanners/BwAuthority/bwauthenv
+ export VIRTUAL_ENV
+ _OLD_VIRTUAL_PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/builder/bin
+ PATH=/usr/home/builder/torflow/NetworkScanners/BwAuthority/bwauthenv/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/builder/bin
+ export PATH
+ [ -n '' ]
+ [ -z '' ]
+ _OLD_VIRTUAL_PS1='$ '
+ [ x != x ]
+ basename /usr/home/builder/torflow/NetworkScanners/BwAuthority/bwauthenv
+ [ bwauthenv = __ ]
+ basename /usr/home/builder/torflow/NetworkScanners/BwAuthority/bwauthenv
+ PS1='(bwauthenv)$ '
+ export PS1
+ alias 'pydoc=python -m pydoc'
+ [ -n '' -o -n '' ]
+ [ -z ../../../SQLAlchemy-0.7.10/lib:../../../Elixir-0.7.1/ ]
+ export PYTHONPATH
+ seq 9
+ nice -n 20 ./bwauthority.py ./data/scanner.1/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.2/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.3/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.4/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.5/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.6/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.7/bwauthority.cfg
+ nice -n 20 ./bwauthority.py ./data/scanner.8/bwauthority.cfg
+ echo 'Launched 9 bandwidth scanners. Job listing: '
Launched 9 bandwidth scanners. Job listing: 
+ jobs -l
+ nice -n 20 ./bwauthority.py ./data/scanner.9/bwauthority.cfg
[1]   26480 Running           
[2]   26482 Running           
[3]   26530 Running           
[4]   26531 Running           
[5]   26532 Running           
[6]   26533 Running           
[7]   26534 Running           
[8]   26535 Running           
[9]   26536 Running           
[10] - 26537 Running          
[11] + 26538 Running          

comment:2 Changed 4 years ago by ioerror

Looking at the data output, I see this - unsure if it is normal or not:

==> data/scanner.9/bw.log <==
./bwauthority.py:70: SyntaxWarning: name 'p' is assigned to before global declaration
  global p
NOTICE[Tue Jun 09 20:33:58 2015]:TorFlow Version: master dd3d01fae752ff0c400ec55a700dfa11c31546d6
NOTICE[Tue Jun 09 20:33:58 2015]:TorCtl Version: detached c8fcb25b079d52a20cafc7f7adf178e90ab76338
NOTICE[Tue Jun 09 20:33:59 2015]:Child Process Spawned...
NOTICE[Tue Jun 09 20:37:06 2015]:Starting slice for percentiles 0.0-10.3305785124

comment:3 Changed 4 years ago by ioerror

I don't see ./data/aggregate-debug.log - where should it be?

comment:4 Changed 4 years ago by TvdW

aggregate-debug will only reveal itself once aggregate.py has been run at least once

comment:5 Changed 21 months ago by teor

Severity: Blocker
Summary: BWAuth on FreeBSD 10Compatibility fixes for BWAuths on FreeBSD 10

Single tickets that deal with multiple issues are really hard to resolve. If anyone wants to keep on working on this, please split up each issue into its own ticket, and make this ticket the parent.

I believe that this patch is no longer necessary, because the issue has been resolved in a recent tor version:

-  make -j4
+  make

comment:6 Changed 21 months ago by teor

Severity: BlockerNormal

Priorities and Severities in torflow are meaningless, setting them all to Medium/Normal.

comment:7 Changed 21 months ago by teor

Owner: aagbsn deleted
Status: newassigned

aagbsn was the default owner, unassigning

comment:8 Changed 20 months ago by teor

Status: assignednew

Mark all tickets that are assigned to nobody as "new".

Note: See TracTickets for help on using tickets.