Opened 4 years ago

Closed 9 months ago

#14177 closed defect (user disappeared)

bwauth spends some 40 minutes "updating rank history". Every hour.

Reported by: ln5 Owned by: aagbsn
Priority: High Milestone:
Component: Core Tor/Torflow Version:
Severity: Blocker Keywords:
Cc: Actual Points:
Parent ID: #13630 Points:
Reviewer: Sponsor:

Description

This is a newly started, well Jan 07 around 01:00, bwauth with default settings.

[bwscanner@salsa ~/torflow/NetworkScanners/BwAuthority]$ tail -100000 data/scanner.1/bw.log  | egrep -A 1 'Updating rank history'
INFO[Mon Jan 12 00:03:37 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 00:42:20 2015]:Consensus history updated.
--
INFO[Mon Jan 12 00:44:36 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 01:23:38 2015]:Consensus history updated.
--
INFO[Mon Jan 12 01:54:48 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 02:34:08 2015]:Consensus history updated.
--
INFO[Mon Jan 12 02:36:32 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 03:16:12 2015]:Consensus history updated.
--
INFO[Mon Jan 12 03:38:51 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 04:18:45 2015]:Consensus history updated.
--
INFO[Mon Jan 12 04:21:06 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 05:00:14 2015]:Consensus history updated.
--
INFO[Mon Jan 12 05:03:46 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 05:43:08 2015]:Consensus history updated.
--
INFO[Mon Jan 12 05:45:15 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 06:25:19 2015]:Consensus history updated.
--
INFO[Mon Jan 12 07:09:36 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 07:48:50 2015]:Consensus history updated.
--
INFO[Mon Jan 12 08:06:39 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 08:46:14 2015]:Consensus history updated.
--
INFO[Mon Jan 12 08:59:29 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 09:39:40 2015]:Consensus history updated.
--
INFO[Mon Jan 12 09:42:26 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 10:22:41 2015]:Consensus history updated.
--
INFO[Mon Jan 12 10:54:23 2015]:Consensus change... Updating rank history

Don't know if this might be related to #4015. What does "freeze" mean there?

Child Tickets

Change History (7)

comment:1 Changed 4 years ago by ln5

$ dpkg -l | egrep python
ii  dh-python                           1.20140128-1ubuntu8              all          Debian helper tools for packaging Python libraries and applications
ii  libpython-stdlib:amd64              2.7.5-5ubuntu3                   amd64        interactive high-level object-oriented language (default python version)
ii  libpython2.7:amd64                  2.7.6-8                          amd64        Shared Python runtime library (version 2.7)
ii  libpython2.7-minimal:amd64          2.7.6-8                          amd64        Minimal subset of the Python language (version 2.7)
ii  libpython2.7-stdlib:amd64           2.7.6-8                          amd64        Interactive high-level object-oriented language (standard library, version 2.7)
ii  libpython3-stdlib:amd64             3.4.0-0ubuntu2                   amd64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.4-minimal:amd64          3.4.0-2ubuntu1                   amd64        Minimal subset of the Python language (version 3.4)
ii  libpython3.4-stdlib:amd64           3.4.0-2ubuntu1                   amd64        Interactive high-level object-oriented language (standard library, version 3.4)
ii  python                              2.7.5-5ubuntu3                   amd64        interactive high-level object-oriented language (default version)
ii  python-apport                       2.14.1-0ubuntu3.6                all          Python library for Apport crash report handling
ii  python-apt                          0.9.3.5ubuntu1                   amd64        Python interface to libapt-pkg
ii  python-apt-common                   0.9.3.5ubuntu1                   all          Python interface to libapt-pkg (locales)
ii  python-chardet                      2.0.1-2build2                    all          universal character encoding detector
ii  python-configobj                    4.7.2+ds-5build1                 all          simple but powerful config file reader and writer for Python
ii  python-crypto                       2.6.1-4build1                    amd64        cryptographic algorithms and protocols for Python
ii  python-dbus                         1.2.0-2build2                    amd64        simple interprocess messaging system (Python interface)
ii  python-dbus-dev                     1.2.0-2build2                    all          main loop integration development files for python-dbus
ii  python-debian                       0.1.21+nmu2ubuntu2               all          Python modules to work with Debian-related data formats
ii  python-elixir                       0.7.1-2                          all          declarative mapper for SQLAlchemy
ii  python-gdbm                         2.7.5-1ubuntu1                   amd64        GNU dbm database support for Python
ii  python-gi                           3.12.0-1ubuntu1                  amd64        Python 2.x bindings for gobject-introspection libraries
ii  python-gnupginterface               0.3.2-9.1ubuntu3                 all          Python interface to GnuPG (GPG)
ii  python-httplib2                     0.8-2build1                      all          comprehensive HTTP client library written for Python
ii  python-keyring                      3.5-1                            all          store and access your passwords safely
ii  python-launchpadlib                 1.10.2+ds-2                      all          Launchpad web services client library
ii  python-lazr.restfulclient           0.13.3-1build1                   all          client for lazr.restful-based web services
ii  python-lazr.uri                     1.0.3-1build1                    all          library for parsing, manipulating, and generating URIs
ii  python-minimal                      2.7.5-5ubuntu3                   amd64        minimal subset of the Python language (default version)
ii  python-newt                         0.52.15-2ubuntu5                 amd64        NEWT module for Python
ii  python-oauth                        1.0.1-3build2                    all          Python library implementing of the OAuth protocol
ii  python-openssl                      0.13-2ubuntu6                    amd64        Python 2 wrapper around the OpenSSL library
ii  python-pam                          0.4.2-13.1ubuntu3                amd64        Python interface to the PAM library
ii  python-pkg-resources                3.3-1ubuntu1                     all          Package Discovery and Resource Access using pkg_resources
ii  python-problem-report               2.14.1-0ubuntu3.6                all          Python library to handle problem reports
ii  python-requests                     2.2.1-1ubuntu0.1                 all          elegant and simple HTTP library for Python, built for human beings
ii  python-secretstorage                2.0.0-1ubuntu1.1                 all          Python module for storing secrets - Python 2.x version
ii  python-serial                       2.6-1build1                      all          pyserial - module encapsulating access for the serial port
ii  python-simplejson                   3.3.1-1ubuntu6                   amd64        simple, fast, extensible JSON encoder/decoder for Python
ii  python-six                          1.5.2-1                          all          Python 2 and 3 compatibility library (Python 2 interface)
ii  python-sqlalchemy                   0.8.4-1build1                    all          SQL toolkit and Object Relational Mapper for Python
ii  python-sqlalchemy-ext               0.8.4-1build1                    amd64        SQL toolkit and Object Relational Mapper for Python - C extension
ii  python-twisted-bin                  13.2.0-1ubuntu1                  amd64        Event-based framework for internet applications
ii  python-twisted-core                 13.2.0-1ubuntu1                  all          Event-based framework for internet applications
ii  python-urllib3                      1.7.1-1build1                    all          HTTP library with thread-safe connection pooling for Python
ii  python-wadllib                      1.3.2-2build1                    all          Python library for navigating WADL files
ii  python-xapian                       1.2.16-2ubuntu1                  amd64        Xapian search engine interface for Python
ii  python-zope.interface               4.0.5-1ubuntu4                   amd64        Interfaces for Python
ii  python2.7                           2.7.6-8                          amd64        Interactive high-level object-oriented language (version 2.7)
ii  python2.7-minimal                   2.7.6-8                          amd64        Minimal subset of the Python language (version 2.7)
ii  python3                             3.4.0-0ubuntu2                   amd64        interactive high-level object-oriented language (default python3 version)
ii  python3-apport                      2.14.1-0ubuntu3.6                all          Python 3 library for Apport crash report handling
ii  python3-apt                         0.9.3.5ubuntu1                   amd64        Python 3 interface to libapt-pkg
ii  python3-commandnotfound             0.3ubuntu12                      all          Python 3 bindings for command-not-found.
ii  python3-dbus                        1.2.0-2build2                    amd64        simple interprocess messaging system (Python 3 interface)
ii  python3-distupgrade                 1:0.220.6                        all          manage release upgrades
ii  python3-gdbm:amd64                  3.4.0-0ubuntu1                   amd64        GNU dbm database support for Python 3.x
ii  python3-gi                          3.12.0-1ubuntu1                  amd64        Python 3 bindings for gobject-introspection libraries
ii  python3-minimal                     3.4.0-0ubuntu2                   amd64        minimal subset of the Python language (default python3 version)
ii  python3-problem-report              2.14.1-0ubuntu3.6                all          Python 3 library to handle problem reports
ii  python3-update-manager              1:0.196.12                       all          python 3.x module for update-manager
ii  python3.4                           3.4.0-2ubuntu1                   amd64        Interactive high-level object-oriented language (version 3.4)
ii  python3.4-minimal                   3.4.0-2ubuntu1                   amd64        Minimal subset of the Python language (version 3.4)
$

comment:2 Changed 4 years ago by ln5

$ ps axuwwww|head -1; ps axuwwww|egrep ^bwscann.\*python
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bwscann+ 13810  0.0  0.1  34772  4716 pts/3    SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.1/bwauthority.cfg
bwscann+ 13811  0.0  0.1  34772  4716 pts/3    SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.2/bwauthority.cfg
bwscann+ 13812  0.0  0.1  34772  4716 pts/3    SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.3/bwauthority.cfg
bwscann+ 13813  0.0  0.1  34772  4712 pts/3    SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.4/bwauthority.cfg
bwscann+ 25153 18.1  5.6 482512 224236 pts/3   SNl  21:02  14:22 python bwauthority_child.py ./data/scanner.4/bwauthority.cfg 48
bwscann+ 25882 13.8  6.5 584072 263780 pts/3   SNl  21:37   6:07 python bwauthority_child.py ./data/scanner.3/bwauthority.cfg 9
bwscann+ 26552 18.2  4.8 417412 194444 pts/3   SNl  21:57   4:31 python bwauthority_child.py ./data/scanner.2/bwauthority.cfg 23
bwscann+ 27307 29.6  3.8 407760 153548 pts/3   SNl  22:18   0:54 python bwauthority_child.py ./data/scanner.1/bwauthority.cfg 3
$

comment:3 Changed 4 years ago by cypherpunks

In the hopes that this will help debugging, here is the configuration from a debian wheezy system

dpkg -l | egrep python
ii  libpython2.7                       2.7.3-6+deb7u2                amd64        Shared Python runtime library (version 2.7)
ii  python                             2.7.3-4+deb7u1                all          interactive high-level object-oriented language (default version)
ii  python-apt                         0.8.8.2                       amd64        Python interface to libapt-pkg
ii  python-apt-common                  0.8.8.2                       all          Python interface to libapt-pkg (locales)
ii  python-central                     0.6.17                        all          register and build utility for Python packages
ii  python-crypto                      2.6-4+deb7u3                  amd64        cryptographic algorithms and protocols for Python
ii  python-dev                         2.7.3-4+deb7u1                all          header files and a static library for Python (default)
ii  python-dumbnet                     1.12-3.1                      amd64        A dumb, portable networking library -- python bindings
ii  python-elixir                      0.7.1-1                       all          Declarative Mapper for SQLAlchemy
ii  python-geoip                       1.3.2-1~bpo70+1               amd64        Python bindings for the GeoIP IP-to-country resolver library
ii  python-httplib2                    0.7.4-2+deb7u1                all          comprehensive HTTP client library written for Python
ii  python-ipaddr                      2.1.10-1                      all          Python module for working with IP addresses, both IPv4 and IPv6
ii  python-jinja2                      2.6-1                         amd64        small but fast and easy to use stand-alone template engine
ii  python-markupsafe                  0.15-1                        amd64        XML/HTML/XHTML Markup safe string for Python
ii  python-minimal                     2.7.3-4+deb7u1                all          minimal subset of the Python language (default version)
ii  python-openssl                     0.13-2+deb7u1                 amd64        Python 2 wrapper around the OpenSSL library
ii  python-pam                         0.4.2-13                      amd64        Python interface to the PAM library
ii  python-paramiko                    1.7.7.1-3.1                   all          Make ssh v2 connections with Python
ii  python-parsley                     1.2-1~bpo70+1                 all          pattern-matching language based on OMeta and Python
ii  python-pkg-resources               0.6.24-1                      all          Package Discovery and Resource Access using pkg_resources
ii  python-pyasn1                      0.1.3-1                       all          ASN.1 library for Python
ii  python-pypcap                      1.1.2+debian-2.2              amd64        object-oriented Python interface for libpcap
ii  python-pyptlib                     0.0.6-1~d70.wheezy+1          all          library for Tor pluggable transports written in Python
ii  python-scapy                       2.2.0-1                       all          Packet generator/sniffer and network scanner/discovery
ii  python-selinux                     2.1.9-5                       amd64        Python bindings to SELinux shared libraries
ii  python-serial                      2.5-2.1                       all          pyserial - module encapsulating access for the serial port
ii  python-sqlalchemy                  0.7.8-1                       all          SQL toolkit and Object Relational Mapper for Python
ii  python-sqlalchemy-ext              0.7.8-1                       amd64        SQL toolkit and Object Relational Mapper for Python - C extension
ii  python-support                     1.0.15                        all          automated rebuilding support for Python modules
ii  python-twisted                     13.2.0-1~bpo70+1              all          Event-based framework for internet applications (dependency package)
ii  python-twisted-bin                 13.2.0-1~bpo70+1              amd64        Event-based framework for internet applications
ii  python-twisted-conch               1:13.2.0-1~bpo70+1            all          Twisted SSH Implementation
ii  python-twisted-core                13.2.0-1~bpo70+1              all          Event-based framework for internet applications
ii  python-twisted-lore                13.2.0-1~bpo70+1              all          Documentation generator with HTML and LaTeX support
ii  python-twisted-mail                13.2.0-1~bpo70+1              all          SMTP, IMAP and POP protocol implementation
ii  python-twisted-names               13.2.0-1~bpo70+1              all          DNS protocol implementation with client and server
ii  python-twisted-news                13.2.0-1~bpo70+1              all          NNTP protocol implementation with client and server
ii  python-twisted-runner              13.2.0-1~bpo70+1              amd64        Process management, including an inetd server
ii  python-twisted-web                 13.2.0-1~bpo70+1              all          HTTP protocol implementation together with clients and servers
ii  python-twisted-words               13.2.0-1~bpo70+1              all          Chat and Instant Messaging
ii  python-txsocksx                    1.13.0.3-1~bpo70+1            all          SOCKS{4,4a,5} endpoints for Twisted
ii  python-txtorcon                    0.11.0-1~bpo70+1              all          Twisted-based asynchronous Tor control protocol implementation
ii  python-yaml                        3.10-4                        amd64        YAML parser and emitter for Python
ii  python-zope.interface              3.6.1-3                       amd64        Interfaces for Python
rc  python2.6                          2.6.8-1.1                     amd64        Interactive high-level object-oriented language (version 2.6)
rc  python2.6-minimal                  2.6.8-1.1                     amd64        Minimal subset of the Python language (version 2.6)
ii  python2.7                          2.7.3-6+deb7u2                amd64        Interactive high-level object-oriented language (version 2.7)
ii  python2.7-dev                      2.7.3-6+deb7u2                amd64        Header files and a static library for Python (v2.7)
ii  python2.7-minimal                  2.7.3-6+deb7u2                amd64        Minimal subset of the Python language (version 2.7)
ps axuwwww|head -1; ps axuwwww|egrep ^bwau.\*python
bwauth    4567  0.0  0.2  34252  2940 ?        SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.1/bwauthority.cfg
bwauth    4568  0.0  0.2  34252  2932 ?        SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.2/bwauthority.cfg
bwauth    4569  0.0  0.2  34252  2940 ?        SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.3/bwauthority.cfg
bwauth    4570  0.0  0.2  34252  2928 ?        SN   Jan23   0:00 python ./bwauthority.py ./data/scanner.4/bwauthority.cfg
bwauth   24794 12.4 17.8 311752 183176 ?       SNl  19:49  14:57 python bwauthority_child.py ./data/scanner.3/bwauthority.cfg 4
bwauth   24819 14.2 17.8 311788 183204 ?       SNl  19:52  16:38 python bwauthority_child.py ./data/scanner.4/bwauthority.cfg 12
bwauth   24828 12.3 17.8 311940 183680 ?       SNl  19:54  14:13 python bwauthority_child.py ./data/scanner.2/bwauthority.cfg 17
bwauth   24833 10.4 18.3 316632 188580 ?       SNl  19:55  12:01 python bwauthority_child.py ./data/scanner.1/bwauthority.cfg 6
NOTE: this is a tweaked bwauthority.cfg, max_fetch_time = 30
grep "Timer exceeded limit: " -r data/scanner.1 | grep "Feb 03" | wc -l
410
grep "Launching stream request for url" -r data/scanner.1 | grep "Feb 03" | wc -l
3728

Sample bwauthority.cfg
_pct ranges are: 0-30, 30-60, 60-80, 80-100
I set circs_per_node to 3 because I was in a hurry. May not be ideal.

[TorCtl]
loglevel=WARN
tor_host = 127.0.0.1
tor_port = 9110
control_host = 127.0.0.1
control_port = 9111
control_pass = 
# XXX: Unused
meta_host = 127.0.0.1
meta_port = 9112

[BwAuthority]
out_dir = ./data/scanner.1/scan-data
pid_file = ./data/scanner.1/bwauthority.pid
db_url =
tor_dir = ./data/tor
start_pct = 0
stop_pct = 30
save_every = 0
nodes_per_slice = 100
circs_per_node = 3
min_streams = 1
max_fetch_time = 30
sleep_start = 01:30
sleep_stop = 01:30

Memory Info

KiB Mem:   1027068 total,   957124 used,    69944 free,     4996 buffers
KiB Swap:  2047996 total,    95808 used,  1952188 free,    69540 cached

Single core VM

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 2
model name      : QEMU Virtual CPU version 1.4.0
stepping        : 3
microcode       : 0x1000065
cpu MHz         : 2799.998
cache size      : 512 KB

comment:4 Changed 4 years ago by cypherpunks

The differences I see:

ii  python-sqlalchemy                  0.7.8-1                       all          SQL toolkit and Object Relational Mapper for Python
ii  python-sqlalchemy-ext              0.7.8-1                       amd64   

vs

ii  python-sqlalchemy                   0.8.4-1build1                    all          SQL toolkit and Object Relational Mapper for Python
ii  python-sqlalchemy-ext               0.8.4-1build1                    amd64        SQL toolkit and Object Relational Mapper for Python - C extension

Note that the "Timer exceeded limit: " WARNs above could indicate that measurements are getting truncated and I should increase max_fetch_time. It might be nice to know how many measurements have stalled before the timer fired. 600 timeouts/day * 30 seconds ~ 5 hours/day

comment:5 Changed 4 years ago by cypherpunks

I am also seeing significant time spent updating rank history:

grep "Consensus change... Updating rank history" data/scanner.1/bw.log | grep "Feb 03" | wc -l
113

import sys
import datetime
tdz = []
f = open(sys.argv[1])
while f:
	line = f.readline()
	if "Consensus change... Updating rank history" in line:
                if not line: break
		n = f.readline()
		s = [datetime.datetime.strptime(y.split("]")[0][5:],"%a %b %d %H:%M:%S %Y") for y in [line,n]]
		td = s[1]-s[0]
		tdz.append(td.total_seconds())
		print sum(tdz)/len(tdz)
print "total %s" % sum(tdz)
f.close()
Last edited 4 years ago by cypherpunks (previous) (diff)

comment:6 Changed 4 years ago by cypherpunks

I see about 4.38975 hours spent each day updating the consensus, for a single scanner, averaged over 10 days of measurements.

I will try disabling

FetchDirInfoExtraEarly 1
FetchDirInfoEarly 1

in torrc and see if this reduces the amount of time spent updating the consensus.

comment:7 Changed 9 months ago by teor

Parent ID: #13630
Resolution: user disappeared
Severity: Blocker
Status: newclosed

This is either no longer an issue, or we'll fix it by moving away from torflow.

Note: See TracTickets for help on using tickets.