bwauth spends some 40 minutes "updating rank history". Every hour.
This is a newly started, well Jan 07 around 01:00, bwauth with default settings.
[bwscanner@salsa ~/torflow/NetworkScanners/BwAuthority]$ tail -100000 data/scanner.1/bw.log | egrep -A 1 'Updating rank history'
INFO[Mon Jan 12 00:03:37 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 00:42:20 2015]:Consensus history updated.
--
INFO[Mon Jan 12 00:44:36 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 01:23:38 2015]:Consensus history updated.
--
INFO[Mon Jan 12 01:54:48 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 02:34:08 2015]:Consensus history updated.
--
INFO[Mon Jan 12 02:36:32 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 03:16:12 2015]:Consensus history updated.
--
INFO[Mon Jan 12 03:38:51 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 04:18:45 2015]:Consensus history updated.
--
INFO[Mon Jan 12 04:21:06 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 05:00:14 2015]:Consensus history updated.
--
INFO[Mon Jan 12 05:03:46 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 05:43:08 2015]:Consensus history updated.
--
INFO[Mon Jan 12 05:45:15 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 06:25:19 2015]:Consensus history updated.
--
INFO[Mon Jan 12 07:09:36 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 07:48:50 2015]:Consensus history updated.
--
INFO[Mon Jan 12 08:06:39 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 08:46:14 2015]:Consensus history updated.
--
INFO[Mon Jan 12 08:59:29 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 09:39:40 2015]:Consensus history updated.
--
INFO[Mon Jan 12 09:42:26 2015]:Consensus change... Updating rank history
INFO[Mon Jan 12 10:22:41 2015]:Consensus history updated.
--
INFO[Mon Jan 12 10:54:23 2015]:Consensus change... Updating rank historyDon't know if this might be related to #4015 (moved). What does "freeze" mean there?
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Link issues together to show that they're related.
Learn more.
Activity
- Author
$ dpkg -l | egrep python ii dh-python 1.20140128-1ubuntu8 all Debian helper tools for packaging Python libraries and applications ii libpython-stdlib:amd64 2.7.5-5ubuntu3 amd64 interactive high-level object-oriented language (default python version) ii libpython2.7:amd64 2.7.6-8 amd64 Shared Python runtime library (version 2.7) ii libpython2.7-minimal:amd64 2.7.6-8 amd64 Minimal subset of the Python language (version 2.7) ii libpython2.7-stdlib:amd64 2.7.6-8 amd64 Interactive high-level object-oriented language (standard library, version 2.7) ii libpython3-stdlib:amd64 3.4.0-0ubuntu2 amd64 interactive high-level object-oriented language (default python3 version) ii libpython3.4-minimal:amd64 3.4.0-2ubuntu1 amd64 Minimal subset of the Python language (version 3.4) ii libpython3.4-stdlib:amd64 3.4.0-2ubuntu1 amd64 Interactive high-level object-oriented language (standard library, version 3.4) ii python 2.7.5-5ubuntu3 amd64 interactive high-level object-oriented language (default version) ii python-apport 2.14.1-0ubuntu3.6 all Python library for Apport crash report handling ii python-apt 0.9.3.5ubuntu1 amd64 Python interface to libapt-pkg ii python-apt-common 0.9.3.5ubuntu1 all Python interface to libapt-pkg (locales) ii python-chardet 2.0.1-2build2 all universal character encoding detector ii python-configobj 4.7.2+ds-5build1 all simple but powerful config file reader and writer for Python ii python-crypto 2.6.1-4build1 amd64 cryptographic algorithms and protocols for Python ii python-dbus 1.2.0-2build2 amd64 simple interprocess messaging system (Python interface) ii python-dbus-dev 1.2.0-2build2 all main loop integration development files for python-dbus ii python-debian 0.1.21+nmu2ubuntu2 all Python modules to work with Debian-related data formats ii python-elixir 0.7.1-2 all declarative mapper for SQLAlchemy ii python-gdbm 2.7.5-1ubuntu1 amd64 GNU dbm database support for Python ii python-gi 3.12.0-1ubuntu1 amd64 Python 2.x bindings for gobject-introspection libraries ii python-gnupginterface 0.3.2-9.1ubuntu3 all Python interface to GnuPG (GPG) ii python-httplib2 0.8-2build1 all comprehensive HTTP client library written for Python ii python-keyring 3.5-1 all store and access your passwords safely ii python-launchpadlib 1.10.2+ds-2 all Launchpad web services client library ii python-lazr.restfulclient 0.13.3-1build1 all client for lazr.restful-based web services ii python-lazr.uri 1.0.3-1build1 all library for parsing, manipulating, and generating URIs ii python-minimal 2.7.5-5ubuntu3 amd64 minimal subset of the Python language (default version) ii python-newt 0.52.15-2ubuntu5 amd64 NEWT module for Python ii python-oauth 1.0.1-3build2 all Python library implementing of the OAuth protocol ii python-openssl 0.13-2ubuntu6 amd64 Python 2 wrapper around the OpenSSL library ii python-pam 0.4.2-13.1ubuntu3 amd64 Python interface to the PAM library ii python-pkg-resources 3.3-1ubuntu1 all Package Discovery and Resource Access using pkg_resources ii python-problem-report 2.14.1-0ubuntu3.6 all Python library to handle problem reports ii python-requests 2.2.1-1ubuntu0.1 all elegant and simple HTTP library for Python, built for human beings ii python-secretstorage 2.0.0-1ubuntu1.1 all Python module for storing secrets - Python 2.x version ii python-serial 2.6-1build1 all pyserial - module encapsulating access for the serial port ii python-simplejson 3.3.1-1ubuntu6 amd64 simple, fast, extensible JSON encoder/decoder for Python ii python-six 1.5.2-1 all Python 2 and 3 compatibility library (Python 2 interface) ii python-sqlalchemy 0.8.4-1build1 all SQL toolkit and Object Relational Mapper for Python ii python-sqlalchemy-ext 0.8.4-1build1 amd64 SQL toolkit and Object Relational Mapper for Python - C extension ii python-twisted-bin 13.2.0-1ubuntu1 amd64 Event-based framework for internet applications ii python-twisted-core 13.2.0-1ubuntu1 all Event-based framework for internet applications ii python-urllib3 1.7.1-1build1 all HTTP library with thread-safe connection pooling for Python ii python-wadllib 1.3.2-2build1 all Python library for navigating WADL files ii python-xapian 1.2.16-2ubuntu1 amd64 Xapian search engine interface for Python ii python-zope.interface 4.0.5-1ubuntu4 amd64 Interfaces for Python ii python2.7 2.7.6-8 amd64 Interactive high-level object-oriented language (version 2.7) ii python2.7-minimal 2.7.6-8 amd64 Minimal subset of the Python language (version 2.7) ii python3 3.4.0-0ubuntu2 amd64 interactive high-level object-oriented language (default python3 version) ii python3-apport 2.14.1-0ubuntu3.6 all Python 3 library for Apport crash report handling ii python3-apt 0.9.3.5ubuntu1 amd64 Python 3 interface to libapt-pkg ii python3-commandnotfound 0.3ubuntu12 all Python 3 bindings for command-not-found. ii python3-dbus 1.2.0-2build2 amd64 simple interprocess messaging system (Python 3 interface) ii python3-distupgrade 1:0.220.6 all manage release upgrades ii python3-gdbm:amd64 3.4.0-0ubuntu1 amd64 GNU dbm database support for Python 3.x ii python3-gi 3.12.0-1ubuntu1 amd64 Python 3 bindings for gobject-introspection libraries ii python3-minimal 3.4.0-0ubuntu2 amd64 minimal subset of the Python language (default python3 version) ii python3-problem-report 2.14.1-0ubuntu3.6 all Python 3 library to handle problem reports ii python3-update-manager 1:0.196.12 all python 3.x module for update-manager ii python3.4 3.4.0-2ubuntu1 amd64 Interactive high-level object-oriented language (version 3.4) ii python3.4-minimal 3.4.0-2ubuntu1 amd64 Minimal subset of the Python language (version 3.4) $ - Author
$ ps axuwwww|head -1; ps axuwwww|egrep ^bwscann.\*python USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND bwscann+ 13810 0.0 0.1 34772 4716 pts/3 SN Jan23 0:00 python ./bwauthority.py ./data/scanner.1/bwauthority.cfg bwscann+ 13811 0.0 0.1 34772 4716 pts/3 SN Jan23 0:00 python ./bwauthority.py ./data/scanner.2/bwauthority.cfg bwscann+ 13812 0.0 0.1 34772 4716 pts/3 SN Jan23 0:00 python ./bwauthority.py ./data/scanner.3/bwauthority.cfg bwscann+ 13813 0.0 0.1 34772 4712 pts/3 SN Jan23 0:00 python ./bwauthority.py ./data/scanner.4/bwauthority.cfg bwscann+ 25153 18.1 5.6 482512 224236 pts/3 SNl 21:02 14:22 python bwauthority_child.py ./data/scanner.4/bwauthority.cfg 48 bwscann+ 25882 13.8 6.5 584072 263780 pts/3 SNl 21:37 6:07 python bwauthority_child.py ./data/scanner.3/bwauthority.cfg 9 bwscann+ 26552 18.2 4.8 417412 194444 pts/3 SNl 21:57 4:31 python bwauthority_child.py ./data/scanner.2/bwauthority.cfg 23 bwscann+ 27307 29.6 3.8 407760 153548 pts/3 SNl 22:18 0:54 python bwauthority_child.py ./data/scanner.1/bwauthority.cfg 3 $ In the hopes that this will help debugging, here is the configuration from a debian wheezy system
dpkg -l | egrep python ii libpython2.7 2.7.3-6+deb7u2 amd64 Shared Python runtime library (version 2.7) ii python 2.7.3-4+deb7u1 all interactive high-level object-oriented language (default version) ii python-apt 0.8.8.2 amd64 Python interface to libapt-pkg ii python-apt-common 0.8.8.2 all Python interface to libapt-pkg (locales) ii python-central 0.6.17 all register and build utility for Python packages ii python-crypto 2.6-4+deb7u3 amd64 cryptographic algorithms and protocols for Python ii python-dev 2.7.3-4+deb7u1 all header files and a static library for Python (default) ii python-dumbnet 1.12-3.1 amd64 A dumb, portable networking library -- python bindings ii python-elixir 0.7.1-1 all Declarative Mapper for SQLAlchemy ii python-geoip 1.3.2-1~bpo70+1 amd64 Python bindings for the GeoIP IP-to-country resolver library ii python-httplib2 0.7.4-2+deb7u1 all comprehensive HTTP client library written for Python ii python-ipaddr 2.1.10-1 all Python module for working with IP addresses, both IPv4 and IPv6 ii python-jinja2 2.6-1 amd64 small but fast and easy to use stand-alone template engine ii python-markupsafe 0.15-1 amd64 XML/HTML/XHTML Markup safe string for Python ii python-minimal 2.7.3-4+deb7u1 all minimal subset of the Python language (default version) ii python-openssl 0.13-2+deb7u1 amd64 Python 2 wrapper around the OpenSSL library ii python-pam 0.4.2-13 amd64 Python interface to the PAM library ii python-paramiko 1.7.7.1-3.1 all Make ssh v2 connections with Python ii python-parsley 1.2-1~bpo70+1 all pattern-matching language based on OMeta and Python ii python-pkg-resources 0.6.24-1 all Package Discovery and Resource Access using pkg_resources ii python-pyasn1 0.1.3-1 all ASN.1 library for Python ii python-pypcap 1.1.2+debian-2.2 amd64 object-oriented Python interface for libpcap ii python-pyptlib 0.0.6-1~d70.wheezy+1 all library for Tor pluggable transports written in Python ii python-scapy 2.2.0-1 all Packet generator/sniffer and network scanner/discovery ii python-selinux 2.1.9-5 amd64 Python bindings to SELinux shared libraries ii python-serial 2.5-2.1 all pyserial - module encapsulating access for the serial port ii python-sqlalchemy 0.7.8-1 all SQL toolkit and Object Relational Mapper for Python ii python-sqlalchemy-ext 0.7.8-1 amd64 SQL toolkit and Object Relational Mapper for Python - C extension ii python-support 1.0.15 all automated rebuilding support for Python modules ii python-twisted 13.2.0-1~bpo70+1 all Event-based framework for internet applications (dependency package) ii python-twisted-bin 13.2.0-1~bpo70+1 amd64 Event-based framework for internet applications ii python-twisted-conch 1:13.2.0-1~bpo70+1 all Twisted SSH Implementation ii python-twisted-core 13.2.0-1~bpo70+1 all Event-based framework for internet applications ii python-twisted-lore 13.2.0-1~bpo70+1 all Documentation generator with HTML and LaTeX support ii python-twisted-mail 13.2.0-1~bpo70+1 all SMTP, IMAP and POP protocol implementation ii python-twisted-names 13.2.0-1~bpo70+1 all DNS protocol implementation with client and server ii python-twisted-news 13.2.0-1~bpo70+1 all NNTP protocol implementation with client and server ii python-twisted-runner 13.2.0-1~bpo70+1 amd64 Process management, including an inetd server ii python-twisted-web 13.2.0-1~bpo70+1 all HTTP protocol implementation together with clients and servers ii python-twisted-words 13.2.0-1~bpo70+1 all Chat and Instant Messaging ii python-txsocksx 1.13.0.3-1~bpo70+1 all SOCKS{4,4a,5} endpoints for Twisted ii python-txtorcon 0.11.0-1~bpo70+1 all Twisted-based asynchronous Tor control protocol implementation ii python-yaml 3.10-4 amd64 YAML parser and emitter for Python ii python-zope.interface 3.6.1-3 amd64 Interfaces for Python rc python2.6 2.6.8-1.1 amd64 Interactive high-level object-oriented language (version 2.6) rc python2.6-minimal 2.6.8-1.1 amd64 Minimal subset of the Python language (version 2.6) ii python2.7 2.7.3-6+deb7u2 amd64 Interactive high-level object-oriented language (version 2.7) ii python2.7-dev 2.7.3-6+deb7u2 amd64 Header files and a static library for Python (v2.7) ii python2.7-minimal 2.7.3-6+deb7u2 amd64 Minimal subset of the Python language (version 2.7)ps axuwwww|head -1; ps axuwwww|egrep ^bwau.\*python bwauth 4567 0.0 0.2 34252 2940 ? SN Jan23 0:00 python ./bwauthority.py ./data/scanner.1/bwauthority.cfg bwauth 4568 0.0 0.2 34252 2932 ? SN Jan23 0:00 python ./bwauthority.py ./data/scanner.2/bwauthority.cfg bwauth 4569 0.0 0.2 34252 2940 ? SN Jan23 0:00 python ./bwauthority.py ./data/scanner.3/bwauthority.cfg bwauth 4570 0.0 0.2 34252 2928 ? SN Jan23 0:00 python ./bwauthority.py ./data/scanner.4/bwauthority.cfg bwauth 24794 12.4 17.8 311752 183176 ? SNl 19:49 14:57 python bwauthority_child.py ./data/scanner.3/bwauthority.cfg 4 bwauth 24819 14.2 17.8 311788 183204 ? SNl 19:52 16:38 python bwauthority_child.py ./data/scanner.4/bwauthority.cfg 12 bwauth 24828 12.3 17.8 311940 183680 ? SNl 19:54 14:13 python bwauthority_child.py ./data/scanner.2/bwauthority.cfg 17 bwauth 24833 10.4 18.3 316632 188580 ? SNl 19:55 12:01 python bwauthority_child.py ./data/scanner.1/bwauthority.cfg 6NOTE: this is a tweaked bwauthority.cfg, max_fetch_time = 30 grep "Timer exceeded limit: " -r data/scanner.1 | grep "Feb 03" | wc -l 410grep "Launching stream request for url" -r data/scanner.1 | grep "Feb 03" | wc -l 3728Sample bwauthority.cfg _pct ranges are: 0-30, 30-60, 60-80, 80-100 I set circs_per_node to 3 because I was in a hurry. May not be ideal.
[TorCtl] loglevel=WARN tor_host = 127.0.0.1 tor_port = 9110 control_host = 127.0.0.1 control_port = 9111 control_pass = # XXX: Unused meta_host = 127.0.0.1 meta_port = 9112 [BwAuthority] out_dir = ./data/scanner.1/scan-data pid_file = ./data/scanner.1/bwauthority.pid db_url = tor_dir = ./data/tor start_pct = 0 stop_pct = 30 save_every = 0 nodes_per_slice = 100 circs_per_node = 3 min_streams = 1 max_fetch_time = 30 sleep_start = 01:30 sleep_stop = 01:30Memory Info
KiB Mem: 1027068 total, 957124 used, 69944 free, 4996 buffers KiB Swap: 2047996 total, 95808 used, 1952188 free, 69540 cachedSingle core VM
processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 2 model name : QEMU Virtual CPU version 1.4.0 stepping : 3 microcode : 0x1000065 cpu MHz : 2799.998 cache size : 512 KBThe differences I see:
ii python-sqlalchemy 0.7.8-1 all SQL toolkit and Object Relational Mapper for Python ii python-sqlalchemy-ext 0.7.8-1 amd64vs
ii python-sqlalchemy 0.8.4-1build1 all SQL toolkit and Object Relational Mapper for Python ii python-sqlalchemy-ext 0.8.4-1build1 amd64 SQL toolkit and Object Relational Mapper for Python - C extensionNote that the "Timer exceeded limit: " WARNs above could indicate that measurements are getting truncated and I should increase max_fetch_time. It might be nice to know how many measurements have stalled before the timer fired. 600 timeouts/day * 30 seconds ~ 5 hours/day
I am also seeing significant time spent updating rank history:
grep "Consensus change... Updating rank history" data/scanner.1/bw.log | grep "Feb 03" | wc -l 113
import sys import datetime tdz = [] f = open(sys.argv[1]) while f: line = f.readline() if "Consensus change... Updating rank history" in line: if not line: break n = f.readline() s = [datetime.datetime.strptime(y.split("]")[0][5:],"%a %b %d %H:%M:%S %Y") for y in [line,n]] td = s[1]-s[0] tdz.append(td.total_seconds()) print sum(tdz)/len(tdz) print "total %s" % sum(tdz) f.close()I see about 4.38975 hours spent each day updating the consensus, for a single scanner, averaged over 10 days of measurements.
I will try disabling
FetchDirInfoExtraEarly 1 FetchDirInfoEarly 1
in torrc and see if this reduces the amount of time spent updating the consensus.
This is either no longer an issue, or we'll fix it by moving away from torflow.
Trac:
Sponsor: N/A to N/A
Status: new to closed
Reviewer: N/A to N/A
Parent: N/A to #13630 (moved)
Resolution: N/A to user disappeared
Severity: N/A to Blocker- Trac closed
closed
Please register or sign in to reply