Sponsor 31

Project Title - Modularization work 

Project Period: December 1, 2018 - November 30, 2019 

Project Goals/Activities

The key deliverables of the modularization project are:

  • Refactored architecture for how Tor modules should communicate, designed to reduce module complexity and maintenance burden.
  • Architectural documentation for how Tor modules work with one another, covering both the actuality and the refactored architecture.
  • New architecture for several key Tor modules.
  • Improved testing for those modules.
  • Improved documentation for those modules.


  1. Determine how to best refactor modules to make them more resilient,segregated, and automatically testable. (288 hours) -- March 2019

1.1. Identify pieces that we have to refactor. (96 hours)

1.2. Draft initial design for each piece. (192 hours)

  1. Identify which parts of the codebase are most important to refactor first. (256 hours) -- April and August 2019

2.1. Discuss which parts to refactor first. (16 hours)

2.2. Research and discuss automated tooling. (48 hours)

2.3. Design tooling as needed. (48 hours)

2.4. Repeat the same tasks 2.1, 2.2 and 2.3 after milestone 4 is completed. (144 hours)

  1. Revise the system of interactions between modules to improve usability and maintainability. (88 hours) -- May 2019

3.1. Review publisher/subscriber design that Tor uses. (16 hours)

3.2. Implement refactoring. (72 hours)

  1. Create and deploy plan to incorporate refactored modules. (184 hours) -- August 2019

4.1. For each module and each architectural change, form a plan to apply that change to that module. This plan should include increased testing, if the module's test coverage is not already high. (160 hours)

4.2. For each plan formed in 4.1, deploy that plan. (24 hours)

  1. Create documentation for how Tor modules work with one another with both old and new architecture. (432 hours) -- December 2018

5.1. Update module documentation and incorporate into Tor. (96 hours)

5.2. Document existing refactoring. (144 hours)

5.3. Document new elements. (144 hours)

5.4. For modules we are not refactoring, document important interactions. (48 hours)

Teams Involved: Network

Project Tracking

Refactoring cross-module communications

#28226 Backend for an async publish-subcribe messaging system for cross-module communications (Tor 0.4.1)

Coding best practices

#29221 Tooling to track code-style/best-practices violations #29219 Write guidelines for Tor coding best practices #29220 Update review guidelines for practices #29223 List canonical abbreviations to use in Tor functions and identifiers

Refactoring modules

#29201 Tor bootstrap hangs when offline - Split up control.c #29209 Reduce visibility of more data type internals #29218 Revise "handles" to be opaque, safe. Document standard usage #29224 Abstractions and best practices for disabled modules #29217 Script to add new tor files and modules in rust and/or C #29227 Audit Tor roles (relay, bridge, etc) and remove needless dependencies where possible


#29216 Document how to make new files/modules in Tor


Sponsor: Sponsor31-must (3 matches)

Ticket Summary Status Owner Keywords Priority
#29215 Document target, modular tor architecture accepted catalyst network-team-roadmap-september, s31-docs Medium
#32206 write high-level outline of target modular tor architecture needs_review catalyst network-team-roadmap-september, s31-docs Medium
#32208 write description of control subsystem architecture accepted catalyst network-team-roadmap-september, s31-docs Medium

Sponsor: Sponsor31-can (89 matches)

Ticket Summary Status Owner Keywords Priority
#25950 Run "accounting_run_housekeeping" with a periodic event. assigned 034-roadmap-subtask, 034-triage-20180328, 034-included-20180328 Medium
#28170 Test stem pull requests against all supported tor versions, and tor nightly builds new ci, travis Medium
#28208 Run bridges+hs-v23 for make test-network in 0.4.2 and later assigned test, fast-fix, teor-backlog-ci, network-team-roadmap-november Medium
#28377 Add libnss to the travis apt and macOS packages assigned fast-fix, tor-ci, travis, postfreeze-ok, teor-backlog-ci Medium
#28378 Add libsystemd-devel to the travis config, if available assigned fast-fix, tor-systemd, 029-backport, 035-backport, postfreeze-ok, tor-ci, teor-backlog-ci, 033-unreached-backport Medium
#29209 Reduce visibility of more data type internals new technical-debt refactoring Medium
#29210 Distribute control.c functionality across various modules merge_ready catalyst network-team-roadmap-september, 042-deferred-20190918 Medium
#29211 Distribute config.c functionality across more modules assigned nickm network-team-roadmap-august Medium
#29212 Create more modularity inside src/core new Medium
#29213 Minimize the boundary between src/core and src/{feature,app} new Medium
#29216 Document how to make new files/modules in Tor new s31-docs Medium
#29217 Script to add new tor files and modules in rust and/or C new network-team-roadmap-september Medium
#29219 Write (more) guidelines for Tor coding best practices new doc, ex-catalyst-20190619, s31-docs, 042-deferred-20190918 Medium
#29220 Update review guidelines to list best practices assigned asn doc, network-team-roadmap-2019-Q1Q2, 042-deferred, s31-docs Medium
#29222 Fix biggest violations of coding best-practices new 042-deferred-20190918 Medium
#29223 List canonical abbreviations to use in Tor functions and identifiers needs_revision nickm doc, s31-docs 042-can Medium
#29224 Abstractions and best practices for disabled modules new 042-deferred-20190918 Medium
#29225 Improve efficiency and flexibility of threadpool/workqueue API new Medium
#29226 Automate application of C formatting to code new network-team-roadmap-october, 042-deferred-20190918 Medium
#29227 Audit Tor roles (relay, bridge, etc) and remove needless dependencies where possible new 042-deferred-20190918 Medium
#29228 Unify some or all of Tor's various message/event/work queues. new Medium
#29437 test-stem times out intermittently assigned tor-ci-fail-sometimes Medium
#29642 Free Windows library handles when we free_all in a module new tor-windows, memory-management, technical-debt Low
#29656 describe global initialization in our tinytest-based unit tests new developer-doc, network-team-roadmap-october, s31-docs Medium
#29689 Make Tor's consistent with chutney's assigned tor-chutney, tor-test, tor-ci, fast-fix, technical-debt, ex-sponsor-19, ex-sponsor19, teor-backlog-ci Medium
#29793 show-distdir-core and friends should fail more quietly new tor-ci, travis Medium
#29821 Make circuit/channel padding correctly respect dormant mode new wtf-pad padding dormant dormant-mode Low
#29881 Create a GitHub bot that makes sure there is a pull request on master for every branch new tor-ci, 041-deferred-20190530, sponsor31-maybe Medium
#29883 Enforce lower practracker limits for every new release new 042-proposed Medium
#29895 avoid storing ints in void* in mainloop event data new portability, technical-debt Medium
#29911 Update once we have some experience with practracker assigned catalyst doc, postfreeze-ok, 041-deferred-20190719, 042-should Medium
#30225 Run clang's scan-build in Tor's CI new tor-ci, correctness, coverity, clang, scan-build Medium
#30235 Stem hangs waiting for control port data needs_information tor-ci Medium
#30243 Stem's tests use fixed ports for tor, which may already be in use new atagar tor-ci-single-failure Medium
#30257 Propagate USR1 and ABRT signals from stem tests through to tor new atagar Medium
#30349 Document member-hiding conventions for structs new technical-debt, refactoring, network-team-roadmap-october, s31-docs Medium
#30651 Make stem's Travis jobs log controller messages from stem and tor assigned ci, travis, teor-backlog-ci Medium
#30695 Consider running tor's stem CI job in --target ONLINE mode, maybe with chutney new Medium
#30720 Use for the examples in Chutney's README assigned doc, easy, teor-backlog-ci, network-team-roadmap-november High
#30745 Document disabled CI assigned teor 043-should process Medium
#30816 Remove ping ::1 from tor's test-network-all and simplify the logic new refactor, simplify, network-team-roadmap-2019-Q1Q2, tor-ci, 042-deferred-20190918 Medium
#30839 Update with our latest end of life process needs_revision teor network-team-roadmap-september, 042-can, s31-docs Medium
#30858 Load geoip and geoip6 files during the unit tests assigned technical-debt, geoip, fast-fix?, easy?, teor-backlog-ci Low
#30890 Make a useful .gitlab-ci.yml mirroring job new tor-ci, gitlab Medium
#30984 Make a key-value line abstraction to output control replies merge_ready catalyst 042-deferred-20190918, early-review Medium
#30991 Auto-tabify makefiles? complain about mistabbed makefiles? new 042-can Low
#31074 Use tor_queue.h macros in config_line_t new refactor technical-debt network-team-roadmap-september Low
#31078 improve docs for config var abstraction assigned nickm 042-should Medium
#31081 GETCONF allows zero arguments, contrary to spec needs_revision tor-control, tor-spec Medium
#31121 Use publish-subscribe system in more places new network-team-roadmap-october Medium
#31122 Use subsystems architecture in more places new network-team-roadmap-november Medium
#31179 Eliminate non tor_queue.h linked lists. new network-team-roadmap-october Medium
#31180 Remove deprecated options in 0.4.3. assigned network-team-roadmap-november Medium
#31181 Deprecate more options in 0.4.3 assigned network-team-roadmap-november Medium
#31182 Add more typesafe containers to tor new network-team-roadmap-november Medium
#31238 Write a nontrivial "preferred" rust module new Medium
#31483 token_bucket_ctr_adjust() does not convert rate to step accepted nickm Very High
#31502 Revise authority and fallback code so that they use the same "defaults" logic as other code new 042-deferred-20190918 Medium
#31530 Shall we require STMT_BEGIN/STMT_END everywhere? new 042-deferred-20190918 Medium
#31631 Write a test for round-trip encode/decode operations on configuration objects. accepted nickm 042-deferred-20190918 Medium
#31634 Check .may_include order and tor subsystem init order are compatible needs_revision nickm diagnostics, 043-should, practracker Medium
#31713 Automatically run the autostyle scripts before every commit new Medium
#31757 apparently not reliable on Appveyor assigned ahf tor-ci 042-should asn-merge High
#31770 Divide connection_edge.c into separate files. assigned nickm 042-can Medium
#31851 Allow Tor to be compiled without support for relay mode new teor tor-design, network-team-roadmap-october Medium
#31869 Run doctest on all of our python files new 042-can Medium
#31908 Unify meaning of routerset NULL and routerset "" to avoid recurrence of #31495 new Medium
#32137 Split feature/ out of core/ new tor-design Medium
#32139 Disable all dirauth options when those modules are disabled assigned nickm tor-design, network-team-roadmap-november Medium
#32162 Make router.c relay-only new tor-design, network-team-roadmap-october Medium
#32249 Speed up practracker, by passing it a list of modified file names new 043-can Medium
#32306 check-local target to enforce doxygen-correctness assigned nickm network-team-roadmap-november, s31-docs Medium
#32372 Add a practracker mode that only regenerates over-broad exceptions new 043-should Medium
#32373 Create a way to scan all configured file paths new network-team-roadmap-? Medium
#32374 Create a test framework for options_act*() new network-team-roadmap-? Medium
#32395 Disable all relay options when those modules are disabled new tor-design, network-team-roadmap-november Medium
#32396 Add state file support to new network-team-roadmap-? Medium
#32407 Fix crash when calling --dump-config with failing set_options merge_ready nickm consider-backport-after-0425, 035-backport 040-backport 041-backport 042-backport Medium
#32408 Handle options_act_reversible() in new config system. assigned nickm Medium
#32447 Refactor our initialization sequence into a logical order new Medium
#32455 Run "make doxygen" in a travis builder new doxygen tor-ci Medium
#32474 Remove once I'm sure that it is subsumed by intro.html assigned nickm network-team-roadmap-november, s31-docs Medium
#32487 Phase 1: Stop compiling "acting as a directory cache" in --disable-module-relay new tor-design, network-team-roadmap-november Medium
#32488 Phase 1: Stop compiling "Responding to CREATE and EXTEND cells" in --disable-module-relay new tor-design, network-team-roadmap-november Medium
#32489 Phase 1: Stop compiling "Responding to BEGIN cells" in --disable-module-relay new tor-design, network-team-roadmap-november Medium
#32490 Phase 1: Stop compiling "Listening for OR and Dir connections " in --disable-module-relay new tor-design, network-team-roadmap-november Medium
#32522 Create better tooling for canonical tor header includes needs_revision teor network-team-roadmap-november Medium
#32610 Add macro spacing checks to "make check-spaces" needs_revision teor network-team-roadmap-november Medium
#32663 Require coccinelle 1.0.4 in needs_review teor 043-must, tor-test Medium

Last modified 5 weeks ago Last modified on Nov 4, 2019, 11:37:58 PM