Changes between Version 5 and Version 6 of org/sponsors/SponsorS/IntegrationTesting


Ignore:
Timestamp:
Jan 27, 2017, 2:18:06 PM (2 years ago)
Author:
isabela
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • org/sponsors/SponsorS/IntegrationTesting

    v5 v6  
    13136 month milestone: 2015 April 1
    1414
    15   Designs and specifications for controller improvements to allow better
    16   monitoring of performance, scalability, and overhead, and to allow better
    17   testing of Tor features. (3)
     15  Designs and specifications for controller improvements to allow better monitoring of performance, scalability, and overhead, and to allow better testing of Tor features. (3)
    1816
    19   Design and specifications for a unified pluggable integration test
    20   framework where developers can add their own network tests. (1,2)
     17  Design and specifications for a unified pluggable integration test framework where developers can add their own network tests. (1,2)
    2118
    22   Design and specifications for improved means to decouple Tor modules for
    23   better module-level testing. (2)
     19  Design and specifications for improved means to decouple Tor modules for better module-level testing. (2)
    2420
    25   Identification of highest-priority areas for integration test coverage, and
    26   specifications for test modules to validate those areas. (2)
     21  Identification of highest-priority areas for integration test coverage, and specifications for test modules to validate those areas. (2)
    2722
    28   Identification of least-testable areas in current code/design, and plans
    29   for replacing/mitigating/testing them. (2)
     23  Identification of least-testable areas in current code/design, and plans for replacing/mitigating/testing them. (2)
    3024
    31   Alpha version of testing framework, possibly missing major areas of
    32   functionality, is implemented, and in use as a regular part of tor
    33   development. (1,2)
     25  Alpha version of testing framework, possibly missing major areas of functionality, is implemented, and in use as a regular part of tor development. (1,2)
    3426
    352712 month milestone: 2015 Oct 1
    3628
    37   Improve chutney templating to better handle complex multi-version
    38   configurations. (1)
     29  Improve chutney templating to better handle complex multi-version configurations. (1)
    3930
    40   Alpha version of unified test framework running, including chutney for
    41   launching a tor network, stem for scripting Tor nodes, a set of
    42   test-clients, and a set of ill-behaved testing clients to check for
    43   correct handling of protocol violations. (2)
     31  Alpha version of unified test framework running, including chutney for launching a tor network, stem for scripting Tor nodes, a set of test-clients, and a set of ill-behaved testing clients to check for correct handling of protocol violations. (2)
    4432
    45   Improvements to chutney to launch more kinds of Tor networks and test more
    46   Tor features.  (1)
     33  Improvements to chutney to launch more kinds of Tor networks and test more Tor features.  (1)
    4734
    48   Alpha version of unified test framework running, including chutney for
    49   launching a tor network, stem for scripting Tor nodes, a set of
    50   test-clients, and a set of ill-behaved testing clients to check for
    51   correct handling of protocol violations. (2)
     35  Alpha version of unified test framework running, including chutney for launching a tor network, stem for scripting Tor nodes, a set of test-clients, and a set of ill-behaved testing clients to check for correct handling of protocol violations. (2)
    5236
    5337  Initial data-driven work to simplify Tor's module interdependencies (*)
    5438
    55   Improvements to chutney and test framework based on early reports from
    56   external developers. (1,2)
     39  Improvements to chutney and test framework based on early reports from external developers. (1,2)
    5740
    5841  Test framework is stable enough to recommend to external developers. (2)
     
    624518 month milestone: 2016 Apr 1
    6346
    64   Initial implementations of some high-value controller features, so we can
    65   start getting experience with how they work. (3) #17284
     47  Initial implementations of some high-value controller features, so we can start getting experience with how they work. (3) #17284
    6648
    67   Further data-driven work to simplify Tor's module interdependencies.
    68   Most functions are extracted from the "spaghetti" core of the program. (*) (DONE)
     49  Further data-driven work to simplify Tor's module interdependencies.  Most functions are extracted from the "spaghetti" core of the program. (*) (DONE)
    6950
    7051  Tor test coverage (integration plus unit tests) at 68% or higher. (*) (DONE)
     
    7253  Alpha version of module-isolation mechanism. (2) #17285
    7354
    74   Test infrastructure includes a stable API and documentation on how to write extensions. All its
    75   components have received stable production-quality releases (2). #17282, #17283
     55  Test infrastructure includes a stable API and documentation on how to write extensions. All its components have received stable production-quality releases (2). #17282 , #17283
    7656
    775724 month milestone: 2016 Oct 1
     
    7959  Tor test coverage (integration plus unit tests) at 75% or higher. (*) #17289
    8060
    81   Testing framework includes ill-behaved servers to verify correct behavior
    82   of network and clients in their presence.  (2) #17290
     61  Testing framework includes ill-behaved servers to verify correct behavior of network and clients in their presence.  (2) #17290
    8362
    84   All highest-priority areas identified by initial research have high (>>80%)
    85   test coverage.  Most low-testability areas have been replaced, mitigated, or
    86   tested. (2) #17288
     63  All highest-priority areas identified by initial research have high (>>80%) test coverage.  Most low-testability areas have been replaced, mitigated, or tested. (2) #17288
    8764
    88   Further controller improvements TBD based on experiences using initial
    89   round of new controller features. (3) #17287
     65  Further controller improvements TBD based on experiences using initial round of new controller features. (3) #17287
    9066
    9167  All desired controller features implemented. (3) #17287
     
    9470
    9571  Further improvements TBD based on first 18 months' experience. (*)
     72 
     73
     74----
     75Tracking our work
     76
     77==== Core Tor: ====
     78''"The first task is ['''1a'''] streamlining and automating the process of launching a complete testing Tor network, including Tor directory authorities, relays, bridges, clients, and hidden services, as well as client applications and destination services like websites. ['''1b'''] We need to continue to identify and resolve bugs in Tor's "testing network" configuration. ['''1c'''] We need to improve packaging and portability for these tools, so other researchers and developers can reproduce and extend our results."''
     79
     80 * We're splitting this into
     81   * 1b. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-tests-integration&group=milestone&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-tests-integration]
     82   * 1a + 1c. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-chutney-usability&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-chutney-usability]
     83
     84''"The second task is designing and scripting an automated test suite to exercise and stress as much of Tor's functionality as possible. These tests should come in the form of ['''2a'''] a) standalone unit tests, ['''2b'''] b) scripted clients that perform specific actions and expect certain results, and c) ['''2c'''] Tor controllers that exercise the control port interface as much as possible to observe and modify Tor's internal state. This test framework will allow us to investigate and reproduce bugs reported in the wild in a safe (with respect to user privacy) and controlled environment. We should also focus on getting our test framework to the point where other developers can devise and run their own tests."''
     85
     86 * 2. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-tests-coverage&group=milestone&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-tests-coverage]
     87 * 2a.[https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~+tor-tests-unit&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-tests-unit]
     88 * 2b. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-client-scripting&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-client-scripting]
     89 * 2c. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-tests-stem&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-tests-stem]
     90
     91''"The third task, in parallel with the first two, is to extend Tor's controller interface to allow better monitoring. So far the control protocol has focused on exposing client state to local controllers (like the graphical interface); we will change it to export more details for relays, directory authorities, and hidden services as well. Combined with build automation (a separate project), the extended interface will also provide a more powerful framework for continuous evaluation of performance, scalability, and bandwidth overhead."''
     92
     93 * 3a. [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-controller-extension&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-controller-extension]
     94 * 3b. The keyword [https://trac.torproject.org/projects/tor/query?status=accepted&status=assigned&status=merge_ready&status=needs_information&status=needs_review&status=needs_revision&status=new&status=reopened&keywords=~tor-modularity&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=milestone&col=points&col=reviewer&col=sponsor&order=priority tor-modularity] tracks some stuff that we ''had'' associated with this area, but none of it is a must-do deliverable.