Changes between Version 12 and Version 13 of doc/OONI/Architecture


Ignore:
Timestamp:
Jan 17, 2013, 9:35:44 PM (6 years ago)
Author:
hellais
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • doc/OONI/Architecture

    v12 v13  
    11= Architecture =
    22
    3 [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/CnCView.png, width=550px, align=right)]]
     3** THIS HAS BEEN MOVED **
    44
    5 The goal of this document is to provide the novice with an overall view of all the pieces inside of OONI and how they relate to one another. This will help them understand and approach OONI development easier.
    6 
    7 The other goal is to give an outline of the implementation units of OONI and provide a sketch useful to developers working on implementing it.
    8 
    9 
    10 OONI is split into two main Modules: '''OONIB''' and '''OONIProbe'''.
    11 
    12 '''OONIB''' is the backend component of OONI. It is responsible for exposing test
    13 helpers and for collecting reports from probes. Anyone may run an instance of
    14 either, and we believe this will ensure that the collection of probe data is
    15 resistant to simplistic denial of service. We plan to provide a public set of
    16 collection services over HTTPS and as Tor Hidden Services.
    17 
    18 '''TestHelpers''' are network services that are exposed to OONIProbe clients. These
    19 could be, for example, an HTTP server, a DNS server, or a Traceroute
    20 server. When using the TestHelpers, the OONIB will also keep track of the
    21 testing session, logging all traffic related to the client’s testing session
    22 and including it in the report. Once the client sends their side of the
    23 report, OONIB will match up the client submitted data with the data it
    24 collected locally.
    25 
    26 '''OONI-probe''' is the measurement tool that will be run on edge networks; here
    27 resides the core of the test logic.
    28 
    29 The '''Preprocessor''' takes files and inputs and makes them into Assets. Assets are
    30 the inputs to the Test. The Test module takes care of writing packets to the
    31 wire based on the given inputs. It is designed to be extensible, and it
    32 exposes an interface allowing third party developers to write tests for it.
    33 
    34 The '''PacketCapture''' module is responsible for collecting the packet dumps of a
    35 test. This can be achieved either by having the test writer specify a rule for
    36 collecting packets relevant to the test (BPF filters) or by creating a tun/tap
    37 device through which all the test related data is routed through.
    38 
    39 The '''ControlChannel''' is used when running tests with the assistance of OONIB
    40 TestHelpers. As such, the ControlChannel is useful for out-of-band
    41 communication with the backend. During a testing session, the client may be
    42 interested in notifying the OONIB of the data they have sent. OONIB will
    43 verify that the data sent matches those received and report the result back to
    44 the client. The WorkProcessor is responsible for accepting inputs from a remote
    45 OONIProbe client and for running tests obtained through it. If enabled, it
    46 exposes an HTTP API to receive remote test inputs.
    47 
    48 
    49 The '''UtilityLib''' provides utility functions which are useful for test writing,
    50 for example, networking libraries. Reporting may occur locally or remotely. In
    51 the case of remote reporting, the OONIProbe client will obtain from an OONIB a
    52 Test submission token. This token will allow them to communicate to the
    53 backend the progress of the Test. The reporting system is designed to accept
    54 data from third party testing systems. OONI does not deal with making
    55 measurements using web technologies (browser based solutions), however, it is
    56 possible to develop such systems and use OONIB’s reporting system for
    57 submitting the data.
    58 
    59 The '''TestState''' is used internally to keep track of test progress and
    60 state. This is useful for handling errors during testing and for providing
    61 resume support.
    62 
    63 == Nodes ==
    64 
    65 The purpose of the Node module is to allow tests to be run on remote
    66 machines. A node can also be another OONIProbe that accepts incoming test
    67 requests. If that is the case, then the only information which needs to be
    68 sent over the network are the inputs to the test – all test logic will be
    69 present serverside.
    70 
    71 Nodes which are not aware of OONI are called Network Nodes. Such Network Nodes
    72 are any kind of device that allows traffic to exit from them, and should be
    73 used as last-resort, impromptu systems for testing censorship in certain
    74 countries.
    75 
    76 OONI plans to develop an extension to the SOCKS protocol that will allow for
    77 the creation of RAW sockets. An OONIProxy will then be Network Nodes which are
    78 not necessarily OONI-aware, i.e. they do not run any OONI software components,
    79 but are able to understand this extension to the SOCKS protocol.
    80 
    81 [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/ModuleView1.png, width=900px, align=middle)]]
    82 
    83 == Detection Procedure ==
    84 
    85   1. The inputs to the test are obtained either from a remote network call or by
    86      passing the user-provided inputs to the Input Processor. This generates an
    87      OONI Asset. These items can be URLs, keywords, IP addresses and will be given
    88      as input to the test.
    89 
    90   2. The inputs are passed to the test logic.
    91 
    92   3. If the user has specified that the test should run on a remote Node, the
    93      specified portion of the inputs is serialized and sent to the remote OONI
    94      Node. If the specified Node is remote, a check, based on the test procedure, is
    95      made to ensure that the Node has the requisite capabilities for running the
    96      test, for example, a normal SOCKS server would not allow for raw socket
    97      access.
    98 
    99   4. If the test requires communication with an OONIB, a new test session token
    100      is created.
    101 
    102   5. The test is run on the network.
    103 
    104   6. Every N results, a report is sent back to the OONIB.
    105 
    106   7. Once the test has finished, the final report is sent to the reporting
    107      backend, and a report receipt is delivered to the client.
    108 
    109 
    110 == Reports ==
    111 
    112 [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/OONIProbeReportDataModel.png, align=right)]]
    113 
    114 There are two kinds of reports, those which are collected with data collected clientside, and those which include data collected from
    115 the backend system.
    116 
    117 For locally collected test the Data Model is as follows:
    118 
    119 Report:start_time, is the time at which the testing has started.
    120 
    121 Report:end_time, is the time at which the testing has ended.
    122 
    123 Report:Tests, are the reports for the individual tests, there is 1 test for every input line that is given to OONI (for example if scanning a list of 100 hosts, there will
    124 be 100 individual tests)
    125 
    126 Report:Tests:Test:start_time, is the start time for the test
    127 
    128 Report:Tests:Test:end_time, is the end time for the test
    129 
    130 Report:Tests:Test:input, is the input line given to the test
    131 
    132 Report:Tests:Test:request, (optional) the request that has been made for the test
    133 
    134 Report:Tests:Test:response, (optional) the response received for the request
    135 
    136 Report:packet_capture, the packet dump of the test
    137 
    138 [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/OONIBackendReportDataModel.png, align=right)]]
    139 
    140 Report:capture_type, if the capture has been filtered (it contains only packets relevant to the test) and through what means. If the packets have been filtered
    141 through bpf filters it should include the filter rule.
    142 
    143 Report:session_id, (optional) when reporting to an OONIB this is the session ID that is returned from it's reporting API.
    144 
    145 Report:asn, the ASN of the network running the test
    146 
    147 Report:ip, (optional) the IP address of the network running the test
    148 
    149 Report:country, the country running the test
    150 
    151 Report:related_report, the ID of a report that is related to this one. (for example I may run a Traceroute test and want to attach that test to this one)
    152 
    153 The backend report is bascially just an aggregate of the client report plus the information that is collected on the backend:
    154 
    155 The only added field compared to the client report is the packet_capture on the backend side.
     5see: https://ooni.torproject.org/docs/architecture.html