Opened 9 years ago

Closed 8 years ago

#5875 closed task (fixed)

Write progress report to date.

Reported by: phobos Owned by: phobos
Priority: Medium Milestone:
Component: Archived/Ooni Version:
Severity: Keywords: SponsorH201208
Cc: hellais Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description


Child Tickets

Change History (3)

comment:1 Changed 8 years ago by karsten

Keywords: SponsorH201208 added
Milestone: Sponsor H: August 2012

Switching from using milestones to keywords for sponsor deliverables. See #6365 for details.

comment:2 Changed 8 years ago by arma

Cc: hellais added

Arturo tells me some tor-reports mail he sent satisfies this ticket. Arturo, can you paste which one and then close?

comment:3 Changed 8 years ago by hellais

Resolution: fixed
Status: newclosed

## August

The month of August was dedicated to laying out the foundations and the overall architecture of the ooniprobe backend and client system.

  • Work on daphn3

https://github.com/hellais/ooni-probe/commit/78fe6dea0b46c4055fbc195cf9329001c062cb17

## September

This month was mainly aimed at designing an API that would be flexible enough for supporting the writing of tests easily from third party developers.

  • Implemented the ooniprobe client API based on twisted trial:

https://github.com/hellais/ooni-probe/commit/98ebfe5e98824b4343a7a03f950f70e8f1c30faf
https://github.com/hellais/ooni-probe/commit/2d31d5f14a0f3a7518d47fe748c1677cbd885828
https://github.com/hellais/ooni-probe/commit/712f665423f24701c93889d32d040eb533065dfe
https://github.com/hellais/ooni-probe/commit/cbfcbdd0344d8e9a80d565ba01b64c7a40c97352
https://github.com/hellais/ooni-probe/commit/7dce07bfc58566d2fdec4eec8a831d4ecd545d5f

  • Write documentation for the API:

https://github.com/hellais/ooni-probe/commit/839564d3620c24e0057c1dfd9d39ef15f8b202ac
https://github.com/hellais/ooni-probe/commit/ffc44ce1cbc595ce8cbcd6522e8962f37b7e6869

  • Implement Scapy test template. This allows developers to write tests using the Scapy packet injection framework:

https://github.com/hellais/ooni-probe/commit/36c8a8e149fa5d975fc0d39bd0e5bf72ab38086a

  • Find bugs in YAML, fix them:

https://github.com/hellais/ooni-probe/commit/532e2e9853b4bdd61c951dc3073d8dc68d35f754

  • Add timeout support to scapy test template:

https://github.com/hellais/ooni-probe/commit/b267f3bd882fb5da6aa20c9a106ff3a10630e4f9

  • Outline what is necessary for having database support in oonibackend:

https://github.com/hellais/ooni-probe/commit/b1ca4257c13dff18664c8b175ed6b57acc8874b3

  • Add basic SSL support in our test helper:

https://github.com/hellais/ooni-probe/commit/ae87c2b74844372586d9c863ab63bb5f5e6bec38

## October

  • Get the captive portal test to work:

https://github.com/hellais/ooni-probe/commit/5cdc37ff80f1c8343e26f94fd7b7a29b3081ed03

  • Rewrite DNS tamper test to use twisted functions instead of default python ones:

https://github.com/hellais/ooni-probe/commit/c904e684ac60f9c25c09d52251cb27966aaab065

  • Add support for getting the probes IP address via check.tpo:

https://github.com/hellais/ooni-probe/commit/67b024a0361a0585ebc3bf9ed615ef8bf7c793fd

  • Work on removing dead code and refactoring:

https://github.com/hellais/ooni-probe/commit/55ce0ae91ff67c8212f80db783a53842e7662198

  • Multiple iterations over getting the documentation ready and the tests working as they should

## November

November was dedicated to making sure that the design layer out and experimented with in the past months worked properly. Focus was placed on the ooniprobe command line tool. The ooniprobe backend was made to do the basic set of tasks it needed to support the running of tests.

Highlights:

  • Alpha support for logging to PCAP
  • Alpha support for starting and stopping of Tor
  • Alpha support for reporting to Tor Hidden Services
  • Revert to a state in which everything was not broken and attempt to port Bridget

https://github.com/hellais/ooni-probe/commit/de0c47f7152b3a0a9dd2baf27ce2b028819e3f63

  • Make the DNS Tamper test use the a lower, more flexible twisted API call:

https://github.com/hellais/ooni-probe/commit/769cd3cbbd8521af505f2b740af8468b1a347f2f

  • Work on the backend system database support:

https://github.com/hellais/ooni-probe/commit/bf37eed44600ea9d307f958e35304ff4a7116032

  • Drop support for the old API:

https://github.com/hellais/ooni-probe/commit/654147c3a92156c35fc1a8ecc0aac8e933f57e59

  • Add GeoIP Lookup support for determining the probes ASN number and country:

https://github.com/hellais/ooni-probe/commit/e056fe9b886c3dfaf9546cd94656eec8117f70f8
https://github.com/hellais/ooni-probe/commit/e10033c72b4cf3355095b3a0bbdfca3a5dbeac4a

  • Improve the scapy test template:

https://github.com/hellais/ooni-probe/commit/f4a065da96af966e01126be4c3013780da1b7579

  • Iterations over existing tests:

https://github.com/hellais/ooni-probe/commit/9bf7fc8d86901bca4061dd82f6ca6af311cabee4

  • Add basic basic capture support by spawning a thread:

https://github.com/hellais/ooni-probe/commit/9e632c5d7c8d7f73add3aa38900898a86f6ca9fc
https://github.com/hellais/ooni-probe/commit/6a3ee55b574adaa8740ccafe2e4a01719dc0e86e

  • Improve txscapy so that it uses a valid file descriptor instead of spawning a thread:

https://github.com/hellais/ooni-probe/commit/12726ca463d9e68e93d49fcb418421648d054744

  • Config file support for oonib:

https://github.com/hellais/ooni-probe/commit/d4adf3e0e177e80ce7bc825f1cb4e461e5551b2f

  • Engage the community in contributing code:

https://github.com/hellais/ooni-probe/commit/28241124e6094b224f4a3b4f6c0a8e8a69a7eeb6
https://github.com/hellais/ooni-probe/commit/c67d69ec432e32479daca2ed65b27529260c9ba5

https://github.com/hellais/ooni-probe/commit/e115d4d0e6c063422fc35cbe1a63dcf0f5917a77

  • First iteration over Tor Hidden Service support for oonib and ooniprobe

https://github.com/hellais/ooni-probe/commit/d0d9662b4cafe11979e8173e6cc03c548f99ecf0

  • First iteration over TCP Test template. This allows to specify a TCP payload from a flat file and it will get sent to the destination address and the response will be recorded.

https://github.com/hellais/ooni-probe/commit/f873cd780a37451810b04526065fbe90d6250ba7

  • Work with somebody in Turkey to get them to run some tests.
  • Implement DNS Spoofing test:

https://github.com/hellais/ooni-probe/commit/4208c6e963ae59380999fae45930656c287a627d

  • First iteration over resume support and test state tracking:

https://github.com/hellais/ooni-probe/commit/659dda6b388ecc658f8060b7bef2aa7b4015dce2

  • Add support for starting Tor via txtorcon:

https://github.com/hellais/ooni-probe/commit/a65b6ebb7e90a4350b1a214189196d11a8371eda

https://github.com/hellais/ooni-probe/commit/615ce75c47aec249b6b5a4c0b58fdf7a93f09582

  • Get aagbsn up to speed on ooniprobe.
  • Implement HTTP Request Line "fuzzer":

https://github.com/hellais/ooni-probe/commit/1de07f659f1393d969a1b3766baffeecb111355d

and document it:

https://github.com/hellais/ooni-probe/commit/d67eecfd05f03002e3a9334db82ab2d8c2658368

  • Add support for ooniprobe test decks:

https://github.com/hellais/ooni-probe/commit/263568f

  • Work on documentation for various tests.
  • Lot's of refactoring:

https://github.com/hellais/ooni-probe/commit/10c63e0fceb0479c5893bfd2cb7e77af2af703cc
https://github.com/hellais/ooni-probe/commit/7d6901f1552067bce9595db6a84f8f5245d8f28c

## December

December has started as a month of craziness and me and aagbsn have been working very hard to make sure that we reach the 0.1 in time for the expected date.

In particular we have started data collection from some machines we have rented.

  • Get aagbsn up to speed with ooniprobe and guide him through the things that need to be done
  • Buy some machines for running of tests
  • Coordinate the migration of ooni.nu to ooni.tpo

https://trac.torproject.org/projects/tor/ticket/7644

  • Talk with the crate.io developer to get a Tor hidden service running to deliver us safe code via pip.
  • Iterations over the bugs that aaron encountered while running the code:

https://github.com/hellais/ooni-probe/commit/18c2dbd0b0491b25e997214fef672a8ebe1b2312

  • Work with twisted developers of a better SOCKS5/4/4a implementation:

https://github.com/hellais/txsocksx

https://github.com/hellais/txsocksx/commit/72475c35db3441c65de243019615e85ac40a476a
https://github.com/hellais/txsocksx/commit/fbd0a68be25acc0d6b891f558737320fe22ff375
https://github.com/hellais/txsocksx/commit/9fb250f712a013fbafb843285795524b868c1688
https://github.com/hellais/txsocksx/commit/f542f4d9b28f6690c0359a2d9a6b10d179d1872e
https://github.com/hellais/txsocksx/commit/dcea1b008b57523e02d0eeecc4c750e671847be2

Note: this is not yet bundled with ooniprobe and at this point will probably only be done post 0.1

  • Normalize the report format so that it follows ISO8601:

https://github.com/hellais/ooni-probe/commit/fb477a0982247974bc5f5979239ed015a17ce3ec
https://github.com/hellais/ooni-probe/commit/4648dd50aa44066eec9de1526f08a2e517ea14fb

  • Setting up of the machines for the collector and test helper
  • Documentation and quality assurance for the 0.1 release:

https://github.com/hellais/ooni-probe/pull/9
https://github.com/hellais/ooni-probe/pull/8
https://github.com/hellais/ooni-probe/pull/10
https://github.com/hellais/ooni-probe/pull/11
https://github.com/hellais/ooni-probe/pull/13
https://github.com/hellais/ooni-probe/pull/14
https://github.com/hellais/ooni-probe/pull/15

Note: See TracTickets for help on using tickets.