Opened 5 years ago

Closed 5 years ago

#9937 closed enhancement (fixed)

Create fake non-sanitised bridge descriptors for BridgeDB testing purposes

Reported by: isis Owned by: isis
Priority: High Milestone:
Component: Obfuscation/BridgeDB Version:
Severity: Keywords: bridgedb bridge-descriptors bridgedb-unittests stem
Cc: sysrqb, isis Actual Points:
Parent ID: #9865 Points:
Reviewer: Sponsor:

Description

Sysrqb has written a script to generate fake descriptors. It should be merged into BridgeDB.

The fake descriptors should be used for testing BridgeDB, for the internal database functionality, as well as for the parsers for adding bridges to the database and for distributing to users, and the web and email interfaces, so that anyone can run it or run the tests without requiring actual bridge descriptors.

Eventually, BridgeDB should have bridge descriptor classes (#9380 and #9462) and unittests to ensure that, given some valid descriptors and false descriptors, only valid descriptors match the bridge descriptor classes and are accepted. This fake bridge descriptor generator should, whatever the stage of BridgeDB's (or Stem's) descriptor parsers, create bridge descriptors which function identically to how BridgeDB treats actual descriptors.

Child Tickets

Change History (6)

comment:1 Changed 5 years ago by isis

Status: newneeds_review

I merged the descriptor generation script and it's history from sysrqb's above mentioned repo into my feature/9937-desc-gen branch in my BridgeDB repo.

I'm planning to merge it for the next release; speak now or forever hold your pisces.

Last edited 5 years ago by isis (previous) (diff)

comment:2 Changed 5 years ago by atagar

Likely more heavy weight than you'd like, but stem also has a test module for generating several varieties of fake descriptors...

https://gitweb.torproject.org/stem.git/blob/HEAD:/test/mocking.py

For instance...

>>> import test.mocking
>>> print test.mocking.get_bridge_server_descriptor(attr = {'router-digest': '204FD46BA75E7735A6A3B8B75FE2E2435A13BDB4'}, content = True)
router Unnamed 10.45.227.253 9001 0 0
router-digest 204FD46BA75E7735A6A3B8B75FE2E2435A13BDB4
published 2012-03-22 17:34:38
bandwidth 409600 819200 5120
reject *:*

comment:3 in reply to:  2 Changed 5 years ago by isis

Status: needs_reviewneeds_information

Replying to atagar:

Likely more heavy weight than you'd like, but stem also has a test module for generating several varieties of fake descriptors...

https://gitweb.torproject.org/stem.git/blob/HEAD:/test/mocking.py

For instance...

>>> import test.mocking
>>> print test.mocking.get_bridge_server_descriptor(attr = {'router-digest': '204FD46BA75E7735A6A3B8B75FE2E2435A13BDB4'}, content = True)
router Unnamed 10.45.227.253 9001 0 0
router-digest 204FD46BA75E7735A6A3B8B75FE2E2435A13BDB4
published 2012-03-22 17:34:38
bandwidth 409600 819200 5120
reject *:*

Cool, will it make the router identity and onion keys, and do signing? Could it make fake pluggable transport bridges (we'd need the extra-info descriptors too)?

comment:4 Changed 5 years ago by atagar

Cool, will it make the router identity and onion keys, and do signing?

Nope. All it does it make a minimally valid descriptor with the given attributes. A fancier bit you might find interesting is sign_descriptor_content (example use), though that's not keygen.

comment:5 Changed 5 years ago by isis

Status: needs_informationneeds_review

I created a bridge descriptor generator here. It creates onion, signing, and identity keys with pyOpenSSL. In another branch I started adding the ability to have descriptors with NTOR keys in them using pynacl (mostly for fun because I wanted to see how pynacl uses CFFI), but then it didn't work after half an hour of playing with it, so I gave up for now on the NTOR keys. Everything else seems to be okay.

Here's a bridge descriptor set generated by the new script:

@purpose bridge
router Unnamed 25.109.253.2 39131 0 0
or-address [4cb8:ab90:4b4:f18d:8f93:6d0e:7e6c:38eb]:39131
platform Tor 0.2.4.8-alpha on Linux
protocols Link 1 2 Circuit 1
published 2013-11-15 19:35:47
fingerprint D947 7CC0 5C61 03EC 4A5F 91FC 97A3 2269 2851 3421
uptime 33839086
bandwidth 314161461 355139043 273183879
extra-info-digest A15098BEE5EC927A481EA12C6F8693EE804E9153
onion-key
-----BEGIN RSA PUBLIC KEY-----
MIGkAgEAMA0GCSqGSIb3DQEBAQUABIGPMIGMAgEAAoGBAORLeW80oFtSbf1yKydi
Hsq5765+p+AkPhttO5HWqymkg+GVBBOfmdFPz7LDBg25SWuiKiqTIlCvy9c2PQ7V
KQ0rtm7FqMpzrJraJ4CwiG2ogl3jVD0Wh7j4KCjlshBGz9U7reyBEmLI/QjVByMT
vgpLjmqLy0DiuP/g5JCqh0UhAgMBAAE=
-----END RSA PUBLIC KEY-----
signing-key
-----BEGIN RSA PUBLIC KEY-----
MIGkAgEAMA0GCSqGSIb3DQEBAQUABIGPMIGMAgEAAoGBAMXTrBaZACyzo/53TM4e
oJ0q+OT/16ZajxIDs4QwaVMgfKKb+H21VY9YJT3ickiswBW5Sj5+Haq/7TB4zAVo
Bn0tCTvxG3iPyG1rwf3v1YV5B4ycyooiynGy3AUlXXP26fF9NKt6a4i5PpKz/1nw
Q6VUFTA+5xZLcqjFNhvdU8MVAgMBAAE=
-----END RSA PUBLIC KEY-----
hidden-service-dir
contact Somebody <somebody@example.com>
reject *:*
router-signature
-----BEGIN SIGNATURE-----
u9HNmBwqxK7dp/8psmbH1l1VCCFI/r+0sn3lPvRzw9vrRzVa9Yy44weRnIaUigfa
rnwwJrDxHqFaX4XzlB+WYavID+nB3BPXfP/ADfekHnaP2CxEGDDGjPhoGisCoVXp
Kz4G2VgWAWKRfzVspdWYV8iKfgyzyO+H2eBAPhcOeTo=
-----END SIGNATURE-----
extra-info Unnamed D9477CC05C6103EC4A5F91FC97A3226928513421
published 2013-11-15 19:35:47
write-history 2013-11-15 19:35:47 (900 s)
3188736,2226176,2866176
read-history 2013-11-15 19:35:47 (900 s)
3891200,2483200,2698240
dirreq-write-history 2013-11-15 19:35:47 (900 s)
1024,0,2048
dirreq-read-history 2013-11-15 19:35:47 (900 s)
0,0,0
geoip-db-digest 5F13E6EA2A9726E63A5A150E71BC5DC60D57539B
geoip6-db-digest 097B0CC6026B157D0811099DA77D9ADECC3563FC
dirreq-stats-end 2013-11-15 19:35:47 (86400 s)
dirreq-v3-ips
dirreq-v3-reqs
dirreq-v3-resp
ok=16,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=0,busy=0
dirreq-v3-direct-dl complete=0,timeout=0,running=0
dirreq-v3-tunneled-dl complete=12,timeout=0,running=0
transport obfs3 25.109.253.2:39132
transport obfs2 25.109.253.2:39133
bridge-stats-end 2013-11-15 19:35:47 (86400 s)
bridge-ips ca=8
bridge-ip-versions v4=8,v6=0
bridge-ip-transports <OR>=8
router-signature
-----BEGIN SIGNATURE-----
id4v5f2zr4VUN/P6rS7pGJIiJtkZ0ZfhvpRPwBWeKh5mLNSFSJVzpe9IQoLLkwy+
d72xS9I2rSSDJrjBT5PDEPPoeiL3v5DCvnc7gTujfQBIDdf1/CtDA7h06K9mEobY
nKWeJ8t2qu3tzXWmTJ6WJe/ZlOHM4BPeNPMZtzD33jE=
-----END SIGNATURE-----
r Unnamed 2Ud8wFxhA+xKX5H8l6MiaShRNCE IMCuSBxuu8aEGKW4lQtWl3SOxU8= 2013-11-15 19:35:47 25.109.253.2 39131 0
a [4cb8:ab90:4b4:f18d:8f93:6d0e:7e6c:38eb]:39131
s Fast Guard Running Stable Valid
w Bandwidth=266781
p reject 1-65535

The code to create the descriptors was made with logic/parameters such that it shouldn't be too difficult to get it to create normal OR descriptors, if some other project wants descriptors for testing purposes.

There are also a few things that are not yet exactly according to tor-spec and dir-spec. One of the things I can think of, off the top of my head, is that the generated networkstatus file (the file that contains the third type from the example descriptors) doesn't sort the entries alphanumerically by base64-encoded identity digest. For BridgeDB purposes, this doesn't matter as far as testing parsers go, though it would matter if we were to start benchmarking parsers.

comment:6 Changed 5 years ago by isis

Resolution: fixed
Status: needs_reviewclosed

My branch feature/9937-desc-gen-r1 was merged into develop at commit 64ff94fe636. Then two hotfix branches were merged into develop for a couple minor errors in the generated descriptors: hotfix/9937-desc-gen_fix-tabs and hotfix/9937-desc-gen_base64-padding-descDigest.

There may still be more minor issues with the descriptor generator. If anyone finds one, I expect that should be a new ticket.

So that Stem (and other Tor Python projects) can use this code too, it would be best to separate out this code into a separate module. I was kind of thinking of naming it leekspin. :) But that can also be another ticket.

Note: See TracTickets for help on using tickets.