Use stem to create test descriptors
Hi Isis. Stem now provides the ability to create and sign descriptors.
Unfortunately I didn't manage to get bridgedb's tests running but its use of leekspin looks pretty simple. Replacing it with Stem. There was a comment warning that descriptor creation takes a while but for what it's worth that doesn't seem to be the case. I'm running on an antiquated ten year old desktop and I can generate nineteen signed descriptors a second...
atagar@odin:~/Desktop/tor/bridgedb$ time ./scripts/create-descriptors 250
real 0m13.087s
user 0m12.753s
sys 0m0.320s
atagar@odin:~/Desktop/tor/bridgedb$ cat test_descriptors/descriptor_15
router Unnamed2006509655 194.39.149.152 9001 0 0
published 2005-06-15 02:26:44
bandwidth 153600 256000 104590
reject *:*
onion-key
-----BEGIN RSA PUBLIC KEY-----
NGExNDZmMDU2YWRkYWUwODhiM2ZkNjg3NmNhNDM0YmMzYjc5ZmE5MDViNTRkYjA1
OGMwOGJmNTNmM2NiNzg3NTRjNTdjMDA1MmZlY2QxNzc5YTAwNTQ1OWYyNmRiN2Jm
YWFkM2RlNDRhOTQ1ZmRiZjExYTdmOTE4NzkyMTIwNTQ2YzY1NTAzNmM1MDM=
-----END RSA PUBLIC KEY-----
signing-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALS1NGORgH3xEVOSd5Zs92z3bcrvjEK6WMYNkUiFUVvi7HhEFOT7vinP
vxUtPRakjDD7Kxg8LbL7PtyEtm29uSL9ggzZTZIHmMd4dIqAYSVfuN2wj/Stk5qp
2286BrmhuWyU/ZKLZXrz3g3JOkx9feJ9UxaxGeTGLK626WwMKuHjAgMBAAE=
-----END RSA PUBLIC KEY-----
router-signature
-----BEGIN SIGNATURE-----
pIEyuX8oUjMu2W1bqZ1KagpKDlRxMGBH8rupoWIuIjkvfBXC4goch9MHsa3S/Mf3
0Tb0PWS/KVFnQyTIPPIi5+iV5J/9ol87HPmrnCP4yP3NlE/nnlpbIOfuB+vdZv6n
BApDNLkWuuZCirNOqeps6BVBsH58v35nQoUY+HYOK60=
-----END SIGNATURE-----Attached is a patch file you should be apple to apply with 'git am stem_descriptor_creation.patch'. Note that this requires the version of stem in the git repo to work. I say 'stem 1.6' in the output despite that version not yet being tagged since that's the next release.
Activity
Oh, on a side note leekspin used pycrypto which is no longer maintained...
https://github.com/dlitz/pycrypto/issues/173
Stem uses cryptography instead, hence that new test dependency...
This is great, thanks atagar! I've applied your patch in my
feature/22755-stem-descgenbranch, and added a couple extra tiny patches to ensure that we use yourscripts/create-descriptorsscript to generate the mocked descriptors in tests/CI instead of leekspin.It's not ready to switch yet, though. One thing that leekspin does, which this doesn't do yet, is it creates the
networkstatus-bridges,bridge-descriptors,cached-extrainfo{.new}files all at once. That is, a mocked bridge whose networkstatus entry is included innetworkstatus-bridgeswill also have a bridge server descriptor in thebridge-descriptorsfile (and a bridge extrainfo in eithercached-extrainfoorcached-extrainfo.new. I'd need a way to do this, since one of the things that the mocked descriptors test for BridgeDB is that BridgeDB's logic on whether or not a bridge should be distributed is functioning in a sane manner.Trac:
Status: new to needs_revision
Keywords: N/A deleted, python, bridgedb-ci, bridgedb-parsers, stem addedTrac:
descriptors.tar.xzBah! Spotted your update right as I was about to send ya a patch. Gotta head to the bus but the attached will help move this along. Note that this needs the new commits I just pushed to stem.
This doesn't bundle all the descriptors into a single file as leekspin does. Also, it creates a full consensus rather than the truncated version in networkstatus-bridges. If you'd care for me to adjust the output to match the attached just let me know.
Hey Damian!
This is great! I changed it a bit to write the descriptors all into the same files, the same way as the files which come in to BridgeDB from the BridgeAuth.
Now that CI is slightly more happy, on to other review:
- Is there a way to make the descriptors less sparse? Currently, the server descriptors created by leekspin look like this:
@purpose bridge router Attenuated 102.137.125.126 46184 0 0 or-address [9246:a438:f18e:8d3b:7057:68e2:3d1b:7945]:46183 platform Tor 0.2.4.5-alpha on Linux protocols Link 1 2 Circuit 1 published 2017-07-06 07:10:48 fingerprint A5E0 18E5 60A0 BF4F 4C58 1774 B7B4 7F9D 2F09 CA49 uptime 35532496 bandwidth 737300203 833469795 641130611 extra-info-digest FDC1467C0EFEBCB9DF72A44222A8902EB65F019C onion-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBAJsZUONLz6jmxwqvBfIU1C/6uAOq9mrjdsjIKztfqg8nCj8SS1yIsJIO cweUOq1c0AjZQQrJPqM70V1IgfdqZ8oVhwFi81OJ3qJE5oJsWtVwHDL0mIfUhJlg IRIqKY4wkl8KmVRlCM90jYqIO04DDrO+B3+/94KmAropRJUS8XHNAgMBAAE= -----END RSA PUBLIC KEY----- signing-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBALhm+utng4Wvm29HgLlsnuczcLE56etkRmkWLWXM/3K7zKS48X6rcWB+ O3O5IeV9caCpvPT4Z1p11IqiFVAeeXqUWbjc338M4H9yzTBeV23aSCFzUiKQOgq5 95atzhSYgi1lyPz1xpIwE/nDFZ8l6WiahDJ3ipFaRhcLJzCJDtchAgMBAAE= -----END RSA PUBLIC KEY----- contact Somebody <somebody@example.com> reject *:* router-signature -----BEGIN SIGNATURE----- TJ0d8FkDeLB0cistB3lBlNKsVyLORZ+yNNyBti8jEnNBKxXXk6Nvma1zC8G/Ksym 34K1DhXmNpkb07QHYJbteiAhxOW1JIQztkdUGQ1YaUMJdUhu6RIbsAY2U79W4FtU 6Sc4SQw24TQ8udvpyUQut09LrEu86KaVjYqMHmrTHmo= -----END SIGNATURE-----Whereas the new Stem-produced ones are like this:
router Unnamed101110983533 158.165.189.89 9001 0 0 published 2011-08-17 12:24:06 bandwidth 153600 256000 104590 reject *:* onion-key -----BEGIN RSA PUBLIC KEY----- bymUV231gjwnd3ao0Qclm2JmvRH7J6pLv5xhWqD53KbRpkO60Fx/NSAvpOEf7+lG gzFLmVp7mmUZI376ahK0FhKnOVO3wU2kdBAhNgecolI0wnjhP5dMu63ZfBKMOojh V8WB8rCZXY++rSQ8c4WHtptWWlAcOj0FjTBtNYgL/MjGwK9jth2PpcEVkj8= -----END RSA PUBLIC KEY----- signing-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBAMwcInNw3myp4w9XJ3dFz43Dwr9bVqVT3L2rmgogU15sycj3Ng/NOvWz ryJWoW8OmeBSuIEeRpBLKPKfB+wejgyFtAPgf82GJnv2jGxh2ISZ1JTH39lMYjzq JySh2cUdDTbBSwJwRfTgfXe39ARWvySH3tubAh3nxmQ9GC8Rsnf7AgMBAAE= -----END RSA PUBLIC KEY----- fingerprint 0B87 6476 CECC 5DB8 3D39 9E3B F9B6 E689 DB2A CA1D router-signature -----BEGIN SIGNATURE----- WX6uMBUPvhuVkGLyJ9R0s6K1lCMXXom9rZd1LYDAMVUhwaytaaGX5HzWsq8bgUUC r7BvN2o+FmjRDdni9C1cHIU3GwNUgAhMAUa14VSdQiawEvegjRQCR8oNydODM1z4 Xf6jP3tTku0L47bhWFLipWVAOCH6z/lnmK1etAjSZ60= -----END SIGNATURE-----And a real bridge server descriptor for noether, one of the default Tor Browser bridges is like so:
@purpose bridge router noether 192.99.11.54 63848 0 0 identity-ed25519 -----BEGIN ED25519 CERT----- AQQABlybAYL6e+WjJYnWXtD2/bEhTQyikmzNUHWR4xxU5igkPzgdAQAgBADFQnrw B0ffcBEUpeWZM2Y8H0qhvgB+r7fh4FcaLQv0EczLikcu5s8hqoKBT8+8o+QT2RRM W7kH0XNRm9QL0vCO1wiXwEXm4nGE9gMCKu5//ttTolCPt2dcNIdyw3PNxQU= -----END ED25519 CERT----- master-key-ed25519 xUJ68AdH33ARFKXlmTNmPB9Kob4Afq+34eBXGi0L9BE platform Tor 0.2.8.7 on Linux protocols Link 1 2 Circuit 1 published 2017-07-06 15:52:40 fingerprint 7B12 6FAB 960E 5AC6 A629 C729 434F F84F B507 4EC2 uptime 26600081 bandwidth 1073741824 1073741824 9520756 extra-info-digest 9AFBEFB604AED8846C433DE19ED5BEAE630F0F40 CLpvIYPSqIh/eBYdTkuzTB4sxOkIiHP5CeJssYvILaw onion-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBAMWQ2jIGdJd6YvpIJSZYy/ELzuXX/FR3QoKpXvsxJNRxNkYmvOAAsm2E puRI2Xznm0q1YUiufDUHfG7J0x/N1AhZrfpYbcbQCtIhDCr2l7b6IpMENdts8bWv T5eXkUwXv/7Eb0Ur2HaLkGuACYN1Sd38/VQQLK0mXBRavkGgrd2HAgMBAAE= -----END RSA PUBLIC KEY----- signing-key -----BEGIN RSA PUBLIC KEY----- MIGJAoGBALbDk5BGi0N8V0eOoQTmOdw6CgkzeRjnMaEOAQpYRCbUnI0XVUjIMqA7 LHo7vbB91YCKZa1zsR2iKh5AnrrWe5wpLujMSRZA2yqwLW3V8/1fltF/1IndGlQD okZr2uRNnDUKRckZzF4Naoo4PAzH9PT4wDSPzkPj+qENUdTWLlsnAgMBAAE= -----END RSA PUBLIC KEY----- onion-key-crosscert -----BEGIN CROSSCERT----- Qvi+cyQnlsThRnW/kiuhNf2LHzTZoPM/aqQbE4gvncEWP8CQc0XmtPkoCHlea3nZ O3Dq6pGK8BDzK2DgYS1ZcpO9aPH9GfycZot1xEkg+z0NOYs3aoRZlM3skkLCWmgo Hym76SzFMefH0vLWHfdSIsoqS+Tx/k59s12IZa5jZx8= -----END CROSSCERT----- ntor-onion-key-crosscert 0 -----BEGIN ED25519 CERT----- AQoABluQAcVCevAHR99wERSl5ZkzZjwfSqG+AH6vt+HgVxotC/QRAKX3/P8JXZGs TeOtNdPAIP96eWyZUBUksMsP535aQiJolw9nFR5bFswdX08GbQ3xwZ4zNzosDi77 qtUaywjC9AA= -----END ED25519 CERT----- hidden-service-dir contact Henry de Valence <tor at hdevalence.ca> ntor-onion-key xxgycE7BKQguv+uVqoAwwkb4tv9BOh5p9vH9MBo8M2w= reject *:* tunnelled-dir-server router-sig-ed25519 ZU/p6qvbgdSdQfXC6/IBzk/gF7WYHXCzzOcQfkw3H3RvYRdICHnzNl0W0/Cty0Ks9hLYo3BWkCYuoMgvfsbeDQ router-signature -----BEGIN SIGNATURE----- grlgvXFB337Sxj5J07Q3cC9sI57JB07dIlFHCWTAS4N30F6GgB/7WDUtpxG9DFwJ ZomIdO5vA9AKfVarlnJFqF8Ks4IfJafqhi5mX+Qgr7ppfuwQc10UNIfJrADZXJP+ 00HSOsQP0T9ZpLQz3BquMIQjHiWkc9fDXu3fZ12EaFM= -----END SIGNATURE-----I'm a bit scared of exercising less of the parsing code if the descriptors are more sparse. 2. Same as above but for the extrainfo descriptors. (I'll attach all of noether's descriptors in a second so that there's some good examples.) 3. Sort of inline with the last two things, we need the
extra-info-digestlines in the server descriptors for BridgeDB to know which extra info is correct. 4. Some CI tests are broken because the extrainfo descriptors don't mocktransportlines. 5. Something is writing all the server and extrainfo descriptors to disk in separate files, e.g.server_descriptor_{0,1,2,…}andextrainfo_descriptor_{0,1,2,…}; is there a way to disable that?Great, thanks Isis!
Is there a way to make the descriptors less sparse?
Certainly, no problem. When not provided with any data stem creates a minimal valid descriptor. If you'd care to provide extra fields then simply specify them. This goes for extrainfo descriptors, extra-info-digest, and transport lines too.
desc = RelayDescriptor.create({ 'router': 'caerSidi 71.35.133.197 9001 0 0', 'published': '2012-02-29 04:03:19', 'fingerprint': '4F0C 867D F0EF 6816 0568 C826 838F 482C EA7C FE44', 'or-address': ['71.35.133.197:9001', '[12ab:2e19:3bcf::02:9970]:9001'], 'onion-key': '\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----' % stem.descriptor.CRYPTO_BLOB, 'signing-key': '\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----' % stem.descriptor.CRYPTO_BLOB, })Internally I use the following five helper methods to concoct random descriptor data. Would you find it helpful for them to be public?
https://gitweb.torproject.org/stem.git/tree/stem/descriptor/init.py#n1044
Something is writing all the server and extrainfo descriptors to disk in separate files...
Yup, just drop the following couple lines...
write_descriptor(server_desc, 'server_descriptor_%i' % i) write_descriptor(extrainfo_desc, 'extrainfo_descriptor_%i' % i)Took a peek over your revisions and looks good, though a couple more quick thoughts...
-OUTPUT_DIR = os.path.join(os.getcwd(), 'test_descriptors') +OUTPUT_DIR = os.path.join(os.getcwd())
With this change the os.path.join() no longer does anything. Actually, if I was in your shoes I'd simply drop this constant and replace the spots that use OUTPUT_DIR with os.getcwd().
- descriptor_file.flush()
You should be able to drop this line. As the last line in the 'with' block this is immediately followed by close() which will flush the content.
If you have any questions or need an example of anything don't heasitate to let me know!
Thanks phw! I'm not familiar enough with bridgedb to review this to the degree I like, but skimmed the stem usage and this looks spot on! Great work.
I authored descriptor generation specifically for BridgeDB. Is there anything I can adjust within Stem (either in the form of new capabilities or api adjustments) that would make this better for BridgeDB?
Replying to atagar:
Thanks phw! I'm not familiar enough with bridgedb to review this to the degree I like, but skimmed the stem usage and this looks spot on! Great work.
Thanks, atagar! No worries, I already asked sysrqb to take a look at this patch.
I authored descriptor generation specifically for BridgeDB. Is there anything I can adjust within Stem (either in the form of new capabilities or api adjustments) that would make this better for BridgeDB?
For what it's worth, this is really useful. The other day, I started working on updating BridgeDB's requirements.txt and retiring leekspin will help a lot with this.
As for functionality, I think we're good for now. I'll let you know if anything comes up in the future!
-
Replying to cohosh:
Do we want to add a make clean step that removes the descriptors from the current directory?
Hmm, I don't think we should because our Makefile's clean target handles build artifacts that are created during installation. The test descriptors are only for testing, and not needed during installation.
I just merged and deployed the branch.
Trac:
Status: merge_ready to closed
Resolution: N/A to implemented
Actualpoints: N/A to 1 mentioned in issue #23043 (moved)
mentioned in issue #23333 (moved)
