Opened 3 months ago

Last modified 2 months ago

#22755 needs_review enhancement

Use stem to create test descriptors

Reported by: atagar Owned by: isis
Priority: Low Milestone:
Component: Obfuscation/BridgeDB Version:
Severity: Minor Keywords: python, stem, bridgedb-parsers, bridgedb-ci
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Hi Isis. Stem now provides the ability to create and sign descriptors.

Unfortunately I didn't manage to get bridgedb's tests running but its use of leekspin looks pretty simple. Replacing it with Stem. There was a comment warning that descriptor creation takes a while but for what it's worth that doesn't seem to be the case. I'm running on an antiquated ten year old desktop and I can generate nineteen signed descriptors a second...

atagar@odin:~/Desktop/tor/bridgedb$ time ./scripts/create-descriptors 250 

real  0m13.087s
user  0m12.753s
sys 0m0.320s

atagar@odin:~/Desktop/tor/bridgedb$ cat test_descriptors/descriptor_15
router Unnamed2006509655 194.39.149.152 9001 0 0 
published 2005-06-15 02:26:44
bandwidth 153600 256000 104590
reject *:* 
onion-key
-----BEGIN RSA PUBLIC KEY-----
NGExNDZmMDU2YWRkYWUwODhiM2ZkNjg3NmNhNDM0YmMzYjc5ZmE5MDViNTRkYjA1
OGMwOGJmNTNmM2NiNzg3NTRjNTdjMDA1MmZlY2QxNzc5YTAwNTQ1OWYyNmRiN2Jm
YWFkM2RlNDRhOTQ1ZmRiZjExYTdmOTE4NzkyMTIwNTQ2YzY1NTAzNmM1MDM=
-----END RSA PUBLIC KEY-----
signing-key
-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBALS1NGORgH3xEVOSd5Zs92z3bcrvjEK6WMYNkUiFUVvi7HhEFOT7vinP
vxUtPRakjDD7Kxg8LbL7PtyEtm29uSL9ggzZTZIHmMd4dIqAYSVfuN2wj/Stk5qp
2286BrmhuWyU/ZKLZXrz3g3JOkx9feJ9UxaxGeTGLK626WwMKuHjAgMBAAE=
-----END RSA PUBLIC KEY-----
router-signature
-----BEGIN SIGNATURE-----
pIEyuX8oUjMu2W1bqZ1KagpKDlRxMGBH8rupoWIuIjkvfBXC4goch9MHsa3S/Mf3
0Tb0PWS/KVFnQyTIPPIi5+iV5J/9ol87HPmrnCP4yP3NlE/nnlpbIOfuB+vdZv6n
BApDNLkWuuZCirNOqeps6BVBsH58v35nQoUY+HYOK60=
-----END SIGNATURE-----

Attached is a patch file you should be apple to apply with 'git am stem_descriptor_creation.patch'. Note that this requires the version of stem in the git repo to work. I say 'stem 1.6' in the output despite that version not yet being tagged since that's the next release.

Child Tickets

Attachments (3)

stem_descriptor_creation.patch (8.5 KB) - added by atagar 3 months ago.
Patch file for using stem to create test descriptors.
descriptors.tar.xz (155.1 KB) - added by isis 3 months ago.
leekspin-generated mock descriptors
create_all_descriptors.patch (3.0 KB) - added by atagar 3 months ago.
Creates extrainfo descriptors and consensus too.

Download all attachments as: .zip

Change History (15)

Changed 3 months ago by atagar

Patch file for using stem to create test descriptors.

comment:1 Changed 3 months ago by atagar

Btw, this should be filed with BridgeDB rather than Leekspin but I can't seem to find a trac component for the former. If this patch looks good to you maybe we should create a BridgeDB trac component and perhaps deprecate Leekspin? Stem should now have all the capabilities it has.

comment:2 Changed 3 months ago by atagar

Oh, on a side note leekspin used pycrypto which is no longer maintained...

https://github.com/dlitz/pycrypto/issues/173

Stem uses cryptography instead, hence that new test dependency...

https://cryptography.io/en/latest/

comment:3 Changed 3 months ago by cypherpunks

Component: Core Tor/LeekspinObfuscation/BridgeDB

comment:4 Changed 3 months ago by atagar

Ahhh, great - thanks cypherpunks. :)

comment:5 in reply to:  4 Changed 3 months ago by cypherpunks

Replying to atagar:

Ahhh, great - thanks cypherpunks. :)

No problem. Quick tip: disabling JavaScript turns the component combo boxes into one and it becomes easier to browse through them.

comment:6 Changed 3 months ago by isis

Keywords: python stem bridgedb-parsers bridgedb-ci added
Status: newneeds_revision

This is great, thanks atagar! I've applied your patch in my feature/22755-stem-descgen branch, and added a couple extra tiny patches to ensure that we use your scripts/create-descriptors script to generate the mocked descriptors in tests/CI instead of leekspin.

It's not ready to switch yet, though. One thing that leekspin does, which this doesn't do yet, is it creates the networkstatus-bridges, bridge-descriptors, cached-extrainfo{.new} files all at once. That is, a mocked bridge whose networkstatus entry is included in networkstatus-bridges will also have a bridge server descriptor in the bridge-descriptors file (and a bridge extrainfo in either cached-extrainfo or cached-extrainfo.new. I'd need a way to do this, since one of the things that the mocked descriptors test for BridgeDB is that BridgeDB's logic on whether or not a bridge should be distributed is functioning in a sane manner.

comment:7 Changed 3 months ago by atagar

Thanks for taking a look Isis! No problem, we can easily make router status entries and extrainfo descriptors too. If you have a tarball with the set of files leekspin generates for you I can ensure create-descriptors matches. If not then no worries - I can make look at it tomorrow.

Changed 3 months ago by isis

Attachment: descriptors.tar.xz added

leekspin-generated mock descriptors

comment:8 Changed 3 months ago by atagar

Bah! Spotted your update right as I was about to send ya a patch. Gotta head to the bus but the attached will help move this along. Note that this needs the new commits I just pushed to stem.

This doesn't bundle all the descriptors into a single file as leekspin does. Also, it creates a full consensus rather than the truncated version in networkstatus-bridges. If you'd care for me to adjust the output to match the attached just let me know.

Changed 3 months ago by atagar

Creates extrainfo descriptors and consensus too.

comment:9 Changed 3 months ago by atagar

Oops! You can drop the base64 and binascii requires this adds. Forgot to drop those when I moved some stuff into stem.

comment:10 Changed 2 months ago by isis

Hey Damian!

This is great! I changed it a bit to write the descriptors all into the same files, the same way as the files which come in to BridgeDB from the BridgeAuth.

Now that CI is slightly more happy, on to other review:

  1. Is there a way to make the descriptors less sparse? Currently, the server descriptors created by leekspin look like this:
    @purpose bridge                                                                                                                                                                                                                   
    router Attenuated 102.137.125.126 46184 0 0                                                                                                                                                                                       
    or-address [9246:a438:f18e:8d3b:7057:68e2:3d1b:7945]:46183                                                                                                                                                                        
    platform Tor 0.2.4.5-alpha on Linux                                                                                                                                                                                               
    protocols Link 1 2 Circuit 1                                                                                                                                                                                                      
    published 2017-07-06 07:10:48                                                                                                                                                                                                     
    fingerprint A5E0 18E5 60A0 BF4F 4C58 1774 B7B4 7F9D 2F09 CA49                                                                                                                                                                     
    uptime 35532496                                                                                                                                                                                                                   
    bandwidth 737300203 833469795 641130611                                                                                                                                                                                           
    extra-info-digest FDC1467C0EFEBCB9DF72A44222A8902EB65F019C                                                                                                                                                                        
    onion-key                                                                                                                                                                                                                         
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    MIGJAoGBAJsZUONLz6jmxwqvBfIU1C/6uAOq9mrjdsjIKztfqg8nCj8SS1yIsJIO                                                                                                                                                                  
    cweUOq1c0AjZQQrJPqM70V1IgfdqZ8oVhwFi81OJ3qJE5oJsWtVwHDL0mIfUhJlg                                                                                                                                                                  
    IRIqKY4wkl8KmVRlCM90jYqIO04DDrO+B3+/94KmAropRJUS8XHNAgMBAAE=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    signing-key                                                                                                                                                                                                                       
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    MIGJAoGBALhm+utng4Wvm29HgLlsnuczcLE56etkRmkWLWXM/3K7zKS48X6rcWB+                                                                                                                                                                  
    O3O5IeV9caCpvPT4Z1p11IqiFVAeeXqUWbjc338M4H9yzTBeV23aSCFzUiKQOgq5                                                                                                                                                                  
    95atzhSYgi1lyPz1xpIwE/nDFZ8l6WiahDJ3ipFaRhcLJzCJDtchAgMBAAE=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    contact Somebody <somebody@example.com>                                                                                                                                                                                           
    reject *:*                                                                                                                                                                                                                        
    router-signature                                                                                                                                                                                                                  
    -----BEGIN SIGNATURE-----                                                                                                                                                                                                         
    TJ0d8FkDeLB0cistB3lBlNKsVyLORZ+yNNyBti8jEnNBKxXXk6Nvma1zC8G/Ksym                                                                                                                                                                  
    34K1DhXmNpkb07QHYJbteiAhxOW1JIQztkdUGQ1YaUMJdUhu6RIbsAY2U79W4FtU                                                                                                                                                                  
    6Sc4SQw24TQ8udvpyUQut09LrEu86KaVjYqMHmrTHmo=                                                                                                                                                                                      
    -----END SIGNATURE-----                                                                                                                                                                                                           
    
    Whereas the new Stem-produced ones are like this:
    router Unnamed101110983533 158.165.189.89 9001 0 0                                                                                                                                                                                
    published 2011-08-17 12:24:06                                                                                                                                                                                                     
    bandwidth 153600 256000 104590                                                                                                                                                                                                    
    reject *:*                                                                                                                                                                                                                        
    onion-key                                                                                                                                                                                                                         
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    bymUV231gjwnd3ao0Qclm2JmvRH7J6pLv5xhWqD53KbRpkO60Fx/NSAvpOEf7+lG                                                                                                                                                                  
    gzFLmVp7mmUZI376ahK0FhKnOVO3wU2kdBAhNgecolI0wnjhP5dMu63ZfBKMOojh                                                                                                                                                                  
    V8WB8rCZXY++rSQ8c4WHtptWWlAcOj0FjTBtNYgL/MjGwK9jth2PpcEVkj8=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    signing-key                                                                                                                                                                                                                       
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    MIGJAoGBAMwcInNw3myp4w9XJ3dFz43Dwr9bVqVT3L2rmgogU15sycj3Ng/NOvWz                                                                                                                                                                  
    ryJWoW8OmeBSuIEeRpBLKPKfB+wejgyFtAPgf82GJnv2jGxh2ISZ1JTH39lMYjzq                                                                                                                                                                  
    JySh2cUdDTbBSwJwRfTgfXe39ARWvySH3tubAh3nxmQ9GC8Rsnf7AgMBAAE=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    fingerprint 0B87 6476 CECC 5DB8 3D39 9E3B F9B6 E689 DB2A CA1D                                                                                                                                                                     
    router-signature                                                                                                                                                                                                                  
    -----BEGIN SIGNATURE-----                                                                                                                                                                                                         
    WX6uMBUPvhuVkGLyJ9R0s6K1lCMXXom9rZd1LYDAMVUhwaytaaGX5HzWsq8bgUUC                                                                                                                                                                  
    r7BvN2o+FmjRDdni9C1cHIU3GwNUgAhMAUa14VSdQiawEvegjRQCR8oNydODM1z4                                                                                                                                                                  
    Xf6jP3tTku0L47bhWFLipWVAOCH6z/lnmK1etAjSZ60=                                                                                                                                                                                      
    -----END SIGNATURE-----                                     
    
    And a real bridge server descriptor for noether, one of the default Tor Browser bridges is like so:
    @purpose bridge                                                                                                                                                                                                                   
    router noether 192.99.11.54 63848 0 0                                                                                                                                                                                             
    identity-ed25519                                                                                                                                                                                                                  
    -----BEGIN ED25519 CERT-----                                                                                                                                                                                                      
    AQQABlybAYL6e+WjJYnWXtD2/bEhTQyikmzNUHWR4xxU5igkPzgdAQAgBADFQnrw                                                                                                                                                                  
    B0ffcBEUpeWZM2Y8H0qhvgB+r7fh4FcaLQv0EczLikcu5s8hqoKBT8+8o+QT2RRM                                                                                                                                                                  
    W7kH0XNRm9QL0vCO1wiXwEXm4nGE9gMCKu5//ttTolCPt2dcNIdyw3PNxQU=                                                                                                                                                                      
    -----END ED25519 CERT-----                                                                                                                                                                                                        
    master-key-ed25519 xUJ68AdH33ARFKXlmTNmPB9Kob4Afq+34eBXGi0L9BE                                                                                                                                                                    
    platform Tor 0.2.8.7 on Linux                                                                                                                                                                                                     
    protocols Link 1 2 Circuit 1                                                                                                                                                                                                      
    published 2017-07-06 15:52:40                                                                                                                                                                                                     
    fingerprint 7B12 6FAB 960E 5AC6 A629 C729 434F F84F B507 4EC2                                                                                                                                                                     
    uptime 26600081                                                                                                                                                                                                                   
    bandwidth 1073741824 1073741824 9520756                                                                                                                                                                                           
    extra-info-digest 9AFBEFB604AED8846C433DE19ED5BEAE630F0F40 CLpvIYPSqIh/eBYdTkuzTB4sxOkIiHP5CeJssYvILaw                                                                                                                            
    onion-key                                                                                                                                                                                                                         
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    MIGJAoGBAMWQ2jIGdJd6YvpIJSZYy/ELzuXX/FR3QoKpXvsxJNRxNkYmvOAAsm2E                                                                                                                                                                  
    puRI2Xznm0q1YUiufDUHfG7J0x/N1AhZrfpYbcbQCtIhDCr2l7b6IpMENdts8bWv                                                                                                                                                                  
    T5eXkUwXv/7Eb0Ur2HaLkGuACYN1Sd38/VQQLK0mXBRavkGgrd2HAgMBAAE=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    signing-key                                                                                                                                                                                                                       
    -----BEGIN RSA PUBLIC KEY-----                                                                                                                                                                                                    
    MIGJAoGBALbDk5BGi0N8V0eOoQTmOdw6CgkzeRjnMaEOAQpYRCbUnI0XVUjIMqA7                                                                                                                                                                  
    LHo7vbB91YCKZa1zsR2iKh5AnrrWe5wpLujMSRZA2yqwLW3V8/1fltF/1IndGlQD                                                                                                                                                                  
    okZr2uRNnDUKRckZzF4Naoo4PAzH9PT4wDSPzkPj+qENUdTWLlsnAgMBAAE=                                                                                                                                                                      
    -----END RSA PUBLIC KEY-----                                                                                                                                                                                                      
    onion-key-crosscert                                                                                                                                                                                                               
    -----BEGIN CROSSCERT-----                                                                                                                                                                                                         
    Qvi+cyQnlsThRnW/kiuhNf2LHzTZoPM/aqQbE4gvncEWP8CQc0XmtPkoCHlea3nZ                                                                                                                                                                  
    O3Dq6pGK8BDzK2DgYS1ZcpO9aPH9GfycZot1xEkg+z0NOYs3aoRZlM3skkLCWmgo                                                                                                                                                                  
    Hym76SzFMefH0vLWHfdSIsoqS+Tx/k59s12IZa5jZx8=                                                                                                                                                                                      
    -----END CROSSCERT-----                                                                                                                                                                                                           
    ntor-onion-key-crosscert 0                                                                                                                                                                                                        
    -----BEGIN ED25519 CERT-----                                                                                                                                                                                                      
    AQoABluQAcVCevAHR99wERSl5ZkzZjwfSqG+AH6vt+HgVxotC/QRAKX3/P8JXZGs                                                                                                                                                                  
    TeOtNdPAIP96eWyZUBUksMsP535aQiJolw9nFR5bFswdX08GbQ3xwZ4zNzosDi77                                                                                                                                                                  
    qtUaywjC9AA=                                                                                                                                                                                                                      
    -----END ED25519 CERT-----                                                                                                                                                                                                        
    hidden-service-dir                                                                                                                                                                                                                
    contact Henry de Valence <tor at hdevalence.ca>                                                                                                                                                                                   
    ntor-onion-key xxgycE7BKQguv+uVqoAwwkb4tv9BOh5p9vH9MBo8M2w=                                                                                                                                                                       
    reject *:*                                                                                                                                                                                                                        
    tunnelled-dir-server                                                                                                                                                                                                              
    router-sig-ed25519 ZU/p6qvbgdSdQfXC6/IBzk/gF7WYHXCzzOcQfkw3H3RvYRdICHnzNl0W0/Cty0Ks9hLYo3BWkCYuoMgvfsbeDQ                                                                                                                         
    router-signature                                                                                                                                                                                                                  
    -----BEGIN SIGNATURE-----                                                                                                                                                                                                         
    grlgvXFB337Sxj5J07Q3cC9sI57JB07dIlFHCWTAS4N30F6GgB/7WDUtpxG9DFwJ                                                                                                                                                                  
    ZomIdO5vA9AKfVarlnJFqF8Ks4IfJafqhi5mX+Qgr7ppfuwQc10UNIfJrADZXJP+                                                                                                                                                                  
    00HSOsQP0T9ZpLQz3BquMIQjHiWkc9fDXu3fZ12EaFM=                                                                                                                                                                                      
    -----END SIGNATURE-----                                  
    
    I'm a bit scared of exercising less of the parsing code if the descriptors are more sparse.
  2. Same as above but for the extrainfo descriptors. (I'll attach all of noether's descriptors in a second so that there's some good examples.)
  3. Sort of inline with the last two things, we need the extra-info-digest lines in the server descriptors for BridgeDB to know which extra info is correct.
  4. Some CI tests are broken because the extrainfo descriptors don't mock transport lines.
  5. Something is writing all the server and extrainfo descriptors to disk in separate files, e.g. server_descriptor_{0,1,2,…} and extrainfo_descriptor_{0,1,2,…}; is there a way to disable that?

comment:11 Changed 2 months ago by atagar

Great, thanks Isis!

Is there a way to make the descriptors less sparse?

Certainly, no problem. When not provided with any data stem creates a minimal valid descriptor. If you'd care to provide extra fields then simply specify them. This goes for extrainfo descriptors, extra-info-digest, and transport lines too.

desc = RelayDescriptor.create({
  'router': 'caerSidi 71.35.133.197 9001 0 0',
  'published': '2012-02-29 04:03:19',
  'fingerprint': '4F0C 867D F0EF 6816 0568 C826 838F 482C EA7C FE44',
  'or-address': ['71.35.133.197:9001', '[12ab:2e19:3bcf::02:9970]:9001'],
  'onion-key': '\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----' % stem.descriptor.CRYPTO_BLOB,
  'signing-key': '\n-----BEGIN RSA PUBLIC KEY-----%s-----END RSA PUBLIC KEY-----' % stem.descriptor.CRYPTO_BLOB,
})

Internally I use the following five helper methods to concoct random descriptor data. Would you find it helpful for them to be public?

https://gitweb.torproject.org/stem.git/tree/stem/descriptor/__init__.py#n1044

Something is writing all the server and extrainfo descriptors to disk in separate files...

Yup, just drop the following couple lines...

write_descriptor(server_desc, 'server_descriptor_%i' % i)
write_descriptor(extrainfo_desc, 'extrainfo_descriptor_%i' % i)

Took a peek over your revisions and looks good, though a couple more quick thoughts...

-OUTPUT_DIR = os.path.join(os.getcwd(), 'test_descriptors')
+OUTPUT_DIR = os.path.join(os.getcwd())

With this change the os.path.join() no longer does anything. Actually, if I was in your shoes I'd simply drop this constant and replace the spots that use OUTPUT_DIR with os.getcwd().

+ descriptor_file.flush()

You should be able to drop this line. As the last line in the 'with' block this is immediately followed by close() which will flush the content.

If you have any questions or need an example of anything don't heasitate to let me know!

comment:12 Changed 2 months ago by atagar

Status: needs_revisionneeds_review
Note: See TracTickets for help on using tickets.