Opened 3 years ago

Last modified 17 months ago

#18797 assigned enhancement

Create a DescriptorGenerator for testing and maybe other purposes

Reported by: iwakeh Owned by: metrics-team
Priority: Low Milestone:
Component: Metrics/Library Version:
Severity: Normal Keywords:
Cc: atagar, isis Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

quote from #16873:

I pondered adding a DescriptorGenerator a while ago, where one would construct a new instance of a Descriptor, call its setters to give it some values, and then have the generator generate a text representation of that descriptor. This would be useful for testing and maybe other things. Now, would the Javadoc of these setters copy the Javadoc of the getters or simply link to them? Or would we use some smarter pattern for building these instances? (And is there a smarter pattern for parsed descriptors than a huge interface with dozens of getters?)

Child Tickets

Attachments (1)

BridgeNetworkStatus.g4 (9.6 KB) - added by karsten 22 months ago.
An ANTLR 4 grammar for Tor bridge network statuses

Download all attachments as: .zip

Change History (17)

comment:1 Changed 3 years ago by iwakeh

Milestone: metrics-lib 1.4.0

comment:2 Changed 3 years ago by iwakeh

Ideas for a descriptor generator and maybe the descriptor itself:

  • use reflection to set descriptor values
  • use a key-value approach for setting the descriptor values
  • a combination of the two above; like reading key-value strings and setting the appropriate fields by reflection.

needs more thinking ...

comment:3 Changed 3 years ago by karsten

Milestone: metrics-lib 1.4.0metrics-lib 1.5.0

Move to 1.5.0 in order to get 1.4.0 out sooner.

comment:4 Changed 2 years ago by iwakeh

Milestone: metrics-lib 1.5.0metrics-lib 2.0.0

comment:5 Changed 22 months ago by iwakeh

While replying to #21934, I had another idea for this ticket:

We could use a grammar and from that generate a descriptor parser and all related code.
I'm thinking of something like antlr.

Last edited 22 months ago by iwakeh (previous) (diff)

comment:6 Changed 22 months ago by karsten

comment:7 in reply to:  6 Changed 22 months ago by iwakeh

Replying to karsten:

Here's a related thread from 1.5 years ago: https://lists.torproject.org/pipermail/tor-dev/2015-November/009855.html

That's interesting! I didn't see a real showstopper (yet) in the mail thread.
Could you make that grammar available again?
All those things discussed in the thread should be possible to write into grammar, but that's only an opinion as I didn't try this, yet.

Changed 22 months ago by karsten

Attachment: BridgeNetworkStatus.g4 added

An ANTLR 4 grammar for Tor bridge network statuses

comment:8 Changed 22 months ago by karsten

Oh, I didn't realize that the file was not available anymore. I attached it to this ticket, so that it will be archived forever.

comment:9 Changed 22 months ago by karsten

Cc: atagar added

I'm copying atagar on this ticket who just recently added the ability for Stem to create descriptors.

atagar, can you give us some background on why you added this ability now, what you expect it will be used for, how much code this adds to the overall codebase, etc.? Like, can you recommend it? Or do you think it's sufficient to have this ability in one Tor descriptor-parsing library and not in all of them? Thanks!

comment:10 Changed 22 months ago by karsten

Milestone: metrics-lib 2.0.0

This might be too much for version 2.0.0 if we still want to put that out before end of June. Pessimistically removing the milestone for now.

comment:11 Changed 22 months ago by atagar

Cc: isis added

Hi Karsten - interesting! I didn't realize you were considering this too. Adding Isis too since she's interested in this topic as well.

Descriptor creation has been on my radar for quite a few years (#10227). Stem and BridgeDB need it for their unit tests. Stem filled this need with test helpers, and Isis did so with her own library called leekspin.

Descriptor generation fits nicely into Stem's API so I'm adding it now so we can deduplicate this effort. So TL;DR is...

  • At present the only use of descriptor generation I'm aware of is testing. That said, we might find other uses too.
  • There's two parts to descriptor generation: creating valid descriptors and signing it. Stem presently supports the former but not the later (however, leekspin can). I'm looking into that next.

Cheers! -Damian

PS. Once again kinda sad we keep duplicating each other's efforts. Seems like a waist to keep repeating each other's work... :/

comment:12 Changed 22 months ago by karsten

Thanks, atagar, for sharing your experience here! I have to admit that this is not top priority for us at the moment, so we might continue using real descriptors or partial descriptor for testing rather than generating our own. But let's keep this ticket open, and if we find out new use cases (or a bunch of free development time), we can start hacking on this!

comment:13 Changed 22 months ago by isis

Hey, thanks for getting this conversation started! Leekspin can do signing, but it's a bit out of date in terms of missing some newly-introduced fields in descriptors. Also, leekspin was primarily meant to generate bridge descriptors, and then later HS descriptors, to fuzz the parsers.

I'm happy to expand leekspin, or combine it into something else, as long as I can still fuzz test BridgeDB's parsers and logic with it.

comment:14 Changed 22 months ago by atagar

Thanks Isis! The trixy scheme I planned was to send you a BrideDB patch to use Stem's new descriptor creation once it had signing support. That way I can make sure the new functions meet your needs as well as mine, and adjust them if not.

comment:15 Changed 18 months ago by karsten

Owner: changed from karsten to metrics-team
Status: newassigned

Handing over to metrics-team, because I'm not currently working on this.

comment:16 Changed 17 months ago by karsten

Summary: create a DescriptorGenerator?Create a DescriptorGenerator for testing and maybe other purposes

Tweak the summary a bit.

Note: See TracTickets for help on using tickets.