Opened 3 years ago

Last modified 11 days ago

#13887 new defect

Pick a reporting format for Chutney

Reported by: nickm Owned by:
Priority: Medium Milestone:
Component: Core Tor/Chutney Version:
Severity: Normal Keywords: SponsorS design bikeshed
Cc: dgoulet, atagar Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

We need some way for Chutney to tell another process about the events it's seeing.

Options include CTF, json, protocolbufs, line-based text.

See https://lists.torproject.org/pipermail/tor-dev/2014-December/007898.html for a synopsis of goals.

I'm especially interested in dgoulet's opinion on CTF in particular.

I'm especially interested in atagar's opinion about what would be friendliest to a stem-based chutney.

Child Tickets

Change History (6)

comment:1 Changed 3 years ago by nickm

WRT ctf and other tracing/reporting formats:

13:56 < nickm> What's the biggest/most interesting other project using it?  Can 
               you generate it and parse it from most interesting languages?  
               What's it based on?  And would you recommend it for chutney 
               reporting to test scripts?

Also, please remember the bikeshed.

comment:2 Changed 3 years ago by atagar

I'm especially interested in atagar's opinion about what would be friendliest to a stem-based chutney.

Depends. Would you like to persist events to disk, then be able to parse them back into Stem Event objects? If so then the text-based format would work well, but it would be nice to find a delimitator that doesn't appear in events (they already end with '\r\n'). Maybe something like '\n-------\n'

To persist...

DIVIDER = '\n%s\n' % '-' * 80

events = my_events()  # list of events you've received from tor

with open('/place/to/persist', 'w') as dump_file:
  dump_file.write('DIVIDER'.join([msg.raw_content() for msg in events]))

... then to get them back...

events = []

with open('/place/to/persist') as dump_file:
  for entry in dump_file.read().split(DIVIDER):
    msg = stem.response.ControlMessage.from_str(entry)
    stem.response.convert('EVENT', msg)
    events.append(msg)

This might need a little twiddling (just wrote it off the top of my head, didn't run it).

comment:3 in reply to:  description Changed 3 years ago by dgoulet

Replying to nickm:

I'm especially interested in dgoulet's opinion on CTF in particular.

So I'll detail a bit the CTF option here. First of all, here is a link to "what this is" :)

http://www.efficios.com/ctf

This is a "tracing" format but can actually be used for a multiple of things. The biggest project using it is LTTng (https://lttng.org), a kernel and user space tracer that produces traces in CTF format. I know that some hardware vendor are starting to adopt the format in order to have one single trace format across all their platforms.

The main advantage is that it's a very very compact format with an available reader and reader/writer library (https://www.efficios.com/babeltrace). Python bindings also exists for *reading* the trace. A python tool exists also that generates C99 code to write CTF given a data description (metadata). It's also a very stable ABI based on a strict specification which is detached from the API producing it thus if the API changes well the underlaying ABI should not.

Now does this makes sense for chutney question. I would say it depends on our use case here. If we expect a lot of data being produced by chutney at a fast paste where a JSON format would dump megabytes and megabytes of data, it could be very useful. However, you need an external tool to parse and read the data since it's not in a human readable form.

Text base logging it great until you get way too much and you start to make scripts that parse long strings which I think is never ideal because well when you add a line or modify it, you need to update your regex.

comment:4 Changed 18 months ago by nickm

Owner: nickm deleted
Status: newassigned

Remove myself as chutney ticket owner. Default owners are trouble.

comment:5 Changed 18 months ago by nickm

Status: assignednew

comment:6 Changed 11 days ago by teor

Severity: Normal

Set all open tickets without a severity to "Normal"

Note: See TracTickets for help on using tickets.