Opened 3 months ago

Closed 2 months ago

#30879 closed defect (fixed)

Log file reprocessing shouldn't abort at first encountered line parsing error

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Archived/Onionperf Version:
Severity: Normal Keywords:
Cc: metrics-team, acute Actual Points:
Parent ID: Points:
Reviewer: irl Sponsor:


I just reprocessed a bunch of log files, some of which apparently containing lines that cannot be parsed. Here's the output:

2019-06-13 15:06:47 1560431207.323223 [onionperf] [INFO] parsing log file at /home/karsten/op-logs/op-nl/onionperf-data/tgen-client/log_archive/onionperf_2019-01-03_23:59:59.tgen.log
2019-06-13 15:06:47 1560431207.668200 [onionperf] [INFO] parsing log file at /home/karsten/op-logs/op-nl/onionperf-data/tor-client/log_archive/onionperf_2018-05-18_23:59:59.torctl.log
2019-06-13 15:06:47 1560431207.929735 [onionperf] [INFO] parsing log file at /home/karsten/op-logs/op-nl/onionperf-data/tor-client/log_archive/onionperf_2018-10-25_23:59:59.torctl.log
2019-06-13 15:06:47 1560431207.971164 [onionperf] [WARNING] TGenParser: skipping line due to parsing error: 2018-04-14 21:10:04 1523740204.809894 [message] [shd-tgen-transfer.c:803] [_tgentransfer_log] [transfer-error] transport TCP,17,NULL:,NULL:,,state=SUCCESS,error=NONE transfer (null),26847,op-nl,NONE,0,(null),0,state=ERROR,error=AUTH total-bytes-read=1 total-bytes-write=0 payload-bytes-write=0/0 (-nan%) usecs-to-socket-create=0 usecs-to-socket-connect=8053676879205 usecs-to-proxy-init=-1 usecs-to-proxy-choice=-1 usecs-to-proxy-request=-1 usecs-to-proxy-response=-1 usecs-to-command=-1 usecs-to-response=-1 usecs-to-first-byte=-1 usecs-to-last-byte=-1 usecs-to-checksum=-1

2019-06-13 15:06:47 1560431207.971545 [onionperf] [INFO] Analysing pair for date 2018-01-27 00:00:00
2019-06-13 15:06:47 1560431207.971635 [onionperf] [INFO] parsing log file at /home/karsten/op-logs/op-nl/onionperf-data/tgen-client/log_archive/onionperf_2018-01-27_23:59:59.tgen.log

It seems like commenting out a raise statement fixes this:

diff --git a/onionperf/ b/onionperf/
index 0bc1811..08f4e81 100644
--- a/onionperf/
+++ b/onionperf/
@@ -532,7 +532,7 @@ class TGenParser(Parser):
                 logging.warning("TGenParser: skipping line due to parsing error: {0}".format(line))
-                raise
+                #raise

Is this the correct fix? If so, would it make sense to add a unit test to test this behavior?

Child Tickets

Change History (3)

comment:1 Changed 3 months ago by acute

It looks like the line would be ignored anyway if the parser was unable to cope with it. Looks to me like the raise was there for debugging purposes. I've submitted a pull request which fixes this and adds a couple of tests to validate the behaviour:

comment:2 Changed 3 months ago by irl

Reviewer: irl
Status: newneeds_review

comment:3 Changed 2 months ago by irl

Resolution: fixed
Status: needs_reviewclosed
Note: See TracTickets for help on using tickets.