aggregate.py should ignore empty scan-data files

added component::core tor/torflow easy intro priority::medium reviewer::dgoulet severity::normal status::needs-review type::defect labels

Trac:
Owner: N/A to aagbsn
Component: - Select a component to Torflow

can you figure out which file was being processed at the time (and perhaps attach it here?)

AFAICT from reading code all bws--done are being processed and there's nothing in the traceback indicating which it's stumbling over. I have 1874 matching files in my system atm.

Furthermore, this error has shown once and that was several hours ago. The problematic file is apparently not problematic any more.

Perhaps a bws-* file was still open and being written to while aggregate.py was trying to read it? We could add exclusive locking (see fcntl) to make sure that the bws-* files are written atomically.

For the record, I'm seeing this again.

ERROR[Wed Apr 02 08:40:04 2014]:Exception during aggregate: empty string for float()
Traceback (most recent call last):
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 876, in <module>
    main(sys.argv)
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 424, in main
    timestamp = float(fp.readline())
ValueError: empty string for float()

Again.

ERROR[Wed Apr 30 21:40:04 2014]:Exception during aggregate: empty string for float()
Traceback (most recent call last):
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 876, in <module>
    main(sys.argv)
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 424, in main
    timestamp = float(fp.readline())
ValueError: empty string for float()

I've started getting this now on longclaw. It appears that I'm not submitting bwauth data because of this. Its slightly different error, but looks like the exact same part of the code:

ERROR[Sat Jan 24 21:45:02 2015]:Exception during aggregate: could not convert string to float:
Traceback (most recent call last):
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 876, in <module>
    main(sys.argv)
  File "/home/bwscanner/torflow/NetworkScanners/BwAuthority/aggregate.py", line 424, in main
    timestamp = float(fp.readline())
ValueError: could not convert string to float:

I can't tell which scanner, or file it is operating, as there are four scanners and hundreds of files. Perhaps it would be useful to add some debug information when this happens to try and narrow it down?

I just added an exception to the part of the code to print out which file it is:

  for da in argv[1:-1]:
    # First, create a list of the most recent files in the                                                                                                                                                                                                                                                                     
    # scan dirs that are recent enough                                                                                                                                                                                                                                                                                         
    for root, dirs, f in os.walk(da):
      for ds in dirs:
        if re.match("^scanner.[\d+]$", ds):
          newest_timestamp = 0
          for sr, sd, files in os.walk(da+"/"+ds+"/scan-data"):
            for f in files:
              if re.search("^bws-[\S]+-done-", f):
                try:
                  fp = file(sr+"/"+f, "r")
                  slicenum = sr+"/"+fp.readline()
                  timestamp = float(fp.readline())
                  fp.close()
                except ValueError:
                  print("ValueError on file: "+f)

and then I ran it, and it said:

ValueError on file: bws-41.4:42.1-done-2015-01-21-17:03:13

I included also the information about which scanner directory was in play, by adding:

print("slicenum: "+slicenum)

that showed me:

ValueError on file: bws-41.4:42.1-done-2015-01-21-17:03:13
slicenum: /home/bwscanner/torflow/NetworkScanners/BwAuthority/data/scanner.3/scan-data/

and so looking at that file, I found:

bwscanner@longclaw:~/torflow/NetworkScanners/BwAuthority/data/scanner.3/scan-data$ ls -l bws-41.4:42.1-done-2015-01-21-17:03:13
-rw-r--r-- 1 bwscanner bwscanner 0 Jan 21 17:03 bws-41.4:42.1-done-2015-01-21-17:03:13

aha! Its zero bytes... probably was truncated before it was written when the system was rebooted.

I removed that file and now I get no cron errors. So perhaps the code should be a little smarter here about dealing with zero byte files?

Trac:
Sponsor: N/A to N/A
Severity: N/A to Blocker
Summary: aggregate.py unhappy with input data to aggregate.py should ignore empty scan-data files
Reviewer: N/A to N/A

Trac:
Status: new to assigned
Severity: Blocker to Normal
Keywords: N/A deleted, easy intro added
Owner: aagbsn to tom

Please see my github branch bug10609, which ignores, logs an error, and tries to remove files that fail parsing.

Trac:
Owner: tom to teor

Trac:
Status: assigned to needs_review

Priorities and Severities in torflow are meaningless, setting them all to Medium/Normal.

Disowning this ticket, I don't think we'll ever fix torflow.

Trac:
Owner: teor to N/A
Status: needs_review to assigned

Trac:
Status: assigned to needs_review

Trac:
Reviewer: N/A to dgoulet

I thought we weren't doing any fixes to torflow?

moved to tpo/network-health/torflow#10609 (closed)

aggregate.py should ignore empty scan-data files

Child items ...

Activity