Opened 4 years ago

Last modified 23 months ago

#19282 assigned defect

Avoid truncating descriptors while storing them

Reported by: karsten Owned by: metrics-team
Priority: Medium Milestone:
Component: Metrics/CollecTor Version:
Severity: Normal Keywords: metrics-2018
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Thomas/oma found this while processing archived votes:

org.torproject.descriptor.DescriptorParseException: r line 'r puffytor' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-07-00-00-vote-14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4-16E97744121B2FFD387C43246F22153741FD22F8
org.torproject.descriptor.DescriptorParseException: 'YbVD6iBkPXGbsypMnn4d/sQA4I3hBxF' in line 'm 18,19,20 sha256=YbVD6iBkPXGbsypMnn4d/sQA4I3hBxF' is not a valid base64-encoded 32-byte value.
    in votes-2016-04/01/2016-04-01-06-00-00-vote-49015F787433103580E3B66A1707A00E60F2D15B-B98C42D8E4178E780E3DA2F048C5D25434E8B66D
org.torproject.descriptor.DescriptorParseException: r line 'r KingJasperTheGreat JW3pOXIv/G' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-08-00-00-vote-49015F787433103580E3B66A1707A00E60F2D15B-2C4AA4675C6BE8DF0D2898A42B1D83DE60FF5348
org.torproject.descriptor.DescriptorParseException: '13,14,' in line 'm 13,14,' is not a valid base64-encoded 32-byte value.
    in votes-2016-04/01/2016-04-01-09-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-4BF4EEF4A8618447D6E363ED7D3E3835CEB5D379
org.torproject.descriptor.DescriptorParseException: Keyword 'directory-signature' is contained 0 times, but must be contained exactly once.
    in votes-2016-04/01/2016-04-01-10-00-00-vote-E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58-81E1E9BEBAA47A305D404AFE0E54A32AD603F648
org.torproject.descriptor.DescriptorParseException: 'J35KTpJ2' in line 'm 18,19,20,22 sha256=J35KTpJ2' is not a valid base64-encoded 32-byte value.
    in votes-2016-04/01/2016-04-01-05-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-636CBFED055E4C21442FC6213FBF172A34355BC4
org.torproject.descriptor.DescriptorParseException: r line 'r seele AAoQ1DAR6kkoo19hBAX5K0QztNw xgEIpqUwO/AGzMqJuybadp0MHDg 2016-04-0' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-09-00-00-vote-23D15D965BC35114467363C165C4F724B64B4F66-01917F128E43315854AC6D3CDDAF5AC0F2C6ACFC
org.torproject.descriptor.DescriptorParseException: '1iWRvv4H9t0LHHLRZL' in line 'm 13,14,15 sha256=1iWRvv4H9t0LHHLRZL' is not a valid base64-encoded 32-byte value.
    in votes-2016-04/01/2016-04-01-07-00-00-vote-49015F787433103580E3B66A1707A00E60F2D15B-8DC4AB2F5E83E5117CB1BF8590658BFB2EEF40B4
org.torproject.descriptor.DescriptorParseException: Keyword 'directory-signature' is contained 0 times, but must be contained exactly once.
    in votes-2016-04/01/2016-04-01-05-00-00-vote-49015F787433103580E3B66A1707A00E60F2D15B-DC09DDCD3B34DF976A0D6D1645DE8BEBAB3646B4
org.torproject.descriptor.DescriptorParseException: r line 'r Unnamed GfWunVflTNsyPhaL2FetR0MO96U 2Zw9flyVAbb0Uc13fWa2+o//rg8 2016-03-31 12:' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-09-00-00-vote-E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58-1F1431417D0CEC7086442D36FB88791998CC1961
org.torproject.descriptor.DescriptorParseException: Keyword 'directory-signature' is contained 0 times, but must be contained exactly once.
    in votes-2016-04/01/2016-04-01-10-00-00-vote-EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97-48C075797027F060F9C2ECA203DA83B519ADD488
org.torproject.descriptor.DescriptorParseException: r line 'r seele AAoQ1DAR6kkoo19hBAX5K0QztNw xgEIpqUwO/AGzMqJuybadp0MHDg 2016-04-0' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-10-00-00-vote-23D15D965BC35114467363C165C4F724B64B4F66-03532324381822931CBDBB8C47BBC23C13A1FDE6
org.torproject.descriptor.DescriptorParseException: Keyword 'directory-signature' is contained 0 times, but must be contained exactly once.
    in votes-2016-04/01/2016-04-01-06-00-00-vote-EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97-5132376E2D44A7DEDAB25D6C2B1E5D168DC39EB5
org.torproject.descriptor.DescriptorParseException: Illegal key-value pair in line 'w Bandwi'.
    in votes-2016-04/01/2016-04-01-11-00-00-vote-D586D18309DED4CD6D57C18FDB97EFA96D330566-7EF062C0774ED2AFA0200770F9DE0141006AD4EB
org.torproject.descriptor.DescriptorParseException: 'GYa' in line 'm 13,14,15 sha256=GYa' is not a valid base64-encoded 32-byte value.
    in votes-2016-04/01/2016-04-01-10-00-00-vote-0232AF901C31A04EE9848595AF9BB7620D4C5B2E-19FD6206F29626F8B94EDDAC25CE17A264DCF998
org.torproject.descriptor.DescriptorParseException: r line 'r seele AAoQ1DAR6kkoo19hBAX5K0QztNw iU9ixCKwJOUEHUO1h+KIEkQf0qU 2' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-05-00-00-vote-23D15D965BC35114467363C165C4F724B64B4F66-CB23F13DC89D5F4A61FF54AC5E2D591DE06A91AE
org.torproject.descriptor.DescriptorParseException: r line 'r yashodara PW/bW8B4bVFa55aETVXD8Q8aR' has fewer space-separated elements than expected.
    in votes-2016-04/01/2016-04-01-08-00-00-vote-14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4-758D6B9D8A8E586DC6E04377669AD429284F96D6

The reason is that the disk ran full on that day, and apparently we only stored truncated votes to disk. We should detect that and only store complete descriptors, maybe by storing to a .tmp file and then renaming. This issue might not be limited to votes.

Child Tickets

Change History (8)

comment:1 Changed 3 years ago by iwakeh

Milestone: CollecTor 1.1.0

Added to second release milestone.

comment:2 Changed 3 years ago by iwakeh

Owner: set to iwakeh
Status: newassigned

assigning these issues.

comment:3 Changed 3 years ago by iwakeh

Milestone: CollecTor 1.1.0CollecTor 1.2.0

comment:4 Changed 2 years ago by karsten

Milestone: CollecTor 1.2.0
Summary: Don't store truncated descriptorsAvoid truncating descriptors while storing them

This still seems like a good idea, though the summary doesn't fully reflect this. It's not that we want to avoid storing truncated descriptors. We want to avoid messing with descriptors as much as we can. But we do want to avoid truncating descriptors ourselves while storing them. Updating summary.

However, this change might be too big for the next release which is supposed to come out next week. Removing milestone for now. Maybe it fits into the end-of-month milestone.

comment:5 Changed 2 years ago by karsten

Keywords: metrics-2018 added

comment:6 Changed 2 years ago by karsten

Keywords: metrics-2017 added; metrics-2018 removed

comment:7 Changed 2 years ago by iwakeh

Keywords: metrics-2018 added; metrics-2017 removed

Will be completed in 2018.

comment:8 Changed 23 months ago by iwakeh

Owner: changed from iwakeh to metrics-team

Move to metrics-team as these are not worked on by me during the next week.

Note: See TracTickets for help on using tickets.