Add a BridgeDB module
While talking about making a PT infographic and interesting stats to point out, asn pointed out that there are no publicly available BridgeDB stats.
This can be useful in many different fronts, such as seeing the balance between usage and available resources.
This ticket is about adding a CollecTor module that will archive the stats exposed in #9316 (moved). The actual stats and format should be discussed in #9316 (moved) and may benefit from the discussion in #29315 (moved).
Activity
This looks like a duplicate of #9316 (moved). Should we close it?
Or is this ticket supposed to cover the publish part of the statistics exported in #9316 (moved)? If so, we should move it to Metrics/CollecTor and set it to needs_information until there's something to publish.
Trac:
Cc: asn to asn, metrics-team
Yes. I'm doing that and setting #9316 (moved) as a parent.
Trac:
Parent: N/A to #9316 (moved)
Component: Obfuscation/BridgeDB to Metrics/CollecTor
Status: assigned to needs_information
Trac:
Summary: Publish BridgeDB stats to Add a BridgeDB module
Keywords: metrics deleted, metrics-roadmap-2019-q2 added
Type: defect to enhancement
Parent: #9316 (moved) to N/A
Cc: asn, metrics-team, gaba, cohosh to asn, metrics-team, gaba, cohosh, ahf, dgoulet, phw
Description: While talking about making a PT infographic and interesting stats to point out, asn pointed out that there are no publicly available BridgeDB stats.This can be useful in many different fronts, such as seeing the balance between usage and available resources.
to
While talking about making a PT infographic and interesting stats to point out, asn pointed out that there are no publicly available BridgeDB stats.
This can be useful in many different fronts, such as seeing the balance between usage and available resources.
This ticket is about adding a CollecTor module that will archive the stats exposed in #9316 (moved). The actual stats and format should be discussed in #9316 (moved) and may benefit from the discussion in #29315 (moved).
Points: 1 to 8
Status: needs_information to new-
#9316 (moved) has been resolved recently which unblocks this ticket, AIUI.
Replying to karsten:
#9316 (moved) has been resolved recently which unblocks this ticket, AIUI.
Yes. How would you like me to expose the metrics file on BridgeDB's host? Should it be available over HTTPS? Or do you want me to rsync it to another host?
-
Replying to phw:
Replying to karsten:
#9316 (moved) has been resolved recently which unblocks this ticket, AIUI.
Yes. How would you like me to expose the metrics file on BridgeDB's host? Should it be available over HTTPS? Or do you want me to rsync it to another host?
Is there anything sensitive in the file that would have to be sanitized on the CollecTor host? If so, we should rsync it over ssh to colchicifolium. But if not, the preferred way would be to expose it on the BridgeDB host, so that others can fetch it, too.
Here's another question, similar to the one about Snowflake stats: Would it be possible to expose more than just the latest BridgeDB statistics? Something like 7 or 14 days, or if it's not much data, everything until it gets too big?
-
Replying to karsten:
Replying to phw:
Replying to karsten:
#9316 (moved) has been resolved recently which unblocks this ticket, AIUI.
Yes. How would you like me to expose the metrics file on BridgeDB's host? Should it be available over HTTPS? Or do you want me to rsync it to another host?
Is there anything sensitive in the file that would have to be sanitized on the CollecTor host? If so, we should rsync it over ssh to colchicifolium. But if not, the preferred way would be to expose it on the BridgeDB host, so that others can fetch it, too.
There's nothing sensitive. We're doing the sanitisation ourselves and have published the data before. I'll look into exposing the files over our apache.
Here's another question, similar to the one about Snowflake stats: Would it be possible to expose more than just the latest BridgeDB statistics? Something like 7 or 14 days, or if it's not much data, everything until it gets too big?
Yes, that's definitely feasible. One week worth of data shouldn't be more than ~100 KB. I'll look into logrotate, so we can expose multiple weeks worth of data at any given time.
-
Replying to karsten:
Any updates here? If possible, I'd like to work on this next week. Thanks!
Not yet. I'll try to get it done by early next week.
-
Replying to karsten:
Replying to phw:
Replying to karsten:
#9316 (moved) has been resolved recently which unblocks this ticket, AIUI.
Yes. How would you like me to expose the metrics file on BridgeDB's host? Should it be available over HTTPS? Or do you want me to rsync it to another host?
Is there anything sensitive in the file that would have to be sanitized on the CollecTor host? If so, we should rsync it over ssh to colchicifolium. But if not, the preferred way would be to expose it on the BridgeDB host, so that others can fetch it, too.
Change of plan: Can we instead rsync BridgeDB's metrics to colchicifolium? Weasel isn't a fan of the idea of exposing BridgeDB's metrics on polyanthum. If CollecTor is archiving the metrics anyway, we might as well just sync them to colchicifolium.
If you are ok with this, I just need a directory to sync the metrics to.
Trac:
Status: assigned to needs_information -
Replying to karsten:
Sure, that works, too. How about
/srv/collector.torproject.org/collector/in/bridgedb-stats/? I'll have to edit a script on the receiving side, but feel free to set up the rsync on the sending side whenever you're ready.Thanks. I updated the script on polyanthum's side. It will sync all available bridgedb-metrics.log files, including the rotated ones. The format of rotated files is the same as for assignments.log:
bridgedb-metrics.log-YYYYMMDD.gz, e.g.,bridgedb-metrics.log-20190905.gz. The file bridgedb-metrics.log is written once per day and also rotated once a day. For now, I configured logrotate to retain 30 rotated files, mostly as a precaution, so we don't lose data in case we run into trouble.All the changes I made are in the task/19332 branch of my bridgedb-admin repository. Here's a patch. Cecylia, can you please review these changes when you get a chance?
Trac:
Reviewer: N/A to cohosh
Status: needs_information to needs_review 
These changes look good to me.
Since all logs (including previously rotated ones) are synced each time with rsync, is there a way to detect if old logs have been corrupted and are overwriting the previously synced logs? Not sure how we want to handle a case where logs that have previously been synced have changed for some reason, or what the easiest way to deal with this is.
Trac:
Status: needs_review to merge_ready-
Replying to cohosh:
These changes look good to me.
Okay, please let me know when I need to do something on colchicifolium's side.
Since all logs (including previously rotated ones) are synced each time with rsync, is there a way to detect if old logs have been corrupted and are overwriting the previously synced logs? Not sure how we want to handle a case where logs that have previously been synced have changed for some reason, or what the easiest way to deal with this is.
Thanks for thinking about such problems beforehand. In this case I think it's fine to just rsync what's on the BridgeDB host to colchicifolium. We can still decide on colchicifolium to not overwrite previously imported statistics, which I think is what we do with all other files. I'd say let's give it a try, and we can change this later if this turns out to be an issue.
-
Replying to karsten:
Okay, please let me know when I need to do something on colchicifolium's side.
The rsync is done from our side. We are already trying to sync data to
/srv/collector.torproject.org/collector/in/bridgedb-stats/. -
Replying to cohosh:
These changes look good to me.
Merged to master.
Since all logs (including previously rotated ones) are synced each time with rsync, is there a way to detect if old logs have been corrupted and are overwriting the previously synced logs? Not sure how we want to handle a case where logs that have previously been synced have changed for some reason, or what the easiest way to deal with this is.
I left the patch as it is according to Karsten's suggestion. I also reassigned the ticket to Karsten because BridgeDB's side is looking good now.
Trac:
Owner: phw to karsten
Status: merge_ready to assigned -
Replying to phw:
Replying to karsten:
Okay, please let me know when I need to do something on colchicifolium's side.
The rsync is done from our side. We are already trying to sync data to
/srv/collector.torproject.org/collector/in/bridgedb-stats/.Hmm, I changed something on colchicifolium's side to accept the rsync, but it doesn't seem to work. Is there anything in the log on your side?
-
Here's our latest rsync attempt:
1568738247:metrics:<36>Sep 17 16:37:27 collector-ssh-wrap[29644]: The SSH_ORIGINAL_COMMAND ('rsync --server -logDtpre.iLsfxC . /srv/collector.torproject.org/collector/in/bridgedb-stats/') is not on the whitelist 1568738247:metrics:rsync: connection unexpectedly closed (0 bytes received so far) [sender] 1568738247:metrics:rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]It looks like the same issue as the one we recently solved in #31515 (moved).
-
Replying to karsten:
I probably asked this before and forgot the answer: where would I find the spec for parsing the format? Thanks in advance!
The spec is still on my todo list. Here's a quick-and-dirty summary, so you don't need to block on me:
- A metrics file starts with
bridgedb-stats-end YYYY-MM-DD HH:MM:SS (SECS s) - The second line determines the version of the format. It currently is
bridgedb-stats-version 1.0 - From here on, we have multiple
bridgedb-metric-countlines. They are structured as follows:bridgedb-metric-count DIST.PROTO.CC.[success|fail].none NUM-
DISTis BridgeDB's distribution mechanism, which currently ishttp,email, ormoat. -
PROTOis the obfuscation protocol, which currently isobfs2,obfs3,obfs4,scramblesuit, orfte. -
CCis our two-letter country code. - The second-to-last field is either
successorfaildepending on if the BridgeDB request succeeded or not. - The last field is currently
nonebut will eventually be a anomaly score, perhaps normalised to [0, 1]. I would suggest to ignore it for now. -
NUMis the approximate number of requests, rounded to the next multiple of 10.
I'll follow up with the spec once it's done. I hope this is good enough to make progress in the meanwhile.
- A metrics file starts with
-
Replying to phw:
I'll follow up with the spec once it's done. I hope this is good enough to make progress in the meanwhile.
Yes, that's a fine start. Thanks!
I wrote some code. irl, please review metrics-lib commit 820c246 and CollecTor commit 3866164.
Trac:
Reviewer: cohosh to irl
Status: assigned to needs_review -
Replying to phw:
I'll follow up with the spec once it's done. I hope this is good enough to make progress in the meanwhile.
For what it's worth, we now have a draft of the spec over at #31780 (moved).
Also, the spec isn't carved in stone, so please let me know if you would like to add or improve anything and I'll get right to it.
-
Replying to karsten:
Replying to phw:
I'll follow up with the spec once it's done. I hope this is good enough to make progress in the meanwhile.
Yes, that's a fine start. Thanks!
I forgot to mention that
CCis actuallyCC/EMAIL. Our draft spec explains the details. Sorry about that. -
Trac:
Parent: N/A to #31274 (moved) -
Moving to needs_revision because of pending spec changes in #31780 (moved).
Trac:
Status: needs_review to needs_revision -
irl, please review metrics-lib commit 2e6d689 and CollecTor commit 2e6d689 with updates to the BridgeDB spec changes.
Trac:
Status: needs_revision to needs_review -
This has been merged and deployed, but I noticed a minor issue with paths and file names of written files. Please also review commit 14cff65 in my task-19332-2 branch with a fix. I'd like to release and deploy that together with #31204 (moved).
Trac:
Status: merge_ready to needs_review mentioned in issue #31204 (moved)
mentioned in issue #32144 (moved)
mentioned in issue #32146 (moved)
mentioned in issue #32441 (moved)
mentioned in issue #31274 (moved)