Skip to content
Snippets Groups Projects

Delete old unsanitized logs

    • Closed (moved) Issue created by David Fifield @dcf

    After #21304 (moved), in the 2019-04-18 team checkin, we agreed to delete the pre-sanitization logs by 2019-06-01. Before then, we have a chance to extract any useful sanitized information from them.

    Cf. comment:8:ticket:30125 regarding the location of logs.

    === Broker

    On the broker, there are three files to delete:

    /var/log/snowflake-broker/unstanitized.tar:: Contains three files: * '''unsanitized.tar.1''', pre-rotation logs from 2017-07-15 to 2019-03-22 and rotated logs from 2017-03-22 to 2019-04-16. * '''@400000005cb771341453839c.s''', logs from 2019-04-16 to 2019-04-17. * '''current''', empty file. /var/log/snowflake-broker/mv:: Probably the result of a typo, it is a tar file that contains a single file unsanitized.tar which is identical to the unsanitized.tar.1 inside /var/log/snowflake-broker/unstanitized.tar. /var/log/snowflake-broker/x.tar:: contains a subset of what /var/log/snowflake-broker/mv contains.

    I think the most complete logs come from /var/log/snowflake-broker/unstanitized.tar, extracting the contained unsanitized.tar.1 and combining it with the contained @400000005cb771341453839c.s.

    === snowflake-server

    /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to 2019-04-11.

    === proxy-go /home/snowflake-proxy/snowflake-proxy-17h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-23h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-29h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.tar:: /home/snowflake-proxy/unsanitized_old.tar::

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • David Fifield added component::circumvention/snowflake priority::medium resolution::fixed severity::normal status::closed type::task labels

      added component::circumvention/snowflake priority::medium resolution::fixed severity::normal status::closed type::task labels

    • D
      David Fifield @dcf ·
      Author

      Replying to dcf:

      === snowflake-server

      /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to 2019-04-11.

      This one is easy. We can just delete the file. We were never permanently storing these logs and they would have been automatically rotated and deleted anyway.

    • D
      David Fifield @dcf ·
      Author

      Trac:
      Cc: N/A to arlolra, phw, cohosh

    • D
      David Fifield @dcf ·
      Author

      Trac:
      broker-logs.zip

    • D
      David Fifield @dcf ·
      Author

      Programs to process and visualize broker logs.

    • D
      David Fifield @dcf ·
      Author

      Trac:
      broker-interactions

    • D
      David Fifield @dcf ·
      Author

      Trac:
      broker-estimated-proxies

    • D
      David Fifield @dcf ·
      Author

      I have prepared a candidate sanitized CSV file extracted from the sanitized logs. I've placed it in /var/log/snowflake-broker/broker.csv.xz for evaluation. It's 6 MB compressed but 1.8 GB uncompressed.

      The sanitized CSV looks like this:

      timestamp,event,proxyid,clientid,additional
2017-07-21 23:40:00,proxy-gets-none,157,,no-clients
2017-07-21 23:40:00,proxy-polls,157,,
2017-07-21 23:40:00,proxy-gets-none,149,,no-clients
2017-07-21 23:40:00,proxy-polls,149,,
2017-07-21 23:40:00,client-offers,,,
2017-07-21 23:40:00,proxy-gets-offer,,,
2017-07-21 23:40:00,proxy-answers,,,
2017-07-21 23:40:00,client-gets-answer,,,
2017-07-21 23:40:00,proxy-polls,160,,
2017-07-21 23:40:00,proxy-gets-none,159,,no-clients
2017-07-21 23:40:00,proxy-polls,159,,
2017-07-21 23:40:00,proxy-gets-none,157,,no-clients
2017-07-21 23:40:00,proxy-polls,157,,

      Timestamps are truncated to multiples of 10 minutes. Client and proxy IDs are replaced by sequential integers.

      The event column can take on these values:

      • start the broker was restarted.
      • client-offers a client connects, sends an offer, and awaits an answer.
      • client-gets-answer a client receives a proxy's answer (successful broker match).
      • client-gets-none a client disconnects without receiving an answer, whether because of a timeout or because there were no proxies.
      • proxy-polls a proxy connects in order to receive an offer.
      • proxy-gets-none a proxy disconnects without receiving a client offer (no clients).
      • proxy-gets-offer a proxy receives a client offer.
      • proxy-answers a proxy sends an answer to the broker.
      • error an error; the most common is "http: TLS handshake error". Other possibilities are "http2: server: error", "http2: received GOAWAY", or a bad HTTP request. The additional column distinguishes these cases.

      Some of these have relations to each other. For example proxy-polls ≈ proxy-gets-none + proxy-gets-answer.

      Using the sanitized CSV, I made a couple of graphs. The first shows shows the number of broker outcomes per day, where an outcome is one of the four possibilities:

      • A client and proxy are successfully linked up.
      • A proxy connects but doesn't get a client.
      • A client connects but doesn't get a proxy.
      • Some other error occurred.

      Click to embiggen. broker-interactions.png,100%

      The second graph shows the estimated number of proxies. This is just 10 × proxy-polls / s. It's based on the assumption that each proxy polls every 10 s. The assumption doesn't hold when there are actually clients, but as you can see the estimate is pretty close to 3, which is the number of fallback proxy-go instances.

      Click to embiggen. broker-estimated-proxies.png,100%

      Trac:
      Status: new to needs_review

    • D
      David Fifield @dcf ·
      Author

      This is done. On the broker I ran:

      shred -v -n 1 -u /var/log/snowflake-broker/{unstanitized.tar,mv,x.tar}

      On the bridge I ran:

      shred -v -n 1 -u /var/log/tor/unsanitized.tar
find /home/snowflake-proxy/ -name 'unsanitized*.tar' -print0 | xargs -0 shred -v -n 1 -u

      I opened #30731 (moved) to discuss publishing the sanitized information that was extracted before deletion.

      Here are the filenames contained in the deleted proxy-go log tar files. The filenames include TAI64 timestamps, in case we need to recover a date range later.

      <pre style="font-size: 50%;">
$ tar -tf /home/snowflake-proxy/snowflake-proxy-17h.log.d/unsanitized.tar 
@400000005c3321b70eae9544.s
@400000005c576da82a020164.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-23h.log.d/unsanitized.tar 
$ tar -tf /home/snowflake-proxy/snowflake-proxy-29h.log.d/unsanitized.tar 
@400000005c33279705673784.s
@400000005c57719b36ba5294.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.tar 
@400000005bf83e01218ae80c.s
@400000005bf903a20591c9f4.s
@400000005bfc35e831e25ce4.s
@400000005c00179f27894cbc.s
@400000005c0113a518c7d194.s
@400000005c02797623c4b724.s
@400000005c02fe1200096c1c.s
@400000005c041ea51d289f5c.s
@400000005c0518260b6d83a4.s
@400000005c06d96f0448b3dc.s
@400000005c08a3270b8cc3a4.s
@400000005c08dfab361807b4.s
@400000005c098451156e55f4.s
@400000005c0ab50128791774.s
@400000005c0c5de01151d48c.s
@400000005c101acc2db97f6c.s
@400000005c10689a17a21a2c.s
@400000005c118fe22634b16c.s
@400000005c13bbb10ce0b314.s
@400000005c142d7b13e577bc.s
@400000005c14339b18a5af4c.s
@400000005c1545cd14de182c.s
@400000005c180f7d1219bc54.s
@400000005c1a3b6138d2becc.s
@400000005c1b9332084a71b4.s
@400000005c1e3add3b67a24c.s
@400000005c1e54800447a26c.s
@400000005c1ed19c01f3ee6c.s
@400000005c22507610af2804.s
@400000005c232bd529124264.s
@400000005c251c3420b3f89c.s
@400000005c254d500563ad44.s
@400000005c25aee522781ab4.s
@400000005c274df81d08a3dc.s
@400000005c2870e137bece2c.s
@400000005c2a0447255c144c.s
@400000005c2b3ead36506564.s
@400000005c2da5562878f064.s
@400000005c304cc200ca3794.s
@400000005c31ab610667b3ac.s
@400000005c3308cb2e191bfc.s
@400000005c3482080d28762c.s
@400000005c36138329f55734.s
@400000005c36f03619fae6dc.s
@400000005c37b44c06bee014.s
@400000005c38bf2922a1323c.s
@400000005c38d65a23df6efc.s
@400000005c3c80d00747b36c.s
@400000005c3cb9d308be0b74.s
@400000005c3dd51629206b64.s
@400000005c3ed9a93089878c.s
@400000005c3f40351f451aa4.s
@400000005c409b4b1f7ce3e4.s
@400000005c424a69029bea54.s
@400000005c42de051acd6594.s
@400000005c43b910314a9954.s
@400000005c445e77115c7f04.s
@400000005c45c51f33403b4c.s
@400000005c46d16437474884.s
@400000005c47223b15924cfc.s
@400000005c48200930d6c4fc.s
@400000005c48ed82258191f4.s
@400000005c492e44199ac57c.s
@400000005c4a16df2c6b930c.s
@400000005c4a331434affbd4.s
@400000005c4b3519368999ec.s
@400000005c4c25e617d255d4.s
@400000005c4d252015b06804.s
@400000005c4ed2ab07bde53c.s
@400000005c4ffb4226518454.s
@400000005c51501c00f6f4b4.s
@400000005c5246fd378a25b4.s
@400000005c5255c62eaabd3c.s
@400000005c53e9bf116c2ea4.s
@400000005c55015a383317b4.s
@400000005c55e7401b96ef54.s
@400000005c562c930b5c1e84.s
@400000005c58849801cf5f0c.s
@400000005c59b2fe374b8674.s
@400000005c59ce8f01db22c4.s
@400000005c5a37ea059cc28c.s
@400000005c5cb8d130c41f8c.s
@400000005c5ea5b916b3a734.s
@400000005c5fbc0e18f30044.s
@400000005c5fc78a34b5d004.s
@400000005c5fcc5d3b1878ac.s
@400000005c60fc7729d80f1c.s
@400000005c61c8b5281f8d94.s
@400000005c629a8021651474.s
@400000005c648429005c9914.s
@400000005c654806096a166c.s
@400000005c659c62229a352c.s
@400000005c65c882158ec6a4.s
@400000005c6715e2211496fc.s
@400000005c67b8d4163eb99c.s
@400000005c68f8c521fcf35c.s
@400000005c6ac8ac1f6f0cec.s
@400000005c6b18e11acebd54.s
@400000005c6b3c5328d1748c.s
@400000005c6c34d0144c20fc.s
@400000005c6f828f2755c16c.s
@400000005c7128681217d7f4.s
@400000005c7256c51fc239e4.s
@400000005c72e6fe309aa274.s
@400000005c7373031e170a84.s
@400000005c761ac103fba54c.s
@400000005c7835ef3afd381c.s
@400000005c783c3307f00d14.s
@400000005c78fa071b3a2184.s
@400000005c7a712e318d2c4c.s
@400000005c7b00fb2cd05fbc.s
@400000005c7d6f5f02f948ac.s
@400000005c7ee2a5306993dc.s
@400000005c802c030abb98c4.s
@400000005c805e9b22a5aac4.s
@400000005c81bde91ffdb474.s
@400000005c828ef81730ca5c.s
@400000005c83d1ad237b3e8c.s
@400000005c8535a837e74974.s
@400000005c86ab1b1a28403c.s
@400000005c8803de2d695bcc.s
@400000005c88f14d1e362374.s
@400000005c89e70605f4874c.s
@400000005c8ab3d631848574.s
@400000005c8c00842b8361a4.s
@400000005c8cc39a0256a00c.s
@400000005c8d50d12e9bffcc.s
@400000005c8e888e12cf837c.s
@400000005c8f517b1b64b7dc.s
@400000005c90891e2483213c.s
@400000005c91b78c0e9014ac.s
@400000005c93bf1b02951c24.s
@400000005c95b42c2c1cabd4.s
@400000005c974f8e12a2418c.s
@400000005c97b6a2066bf96c.s
@400000005c98cdca26efeb1c.s
@400000005c994f4d093191ac.s
@400000005c9a3da302ad2c4c.s
@400000005c9b61731fa5c0d4.s
@400000005c9bab130604f654.s
@400000005c9d3e951366cdf4.s
@400000005c9e4cca0b49a7f4.s
@400000005c9e78a229182e04.s
@400000005c9e9fa43b663704.s
@400000005c9f961c1e1a2b4c.s
@400000005c9fdf2507b292cc.s
@400000005ca06669390fbc14.s
@400000005ca0a3e00f224a5c.s
@400000005ca0ab4912083bdc.s
@400000005ca0b3a720def09c.s
@400000005ca0dd5e38b79d7c.s
@400000005ca0fa54173761dc.s
@400000005ca109e020741524.s
@400000005ca1a9071309c1a4.s
@400000005ca31bb910125f4c.s
@400000005ca3426c1e9e014c.s
@400000005ca3b4d81f678b0c.s
@400000005ca412f00da7b84c.s
@400000005ca5029d039fe504.s
@400000005ca61de402669dcc.s
@400000005ca661932b87ca8c.s
@400000005ca718cb19de4e8c.s
@400000005ca786480e75d204.s
@400000005ca7d29812bd7664.s
@400000005ca8662627e3b15c.s
@400000005ca8842d27423d54.s
@400000005ca8bf1c0daaf084.s
@400000005ca8eaa42bd83094.s
@400000005ca9740c200b9724.s
@400000005ca9c14117da318c.s
@400000005caa3aca24644ab4.s
@400000005caa4f3a37b17434.s
@400000005cab502b0d824a44.s
@400000005caba71a1521919c.s
@400000005cac60a5197da85c.s
@400000005cacc3701a379dd4.s
@400000005cace73e2a6e4824.s
@400000005cad0d1b1ea67174.s
@400000005cadfad91ae3f304.s
@400000005cae2a7a05e2d7f4.s
@400000005cae5c3f0649757c.s
@400000005caee27b01f4b5a4.s
@400000005caf503c0a4f9c3c.s
@400000005caf6aa814eb666c.s
@400000005caf766e1e9a2504.s
@400000005caf90e40e39421c.s
@400000005cb043de2f459dac.s
@400000005cb09c35115fda64.s
@400000005cb0b0c8089f4c5c.s
@400000005cb0bc5e07cce8fc.s
@400000005cb0de4e2495ce7c.s
@400000005cb11ace12994c94.s
@400000005cb1d6ca09e8ef14.s
@400000005cb2045624ae9a24.s
@400000005cb23e291fd9212c.s
@400000005cb246102087434c.s
@400000005cb2e0a4038a6904.s
@400000005cb3000a29953da4.s
@400000005cb33635067bb0dc.s
@400000005cb3429e02489e1c.s
@400000005cb356b60085339c.s
@400000005cb43ca61bf3c4f4.s
@400000005cb4c655217eb6f4.s
@400000005cb5959b045fed2c.s
@400000005cb5b05f2678d2d4.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.2.tar 
@400000005cb758970c1b059c.s
@400000005cb76e6301912d2c.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.tar 
@400000005bf8f47003017e3c.s
@400000005bfa68862cac0af4.s
@400000005bfe9c161507d7ac.s
@400000005bff740f27755b44.s
@400000005c0121bd20f274b4.s
@400000005c019f2c03926fb4.s
@400000005c02254a0ad8c19c.s
@400000005c02f9133aed2ea4.s
@400000005c043eaf06d1fab4.s
@400000005c054b140d4bd4dc.s
@400000005c07c28c1d8e17ec.s
@400000005c089b34184bbbf4.s
@400000005c09400f17ba2a54.s
@400000005c0a65b218c5a6e4.s
@400000005c0c090b312f02d4.s
@400000005c0d93192c1dbd44.s
@400000005c105c4d068e6204.s
@400000005c1093a9103f437c.s
@400000005c122e9c0b6048ec.s
@400000005c13a74628489964.s
@400000005c1486c934309a74.s
@400000005c1521961fbad35c.s
@400000005c178fe02c0883ac.s
@400000005c1a7aa8318d5f14.s
@400000005c1cee7b1b151524.s
@400000005c1d79e9046e644c.s
@400000005c1d8c8c395160c4.s
@400000005c233a901ca922a4.s
@400000005c26157218908554.s
@400000005c27418624f7dff4.s
@400000005c291d9a0bd5f5ec.s
@400000005c2c626f27e733cc.s
@400000005c2e20b53974a41c.s
@400000005c2fdda625e32284.s
@400000005c325c7c0b80c16c.s
@400000005c3450e7006ce4f4.s
@400000005c35236837eb50b4.s
@400000005c35d57f1efe595c.s
@400000005c375d011622cd2c.s
@400000005c3775420f1e1c0c.s
@400000005c380ed722827324.s
@400000005c38ce1c36f9ebd4.s
@400000005c3c27f51b7a81fc.s
@400000005c3c939436a86c8c.s
@400000005c3f175701a99344.s
@400000005c3f76561940419c.s
@400000005c40c96000d1482c.s
@400000005c41f8f023e29794.s
@400000005c42bbe91d9036e4.s
@400000005c4336f735877014.s
@400000005c443f403468e49c.s
@400000005c4587fb1c8a9e24.s
@400000005c46293c392f2edc.s
@400000005c47b1391af5e8ac.s
@400000005c48c9f811761d9c.s
@400000005c499dc43a03381c.s
@400000005c4b19e20492d46c.s
@400000005c4c4c9d1b381de4.s
@400000005c4dc1d1380d1154.s
@400000005c4feb8b28b764ac.s
@400000005c509e7133011354.s
@400000005c50f5df06c29164.s
@400000005c51587d05a0dd54.s
@400000005c51bcff0fcf2074.s
@400000005c529191126c3984.s
@400000005c5316d513720cdc.s
@400000005c557598163c024c.s
@400000005c56c5cf28abf6e4.s
@400000005c59843d05cb9abc.s
@400000005c5ad89c3899865c.s
@400000005c5d8aa913f14344.s
@400000005c5ea7080d681b24.s
@400000005c5fdf28225859cc.s
@400000005c6180aa2e1d59ec.s
@400000005c62dd0211300c1c.s
@400000005c63cfff1f8ab6f4.s
@400000005c647e310a6aec6c.s
@400000005c65b9a8347fb474.s
@400000005c660dbb28e6350c.s
@400000005c66503323f59eac.s
@400000005c672aa21e98d8fc.s
@400000005c67c99e38078f2c.s
@400000005c6985a92f0fef7c.s
@400000005c6ab35d3afddc2c.s
@400000005c6ab9bc37406ab4.s
@400000005c6ac0180951085c.s
@400000005c6b32a006c11294.s
@400000005c6d943f184a299c.s
@400000005c6eed2127f98b1c.s
@400000005c70f66c273836ec.s
@400000005c718ccc07a3026c.s
@400000005c733fa308b9dd24.s
@400000005c759865115537bc.s
@400000005c76a0c32ab9d014.s
@400000005c79f87c1efae2a4.s
@400000005c7a477930e3b964.s
@400000005c7a988e021bbdd4.s
@400000005c7aef010c324aa4.s
@400000005c7bc4af2e750354.s
@400000005c7c4d4d068814bc.s
@400000005c7d78470282920c.s
@400000005c7ef2ec22105c1c.s
@400000005c8141751810dda4.s
@400000005c81d686007b5c14.s
@400000005c82c06e043097fc.s
@400000005c84c5dc39e8badc.s
@400000005c869a8e0c069b0c.s
@400000005c8871a416c85814.s
@400000005c8b34a828b9972c.s
@400000005c8c96fa395c2e64.s
@400000005c8ce7691451f144.s
@400000005c8df53919789b64.s
@400000005c8ea80b04015e24.s
@400000005c90089e33fca994.s
@400000005c91324634bbb7bc.s
@400000005c91d63b3507e7a4.s
@400000005c9337a1350d1bac.s
@400000005c958bf000a369fc.s
@400000005c96dc6a340f4734.s
@400000005c9854f437374eac.s
@400000005c9910e72fcce67c.s
@400000005c99359d0a8ecfec.s
@400000005c99d2fb0e40df54.s
@400000005c99f74a0dc3d39c.s
@400000005c9a47e304a0b334.s
@400000005c9a7892231c3284.s
@400000005c9bbb7e31efda04.s
@400000005c9ce5a826014174.s
@400000005c9d21932fdfc29c.s
@400000005c9db93733a375a4.s
@400000005c9edb8730eae16c.s
@400000005c9f624f27b8154c.s
@400000005c9fa11d26c3744c.s
@400000005ca091ef3adc6d94.s
@400000005ca2161122a46e5c.s
@400000005ca322b918dcff74.s
@400000005ca33fbd12813c6c.s
@400000005ca3b05807ad5adc.s
@400000005ca4a76f1c218bb4.s
@400000005ca5075520df3ad4.s
@400000005ca58a251d024ec4.s
@400000005ca5f24b2a747244.s
@400000005ca6672639b13bd4.s
@400000005ca72ee2304060fc.s
@400000005ca76552173d9f84.s
@400000005ca7b88f160c013c.s
@400000005ca833752e752a64.s
@400000005ca840f0046d5aac.s
@400000005ca8504127900f34.s
@400000005ca870ee16ba988c.s
@400000005ca8a0b72db81424.s
@400000005ca8fa4a19b7753c.s
@400000005ca9889d2bac3e94.s
@400000005ca9c25b392835b4.s
@400000005ca9eb1125ed1564.s
@400000005caacf42107f889c.s
@400000005cab333d371e503c.s
@400000005cab79460ef932d4.s
@400000005cac487736ad2394.s
@400000005cacbaa334ab73ac.s
@400000005cacea7e1043816c.s
@400000005cadb5940690bf7c.s
@400000005cae491d2ce602cc.s
@400000005caeae37286a36dc.s
@400000005caf7e691713d834.s
@400000005caf8a672f657dd4.s
@400000005cafc0ef172eda44.s
@400000005cb09e66261f450c.s
@400000005cb0a6b727dfe0cc.s
@400000005cb0acb12fce1b14.s
@400000005cb11e20188cd01c.s
@400000005cb1b8920564b2fc.s
@400000005cb1f4830411773c.s
@400000005cb213931f5124ac.s
@400000005cb24a5a2dcec8a4.s
@400000005cb25ce90564ed94.s
@400000005cb322c51017126c.s
@400000005cb3659e2570e46c.s
@400000005cb495c00da0a7b4.s
@400000005cb55d2530e8b2d4.s
@400000005cb5d98c2fed43a4.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.2.tar 
@400000005cb7709112673c2c.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.tar 
@400000005bf7166837c1375c.s
@400000005bf71c4f333b3df4.s
@400000005bf8e627276ecf7c.s
@400000005bf9c5b02e52064c.s
@400000005bfcd0502db8a0c4.s
@400000005bfdb0af3606eccc.s
@400000005c0024e632d1e91c.s
@400000005c0176eb2f2b85fc.s
@400000005c03ab1729a33bac.s
@400000005c03c4c11635b8ec.s
@400000005c03edcf0ef4e92c.s
@400000005c061505218398f4.s
@400000005c08e51436a130fc.s
@400000005c096aba10f74cc4.s
@400000005c0ac37e024e724c.s
@400000005c0bf61c387c3e44.s
@400000005c0cb36f0272d6b4.s
@400000005c0ccd9d281f37a4.s
@400000005c0f6d0314a8f69c.s
@400000005c0ffba50e0414d4.s
@400000005c1041242beee12c.s
@400000005c104dc305ac3f64.s
@400000005c10793b22043aa4.s
@400000005c111d8408f10a24.s
@400000005c116118128a066c.s
@400000005c129e1c3a0606dc.s
@400000005c15e8fa3aa60f9c.s
@400000005c1a45760c2d60d4.s
@400000005c1cfd3412bf04d4.s
@400000005c1f326529cf9b0c.s
@400000005c22602418383bc4.s
@400000005c2558d02b7b10bc.s
@400000005c25698621b6841c.s
@400000005c26303329154004.s
@400000005c27b53a0dd35844.s
@400000005c2a726435292374.s
@400000005c2b7a7d2c2769d4.s
@400000005c2d1cbe2013ccb4.s
@400000005c2e400e2b7d7dd4.s
@400000005c3195aa01775bcc.s
@400000005c32981b171f2e8c.s
@400000005c34472a1f39762c.s
@400000005c350c670ec1cb3c.s
@400000005c36dfb213c6ef54.s
@400000005c3788f003e77554.s
@400000005c38b2160f32a1f4.s
@400000005c3903df1f32a7fc.s
@400000005c3d27b52ca1b66c.s
@400000005c3ee0fc1fcb7914.s
@400000005c4025eb37a37a14.s
@400000005c4083da0d9808ac.s
@400000005c40e6651249638c.s
@400000005c4331233b86bf24.s
@400000005c4416781759af1c.s
@400000005c4583082e7351bc.s
@400000005c45d41e10bbddec.s
@400000005c46e4303a70ee0c.s
@400000005c475b831c35467c.s
@400000005c477e121da98374.s
@400000005c48878324169fe4.s
@400000005c4894093a9a114c.s
@400000005c48a8dd3b251724.s
@400000005c49e20e25d0a424.s
@400000005c4a2a6b24676794.s
@400000005c4b5ef734c8cf4c.s
@400000005c4c0f7234ca88b4.s
@400000005c4cffd520816b34.s
@400000005c4d0dcb1f731bfc.s
@400000005c4e1b293020cb0c.s
@400000005c4edb6f1789b02c.s
@400000005c4f398a13d40acc.s
@400000005c5033241ae258dc.s
@400000005c508e0c14c519bc.s
@400000005c50ce520c2ad864.s
@400000005c50e08436935dec.s
@400000005c51ba6d335cbc2c.s
@400000005c52ef041de4bf84.s
@400000005c5411c203d623bc.s
@400000005c54a88a12fcfc1c.s
@400000005c556c8408c6930c.s
@400000005c56ae612071d304.s
@400000005c57715a212ec234.s
@400000005c58dd8426b13854.s
@400000005c5a0ebc1034faac.s
@400000005c5ad4fb076ecf24.s
@400000005c5d4fcc01d8cd1c.s
@400000005c5ddd7f31b793c4.s
@400000005c5ebf03306ded24.s
@400000005c5f32fc0f941344.s
@400000005c61abb801152b14.s
@400000005c63b0d5251fc2e4.s
@400000005c64f86507654d8c.s
@400000005c6730e73b4adb1c.s
@400000005c673a1b03d5c214.s
@400000005c67cd2a19d3f61c.s
@400000005c6881c32c5e8734.s
@400000005c694c4b24938c5c.s
@400000005c6ad2621e66a184.s
@400000005c6c19860e8cff9c.s
@400000005c6e54f22053fa64.s
@400000005c705616397a2a2c.s
@400000005c70652111f0c7f4.s
@400000005c72420e194c3c04.s
@400000005c73b7f2187f0cac.s
@400000005c757ab70370e1dc.s
@400000005c76adc6363c3d3c.s
@400000005c798a2625019454.s
@400000005c7acd8b1d0de39c.s
@400000005c7b7ae80ea250a4.s
@400000005c7be7b103cfaf64.s
@400000005c7d61970801b884.s
@400000005c7eb6d83205e2a4.s
@400000005c7f7f912794ce0c.s
@400000005c808b5d29c90f44.s
@400000005c814fa40a90086c.s
@400000005c8257ae17b5a614.s
@400000005c82d157197bd39c.s
@400000005c83570939179f9c.s
@400000005c84007a23589f44.s
@400000005c84d63e15f98e94.s
@400000005c857f191f27e9fc.s
@400000005c876bae3465df2c.s
@400000005c8805de11cf4da4.s
@400000005c89981e14493acc.s
@400000005c8a9c761b4ca7b4.s
@400000005c8b3922196d6064.s
@400000005c8bb88e29b1aee4.s
@400000005c8bfc3720f98164.s
@400000005c8c752c2595ece4.s
@400000005c8cd83d27c11214.s
@400000005c8d5d1c040044e4.s
@400000005c8e2bc92dc6299c.s
@400000005c8f1e56244bd4fc.s
@400000005c8fa1120808b97c.s
@400000005c93a9ef08f0799c.s
@400000005c969e262d2c52cc.s
@400000005c98434a1f7fa6ec.s
@400000005c997e8b1f2968cc.s
@400000005c9aa1130651f544.s
@400000005c9bcfdd2b0a8054.s
@400000005c9bddbb1f8feafc.s
@400000005c9c584e1fb24fac.s
@400000005c9e413d051064f4.s
@400000005c9e6bb603fa073c.s
@400000005c9f21e30c60154c.s
@400000005c9f9c5f1b055db4.s
@400000005ca0609e0bc461ec.s
@400000005ca0fb482215fd84.s
@400000005ca1ce2d0dc09b64.s
@400000005ca21ddc281e224c.s
@400000005ca2272635598a14.s
@400000005ca22dbe2a0d4434.s
@400000005ca23b90331edc54.s
@400000005ca25f771d3d021c.s
@400000005ca3ba3e21eab764.s
@400000005ca4999b15a538bc.s
@400000005ca5b6010b88bc64.s
@400000005ca77511361bbcec.s
@400000005ca7a9840275a574.s
@400000005ca86c9a21913d24.s
@400000005ca8dad217026f2c.s
@400000005ca9bbc527c25264.s
@400000005caa009d30c2eedc.s
@400000005caa35973035cdf4.s
@400000005cab014d296023e4.s
@400000005cab419b2520f394.s
@400000005cab892f1a7231ec.s
@400000005cab92c70b44b26c.s
@400000005cab9c4f3420e6ec.s
@400000005cac21a102ae1a94.s
@400000005cac78c60181cbac.s
@400000005caca29f1d202b4c.s
@400000005cacb15a04e0a264.s
@400000005cacbc911004652c.s
@400000005cacea412b8cfaac.s
@400000005cad1144003edbcc.s
@400000005cae0bdf143fa1c4.s
@400000005cae593719efe28c.s
@400000005cae5eee35a77364.s
@400000005cae657327e470c4.s
@400000005cae9c412ceba04c.s
@400000005caedea6305f915c.s
@400000005caf582132a399a4.s
@400000005cafa9450ba4c044.s
@400000005cb0dec62696befc.s
@400000005cb1b0710cb6dc24.s
@400000005cb2188c3143f61c.s
@400000005cb220be02ec92c4.s
@400000005cb2d7a71aca0e1c.s
@400000005cb35ee9181830a4.s
@400000005cb432f329b23f6c.s
@400000005cb45c79122341d4.s
@400000005cb4c5423768edcc.s
@400000005cb4e0851b3bbbac.s
@400000005cb534ac00295414.s
@400000005cb5c34816c235c4.s
@400000005cb5e927297d9adc.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.2.tar 
@400000005cb6e26501db70e4.s
@400000005cb735731d9f6d6c.s
@400000005cb76ac608a76694.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.tar 
@400000005bf70ad02f3e46c4.s
@400000005bf9a42b3469cefc.s
@400000005bfab5e52afb0b4c.s
@400000005c0f076e23a04aec.s
@400000005c5172e43b4bf45c.s
@400000005c6b47e62dbd34a4.s
@400000005c6c275e03385164.s
@400000005cafce9b2d17c8fc.s
@400000005cb08faa16352c4c.s
@400000005cb20dcb30e94744.s
@400000005cb30bec1437dd7c.s
@400000005cb4eb470912b354.s
@400000005cb5c125193d6ef4.s
current
$ tar -tf /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.2.tar 
current
$ tar -tf /home/snowflake-proxy/unsanitized_old.tar 
snowflake-proxy-17h.log
snowflake-proxy-23h.log
snowflake-proxy-29h.log
snowflake-proxy-restartless.log
snowflake-proxy-standalone-17h.log
snowflake-proxy-standalone-23h.log
snowflake-proxy-standalone-29h.log
snowflake-proxy-10h.log.xz
snowflake-proxy-17h.20180705.log.xz
snowflake-proxy-17h.20181121.log.xz
snowflake-proxy-1h.log.xz
snowflake-proxy-23h.20180705.log.xz
snowflake-proxy-23h.20181121.log.xz
snowflake-proxy-29h.20180705.log.xz
snowflake-proxy-29h.20181121.log.xz
snowflake-proxy-2h.log.xz
snowflake-proxy-a.log.xz
snowflake-proxy-b.log.xz
snowflake-proxy-c.log.xz
snowflake-proxy-standalone-17h.20180705.log.xz
snowflake-proxy-standalone-17h.20181121.log.xz
snowflake-proxy-standalone-23h.20180705.log.xz
snowflake-proxy-standalone-23h.20181121.log.xz
snowflake-proxy-standalone-29h.20180705.log.xz
snowflake-proxy-standalone-29h.20181121.log.xz
snowflake-proxy-standalone-a.log.xz
snowflake-proxy-standalone-b.log.xz
snowflake-proxy-standalone-c.log.xz
snowflake-proxy-test-10h.log.xz
snowflake-proxy-test-1h.log.xz
snowflake-proxy-test-2h.log.xz
</pre>

      Trac:
      Description: After #21304 (moved), in the 2019-04-18 team checkin, we agreed to delete the pre-sanitization logs by 2019-06-01. Before then, we have a chance to extract any useful sanitized information from them.

      Cf. comment:8:ticket:30125 regarding the location of logs.

      === Broker

      On the broker, there are three files to delete:

      /var/log/snowflake-broker/unstanitized.tar:: Contains three files: * '''unsanitized.tar.1''', pre-rotation logs from 2017-07-15 to 2019-03-22 and rotated logs from 2017-03-22 to 2019-04-16. * '''@400000005cb771341453839c.s''', logs from 2019-04-16 to 2019-04-17. * '''current''', empty file. /var/log/snowflake-broker/mv:: Probably the result of a typo, it is a tar file that contains a single file unsanitized.tar which is identical to the unsanitized.tar.1 inside /var/log/snowflake-broker/unstanitized.tar. /var/log/snowflake-broker/x.tar:: contains a subset of what /var/log/snowflake-broker/mv contains.

      I think the most complete logs come from /var/log/snowflake-broker/unstanitized.tar, extracting the contained unsanitized.tar.1 and combining it with the contained @400000005cb771341453839c.s.

      === snowflake-server

      /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to 2019-04-11.

      === proxy-go

      /home/snowflake-proxy/unsanitized_old.tar:: contains 31 individual log files:

      • snowflake-proxy-17h.log
      • snowflake-proxy-23h.log
      • snowflake-proxy-29h.log
      • snowflake-proxy-restartless.log
      • snowflake-proxy-standalone-17h.log
      • snowflake-proxy-standalone-23h.log
      • snowflake-proxy-standalone-29h.log
      • snowflake-proxy-10h.log.xz
      • snowflake-proxy-17h.20180705.log.xz
      • snowflake-proxy-17h.20181121.log.xz
      • snowflake-proxy-1h.log.xz
      • snowflake-proxy-23h.20180705.log.xz
      • snowflake-proxy-23h.20181121.log.xz
      • snowflake-proxy-29h.20180705.log.xz
      • snowflake-proxy-29h.20181121.log.xz
      • snowflake-proxy-2h.log.xz
      • snowflake-proxy-a.log.xz
      • snowflake-proxy-b.log.xz
      • snowflake-proxy-c.log.xz
      • snowflake-proxy-standalone-17h.20180705.log.xz
      • snowflake-proxy-standalone-17h.20181121.log.xz
      • snowflake-proxy-standalone-23h.20180705.log.xz
      • snowflake-proxy-standalone-23h.20181121.log.xz
      • snowflake-proxy-standalone-29h.20180705.log.xz
      • snowflake-proxy-standalone-29h.20181121.log.xz
      • snowflake-proxy-standalone-a.log.xz
      • snowflake-proxy-standalone-b.log.xz
      • snowflake-proxy-standalone-c.log.xz
      • snowflake-proxy-test-10h.log.xz
      • snowflake-proxy-test-1h.log.xz
      • snowflake-proxy-test-2h.log.xz

      to

      After #21304 (moved), in the 2019-04-18 team checkin, we agreed to delete the pre-sanitization logs by 2019-06-01. Before then, we have a chance to extract any useful sanitized information from them.

      Cf. comment:8:ticket:30125 regarding the location of logs.

      === Broker

      On the broker, there are three files to delete:

      /var/log/snowflake-broker/unstanitized.tar:: Contains three files: * '''unsanitized.tar.1''', pre-rotation logs from 2017-07-15 to 2019-03-22 and rotated logs from 2017-03-22 to 2019-04-16. * '''@400000005cb771341453839c.s''', logs from 2019-04-16 to 2019-04-17. * '''current''', empty file. /var/log/snowflake-broker/mv:: Probably the result of a typo, it is a tar file that contains a single file unsanitized.tar which is identical to the unsanitized.tar.1 inside /var/log/snowflake-broker/unstanitized.tar. /var/log/snowflake-broker/x.tar:: contains a subset of what /var/log/snowflake-broker/mv contains.

      I think the most complete logs come from /var/log/snowflake-broker/unstanitized.tar, extracting the contained unsanitized.tar.1 and combining it with the contained @400000005cb771341453839c.s.

      === snowflake-server

      /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to 2019-04-11.

      === proxy-go /home/snowflake-proxy/snowflake-proxy-17h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-23h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-29h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-restartless.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-17h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d/unsanitized.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.2.tar:: /home/snowflake-proxy/snowflake-proxy-standalone-29h.log.d/unsanitized.tar:: /home/snowflake-proxy/unsanitized_old.tar::
      Status: needs_review to closed
      Resolution: N/A to fixed

    • Trac closed

      closed

    • David Fifield mentioned in issue #30731 (moved)

      mentioned in issue #30731 (moved)

    • Cecylia Bocovich mentioned in issue #30830 (moved)

      mentioned in issue #30830 (moved)

    • Trac moved to tpo/anti-censorship/pluggable-transports/snowflake#30693 (closed)

      moved to tpo/anti-censorship/pluggable-transports/snowflake#30693 (closed)

    • Trac mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#30731 (closed)

      mentioned in issue tpo/anti-censorship/pluggable-transports/snowflake#30731 (closed)

    Please register or sign in to reply