Opened 11 months ago

Closed 2 months ago

Last modified 2 months ago

#31690 closed project (fixed)

study trac.torproject.org archival possibilities

Reported by: anarcat Owned by: qbi
Priority: Medium Milestone:
Component: Internal Services/Service - trac Version:
Severity: Normal Keywords: tickets-migration
Cc: gk, intrigeri, hiro, gaba, mcs, boklm Actual Points:
Parent ID: #30857 Points:
Reviewer: Sponsor:

Description (last modified by anarcat)

this is a split out of #30857 to discuss specifically the question of if/how to archive trac.torproject.org.

As mentioned in that ticket, there are a few options on how to deal with trac, provided we have another system we want to use:

  1. the golden redirect set: every migrated ticket and wiki page has a corresponding ticket/wiki page in GitLab and a gigantic set of redirection rules makes sure they are mapped correctly. probably impractical, but solves the maintenance problem possibly forever.
  1. read-only Trac: user creation is disabled and existing users are locked from making any change to the site. only a temporary or intermediate measure.
  1. fossilization: Trac is turned into a static HTML site that can be mirrored like any other site. can be a long term solution and a good compromise with a possibly impossible to design and therefore failing (because incomplete) set of redirection rules.
  1. destruction: we hate the web and pretend link rot is not a problem and just get rid of the old site, assuming everything is migrated and people will find their stuff eventually. probably not an option.
  1. redirect to the wayback machine: like fossilization, but delegate to the internet archive and hope for the best

Archive team work

With my archive team hat, I was able to coordinate a first archival of the website during the summer of 2019, as documented in #30857. This is an attempt at doing "3. fossilization".

All those jobs end up populating the wayback machine at web.archive.org, but are also available as WARC files, an archival format for web pages.

A first archival of all tickets up to #30856 has been performed here:

https://archive.fart.website/archivebot/viewer/job/5vytc

It's about 600MB of compressed HTML (more or less).

Then a full archival job of the entire site was performed here:

https://archive.fart.website/archivebot/viewer/job/bpu6j

It created about 10GB of WARC files, crawled over 730,000 links (including external sites linked from Trac) and 105.34GiB of data. It took over 5 days:

2019-06-17 01:49:02,514 - wpull.application.tasks.stats - INFO - Duration: 5 days, 7:32:55. Speed: 0.0 B/s.
2019-06-17 01:49:02,514 - wpull.application.tasks.stats - INFO - Downloaded: 732488 files, 105.4 GiB.

Other statistics

Archiving the server itself means dealing with:

  • ~1GB of attachments
  • 4GB PostgreSQL database

The actual server uses around 25GB of disk space because of random junk here and there but that's the very minimum it can be trimmed down to. naturally, we can keep *that* data forever, the problem is keeping the app running on top of that... That would be some incarnation of "4. destruction".

Child Tickets

Change History (3)

comment:1 Changed 11 months ago by gaba

Keywords: tickets-migration added

comment:2 Changed 3 months ago by anarcat

Description: modified (diff)

a conversation is happening over tor-internal about which option to choose between fossilization (3), destruction (4) and "just use the internet archive (added as option 5) in the above list. so far (3) has two votes, but I've warned about the complexity of the problem and personally favor option 5 at this stage.

the golden redirect set (option 1) has been ruled out for now because the gitlab migration has stalled and we are unsure that we can actually reliably migrate those tickets.

comment:3 Changed 2 months ago by anarcat

Resolution: fixed
Status: newclosed

we are going to do option 1 and 2, that is:

  • the golden redirect set: content from trac (issues and wiki) *will* be migrated to gitlab, and redirects will be deployed (#34373)
  • trac readonly: on top of that, we'll set trac readonly for 6 months for the migration to occur and people to refer back to the tickets in case of problems with the migration (#34374)

this ticket will be closed in favor of action for the above.

Last edited 2 months ago by anarcat (previous) (diff)
Note: See TracTickets for help on using tickets.