Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#19651 closed task (implemented)

add missing scripts to collector.git

Reported by: iwakeh Owned by: iwakeh
Priority: Medium Milestone: CollecTor 1.0.0
Component: Metrics/CollecTor Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

known to be missing:

create-tarball.sh

Child Tickets

Change History (8)

comment:1 Changed 3 years ago by iwakeh

Status: newneeds_review

Please review my task-19651-create-tar-script branch.

Initial check-in and some tweaks to make it easier to run the script on different instances.

comment:2 Changed 3 years ago by karsten

Status: needs_reviewneeds_revision

Commits until b0a7e58 look good except for one change: Can we set WORKDIR="tarballs"? It's certainly good to make paths configurable, but picking different defaults than what's currently hard-coded is a bit unexpected. Think of the current operator who might pull in this change in a couple of weeks or months and doesn't remember that paths have changed. Happy to make this change, if you agree, and then push changes to master.

comment:3 Changed 3 years ago by iwakeh

Thanks for spotting this!
I intended to use all the paths from the initial script, but this slipped by.
Please correct the WORKDIR setting.

comment:4 Changed 3 years ago by karsten

Status: needs_revisionnew

Fixed and pushed. Leaving this ticket open for any other scripts we might still be missing. Thanks!

comment:5 Changed 3 years ago by iwakeh

Status: newneeds_review

The second script is now added to the create tarball one.
The reasoning behind is that the current use-case runs the scripts together anyway.

One variable setting I could only guess: I assume that archive is the same as the absolute path /srv/collector.torproject.org/collector/archive on main CollecTor.

With the configurable paths I let it run on my mirror and everything worked fine, but took ages, well hours. Maybe, we should up the priority on the improve create tarball topic (mentioned here)?

In addition, I'd like to integrate this script process in the java application, i.e., replace the script and let java handle all. This could also pause the tarball-creation momentarily when new downloads take place and start it as soon the download stops, in general there would be more control.

Well, some thoughts while waiting for the script to finish ;-)

Please review the extended create tarballs script.

comment:6 in reply to:  5 Changed 3 years ago by karsten

Replying to iwakeh:

The second script is now added to the create tarball one.
The reasoning behind is that the current use-case runs the scripts together anyway.

One variable setting I could only guess: I assume that archive is the same as the absolute path /srv/collector.torproject.org/collector/archive on main CollecTor.

Yep, that's the directory.

With the configurable paths I let it run on my mirror and everything worked fine, but took ages, well hours. Maybe, we should up the priority on the improve create tarball topic (mentioned here)?

It does take ages to produce tarballs, but that's also because we're using xz -9e which saves a lot of disk space and bandwidth, at the price of long creation time.

In addition, I'd like to integrate this script process in the java application, i.e., replace the script and let java handle all. This could also pause the tarball-creation momentarily when new downloads take place and start it as soon the download stops, in general there would be more control.

Sure, sounds useful. In the past I tried to append to existing tarballs, which worked for a while but then produced some weird problems. Happy to dig out what that was when we improve this.

However, I wouldn't want to prioritize this over, say, syncing between CollecTor instances.

Well, some thoughts while waiting for the script to finish ;-)

Please review the extended create tarballs script.

Looks good, merged! Thanks.

comment:7 Changed 3 years ago by iwakeh

Resolution: implemented
Status: needs_reviewclosed

I think these were all missing puzzle pieces. If not, we can reopen.

Closing now.
Thanks!

comment:8 Changed 3 years ago by iwakeh

Milestone: CollecTor 1.0.0

Added to milestone for first release.

Note: See TracTickets for help on using tickets.