Opened 8 years ago

Closed 7 years ago

#4342 closed task (implemented)

move gettor to a tpo machine

Reported by: erinn Owned by: kaner
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Keywords:
Cc: kaner, weasel, phobos Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

Would it be possible to migrate the existing gettor service to a tpo machine, possibly on our new machine? I don't know if that has all its VMs pre-provisioned, but if it doesn't, it would be awesome if we could put gettor on it, since kaner is having issues with diskspace and bandwidth (i.e., TBBs take up a lot of room and take a long time to transfer.) Adding him to Cc since he can more clearly explain what kinds of resources it needs better than I can.

Child Tickets

Attachments (2)

0001-fix-the-url-to-the-code.patch (867 bytes) - added by phobos 8 years ago.
0002-point-at-the-right-rsync-host.patch (726 bytes) - added by phobos 8 years ago.

Download all attachments as: .zip

Change History (39)

comment:1 Changed 8 years ago by phobos

gettor is on a tpo machine. There are three problems with the current server;

  1. it mirrors all of torproject.org/dist. This is a problem when we have 2+ versions of TBB on the system. I manually purge all of the old versions to not waste disk space.
  2. we log everything and keep it forever.
  3. we seem to have three versions of gettor on the machine, each with past logs, etc.

24GB of disk space should be plenty for a mailing bot. I suggest we purge logs after some time period, like 14 days, or at least rotate and compress them daily, and then purge after 14 days.

As for torproject.org/dist being full, i'll purge it now and free up 12GB of space or so.

As for transfer time, the two machines (vescum and dnsel2) are on the same physical host, all transfers happen over virtual interfaces.

comment:2 Changed 8 years ago by phobos

vescum purge done. dnsel2 sync done.

comment:3 Changed 8 years ago by phobos

We could also free up 9GB if we could not store all of the bridge desc on the server too.

comment:4 in reply to:  1 ; Changed 8 years ago by kaner

Replying to phobos:

gettor is on a tpo machine. There are three problems with the current server;

  1. it mirrors all of torproject.org/dist. This is a problem when we have 2+ versions of TBB on the system. I manually purge all of the old versions to not waste disk space.
  2. we log everything and keep it forever.
  3. we seem to have three versions of gettor on the machine, each with past logs, etc.

24GB of disk space should be plenty for a mailing bot. I suggest we purge logs after some time period, like 14 days, or at least rotate and compress them daily, and then purge after 14 days.

As for torproject.org/dist being full, i'll purge it now and free up 12GB of space or so.

As for transfer time, the two machines (vescum and dnsel2) are on the same physical host, all transfers happen over virtual interfaces.

The logs are 140MB currently, and not much of a problem.

As far as I know, there are two versions of GetTor on the host. One is the productive deployed GetTor, the other is a version for testing that is necessary. Where is the third version located? We should purge it.

The problem really was the /dist/ dir getting larger and larger over the past time. Syncing takes ages (even though both host seem to be running on the same hardware), and disk space was getting low. If the hosts are on the same machine, maybe we could magically mount or link /dist/ into GetTor's file system somehow?

If we can get rid of the 9GB bridge descs and /dist/ is kept clean, we can probably stay on that host. Thanks for freeing /dist/ for now.

comment:5 Changed 8 years ago by phobos

The sync I did last night took 63 seconds. Not sure if that counts as 'forever' to you. I don't suggest we mount file systems across hosts.

comment:6 in reply to:  1 ; Changed 8 years ago by erinn

Replying to phobos:

gettor is on a tpo machine.

Ah, my mistake. I thought it wasn't because kaner is the one setting up accounts and it didn't seem to be administered the same as the other machines.

comment:7 in reply to:  5 Changed 8 years ago by kaner

Replying to phobos:

The sync I did last night took 63 seconds. Not sure if that counts as 'forever' to you. I don't suggest we mount file systems across hosts.

Just because I don't want to look like a fool, this is from a sync I just did:

real 82m10.373s
user 0m18.961s
sys 0m34.434s

:-)

comment:8 in reply to:  6 Changed 8 years ago by phobos

Status: newaccepted

Replying to erinn:

Ah, my mistake. I thought it wasn't because kaner is the one setting up accounts and it didn't seem to be administered the same as the other machines.

It's a tpo machine, just not an ldap machine.

comment:9 Changed 8 years ago by kaner

So, should we close this ticket?

Changed 8 years ago by phobos

Changed 8 years ago by phobos

comment:10 Changed 8 years ago by phobos

I attached two patches to the gettor code. one fixes a documentation issue, the other uses the better rsync host for website mirroring. Not sure if packages.py will like the full rsync url, but you get the idea. rsync.tpo is hosted in austria and is frequently bandwidth constrained. aroides is hosted on the same hardware as gettor. aroides is not bandwidth constrained.

comment:11 Changed 8 years ago by kaner

Nice:

real 1m28.840s
user 0m15.173s
sys 0m25.142s

comment:12 Changed 8 years ago by kaner

Patches added and deployed.

comment:13 in reply to:  12 ; Changed 8 years ago by phobos

Replying to kaner:

Patches added and deployed.

Great.

Just a nitpick, you changed one of my patches. The copyright notice is only the word copyright or the encircled c symbol, the year of first publication, and the owner. See http://www.copyright.gov/circs/circ03.pdf. The copyright automatically lasts 120 years from the year of first publication. Anyway, great. rsync is faster.

comment:14 in reply to:  4 ; Changed 8 years ago by phobos

Replying to kaner:

As far as I know, there are two versions of GetTor on the host. One is the productive deployed GetTor, the other is a version for testing that is necessary. Where is the third version located? We should purge it.

There's /home/gettor, /home/gettor2, and then possibly another gettor inside /home/gettor.

If we can get rid of the 9GB bridge descs and /dist/ is kept clean, we can probably stay on that host. Thanks for freeing /dist/ for now.

I moved these off the system tonight. 14GB usable space now.

comment:15 in reply to:  13 Changed 8 years ago by kaner

Replying to phobos:

Just a nitpick, you changed one of my patches. The copyright notice is only the word copyright or the encircled c symbol, the year of first publication, and the owner. See http://www.copyright.gov/circs/circ03.pdf. The copyright automatically lasts 120 years from the year of first publication. Anyway, great. rsync is faster.

Changed it back.

comment:16 in reply to:  14 Changed 8 years ago by kaner

Replying to phobos:

Replying to kaner:

As far as I know, there are two versions of GetTor on the host. One is the productive deployed GetTor, the other is a version for testing that is necessary. Where is the third version located? We should purge it.

There's /home/gettor, /home/gettor2, and then possibly another gettor inside /home/gettor.

/home/gettor is the deployed, productive version of GetTor.
/home/gettor2 contains a testing version. It is necessary to test changes so they don't break the productive version.

There isn't another one inside /home/gettor that I could find.

If we can get rid of the 9GB bridge descs and /dist/ is kept clean, we can probably stay on that host. Thanks for freeing /dist/ for now.

I moved these off the system tonight. 14GB usable space now.

Excellent. Turned on the GetTor package fetching & packaging cron job.

comment:17 Changed 8 years ago by phobos

Ok, time to create a new machine for a new gettor instance. Kaner, when would be a good time to have you setup gettor on the new machine? I'll schedule a migration for when you are around. No rush.

comment:18 Changed 8 years ago by phobos

The next steps here are to:

  1. create a new vm
  2. see if gettor likes current python in debian 6.0.4
  3. standardize gettor such that all configs and changes live in /srv/gettor.torproject.org/ rather than strewn about the system
  4. migrate dns entries over to new VM
  5. shut down old gettor and kill the old system.

comment:19 Changed 8 years ago by kaner

I can do steps 2 and 3. [3 should actually be a matter of configuration]

comment:20 Changed 8 years ago by weasel

Cc: weasel added

comment:21 Changed 8 years ago by weasel

There is now a getulum.torproject.org.

It has an /srv/gettor.torproject.org.

There also is a gettor role account, accessible using sudo by people in the corresponding group.

~gettor/.procmailrc is a sample procmailrc that might just do all you want it to do, including dkim checking.

Now wants installing gettor into /srv/gettor.torproject.org/gettor or wherever.

Once that works, we uppate DNS.

comment:22 Changed 8 years ago by kaner

Please install the package gettext on getulum. TIA

comment:23 Changed 8 years ago by weasel

So what's the status of the move?

comment:24 in reply to:  23 ; Changed 8 years ago by kaner

Replying to weasel:

So what's the status of the move?

Ongoing. Usually I'm able of moving things forward on weekends. Is there any rush or deadlines I should be aware of?

comment:25 in reply to:  24 Changed 8 years ago by phobos

Owner: changed from phobos to kaner
Status: acceptedassigned

Replying to kaner:

Replying to weasel:

So what's the status of the move?

Ongoing. Usually I'm able of moving things forward on weekends. Is there any rush or deadlines I should be aware of?

no rush nor deadlines. carry on. remain calm.

comment:26 Changed 7 years ago by kaner

Can we do this on the weekend? I think the gettor installation on getulum is ready. But I'd like to do the switch at a time when I have more free time at hand to fix things if they break.

comment:27 Changed 7 years ago by kaner

Update: I'll test Karsten's stats script and after that I think we're okay to move the DNS entries.

comment:28 Changed 7 years ago by kaner

Seems like the script also wants the package "python-tk" installed. I'll open a ticket for that.

comment:29 in reply to:  28 Changed 7 years ago by rransom

Replying to kaner:

Seems like the script also wants the package "python-tk" installed. I'll open a ticket for that.

Which script are you talking about? Are you absolutely certain that it will not run without python-tk installed?

comment:30 Changed 7 years ago by kaner

{{{Traceback (most recent call last):

File "./PlotStat.py", line 5, in <module>

import pylab

File "/usr/lib/pymodules/python2.6/pylab.py", line 1, in <module>

from matplotlib.pylab import *

File "/usr/lib/pymodules/python2.6/matplotlib/pylab.py", line 247, in <module>

from matplotlib.pyplot import *

File "/usr/lib/pymodules/python2.6/matplotlib/pyplot.py", line 78, in <module>

new_figure_manager, draw_if_interactive, show = pylab_setup()

File "/usr/lib/pymodules/python2.6/matplotlib/backends/init.py", line 25, in pylab_setup

globals(),locals(),[backend_name])

File "/usr/lib/pymodules/python2.6/matplotlib/backends/backend_tkagg.py", line 7, in <module>

import Tkinter as Tk, FileDialog

File "/usr/lib/python2.6/lib-tk/Tkinter.py", line 42, in <module>

raise ImportError, str(msg) + ', please install the python-tk package'

ImportError: No module named _tkinter, please install the python-tk package}}}

comment:31 in reply to:  30 Changed 7 years ago by rransom

Replying to kaner:

  File "/usr/lib/pymodules/python2.6/matplotlib/backends/__init__.py", line 25, in pylab_setup
    globals(),locals(),[backend_name])
  File "/usr/lib/pymodules/python2.6/matplotlib/backends/backend_tkagg.py", line 7, in <module>
    import Tkinter as Tk, FileDialog
  File "/usr/lib/python2.6/lib-tk/Tkinter.py", line 42, in <module>
    raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package

Ah. Try using the ‘Agg’ backend instead. (See page 24 (PDF page 34) of /usr/share/doc/python-matplotlib-doc/Matplotlib.pdf .)

comment:32 Changed 7 years ago by phobos

Our hosting donor would like to reclaim this IP range and shut down the old cluster on which the current gettor lives. What are next steps towards migrating this to the tpo vm?

comment:33 Changed 7 years ago by phobos

Cc: phobos added

comment:34 Changed 7 years ago by phobos

Component: CompanyTor Sysadmin Team

comment:35 in reply to:  32 Changed 7 years ago by kaner

Replying to phobos:

Our hosting donor would like to reclaim this IP range and shut down the old cluster on which the current gettor lives. What are next steps towards migrating this to the tpo vm?

Just do it. :-) (Switch DNS entries)

comment:36 Changed 7 years ago by phobos

Ok, done. gettor is now running on the new vm, getulum. The old VM is still up just in case.

comment:37 Changed 7 years ago by phobos

Resolution: implemented
Status: assignedclosed

and we've been running for three weeks. calling this done.

Note: See TracTickets for help on using tickets.