Opened 3 months ago

Closed 12 days ago

#31718 closed defect (fixed)

Update DNS records for .ooni.torproject.org domains

Reported by: hellais Owned by: anarcat
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: gaba Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

To make it easier for us to manage where these domains point to it would be great if the records for the domain explorer.ooni.torproject.org were to point to explorer.ooni.io and the record for ooni.torproject.org pointed to ooni.io.

The most high priority is the update of explorer.ooni.torproject.org as we are launching that today and we still have places where we link to explorer.ooni.torproject.org instead of explorer.ooni.io.

Child Tickets

Change History (27)

comment:1 Changed 3 months ago by weasel

I'm not convinced we want to point more of our torproject.org namespace to the outside. I'll bring this up with the rest of the team.

comment:2 Changed 3 months ago by arma

Another option, if we don't like having .torproject.org sites running on non-TPA machines, would be to run a tiny webserver as the .torproject.org site, which sends an http-level redirect to the external site?

I mention it because that http redirect is happening now already, just on the remote site.

comment:3 Changed 3 months ago by anarcat

hellais, do you want an actual CNAME (ie. that the user doesn't know they get redirected to ooni) or a redirect (that the user *does* end up on ooni.io)?

i do agree that it's unconventional to do those things for us. we usually point *.torproject.net at external resources.

is this domain used by non-HTTP clients?

comment:4 Changed 3 months ago by hellais

Currently this is already happening though, explorer.ooni.torproject.org being a CNAME to explorer.ooni.io, I mistakenly thought this was not currently the case when opening the ticket, but this is already happening and no change is necessary on this front.

It was done this way specifically to make it easier for us to more independently handle how we serve requests to users hitting out website from the various domains that were distributed.

The website ooni.org & ooni.io & ooni.torproject.org is still running on tpo infrastructure, but we would like to change that to reduce the complexity of having something hosted on system where people need LDAP access to administer it.

Our preference would be that we setup a CNAME record for ooni.torproject.org that points to ooni.io or ooni.org so that we are able to on our own setup a redirect, if desirable, or handle the requests directly by keeping the ooni.torproject.org domain (we probably will do this in the beginning).

would be to run a tiny webserver as the .torproject.org site, which sends an http-level redirect to the external site?

hellais, do you want an actual CNAME (ie. that the user doesn't know they get redirected to ooni) or a redirect (that the user *does* end up on ooni.io)?

It would be preferable if we could get a CNAME record, so that we can manage how redirects are handled autonomously.

is this domain used by non-HTTP clients?

It's the domain used for our primary website. We don't make any assumption as to what type of client is going to access it. I suppose most modern browsers will do HTTPS.

comment:5 Changed 3 months ago by hellais

Another record which is currently setup in a similar fashion is the CNAME for

get.ooni.torproject.org. 3599	IN	CNAME	get.ooni.io.

To be clear it's not a big problem if the policy WRT to setting up CNAME records has changed, I just need to be aware of it and plan according to it.

This is probably also a good opportunity to do some cleanup of other *.ooni.torproject.org domains as we are trying to simplify our infrastructure and reduce our devops cognitive load by simplifying our infrastructure.

comment:6 Changed 3 months ago by anarcat

i don't exactly know what the policy is regarding CNAMEs, to be honest. :) the best source I know of is this:

https://help.torproject.org/tsa/doc/naming-scheme/

... which outlines the distinction between TPO (torproject.org) and TPN (torproject.net) that weasel was refering to. The problem might not be CNAMEs per se, but pointing to outside stuff.

Another thing is that CNAMEs are not a great way to move stuff around, because they are transparent to clients. An web browser or crawler will not treat a CNAME as "this is now hosted over there", it's just an alias. For those kind of transitions, you want to do a HTTP redirect, that is respond with a 301 (Moved Permanently) or 302 (Found) status code:

https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection

Then we can deprecate the *.ooni.tpo namespace and eventually transition to ooni.io cleanly.

This is why I was asking about non-HTTP (and non-HTTPS) clients: those redirections will work only for HTTP clients. If you have people using this over SSH or Git or whatever non-HTTP protocol, those would break of course.

(Sorry if you already know all of this about HTTP status codes vs CNAMEs, but I thought it was useful to get back to the specs to clarify my thoughts.)

comment:7 Changed 3 months ago by anarcat

Owner: changed from tpa to anarcat
Status: newassigned

we agreed that we'd add a CNAME record and keep the CNAMEs until may 7th 2020, at which point they'd turn into HTTP redirects. we'll do this tomorrow at 1200 EDT (1600 UTC).

comment:8 Changed 3 months ago by anarcat

seems to me that just adding the CNAME will not be enough, as there are many other things to cleanup. the main procedure should be:

  1. remove ooni.torproject.org from tor-puppet/modules/roles/misc/static-components.yaml
  2. remove ooni from auto-dns, push
  3. add the CNAME, push

Other things to cleanup include:

letsencrypt-domains/domains:46:ooni.torproject.org
tor-nagios/config/nagios-master.cfg:1330:    name: mirror static sync - ooni
tor-nagios/config/nagios-master.cfg:1331:    check: "dsa_check_staticsync!ooni.torproject.org"
tor-puppet/modules/sudo/files/sudoers:63:%ooni			STATICMASTER=(ooni)			ALL
tor-puppet/modules/sudo/files/sudoers:95:%ooni			STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component ooni.torproject.org, /usr/local/bin/static-update-component ooni.torproject.org
tor-puppet/modules/roles/manifests/static_mirror_web.pp:74:  ssl::service { 'ooni.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
tor-puppet/modules/roles/manifests/static_mirror_onion.pp:37:      'ooni.torproject.org',
tor-puppet/onions/onionbalance-services.yaml:17: [...]

I'm particularly concerned about let's encrypt - wouldn't adding the cname break the X509 cert, as we would now point to another server?

Last edited 3 months ago by anarcat (previous) (diff)

comment:9 Changed 3 months ago by hellais

So we looked into this with @anarcat and encountered the following issues:

  • The current setup has both HSTS and certificate pinning enabled for the ooni.torproject.org website
  • It is not straightforward to do custom HTTPS changes on the current ooni hosting service (netlify)

Since the maxage for the certificate pinning is set to 60 days we will need to wait for that amount of time before we are able to migrate over.

In the meantime @anarcat is going to see how to disable the certificate pinning headers from the ooni.torproject.org host config, so that we can begin waiting the 60 days after which we can proceed with the CNAME plan as mentioned above.

comment:10 Changed 3 months ago by anarcat

i have disabled certificate pinning on ooni.torproject.org around 15 minutes ago. it should therefore expire in 60 days exactly, which is about on saturday november 16th at 19:30UTC. assuming we don't want to do this transition on a saturday, we should probably look into this again on november 18th.

i documented a bit how HPKP works in:

https://help.torproject.org/tsa/howto/letsencrypt/#index3h1

comment:11 Changed 2 months ago by gaba

Cc: gaba added

comment:12 Changed 4 weeks ago by anarcat

we're getting ready for this transition again, which should happen some time next week.

@hellais, are you around next week? should we carry on the plan as expected?

where should the CNAME point to? ooni.org? `www.ooni.org?

thanks!

comment:13 in reply to:  12 Changed 4 weeks ago by hellais

Replying to anarcat:

we're getting ready for this transition again, which should happen some time next week.

Thanks for following up on this @anarcat!

I had marked on my calendar Nov 18th as the date we can do the migrateion

@hellais, are you around next week? should we carry on the plan as expected?

Yes I am going to be around and let's proceed as planned with doing the migration on Nov 18th. Does that work for you?

where should the CNAME point to? ooni.org? `www.ooni.org?

The CNAME should point to ooni.org and in theory that would work with out netlify based host. I don't think I have ever done this, though, so it's useful if we are both online to coordinate on this in realtime.

I am going to be mostly online on Nov 18th from ~10:00 UTC - 18:00 UTC. Should we try to meet online at around 16:00 UTC?

comment:14 Changed 4 weeks ago by anarcat

sounds good, let's meet on monday 1600UTC, which is 17:00 in paris and 11:00 in montreal.

comment:15 Changed 3 weeks ago by anarcat

Status: assignedaccepted

this has now been deployed, with the following three patches, in dns/domains.git:

From 471f529240673d324a66a1258f6acc257857f964 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Mon, 18 Nov 2019 11:50:50 -0500
Subject: [PATCH] add ooni.tpo CNAME (#31718)

---
 torproject.org | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/torproject.org b/torproject.org
index 8e62797..87623f1 100644
--- a/torproject.org
+++ b/torproject.org
@@ -100,6 +100,7 @@ rsync.media		IN	CNAME	listera
 metrics			IN	CNAME	meronense
 munin			IN	CNAME	schmitzi
 nagios			IN	CNAME	hetzner-hel1-01
+ooni			IN	CNAME	ooni.io.
 get.ooni		IN	CNAME	get.ooni.io.
 measurements.ooni	IN	CNAME	measurements.ooni.io.
 explorer.ooni		IN	CNAME	explorer.ooni.io.
@@ -168,7 +169,6 @@ help				IN	CNAME	static
 lektor-staging			IN	CNAME	static
 newsletter			IN	CNAME	static
 nyx				IN	CNAME	static
-; ooni A/AAAA records via services-auto
 openpgpkey			IN	CNAME	static
 rbm				IN	CNAME	static
 rpm				IN	CNAME	static
-- 
2.20.1

in dns/auto-dns.git:

From 7a1229bc1d0e4b92ee75712942eba146db9adee9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Mon, 18 Nov 2019 11:48:59 -0500
Subject: [PATCH] retire ooni.tpo, will be a CNAME (#31718

---
 services/ooni.torproject.org.service | 7 -------
 1 file changed, 7 deletions(-)
 delete mode 100644 services/ooni.torproject.org.service

diff --git a/services/ooni.torproject.org.service b/services/ooni.torproject.org.service
deleted file mode 100644
index ec2a1f2..0000000
--- a/services/ooni.torproject.org.service
+++ /dev/null
@@ -1,7 +0,0 @@
----
-ttl: 150
-hosts:
-  default:
-    - hetzner-hel1-03.torproject.org
-    - listera.torproject.org
-# vim:syn=yaml:
-- 
2.20.1

... and tor-puppet.git:

From 9cc7af7889ba9b7fd9b167591c30e5baa395acf6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Mon, 18 Nov 2019 11:44:10 -0500
Subject: [PATCH] retire ooni.tpo, will be a CNAME (#31718)

---
 modules/roles/misc/static-components.yaml | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/modules/roles/misc/static-components.yaml b/modules/roles/misc/static-components.yaml
index 9151c5f6..a810c5bd 100644
--- a/modules/roles/misc/static-components.yaml
+++ b/modules/roles/misc/static-components.yaml
@@ -47,9 +47,6 @@ components:
   help.torproject.org:
     master: staticiforme.torproject.org
     source: staticiforme.torproject.org:/srv/help-master.torproject.org/output
-  ooni.torproject.org:
-    master: staticiforme.torproject.org
-    source: staticiforme.torproject.org:/home/ooni/website
   openpgpkey.torproject.org:
     master: staticiforme.torproject.org
     source: alberti.torproject.org:/srv/db.torproject.org/keyrings/openpgpkey
-- 
2.20.1

@hellais renewed the domain with netlify and the new site seems to be online and working.

i still have my own cleanup to do, but the synchronous, "OMG IS THIS GOING TO WORK" step is over, i believe.

comment:16 Changed 3 weeks ago by anarcat

i removed the nagios check and let's encrypt cert, then also cleaned this up in puppet:

From b8e3ebc8f10c9b2e6654c84e85291c277b861637 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= <anarcat@debian.org>
Date: Mon, 18 Nov 2019 12:08:12 -0500
Subject: [PATCH] remove remaining traces of ooni.tpo mirror (#31718)

---
 modules/roles/manifests/static_mirror_onion.pp                 | 3 ++-
 modules/roles/manifests/static_mirror_web.pp                   | 2 +-
 .../roles/templates/static-mirroring/vhost/static-vhosts.erb   | 1 -
 modules/sudo/files/sudoers                                     | 2 --
 4 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/modules/roles/manifests/static_mirror_onion.pp b/modules/roles/manifests/static_mirror_onion.pp
index d9c15fce..706783cd 100644
--- a/modules/roles/manifests/static_mirror_onion.pp
+++ b/modules/roles/manifests/static_mirror_onion.pp
@@ -34,7 +34,6 @@ class roles::static_mirror_onion {
       'nyx.torproject.org',
       'onion.torproject.org',
       'onionperf.torproject.org',
-      'ooni.torproject.org',
       'openpgpkey.torproject.org',
       'rbm.torproject.org',
       'research.torproject.org',
@@ -56,5 +55,7 @@ class roles::static_mirror_onion {
       ensure => 'ifstatic';
     'spec.torproject.org':
       ensure => 'present';
+    'ooni.torproject.org':
+      ensure => 'absent';
   }
 }
diff --git a/modules/roles/manifests/static_mirror_web.pp b/modules/roles/manifests/static_mirror_web.pp
index 997140b7..73859c41 100644
--- a/modules/roles/manifests/static_mirror_web.pp
+++ b/modules/roles/manifests/static_mirror_web.pp
@@ -65,7 +65,7 @@ class roles::static_mirror_web {
   ssl::service { 'nyx.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
   ssl::service { 'onion.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
   ssl::service { 'onionperf.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
-  ssl::service { 'ooni.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
+  ssl::service { 'ooni.torproject.org': ensure => 'absent', notify  => Exec['service apache2 reload'], key => true, }
   ssl::service { 'openpgpkey.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
   ssl::service { 'rbm.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
   ssl::service { 'research.torproject.org': ensure => 'ifstatic', notify  => Exec['service apache2 reload'], key => true, }
diff --git a/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb b/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb
index a49d64b5..30fd426b 100644
--- a/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb
+++ b/modules/roles/templates/static-mirroring/vhost/static-vhosts.erb
@@ -152,7 +152,6 @@ vhost(lines, "newsletter.torproject.org")
 vhost(lines, "nyx.torproject.org")
 vhost(lines, "onion.torproject.org")
 vhost(lines, "onionperf.torproject.org")
-vhost(lines, "ooni.torproject.org")
 vhost(lines, "openpgpkey.torproject.org", :extra => true)
 vhost(lines, "rbm.torproject.org")
 vhost(lines, "research.torproject.org")
diff --git a/modules/sudo/files/sudoers b/modules/sudo/files/sudoers
index 39156276..90b2bcbc 100644
--- a/modules/sudo/files/sudoers
+++ b/modules/sudo/files/sudoers
@@ -59,7 +59,6 @@ letsencrypt		nevii=(dnsadm)				NOPASSWD: /srv/dns.torproject.org/bin/update
 %metrics		meronense=(metrics)			ALL
 %onionoo		ONIONOOHOSTS=(onionoo)			ALL
 %onionoo		ONIONOOHOSTS=(onionoo-unpriv)		ALL
-%ooni			STATICMASTER=(ooni)			ALL
 %stem			STATICMASTER=(stem)			ALL
 %nyx			STATICMASTER=(nyx)			ALL
 %rtfolks		rude=(rtstuff)				ALL
@@ -89,7 +88,6 @@ exonerator		materculae=(exonerator-web)		NOPASSWD:		ALL
 %globe			STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component globe.torproject.org, /usr/local/bin/static-update-component globe.torproject.org
 %consensus-health	henryi=(mirroradm)		NOPASSWD: /usr/local/bin/static-master-update-component consensus-health.torproject.org, /usr/local/bin/static-update-component consensus-health.torproject.org
 %torwww,%metrics		STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component onionperf.torproject.org, /usr/local/bin/static-update-component onionperf.torproject.org
-%ooni			STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component ooni.torproject.org, /usr/local/bin/static-update-component ooni.torproject.org
 %snowflake		STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component snowflake.torproject.org, /usr/local/bin/static-update-component snowflake.torproject.org
 %stem			STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component stem.torproject.org, /usr/local/bin/static-update-component stem.torproject.org
 %nyx			STATICMASTER=(mirroradm)	NOPASSWD: /usr/local/bin/static-master-update-component nyx.torproject.org, /usr/local/bin/static-update-component stem.torproject.org
-- 
2.20.1

finally, i need to do documentation and we need to decide if/when we do HTTP redirects instead of CNAMEs here to finalize this transition. but i guess that OONI can do those redirects themselves, when they want to as well...

comment:17 Changed 3 weeks ago by anarcat

i updated the documentation on how to remove a static component here:

https://help.torproject.org/tsa/howto/static-component/

the only thing remaining is to remove the user/group, and the actual files on staticiforme (and mirrors?)

@hellais, can i remove the actual site? how about the users? shouldn't i be removing users that were created just for the purpose of managing this website?

thanks!

comment:18 Changed 3 weeks ago by hellais

@hellais, can i remove the actual site? how about the users? shouldn't i be removing users that were created just for the purpose of managing this website?

Yes go ahead and remove the actual site and the user (I assume you mean the ooni user, are there others?).

comment:19 Changed 3 weeks ago by anarcat

no, i also mean the actual users that have ooni as a group. those are:

  • art
  • aagbsn
  • andz
  • darkk
  • agrabeli

I assume at least *some* of them should keep their accesses, but I was wondering if some were created *just* for the purpose of updating the website and should be removed...

comment:20 Changed 3 weeks ago by gaba

Some of those users are not longer with Tor/OONI like darkk.

@anarcat, should we check on servers for old accounts that are no longer being use?

comment:21 Changed 3 weeks ago by hellais

@anarcat the only ones you should keep of the above list are:

  • art
  • agrabeli

comment:22 Changed 3 weeks ago by anarcat

alright, i'll retire the other three accounts, thanks!

comment:23 Changed 3 weeks ago by anarcat

I have disbled ("locked") the following users:

  • aagbsn
  • andz
  • darkk

i've moved the ooni home directory on staticiforme and mirrors, and scheduled it for deletion in 7 days, just as a safety measure.

when that is done, the ooni group and users can be removed.

Last edited 3 weeks ago by anarcat (previous) (diff)

comment:24 Changed 3 weeks ago by anarcat

i should note that I created #32558 to followup on what happens with such email accounts after lockout.

comment:25 Changed 2 weeks ago by anarcat

it turns out locking out those users was probably a mistake, as some if not all of them are still on tor-internal. my mistake. i have restored their accesses, although I have lost their passwords for now. i'm looking in the backups to see if i can restore those hashes as well. what i specifically did was:

  • restore keyFingerPrint (based on the account-keyring git repo)
  • delete accountStatus
  • delete shadowExpire
  • remove the ooni group membership from all users

I'm trying to see if i can coerce our backup system to give us a view on those old hashes now.

comment:26 Changed 2 weeks ago by anarcat

i restored the passwords as well now.

comment:27 Changed 12 days ago by anarcat

Resolution: fixed
Status: acceptedclosed

i removed the ooni user and group now as well, and the files are gone. we're all clear now, closing.

Note: See TracTickets for help on using tickets.