Opened 6 months ago

Closed 6 months ago

Last modified 5 months ago

#30419 closed defect (fixed)

Apache's server-status page accessible via TPO onion services

Reported by: Parckwart Owned by: anarcat
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Normal Keywords:
Cc: Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

The following Apache server-status pages are accessible, leaking IP addresses of TPO website visitors:

2d5quh2deowe4kpd.onion/server-status
2iqyjmvrkrq5h5mg.onion/server-status
4bflp2c4tnynnbes.onion/server-status
52g5y5karruvc7bz.onion/server-status
54nujbl4qohb5qdp.onion/server-status
bn6kma5cpxill4pe.onion/server-status
bo7uextohjpuqvrh.onion/server-status
bogdyardcfurxcle.onion/server-status
buqlpzbbcyat2jiy.onion/server-status
c5qrls2slxqz6vdw.onion/server-status
dgvdmophvhunawds.onion/server-status
ea5faa5po25cf7fb.onion/server-status
ebxqgaz3dwywcoxl.onion/server-status
expyuzz4wqqyqhjn.onion/server-status
fhny6b7b6sbslc2b.onion/server-status
fqnqc7zix2wblwex.onion/server-status
fr6scuhdp5dqvy7d.onion/server-status
fylvgu5r6gcdadeo.onion/server-status
hzmun3rnnxjhkyhg.onion/server-status
icxe4yp32mq6gm6n.onion/server-status
jqs44zhtxl2uo6gk.onion/server-status
klbl4glo2btuwyok.onion/server-status
krkzagd5yo4bvypt.onion/server-status
kzcx36ytbsm5iogs.onion/server-status
l3xrunzkfufzvw2c.onion/server-status
lfdhmyq24uacliu5.onion/server-status
llhb3u5h3q66ha62.onion/server-status
n46o4uxsej2icp5l.onion/server-status
ngp5wfw5z6ms3ynx.onion/server-status
nraswjtnyrvywxk7.onion/server-status
nwoyhtkk4tloji3j.onion/server-status
qigcb4g4xxbh5ho6.onion/server-status
qrmfuxwgyzk5jdjz.onion/server-status
rqef5a5mebgq46y5.onion/server-status
s2bweojt5vg52e5i.onion/server-status
sbe5fi5cka5l3fqe.onion/server-status
sdscoq7snqtznauu.onion/server-status
tgnv2pssfumdedyw.onion/server-status
tngjm3owsslo3wgo.onion/server-status
vhbbidwvzwhahsrg.onion/server-status
vijs2fmpd72nbqok.onion/server-status
vt5hknv6sblkgf22.onion/server-status
wcgqzqyfi7a6iu62.onion/server-status
x3nelbld33llasqv.onion/server-status
y7pm6of53hzeb7u2.onion/server-status
yabd3wlpvybdnvzg.onion/server-status
yjuwkcxlgo7f7o6s.onion/server-status
yz7lpwfhhzcdyc5y.onion/server-status

Child Tickets

Change History (4)

comment:1 Changed 6 months ago by anarcat

Owner: changed from tpa to anarcat
Status: newassigned

taking a look

comment:2 Changed 6 months ago by anarcat

Resolution: fixed
Status: assignedclosed

The problem was fixed at around midnight UTC. More details might follow.

comment:3 Changed 6 months ago by arma

Thanks Parckwart! Good find.

We believe anarcat fixed it -- if you find anyplace in Tor infrastructure land that still has the issue, please reopen this ticket.

It looks like we added in the problem on March 19, during an apache config file update for apache 2.4.

We've begun the process of trying to figure out if we can learn whether people exploited this issue much in the past six weeks. Our webservers don't really keep logs that help much here (which is a feature in other circumstances: #20928) so it's not straightforward.

anarcat: this seems like the sort of security audit we should want to set up an automated check for, so that it can squeal if some future configuration ever starts revealing this content again. And while I'm thinking of follow-up steps, take a look at
https://riseup.net/en/security/network-security/tor/onionservices-best-practices#be-careful-of-localhost-bypasses

comment:4 Changed 5 months ago by anarcat

Context

As documented in the metrics pages, Tor webservers do not keep logs of visitors. The webserver (Apache) itself keeps those IP addresses in memory during the lifetime of the connection. This information can be disclosed on a /server-status page that is usually visible only to inside monitoring systems. A configuration error was introduced on March 19th 2019 which allowed onion services to access that page which could be used to access that information. This issue was reported in the Trac bugtracker (issue #30419) on May 6th 2019 and was fixed within the hour.

Mitigation

The server-status page was checked and the issue was confirmed. A check in the git history found the bug and resolved it, and an audit was performed to see if the issue was correctly resolved. Analysis of the logs suggests there wasn't a significant increase in requests to /server-status.

Patch that introduced the bug:

commit 8ba7d37b9b2e2431e201752a1eb69a9bcce483e1
Date:   Tue Mar 19 16:50:44 2019 -0400

    port server-status configuration to Apache 2.4
    
    I verified that all hosts run at least Apache 2.4.10-10+deb8u13,
    shipped with Debian jessie.
    
    This is necessary for the apache collector to work.

diff --git a/modules/apache2/files/common/etc/apache2/conf.d/server-status b/modules/apache2/files/common/etc/apache2/conf.d/server-status
index 1a44e9b9..9362bcc2 100644
--- a/modules/apache2/files/common/etc/apache2/conf.d/server-status
+++ b/modules/apache2/files/common/etc/apache2/conf.d/server-status
@@ -11,8 +11,6 @@
     ExtendedStatus on
     <Location /server-status>
         SetHandler server-status
-        Order deny,allow
-        Deny from all
-        Allow from 127.0.0.1
+        Require local
     </Location>
 </IfModule>

Patch that fixed it:

commit 19d5a30ca88fba4aa57d2574774c72d344114b1a
Date:   Mon May 6 19:47:36 2019 -0400

    hide server-status from tor hidden services
    
    This is a hotfix for bug #30419 which correctly identified that the
    server-status pages are accessible when the webserver is accessed
    through the hidden service. It's unclear to me why "local" isn't
    equivalent to "127.0.0.1" but this fixes the problem on
    troodi/trac/ea5faa5po25cf7fb.onion so I'm satisfied.
    
    This was a regression introduced since march 19th, in commit
    8ba7d37b9b2e2431e201752a1eb69a9bcce483e1.

diff --git a/modules/apache2/files/common/etc/apache2/conf.d/server-status b/modules/apache2/files/common/etc/apache2/conf.d/server-status
index 9362bcc2..0da33673 100644
--- a/modules/apache2/files/common/etc/apache2/conf.d/server-status
+++ b/modules/apache2/files/common/etc/apache2/conf.d/server-status
@@ -11,6 +11,6 @@
     ExtendedStatus on
     <Location /server-status>
         SetHandler server-status
-        Require local
+        Require ip 127.0.0.1
     </Location>
 </IfModule>

Timeline

All times in UTC starting on 2019-05-07:

  • 22:56:49 bug #30419 opened
  • 23:36:04 noticed by anarcat
  • 23:38:00 source of the problem identified
  • 23:47:36 patch implemented
  • 23:50:46 patch pushed to puppet
  • 23:50:57 issue claimed by anarcat
  • 00:00:00 (approximate) fix deployed everywhere, checks started
  • 00:38:30 all .onion sites from onion.tpo and ticket audited
  • 00:49:00 this report started
  • 01:14:00 audit of the webserver logs completed
Note: See TracTickets for help on using tickets.