Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#23325 closed defect (fixed)

Torproject mirror cloud.ipnett.se has connectivity issues

Reported by: hellais Owned by: tpa
Priority: Medium Milestone:
Component: Internal Services/Tor Sysadmin Team Version:
Severity: Major Keywords:
Cc: darkk, weasel, ln5 Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description

I am experiencing issues when connecting to the torproject.org mirror 89.45.235.21 (cloud.ipnett.se) when connecting to it from Vodafone Italia.

Here is the result of a recent mtr:

| mtr -b -w 89.45.235.21
Start: 2017-08-25T12:28:28+0200
HOST: sony-vaio.local                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                                        0.0%    10    7.3   3.0   1.3   7.3   2.2
  2.|-- net-37-119-78-1.cust.vodafonedsl.it (37.119.78.1)  0.0%    10   11.8  14.1  11.7  26.2   4.5
  3.|-- 83.224.40.134                                      0.0%    10   12.6  14.5  12.0  29.5   5.3
  4.|-- 83.224.40.133                                      0.0%    10   13.2  17.3  12.0  27.0   5.7
  5.|-- 83.224.40.225                                     10.0%    10   22.7  23.0  21.6  27.4   1.8
  6.|-- 85.205.14.101                                      0.0%    10   32.3  31.6  23.9  50.9   7.4
  7.|-- 85.205.30.177                                      0.0%    10   35.1  34.3  32.5  37.0   1.4
  8.|-- 92.79.209.89                                       0.0%    10   46.0  41.6  32.5  77.5  13.1
  9.|-- 145.254.2.211                                      0.0%    10   32.7  33.4  31.6  37.4   2.0
 10.|-- 145.254.2.187                                      0.0%    10   41.2  40.1  38.6  43.5   1.5
 11.|-- 145.254.2.187                                      0.0%    10   39.1  43.5  38.3  55.6   7.0
 12.|-- ???                                               100.0    10    0.0   0.0   0.0   0.0   0.0

Here are some mtr results when I first reported this isssue on IRC:

| mtr -b -w 89.45.235.21 
| Start: 2017-07-24T13:47:47+0200 
| HOST: sony-vaio.station Loss% Snt Last Avg Best Wrst StDev 
| 1.|-- vodafone.station (192.168.1.1) 0.0% 10 5.1 3.2 1.7 7.4 1.9 
| 2.|-- net-5-89-113-1.cust.vodafonedsl.it (5.89.113.1) 0.0% 10 10.5 16.0 8.0 36.3 11.4 
| 3.|-- 83.224.40.134 0.0% 10 13.2 9.7 8.4 13.2 1.5 
| 4.|-- 83.224.40.133 0.0% 10 10.5 10.0 8.2 13.2 1.7 
| 5.|-- 83.224.40.225 0.0% 10 18.1 29.1 17.6 68.6 18.0 
| 6.|-- 85.205.14.101 0.0% 10 21.3 24.5 19.7 32.6 4.2 
| 7.|-- 85.205.30.177 0.0% 10 33.8 33.5 29.0 47.9 5.7 
| 8.|-- 92.79.209.89 0.0% 10 33.9 31.3 29.1 33.9 1.7 
| 9.|-- 145.254.2.211 0.0% 10 31.8 33.9 31.6 47.1 4.7 
| 10.|-- 145.254.2.187 0.0% 10 35.8 38.6 35.2 44.6 3.8 
| 11.|-- 145.254.2.187 0.0% 10 35.4 36.6 34.4 48.0 4.1 
| 12.|-- ??? 

| weasel@weschniakowii:~$ mtr -b 5.89.113.15 -w 
| Start: Mon Jul 24 11:49:16 2017 
| HOST: weschniakowii Loss% Snt Last Avg Best Wrst StDev 
| 1.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 
| 2.|-- com-mx80-dc1-r1-1.cloud.ipnett.se (193.11.88.1) 0.0% 10 0.6 2.5 0.4 17.0 5.1 
| 3.|-- 193.11.88.82 0.0% 10 0.5 2.8 0.3 24.3 7.5 
| 4.|-- stockholm-tug-r2.sunet.se (130.242.82.196) 0.0% 10 0.6 0.8 0.5 2.4 0.5 
| 5.|-- stockholm-tug-r1.sunet.se (130.242.4.96) 0.0% 10 0.7 2.8 0.6 17.1 5.1 
| 6.|-- se-tug.nordu.net (109.105.102.17) 0.0% 10 0.7 1.8 0.6 11.6 3.4 
| 7.|-- 109.105.97.25 0.0% 10 0.9 0.9 0.8 1.0 0.0 
| 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 

I hear that I am actually not the only person experiencing this issue (darkk also had similar issues and reported that RIPE atlas shows connectivity issues globally).

This is actually starting to cause quite a few problems for being able to work properly as when I hit this mirror, I end up having to reload the page many times before I am able to get the content I need.

Child Tickets

Change History (16)

comment:1 Changed 2 years ago by ln5

Cc: ln5 added

comment:2 Changed 2 years ago by darkk

!N in tracepath stands for ICMP Network Unreachable

[23:06] *@* ~ $ tracepath 89.45.235.21
 1?: [LOCALHOST]                                         pmtu 1500
 1:  192.168.100.1                                         3.153ms 
 1:  192.168.100.1                                         1.714ms 
 2:  192.168.100.1                                         3.015ms pmtu 1492
 2:  pppoe.95-55-130-1.dynamic.avangarddsl.ru              7.129ms 
 3:  bbn.212-48-204-180.nwtelecom.ru                       6.650ms 
 4:  bbn.212-48-204-197.nwtelecom.ru                       5.952ms 
 5:  rascom-szt-gw.rascom.as20764.net                      7.290ms 
 6:  80-64-96-193.rascom.as20764.net                      16.788ms asymm  9 
 7:  80-64-96-193.rascom.as20764.net                      14.639ms asymm  9 
 8:  m9-cr04-be38.msk.stream-internet.net                 18.960ms !N
     Resume: pmtu 1492 

[23:38] *@* ~ $ mtr -b -w 89.45.235.21
Start: Thu Aug 24 23:38:20 2017
HOST: darkk-ya-laptop                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.100.1                                           0.0%    10    1.3   1.9   0.9   5.3   1.2
  2.|-- pppoe.95-55-130-1.dynamic.avangarddsl.ru (95.55.130.1)  0.0%    10    7.5   8.2   3.4  28.7   7.8
  3.|-- bbn.212-48-204-174.nwtelecom.ru (212.48.204.174)        0.0%    10    4.0   4.5   3.1  11.0   2.2
  4.|-- bbn.212-48-204-197.nwtelecom.ru (212.48.204.197)        0.0%    10    4.1   4.1   2.9   6.0   0.6
  5.|-- rascom-szt-gw.rascom.as20764.net (80.64.101.141)        0.0%    10    5.3   5.2   4.5   7.1   0.6
  6.|-- 80-64-96-118.rascom.as20764.net (80.64.96.118)          0.0%    10   12.4  12.7  11.5  17.8   1.7
  7.|-- ???                                                    100.0    10    0.0   0.0   0.0   0.0   0.0
Last edited 2 years ago by darkk (previous) (diff)

comment:3 Changed 2 years ago by ln5

lg.mtu.ru says MTU is blackholing this particular IP address (/32):

inet.0: 762959 destinations, 2089556 routes (762259 active, 3 holddown, 732 hidden)
Restart Complete
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

89.45.235.21/32    *[BGP/170] 14w5d 14:31:22, MED 0, localpref 200, from 195.34.52.181
                      AS path: ?, validation-state: unverified
                    > to 212.188.29.249 via ae14.66, Push 24125
                    [BGP/170] 14w5d 14:31:22, MED 0, localpref 200, from 195.34.52.182
                      AS path: ?, validation-state: unverified
                    > to 212.188.29.249 via ae14.66, Push 24125
                    [BGP/170] 14w5d 14:31:22, MED 0, localpref 200, from 195.34.52.189
                      AS path: ?, validation-state: unverified
                    > to 212.188.29.249 via ae14.66, Push 24125

comment:4 Changed 2 years ago by ln5

MTU is blocking one more (out of six) dist.tpo addresses: 38.229.72.16

comment:5 Changed 2 years ago by ln5

Regarding Vodafone, their looking glass (or CW's, at https://support.cw.com/index.html?TESTDRIVE=1&LG=1) shows a similar route for this particular /32 (since almost six weeks "Age: 5w4d 19:40:02"):

inet.0: 657301 destinations, 8546599 routes (656813 active, 1 holddown, 4338 hidden)
89.45.235.21/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-201
                Next hop type: Indirect, Next hop index: 0
                Address: 0x2c064dec
                Next-hop reference count: 7327
                                Next hop type: Router, Next hop index: 1049157
                Next hop: 195.2.10.246 via ae6.0 weight 0x1
                Label-switched-path EU_AS-XCR1.FIX-XCR2.SGS-1
                Label operation: Push 693543
                Label TTL action: prop-ttl
                Load balance label: Label 693543: None; 
                Label element ptr: 0xac4be480
                Label parent element ptr: 0x0
                Label element references: 5
                Label element child references: 2
                Label element lsp id: 200
                Session Id: 0x2968
                Next hop: 195.2.10.246 via ae6.0 weight 0x1, selected
                Label-switched-path EU_AS-XCR1.FIX-XCR2.SGS-2
                Label operation: Push 711623
                Label TTL action: prop-ttl
                Load balance label: Label 711623: None; 
                Label element ptr: 0xb02ac100
                Label parent element ptr: 0x0
                Label element references: 5
                Label element child references: 2
                Label element lsp id: 201
                Session Id: 0x2968
                Next hop: 195.2.10.246 via ae6.0 weight 0x1
                Label-switched-path EU_AS-XCR1.FIX-XCR2.SGS-3
                Label operation: Push 716519
                Label TTL action: prop-ttl
                Load balance label: Label 716519: None; 
                Label element ptr: 0x859a0ea80
                Label parent element ptr: 0x0
                Label element references: 5
                Label element child references: 2
                Label element lsp id: 202
                Session Id: 0x2968
                Protocol next hop: 195.2.1.103
                Indirect next hop: 0x56fdb18 1048910 INH Session ID: 0x20f1
                State: 
                Local AS:  1273 Peer AS:  1273
                Age: 5w4d 19:40:02 	Metric: 0 	Metric2: 178 
                Validation State: unverified 
                Announcement bits (5): 0-KRT 7-RT 8-BGP_RT_Background 9-Resolve tree 5 10-Resolve tree 7 
                AS path: 65519 I
                AS path: Recorded
                Communities: 1273:40001
                Accepted
                Localpref: 200
                Router ID: 195.2.1.103

comment:6 Changed 2 years ago by ln5

None of the other five addresses for dist.tpo is blocked like this at Vodafone.

comment:7 Changed 2 years ago by darkk

(for the record) web archive snapshots of LG: m9-cr02.msk with community: 8359:666 & m9-cr03.msk.

comment:8 Changed 2 years ago by cypherpunks

Last edited 2 years ago by cypherpunks (previous) (diff)

comment:9 Changed 2 years ago by darkk

first reported on 13 May 2017

That's interesting coincidence, the route appeared on ~ May 13 22:00 UTC at m9-cr03.msk if I read show bgp output correctly.

comment:10 Changed 2 years ago by cypherpunks

Last edited 2 years ago by cypherpunks (previous) (diff)

comment:11 Changed 2 years ago by cypherpunks

Last edited 2 years ago by cypherpunks (previous) (diff)

comment:12 Changed 2 years ago by cypherpunks

Last edited 2 years ago by cypherpunks (previous) (diff)

comment:13 Changed 2 years ago by ln5

Vodafone has removed their 89.45.235.21/32 route now. hellais, please verify that you don't experience any issues with 89.45.235.21. Thanks.

comment:14 Changed 2 years ago by weasel

Resolution: fixed
Status: newclosed

Thanks for tracking this down and getting it resolved.

comment:15 Changed 2 years ago by darkk

resolved

Well. Number of failures went down from 64 to 40 (same probes were used in both measurements), but IMHO it's still far from "resolved" as connectivity for other addresses is way better.

Two more measurements towards the host using different set of probes: #9262533 and #9263677.

(irrelevant sidenote, couple of probes: in Spain in Norway show TLS MITM from FortiGate)

comment:16 Changed 2 years ago by hellais

FWIW I have stopped experiencing issues from Vodafone Italia, however as @darkk points out, probably there are still users out there having issues.

Note: See TracTickets for help on using tickets.