Tor stuck in 25% Loading networkstatus consensus

changed milestone to %Tor: 0.3.4.x-final

added component::core tor/tor milestone::Tor: 0.3.4.x-final priority::medium reporter::loskiq resolution::fixed reviewer::nickm severity::normal status::closed type::defect version::tor 0.3.4.9 labels

Trac:
Username: loskiq
Component: - Select a component to Core Tor/Tor
Cc: N/A to loskiq@gmail.com

P.S. The attached logs belongs to the Tor client, not to the bridge

Trac:
Username: loskiq

What do the logs on the bridge say? Has the bridge bootstrapped?

Can you attach your bridge and client torrcs?

Trac:
Milestone: N/A to Tor: unspecified

Trac:
Username: loskiq

torrc_of_the_client.txt

torrc of the client

Trac:
Username: loskiq

torrc_of_the_bridge.txt

torrc of the bridge

Trac:
Username: loskiq

log_of_the_bridge.txt

log of the bridge

Trac:
Username: loskiq

Attached. Yes, bridge is bootstrapped

Replying to teor:

What do the logs on the bridge say? Has the bridge bootstrapped?

Can you attach your bridge and client torrcs?

Trac:
Username: loskiq

This ip address belongs to my bridge. I wanted to hide it and therefore I changed the address in the logs, but missed this line.

Replying to cypherpunks3:

Guessed our IP address as 54.93.104.200 (source: 79.137.112.4).

This.

54.93.104.200 works for me

Trac:
Username: loskiq

It looks like your obfs4 is only listening on IPv6:

Dec 04 08:12:04.000 [notice] Registered server transport 'obfs4' at '[::]:40635'

But your client is configured to connect via IPv4:

Bridge obfs4 79.103.124.21:40635 3E0CFCEE7183970DCC70ABC2D10518BC288BF0DE cert=ZzN5WrKqUZCkHYlb8gh0Ew1B5tMgO+oP60jxfar1r3A8yMH/syZ0T3td4x13VbEj1+G4EQ iat-mode=0

Unless you changed the IPv4 address to an IPv6 address when you redacted it. (Please don't change logs without telling us what you changed, it makes debugging much harder.)

Can you check if port 40635 is open on IPv4, IPv6, or both? If it's open on IPv4, tor has a logging bug.

If it's not, you can use ServerTransportListenAddress to set the correct address. And tor has a default IP version bug.

Also, you probably don't need this torrc option, it doesn't do anything on most systems: HardwareAccel 1

I checked port 40635 on IPv4 via nmap, and it is open. I just changed the IP address of my bridge. Sorry. The correct IP address of my bridge is 54.93.104.200, not 79.103.124.21

loskiq@loskiq-work:~$ nmap 54.93.104.200 -p 40635

Starting Nmap 7.40 ( https://nmap.org ) at 2018-12-05 17:33 MSK
Nmap scan report for ec2-54-93-104-200.eu-central-1.compute.amazonaws.com (54.93.104.200)
Host is up (0.070s latency).
PORT      STATE SERVICE
40635/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

And I removed the option HardwareAccel. Thanks for this.

Replying to teor:

Trac:
Username: loskiq

Ok, tor has a logging bug.

And it's still not clear why your client can't connect to your bridge. Can you connect to the obfs4 port from your client's IP address? Is the bridge line correct? Can you collect info-level logs on the client, and post them here. (You can redact if you want to.)

Trac:
Username: loskiq

tor.log

log on the client

Trac:
Username: loskiq

The client can connect to your bridge, but your bridge has no descriptor:

Dec 05 17:57:29.000 [info] handle_response_fetch_desc(): Received server info (body size 0) from server '54.93.104.200:40635'
Dec 05 17:57:29.000 [info] handle_response_fetch_desc(): Received http status code 404 ("Servers unavailable") from server '54.93.104.200:40635' while fetching "/tor/server/authority.z". I'll try again soon.

That's really strange, because your bridge has a descriptor:

Dec 04 09:07:05.000 [notice] Guessed our IP address as 54.93.104.200 (source: 79.137.112.4).
Dec 04 09:07:05.000 [notice] Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor.

Can you please post the logs from your bridge, after you changed the IP address and restarted the bridge?

Can you post info-level logs from your client and bridge, that contain the lines that your client and bridge log when the client tries to connect?

Trac:
Username: loskiq

tor_bridge.log

log on bridge

Trac:
Username: loskiq

tor_client.log

log on client

Trac:
Username: loskiq

Replying to loskiq:

Dec 04 10:47:39.000 [notice] Delaying directory fetches: No running bridges
}}}

I have seen this error before while testing PTs. I think it is a tor bug. I don't know the exact cause, but after a few bridge failures, tor will cache the fact that it thinks all bridges are down, and refuse even to try connecting to them. See:

comment:3:ticket:26891
#11301 (moved)

My usual workaround is to delete the /state file and restart tor. You can also try adding a line like this to your torrc: {{{ DataDirectory tmp-datadir

If that works, then the problem is likely the one I described.

Thank you for your answer, but unfortunately it did not help me. First I deleted /state and restarted bridge, then I changed DataDirectory to /tmp, restarted bridge and wrote the necessary changes to torrc of client. Client still unable to connect and stuck in 25%.

Replying to dcf:

Trac:
Username: loskiq

By the way, with the same configuration, the client successfully works with other bridges.

Trac:
Username: loskiq

Replying to loskiq:

By the way, with the same configuration, the client successfully works with other bridges.

I see. This means my guess was wrong.

I don't have any other good ideas, except to turn on obfs4proxy logging. In the client torrc, use

ClientTransportPlugin obfs4 exec /usr/local/bin/obfs4proxy --enableLogging --logLevel=DEBUG

In the bridge torrc, use

ServerTransportPlugin obfs4 exec /usr/local/bin/obfs4proxy --enableLogging --logLevel=DEBUG

In both cases, the log file will appear in /pt_state/obfs4proxy.log.

Trac:
Username: loskiq

obfs4proxy_server.log

obfs4proxy log of server

Trac:
Username: loskiq

obfs4proxy_client.log

obfs4proxy log of client

Trac:
Username: loskiq

It seems with obfs4proxy all right

Replying to dcf:

Trac:
Username: loskiq

I have just tested working with another obfs4 bridge received from bridges@torproject.org. The problem persists. Stuck in 25%. But there is a bridge with which the client works correctly.

The bridges I checked:

Not worked

obfs4 154.16.245.120:443 072AF4D16146012D6E8DFA8518B169345D8CEA51 cert=/nZrKfIZrw39ij9PLfoY0Uq+CCyQIYpP0BefZJbe2yf9cbJtXynlsWLwV77pVbKK4xDvHw iat-mode=0

Worked

obfs4 93.190.138.248:41248 577059FF2CADFB6AEFBCC78FFAB4DBEC1AF8A57B cert=d5ijyl75isX32Vx9yQkSDZicYPiamD413fnyKf/5TQxBTytobCVrzX7ATtBKZGOwIQ4IYg iat-mode=0

I think this problem is related to bridges, not clients...

Trac:
Username: loskiq

I just installed the bridge version 0.2.9.17, and the client immediately started working. I believe that the problem of 25% is related to the new version of the bridge.

Trac:
Username: loskiq

Replying to loskiq:

I just installed the bridge version 0.2.9.17, and the client immediately started working. I believe that the problem of 25% is related to the new version of the bridge.

Oh! How surprising. I see from tor_bridge.log that formerly you were using 0.3.4.9. So downgrading from 0.3.4.9 to 0.2.9.17 on the bridge (keeping the client at 0.3.4.9) made the bridge start working.

This also matches with the bridges you posted in comment:16.

154.16.245.120:443	not working	0.3.4.9	relay search (archive)
93.190.138.248:41248	working	0.2.9.17	relay search (archive)

I'll ask and see if any core-tor developer have an idea about what it wrong. If you have time, you can help by testing versions between 0.2.9.17 and 0.3.4.9 to see which versions work.

Replying to dcf:

If you have time, you can help by testing versions between 0.2.9.17 and 0.3.4.9 to see which versions work.

I tested different versions of Tor, and here is the result:

Tor version 0.2.9.17.	working
Tor version 0.3.2.10 (git-31cc63deb69db819).	working
Tor version 0.3.3.10 (git-2e94df92caee0fca).	working
Tor version 0.3.3.6 (git-7dd0813e783ae16e).	working
Tor version 0.3.3.7 (git-035a35178c92da94).	working
Tor version 0.3.3.9 (git-45028085ea188baf).	working
Tor version 0.3.4.1-alpha (git-deb8970a29ef7427).	not working
Tor version 0.3.4.2-alpha (git-bc951e83aac770d1).	not working
Tor version 0.3.4.6-rc (git-6045c26d8442913e).	not working
Tor version 0.3.4.7-rc (git-8465a8d84647c349).	not working
Tor version 0.3.4.8 (git-da95b91355248ad8).	not working
Tor version 0.3.4.9 (git-4ac3ccf2863b86e7).	not working

Trac:
Username: loskiq

Dieses problem wurde bei linked-verbindungen aufgespürt. Dies war nie richtig funktioniert aber bei älteren versionen und normal geladenen relais.

diff --git a/src/or/main.c b/src/or/main.c
index bc01e07..dd1f0d6 100644
--- a/src/or/main.c
+++ b/src/or/main.c
@@ -404,6 +404,9 @@ connection_unlink(connection_t *conn)
   connection_free(conn);
 }
 
+/** Event that invokes schedule_active_linked_connections_cb. */
+static mainloop_event_t *schedule_active_linked_connections_event = NULL;
+
 /**
  * Callback: used to activate read events for all linked connections, so
  * libevent knows to call their read callbacks.  This callback run as a
@@ -420,11 +423,10 @@ schedule_active_linked_connections_cb(mainloop_event_t *ev
    * so that libevent knows to run their callbacks. */
   SMARTLIST_FOREACH(active_linked_connection_lst, connection_t *, conn,
                     event_active(conn->read_event, EV_READ, 1));
+  if (smartlist_len(active_linked_connection_lst)) //QQQ: vvv safe?
+    mainloop_event_activate(schedule_active_linked_connections_event);
 }
 
-/** Event that invokes schedule_active_linked_connections_cb. */
-static mainloop_event_t *schedule_active_linked_connections_event = NULL;
-
 /** Initialize the global connection list, closeable connection list,
  * and active connection list. */
 STATIC void

Hmmm... I'm able to reproduce on 154.16.245.120:443 but it is working on my bridge listed in TB that is running 0.3.4.9 (Lisbeth, https://metrics.torproject.org/rs.html#details/D9C805C955CB124D188C0D44F271E9BE57DE2109). Bridge line:

Bridge obfs4 192.95.36.142:443 CDF2E852BF539B82BD10E27E9115A31734E378C2 cert=qUVQ0srL1JI/vO6V6m/24anYXiJD3QP2HgzUKQtQ7GRqqUvs7P+tG43RtAqdhLOALP7DJQ iat-mode=0

I think there is an issue with the bridge 154.16.245.120 itself, some sort of heavy throttling maybe. In the debug logs of the tor client, I see bursts of cells (no clear patterns but between 30 and 60 cells every 2 to 5 minutes) and then it get stuck waiting for more, tor just sits there idle waiting for I'm guessing the download to complete... I bet if I let it sit there long enough, the download would finish.

an irc user reports being happy with this patch on #28912 (moved) (which sounds from the ticket titles like the same issue)

Trac:
Status: new to needs_review

I am running this patch on moria1 now.

(I wonder if weasel's surprisingly varying bootstrap times for tor clients has to do with directory authorities, or fallbackdirs, now running substantially on Tor 0.3.4.x or later.)

See old code for "called_loop_once = smartlist_len(active_linked_connection_lst) ? 1 : 0;" Stuck at:

diff --git a/src/or/connection.c a/src/or/connection.c
index 0a2a635..0e051a5 100644
--- a/src/or/connection.c
+++ b/src/or/connection.c
@@ -3428,6 +3428,8 @@ connection_handle_read_impl(connection_t *conn)
 
     if (!buf_datalen(linked->outbuf) && conn->active_on_link)
       connection_stop_reading_from_linked_conn(conn);
+    /* Now. Now. If code still reading from connection then code */
+    /* must to reactivate event. It's linked connection. */
   }
   /* If we hit the EOF, call connection_reached_eof(). */
   if (!conn->marked_for_close &&

See https://trac.torproject.org/projects/tor/ticket/28912#comment:10

We've used the mainloop patch for now instead of going with reactivating the linked connection if it still has data to send. The reason is that it is a safer fix because <= 0.3.3 has that behavior. Doing the latter would require new tricky code that could introduce more issues :S.

I compiled the latest stable version of Tor (0.3.4.9), replacing the main.c file with yours. It works!

Replying to [comment:29 dgoulet]

Trac:
Username: loskiq

Trac:
Reviewer: N/A to nickm

I believe that the proposed patch above is the same patch as for #28912 (moved), which should be fixed in 0.3.4.10 and 0.3.5.7. Please reopen if this happens between a client and a relay running those versions or later?

Trac:
Resolution: N/A to fixed
Milestone: Tor: unspecified to Tor: 0.3.4.x-final
Status: needs_review to closed

closed

mentioned in issue #28742 (moved)

mentioned in issue #28912 (moved)

moved to tpo/core/tor#28717 (closed)

Tor stuck in 25% Loading networkstatus consensus

Child items 0

Activity