Opened 21 months ago

Closed 21 months ago

Last modified 21 months ago

#28390 closed defect (fixed)

Snowflake stuck at "Loading relay information..."

Reported by: cypherpunks3 Owned by:
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords:
Cc: dcf, arlolra Actual Points:
Parent ID: Points:
Reviewer: Sponsor:


Happened this morning and snowflake no longer works for me after that.

Child Tickets

Change History (8)

comment:1 Changed 21 months ago by cypherpunks3

Resolution: worksforme
Status: newclosed

It's working again.

comment:2 Changed 21 months ago by cypherpunks3

Resolution: worksforme
Status: closedreopened

Again seems to not work. Maybe not enough proxies I guess?

comment:3 Changed 21 months ago by cypherpunks3

Yeah not enough proxies it seems.

comment:4 in reply to:  2 Changed 21 months ago by arlolra

Replying to cypherpunks3:

Again seems to not work. Maybe not enough proxies I guess?

Yes, sorry. We've been running some proxies to ensure there's always some capacity while the transport is in development but those have been having some issues.

We really need to get on #20813

comment:6 Changed 21 months ago by dcf

Resolution: fixed
Status: reopenedclosed

The problem was a full disk, just like in #26661. The bridge and the fallback proxies are currently hosted on the same machine, so when the proxies filled up the disk with logs, it stopped the bridge running as well. The end of one of the log files was (timestamps are UTC)

2018/11/16 15:10:27 broker returns: 504
2018/11/16 15:10:38 broker returns: 504
2018/11/16 15:10:48 broker returns: 504
2018/11/16 15:10:58 broker returns: 504
2018/11/16 15:11:08 broker returns: 504
2018/11/16 15:11:18 broker returns: 504
2018/11/16 15:11:28 broker returns: 504
2018/11/16 15:11

I compressed the logs to make some space (they compress to around 1% of their original size, as they mostly contain those broker returns: 504 lines). Then I did what I should have done after #26661, which is activate log rotation to prevent this from happening again. We're using runit to manage the proxies, which comes with a program svlogd to manage log files. I added a log/run script to each service directory containing:

exec chpst -u snowflake-proxy svlogd snowflake-proxy-xxx.log.d

In each service run script, I removed the -log option from the snowflake-proxy command, so that it will log to stderr. Then, in each snowflake-proxy-xxx.log.d directory, I made a file config that contains:

# size of log file before rotation (10 MB)
# don't delete old logs
# unless the filesystem is full
# compress old logs

What this means is, when a log file reaches 10 MB, it will be compressed and archived. Keep archived log files, but delete the oldest ones if the disk is full.

comment:7 Changed 21 months ago by dcf

Oh, and an additional problem I didn't notice and fix until 2018-11-22 04:00:00. Because of #18356 and, I have to remember to do these steps whenever systemd is upgraded (which I did while making space on the filesystem):

setcap 'cap_net_bind_service=+ep' /usr/local/bin/snowflake-server
Set NoNewPrivileges=no in /lib/systemd/system/tor@*.service
systemctl daemon-reload

Without these changes, the websocket PT plugin fails with error opening HTTP-01 ACME listener: listen tcp bind: permission denied.

comment:8 Changed 21 months ago by cypherpunks3

I love you so much snowflake folks <3

Note: See TracTickets for help on using tickets.