Opened 3 months ago

Closed 3 months ago

Last modified 3 months ago

#30205 closed defect (fixed)

After log sanitization, proxy-go didn't restart, broker isn't logging

Reported by: cypherpunks Owned by: cohosh
Priority: Medium Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords:
Cc: dcf, arlolra, cohosh Actual Points:
Parent ID: Points:
Reviewer: Sponsor:

Description


Child Tickets

Change History (9)

comment:1 Changed 3 months ago by cohosh

Owner: set to cohosh
Status: newassigned

Looking into this... it looks like since the update for #30125 there are no new logs being created for the broker or proxy-go instances.

comment:2 Changed 3 months ago by cohosh

I've rolled back the versions of both proxy-go and broker to commit 88f282c with no luck. I think something about the log rotations in https://trac.torproject.org/projects/tor/ticket/30125#comment:8 are causing problems.

comment:3 Changed 3 months ago by cohosh

Okay not sure if this was causing the weird behaviour but the snowflake broker wasn't logging. The following additional commands fixed this:

$ sv d snowflake-broker/log
$ sv start snowflake-broker/log
ok: run: snowflake-broker/log: (pid 17524) 0s

Trying this for the proxy-go instances now (there's one running)

comment:4 Changed 3 months ago by dcf

Errors from runit appear in the argv of the main runsvdir process, which you can see with ps axwww:

runsvdir -P /etc/service log: alone-restartless.log.d: file does not exist svlogd: fatal: no functional log directories. svlogd: pausing: unable to rename current: /home/snowflake-proxy/snowflake-proxy-standalone-23h.log.d: file does not exist svlogd: warning: unable to open log directory: /home/snowflake-proxy/snowflake-proxy-standalone-restartless.log.d: file does not exist svlogd: fatal: no functional log directories. 

/home/snowflake-proxy/snowflake-proxy-standalone-restartless.log.d is actually missing, but I'm not sure about the errors about "current" not existing.

comment:5 Changed 3 months ago by dcf

Looking at the status of the processes gives me an error I don't know how to interpret:

# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 1372) 4940035s, got TERM

But a kill and a restart seems to have gotten that one working again:

# sv kill snowflake-proxy-standalone-17h/log
# sv restart snowflake-proxy-standalone-17h/log
ok: run: snowflake-proxy-standalone-17h/log: (pid 17809) 0s
# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 17809) 156s

I forgot that the svlogd log processes have to be stopped and started separately when I wrote the instructions in comment:4:ticket:30125.

Last edited 3 months ago by dcf (previous) (diff)

comment:6 in reply to:  5 Changed 3 months ago by cohosh

Resolution: fixed
Status: assignedclosed

Replying to dcf:

Looking at the status of the processes gives me an error I don't know how to interpret:

# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 1372) 4940035s, got TERM

But a kill and a restart seems to have gotten that one working again:

# sv kill snowflake-proxy-standalone-17h/log
# sv restart snowflake-proxy-standalone-17h/log
ok: run: snowflake-proxy-standalone-17h/log: (pid 17809) 0s
# sv status snowflake-proxy-standalone-17h/log
run: snowflake-proxy-standalone-17h/log: (pid 17809) 156s

I forgot that the svlogd log processes have to be stopped and started separately when I wrote the instructions in comment:4:ticket:30125.

Awesome, thanks! Setting this as fixed.

comment:7 Changed 3 months ago by dcf

Summary: Snowflake is not working againAfter log sanitization, proxy-go didn't restart, broker isn't logging

I restarted the instances at these times:

  • snowflake-proxy-standalone-17h 2019-04-16 22:23
  • snowflake-proxy-standalone-23h 2019-04-16 22:58
  • snowflake-proxy-standalone-29h 2019-04-16 22:58

Additionally, I created a log directory for snowflake-proxy-standalone-restartless and started it at 2019-04-16 23:14.

comment:8 in reply to:  7 Changed 3 months ago by dcf

Replying to dcf:

I restarted the instances at these times:

  • snowflake-proxy-standalone-17h 2019-04-16 22:23
  • snowflake-proxy-standalone-23h 2019-04-16 22:58
  • snowflake-proxy-standalone-29h 2019-04-16 22:58

Additionally, I created a log directory for snowflake-proxy-standalone-restartless and started it at 2019-04-16 23:14.

On second thought, the existing snowflake-proxy-restartless was already using the standalone broker. I'm not sure where snowflake-proxy-standalone-restartless came from, but I removed it from /etc/service so it doesn't run.

comment:9 Changed 3 months ago by cohosh

That's my bad, I started the snowflake-proxy-standalone-restartless but we can use snowflake-proxy-restartless for now.

I will probably make another ticket for renaming the proxy-go instances at some point since we shouldn't need to have them restart anymore.

Note: See TracTickets for help on using tickets.