Opened 7 months ago

Closed 7 months ago

#29426 closed defect (duplicate)

proxy-go instances not available

Reported by: cohosh Owned by:
Priority: High Milestone:
Component: Circumvention/Snowflake Version:
Severity: Normal Keywords: snowflake
Cc: dcf, arlolra, cohosh Actual Points:
Parent ID: Points:
Reviewer: Sponsor: Sponsor19

Description (last modified by cohosh)

The broker is reporting no available snowflakes despite the fact that the proxies are running. Not sure if this is due to the deadlock problem or if these 504 errors indicate a problem with the proxy-broker communication.

Child Tickets

Change History (6)

comment:1 Changed 7 months ago by dcf

I happened to be testing against the broker yesterday, and then there were 3 proxies reported at https://snowflake-broker.bamsoftware.com/debug. Checking just now, I see either 0 or 1 as I refresh. I suspect, as you suggested, that something went wrong with the proxy-go instances.

The proxy-go instances are set to restart themselves periodically (about once per day). However there is also a special proxy-go-restartless instance that doesn't get restarted--you could check that one with gdb to see where it's stuck, or create more instances like that to get more samples.

comment:2 Changed 7 months ago by dcf

I might have misinterpreted the ticket. I read the title as "clients receiving 504 errors from broker".

The 504 code itself isn't an error. That's the way the broker indicates to a client that no proxy is available, or to a proxy that no clients are available (StatusGatewayTimeout is 504):

I think the intention behind using status code 504 was a kind of REST thing, but IMO it was a questionable decision to use an "internal server" error code to stand for a common occurrence, and could stand to be redesigned. It's really confusing.

The dearth of proxies I noted in comment:1 is a (possibly recurring) problem, but if you run proxy-go and it logs broker returns: 504, that in itself isn't an error.

comment:3 Changed 7 months ago by cohosh

Description: modified (diff)
Summary: proxy-go instances receiving 504 errors from brokerproxy-go instances not available

Thanks for the clarification. I updated the ticket summary to not focus on the 504 message but rather the fact they aren't reachable.

Is this likely a duplicate of #25688?

comment:4 Changed 7 months ago by cohosh

Added a comment to #29207 RE the log messages

comment:5 in reply to:  3 Changed 7 months ago by dcf

Replying to cohosh:

Is this likely a duplicate of #25688?

It's possible--though if proxy-go is still writing log messages, then it's likely not the same as #25688--at least I think proxy-go stops logging if that kind of deadlock happens.

I'm not sure what you're looking at, whether the process is still actively logging or not. I would check the most recent timestamp in each of the logs; it should be within the last 30 seconds (UTC), and if you tail -f it, it should keep scrolling.

comment:6 Changed 7 months ago by cohosh

Resolution: duplicate
Status: newclosed

Okay 2/3 instances are reported up again and logging. The one that's still down isn't logging so I'm marking this as duplicate of #25688.

Note: See TracTickets for help on using tickets.