feat(webapp,redis): handle UNBLOCKED during ElastiCache role change (#3549)
## Summary When ElastiCache demotes a primary to replica — during a Multi-AZ failover or a vertical node-type change — the demoting primary issues an `UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`, `BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role flips. ioredis surfaces these as `ReplyError` to caller code. The shared `defaultReconnectOnError` added in #3548 only matches `READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same way the existing two cases handle non-blocking-command errors. ## Fix ```ts export function defaultReconnectOnError(err: Error): boolean | 1 | 2 { const msg = err.message ?? ""; if ( msg.startsWith("READONLY") || msg.startsWith("LOADING") || msg.startsWith("UNBLOCKED") ) { return 2; } return false; } ``` Returning `2` tells ioredis to disconnect, reconnect, and re-issue the command. For a BLPOP that means a fresh BLPOP against the new primary instead of the `UNBLOCKED` error escaping to the caller. ## Test plan - [ ] CI green - [ ] Trigger a Multi-AZ failover or a vertical scale event on an ElastiCache replication group whose clients are running blocking commands and confirm no `UNBLOCKED` errors surface to caller code during the cutover.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
---
|
||||
area: webapp
|
||||
type: improvement
|
||||
---
|
||||
|
||||
Extend the shared ioredis `reconnectOnError` hook (PR #3548) to also match `UNBLOCKED` reply errors so blocking commands like BLPOP transparently reconnect-and-retry when the ElastiCache primary forces them to unblock during a node role change.
|
||||
@@ -9,6 +9,12 @@ export { Redis, type Callback, type RedisOptions, type Result, type RedisCommand
|
||||
* reply errors to caller code over a healthy TCP/TLS connection (the
|
||||
* client keeps talking to a node whose role swapped underneath it).
|
||||
*
|
||||
* UNBLOCKED is the BLPOP-shaped case: the Redis primary forcibly
|
||||
* unblocks any blocking command on a connection whose node is about
|
||||
* to be demoted, returning an UNBLOCKED reply. Surfaced 65 times on
|
||||
* engine/v1/worker-actions/dequeue at the cutover instant during the
|
||||
* TRI-8873 test-cloud scale-up dry-run.
|
||||
*
|
||||
* Returning 2 tells ioredis to disconnect, reconnect, and retry the
|
||||
* command that triggered the error. After reconnect, DNS / SG routing
|
||||
* should land on a writable primary.
|
||||
@@ -18,7 +24,13 @@ export { Redis, type Callback, type RedisOptions, type Result, type RedisCommand
|
||||
*/
|
||||
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
|
||||
const msg = err.message ?? "";
|
||||
if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
|
||||
if (
|
||||
msg.startsWith("READONLY") ||
|
||||
msg.startsWith("LOADING") ||
|
||||
msg.startsWith("UNBLOCKED")
|
||||
) {
|
||||
return 2;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user