fix: do not close connection on ReceiveWithTimeout deadline - #698
fix: do not close connection on ReceiveWithTimeout deadline#698Solaris-star wants to merge 4 commits into
Conversation
|
The problem with this is on timeout you can't guarantee the wire state, specifically there could have been a partial send or inflight response and even if there wasn't already there could be one in the future which could result unexpected data corrupting the response processing. How is this fix preventing that from happening? |
A ReceiveWithTimeout deadline that fires before any bytes of a reply are consumed leaves the wire at a reply boundary, so the connection can be safely reused (idle pub/sub polling, gomodule#676). Once the first line has been read, a mid-reply timeout (partial bulk body or array element) desynchronises the stream; such cases remain fatal and close the connection to avoid returning stale bytes to a later Receive. Addresses the wire-state concern raised in review.
…light A clean reply boundary at timeout is only safe to reuse when nothing is outstanding. If a command was sent but its reply has not yet arrived, the late reply could be misread as the answer to a later receive. Guard the connection-preserving path with pending==0 so only truly idle waits (e.g. pub/sub) keep the connection. Addresses @stevenh's wire-state review.
|
@stevenh good catch — you're right that a clean boundary now isn't sufficient on its own, so the original version of this PR did not fully address your concern. I've reworked it. The connection is now only preserved on timeout when both hold:
So the only path that keeps the connection is an idle wait with nothing outstanding — e.g. a pub/sub poll waiting for a server push, which is exactly what #676 needs. Any partial send / in-flight response (present or future) falls through to Two tests cover both directions:
Does this address the corruption scenario you had in mind? |
| line, err := c.readLine() | ||
| if err != nil { | ||
| return nil, err | ||
| return nil, true, err |
There was a problem hiding this comment.
bug: I don't see how just because we got an error on readLine it indicates that it we're at a boundary.
The underling socket is read through bufio which states:
// If ReadSlice encounters an error before finding a delimiter,
// it returns all the data in the buffer and the error itself (often io.EOF).
So we could have experienced a partial read and a timeout meaning the connection state is actually unknown.
I don't think what you're trying to do is achievable, as there is no way to tell that there's no bytes on the wire when a timeout occurs.
There was a problem hiding this comment.
You are right: the previous atBoundary=true on every readLine error was incorrect because bufio.Reader.ReadSlice can return partial bytes together with the timeout.
I pushed 09cf40c to preserve the slice returned by ReadSlice on error and now report a clean boundary only when len(line) == 0. Any timeout after one or more bytes of the first RESP line have been consumed remains fatal. The existing guards still require pending == 0, so this only preserves an actually idle pub/sub poll.
I also added TestReceiveWithTimeoutClosesConnAfterPartialReplyLine, which writes +PARTIAL without a terminator and waits for the read deadline. Applied to the prior commit, it fails 10/10 because the connection remains open; with 09cf40c, it passes 10/10. Full go test ./... and go test -race ./redis pass.
This avoids claiming that no bytes exist on the wire; it only uses the bytes ReadSlice proves were consumed by this read.
There was a problem hiding this comment.
How does this handle:
- Request
- Reply timeout no bytes but response is actually inflight from server to client it has just not received yet
If that you allowed the connection to be reused the next request would get the previous request's response, causing corruption.
ReadSlice returns bytes consumed before a timeout together with the error. Preserve that partial slice so ReceiveWithTimeout only considers a timeout to be at a clean RESP boundary when zero bytes were consumed. Add a regression that sends a partial first line and verifies the connection is closed. Signed-off-by: Solaris-star <820622658@qq.com>
|
Good question — the fix guards against exactly that with two independent checks before preserving the connection:
Both conditions must hold simultaneously. If either fails, the connection is closed exactly as before. The stale read deadline is also cleared so the next The test |
Summary
ReceiveWithTimeouttreated every read error as fatal and closed the connection. Pub/sub poll loops that use a short timeout therefore permanently poison an otherwise idle connection after the first poll deadline.Fix
Preserve the connection only when all of these hold:
ReceiveWithTimeoutAPI.bufio.Reader.ReadSliceconsumed zero bytes of the reply. If it returns a partial first line with the timeout, the connection is closed.c.pending == 0, so no command response can arrive later and be misattributed to a subsequent receive.Connection-level
Receive()/ReceiveContexttimeouts remain fatal. The stale explicit read deadline is cleared only for the safe idle-poll case.Test plan
TestReceiveWithTimeoutDoesNotCloseConn: idle poll stays reusable.TestReceiveWithTimeoutClosesConnWhenRequestInFlight: timeout with a pending response closes the connection.TestReceiveWithTimeoutClosesConnAfterPartialReplyLine: timeout after+PARTIALcloses the connection.go test ./...go test -race ./redisFixes #676