Repository navigation
fix(storage): gate async flush completion on sent offset instead of persisted_size - #16596
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the asynchronous writer connections (writer_connection_buffered and writer_connection_resumed) to correctly handle scenarios where the service's persisted size lags behind the sent bytes, such as on regular GCS buckets that only persist data in 256 KiB multiples. Specifically, Flush() now completes immediately upon receiving the first response carrying a persisted_size rather than waiting for it to match the expected offset, preventing potential hangs. Internal state checks like NeedsFlush and ClearHandlersIfEmpty have been updated to track unsent data using the write offset, and extensive unit and integration tests have been added to verify correct behavior with unaligned uploads. No review comments were provided, so there is no additional feedback to address.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #16596 +/- ##
==========================================
- Coverage 92.39% 92.37% -0.03%
==========================================
Files 2266 2266
Lines 220143 220810 +667
==========================================
+ Hits 203409 203965 +556
- Misses 16734 16845 +111 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
88a34c9 to
33c56e0
Compare
v-pratap
left a comment
There was a problem hiding this comment.
The fix looks right to me. Completing Flush() at the sent offset matches the AsyncWriter::Flush() docs, and the unsent-data check stops the empty-flush loop. All three writer unit tests pass under ASan too.
One doc gap: the public BufferedUploadHwmOption comment still says flow control is released only at the LWM. That is no longer true when the unpersisted tail is at or above the LWM. Please update it (and the stale buffer_size_hwm_ member comments).
AsyncWriterConnectionImpl::Flush()sends its payload withflush: true, state_lookup: true, and the service answers with a single response carryingpersisted_size. On regular (non-appendable) objects, the service only persists 256 KiB multiples before the upload is finalized, so an unaligned flush returns apersisted_sizesmaller than the bytes sent (0for the 15,890-byte upload in theStartBufferedUploadsample). BecauseOnFlushkept reading untilpersisted_size >= offset_, its nextRead()blocked until the stall timeout cancelled the stream, affecting bothAsyncWriter::Flush()on unbuffered uploads and the flushes issued byAsyncWriterConnectionBufferedStateandAsyncWriterConnectionResumedState. Those two classes also waited fortarget_offset <= persisted_sizebefore completingFlush()promises and checked!resend_buffer_.empty()inNeedsFlush()andClearHandlersIfEmpty(), leavingFlush()futures and high-water-mark waiters pending (and looping on empty flushes when idle) whenever an non-persisted tail remained in the buffer.This PR does the following:
OnFlushnow completes on the first response carryingpersisted_sizeorresourcewhile still recording that value inpersisted_state_.SetFlushed()drops its unusedpersisted_sizeparameter and completes pendingFlush()promises up to the sent offset (buffer_offset_ + write_offset_), matching theAsyncWriter::Flush()contract that flushed data may not be persisted yet.NeedsFlush()andClearHandlersIfEmpty()now shareHasUnsentDataAtOrAboveLwm()(write_offset_ < resend_buffer_.size() && resend_buffer_.size() >= buffer_size_lwm_), so once all buffered bytes have been sent the writer goes idle and releases high-water-mark waiters instead of re-flushing data the service already received.OnQuery()still trimsresend_buffer_bypersisted_size, so any unpersisted bytes are replayed onResume()as before.