Repository navigation
feat(storage): log transient errors retried on the sync paths - #16467
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new retry logging mechanism (retry_logging.h and retry_logging.cc) to provide diagnostics for silently retried transient errors in GCS operations, integrating it into ReadObject, UploadChunk, and RetryObjectReadSource. Feedback on the changes highlights style guide violations, specifically recommending the use of std::string_view instead of absl::string_view and avoiding auto for primitive types.
0958b42 to
f154e51
Compare
f154e51 to
1b80cff
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16467 +/- ##
==========================================
+ Coverage 92.34% 92.35% +0.01%
==========================================
Files 2249 2252 +3
Lines 214649 215038 +389
==========================================
+ Hits 198212 198601 +389
Misses 16437 16437 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kalragauri
left a comment
There was a problem hiding this comment.
Regarding your open question on log volume, logging every retried attempt (once ReadObject/open is gated on retry_policy.OnFailure() returning true) is the right choice IMHO rather than sampling powers of two. IIUC, ExponentialBackoffPolicy (the default retry policy) already spaces consecutive attempts exponentially in wall-clock time.
1b80cff to
000c4a5
Compare
|
Thanks — agreed, and I have dropped the throttling idea. Your reasoning holds up when I check it against the code: The decorator you asked for in the So the behaviour is unchanged from what is in the PR body — one |
000c4a5 to
934ec89
Compare
|
Update: I re-ran this against the storage-testbench, and it found something that every unit test missed. Flagging it because it is security relevant and it extends an already-answered thread. The gRPC upload id was being logged in fullIn the earlier discussion about redacting the upload id, the shape under consideration was the resource name Against a real gRPC resumable upload, the session id turned out to be a bare opaque 64-character hex string with no structure at all, so it fell into the unchanged branch and the log published it whole: That is the entire bearer capability for the upload session. The default was fail-open for a credential, which is the worst way to be wrong. Fix
REST is unchanged and still keeps the useful part of the URL: The header comment now explains why the default is fail-closed, so it does not get "helpfully" relaxed later. The unit test that asserted the pass-through behaviour has been deleted and replaced with tests for the opaque shapes, and Full testbench matrix is in the PR description. |
When GCS returns a transient error the client backs off and retries with
no output at all. From the application's point of view a read just takes
a long time. There is nothing to grep for, so an operator cannot tell a
slow network from a server that is shedding load, and the only way to
see the retries is to enable the `rawclient` tracing component and pay
for a full dump of every request and response.
This adds one WARNING record per retried attempt on the synchronous
paths, which cover both the JSON/REST and the gRPC transports (GCS gRPC
sync goes through `RestRetryLoop`, because `GenericStub` takes a
`RestContext&` regardless of transport).
[gcs-retry] ReadObject/open: transient error on attempt 1,
resource=my-bucket/my-object, status=UNAVAILABLE: ...
Call sites:
- `ConnectionImpl::ReadObject` -- an attempt failed to open the stream.
- `RetryObjectReadSource::Read` -- the stream broke mid-download and is
being resumed from the last received byte.
- `ConnectionImpl::UploadChunk` -- an attempt to upload a chunk of a
resumable upload failed.
Every record comes from the retry policy actually in force, never from a
second opinion about which errors are retryable. `ReadObject` wraps the
caller's policy in a `LoggingRetryPolicy` decorator and hands that to
`RestRetryLoop`; the other two sites already sit behind an `OnFailure()`
that returned true. So a line is published only when another attempt
really does follow. A permanent error, the final attempt of an exhausted
policy, and a non-idempotent request are all silent, because each is
reported to the caller through the returned `Status` instead. It also
means a caller who overrides the retry policy gets logging that matches
their policy, with no classification logic to drift out of step.
The severity is WARNING and the record is not gated behind a tracing
component. `StdClogBackend`, the default backend, drops everything below
FATAL, so this is invisible to a default OSS build; it becomes visible
to anyone who installs a `LogBackend`. A log that has to be turned on
first is not useful during the first occurrence of an incident, which is
exactly when it is needed.
Two details worth calling out:
- The status is flattened to a single line. GCS error bodies end with a
newline, which would otherwise push the backend's `(file:line)` suffix
onto a second line and make the record unusable with `grep`. It also
removes a log-injection hazard, since the message is service-supplied.
- `UploadChunkRequest` only exposes the resumable session identifier,
and the token in it is a bearer capability: anyone holding it can
append to or finalize the upload for the lifetime of the session. Both
transport forms are redacted to a short prefix -- the `upload_id`
query parameter on REST, and the trailing token of
`projects/<project>/buckets/<bucket>/<token>` on gRPC.
Also makes `LoggingStub::ReadObject` report the status on failure. It
cannot use `LogWrapper` because `unique_ptr<ObjectReadSource>` has no
`operator<<`, so it previously logged nothing about why the call failed.
Verified against the storage-testbench using its fault-injection API, in
addition to the unit tests.
934ec89 to
98cadb6
Compare
Problem
When GCS returns a transient error (429, 503, 500,
DEADLINE_EXCEEDED) theclient backs off and retries with no output at all. From the application's
point of view a read just takes a long time.
There is nothing to grep for, so an operator cannot tell a slow network from a
server that is shedding load. The only way to see the retries today is to enable
the
rawclienttracing component and pay for a full dump of every request andresponse — which you can only do after you already suspect retries, i.e. not
during the first occurrence of an incident.
This came out of a real incident where reads on a large fleet appeared to stall
and the client produced nothing to correlate against.
What this does
One
WARNINGrecord per retried attempt, nothing on success:This PR covers the synchronous paths, which serve both the JSON/REST and the
gRPC transports — GCS gRPC sync also goes through
RestRetryLoop, becauseGenericStubtakes aRestContext®ardless of transport.whereConnectionImpl::ReadObject(retry lambda)ReadObject/openRetryObjectReadSource::ReadReadObject/resumeConnectionImpl::UploadChunkUploadChunkNew helper
google/cloud/storage/internal/retry_logging.{h,cc}(
LogTransientRetry,LoggingRetryPolicy,RetryLogResource,RetryLogUploadResource) so the call sites stay one line each.Also makes
LoggingStub::ReadObjectreport the status on failure. It cannot useLogWrapperbecauseunique_ptr<ObjectReadSource>has nooperator<<, so itpreviously logged nothing about why the call failed.
Design notes
Why WARNING, and why not behind a tracing component
LogSink::minimum_severity_defaults toTRACE, but the defaultStdClogBackenddrops everything belowFATAL(log.cc#L228-L231).
So this is invisible in a default OSS build and becomes visible to anyone
who installs their own
LogBackend. A diagnostic that has to be turned on firstis not useful during the first occurrence of an incident, which is exactly when
it is needed.
No changes to
google/cloud/internal/**orlog.cc; everything is insidegoogle/cloud/storage/**.The status is flattened to one line
GCS error bodies end with a newline. Streaming the
Statusstraight intoGCP_LOGcarried that newline through, pushing the backend's(file:line)suffix onto a second line — so
grep '\[gcs-retry\]'returned a truncatedrecord, defeating the point of the greppable prefix.
It is also a small log-injection hazard, since the message is
service-supplied.
OneLine()flattens\n,\r,\t.Pinned by
RetryLogging.StatusIsFlattenedToASingleLine.The upload session URL is redacted
UploadChunkRequestonly exposesupload_session_url(), and theupload_idinit is a bearer capability — anyone holding it can append to or finalize that
upload with no other credential, for the lifetime of the session (~7 days).
The request's own
operator<<does print it, but that is behind the opt-inrawclientcomponent, whereas this record is always on and lands in logaggregators that are usually readable by more people than the bucket itself.
RetryLogUploadResource()keeps the bucket and an 8-character prefix of the id— enough to tell concurrent uploads apart, not enough to hijack one.
Both transports are handled, because
UploadChunkRequestcarries whichever formthe transport produced: REST passes a URL with an
upload_id=query parameter,gRPC may pass
projects/<project>/buckets/<bucket>/<token>, where the token mayitself contain
/. Anything matching neither shape is redacted whole, onthe grounds that gRPC also issues wholly opaque tokens (see the testbench
section) and treating an unfamiliar value as safe would leak the entire
capability the first time the service changed format.
Every record comes from the retry policy in force
ReadObject/openhas no natural place to count attempts: the work happens insidea lambda that
RestRetryLoop()calls. Rather than classifying the statusseparately, it wraps the caller's policy in
LoggingRetryPolicy, agoogle::cloud::RetryPolicydecorator whoseOnFailure()asks the wrappedpolicy first and logs only when the answer is "retry".
A record is therefore emitted only when another attempt really does follow. A
permanent error, the final attempt of an exhausted policy, and a non-idempotent
request are all silent — each is already reported to the caller through the
returned
Status. There is also no second copy of "which errors are retryable"to drift away from the policy, so a caller who overrides
RetryPolicyOptionautomatically gets matching logs.
ReadObject/resumeandUploadChunkalready sat after their policy'sOnFailure(), so all three sites are now exact.The backoff delay is deliberately not logged: reading it requires
backoff_policy.OnCompletion(), which advances the policy.Testbench verification
End-to-end against
storage-testbench
v0.61.0using its fault-injection API (
POST /retry_test, id passed asx-retry-test-idon REST and as call metadata on gRPC), withGOOGLE_CLOUD_CPP_ENABLE_CLOG=yes. Re-run in full after switching to theLoggingRetryPolicydecorator. Each scenario asserts an exact record count.objects.get: 503 ×3, 2 retriesobjects.get: 503, 0 retriesobjects.get: 503 ×2, 2 retriesobjects.get: 400objects.insert: return-503-after-256Kobjects.insert: return-503-after-256KScenarios 1 and 2 are the ones the decorator changed. Previously the give-up
attempt was reported as though it were retried, and a policy granting no
retries still produced a record.
The trailing
(retry_logging.cc:144)is added by the library's own log backend(
operator<<(std::ostream&, LogRecord const&)inlog.cc), not by this change.Important
Scenario 7 is why this section exists. The gRPC upload session id is a bare
opaque 64-character hex string, not the
projects/_/buckets/<bucket>/<token>resource name. The earlier code returned unrecognised shapes unchanged and so
logged the whole token — a bearer credential — in clear. Redaction is now
fail-closed: anything not structurally recognised is redacted whole, keeping
only a short prefix for correlation. No unit test caught this.
Tests
bazel test //google/cloud/storage:all→ 175 / 175 PASSED on this branchalone.
New coverage in
retry_logging_test.cc,connection_impl_object_test.cc,retry_object_read_source_test.cc,logging_stub_test.cc— including negativetests asserting nothing is logged on success, on a permanent error, on the
final attempt of an exhausted policy, or on a non-idempotent request.
UploadChunkLogsTransientFailuresnow uses a realistic resumable session URLand asserts the bucket survives while the
upload_iddoes not.Follow-ups
UploadChunk. ARetryLoggingStubGenericStubdecorator installed ingeneric_stub_factory.ccwould cover every unary RPC from a single place;worth doing separately rather than growing this PR.
kept.
ExponentialBackoffPolicyalready spaces attempts exponentially inwall-clock time, and the decorator bounds the count at exactly the number of
attempts the policy granted.
Internal ref: b/547926601