Skip to content

Commit

Permalink
[exporter/datadog] make error retryable if logs sender received nil r…
Browse files Browse the repository at this point in the history
…esponse (open-telemetry#28672)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
The Datadog exporter threats network/connectivity errors (HTTP client
doesn't receive a response) as permanent errors, which can lead to log
records loss. This change makes these errors retryable.

**Link to tracking Issue:** open-telemetry#24550

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>
  • Loading branch information
siarhei-kharchanka-cko authored and jmsnll committed Nov 12, 2023
1 parent 39db4f3 commit 263feaa
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 4 deletions.
27 changes: 27 additions & 0 deletions .chloggen/retry-on-connectivity-issue.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: bug_fix

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: datadogexporter

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Fixes potential log records loss on a transient network/connectivity error

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [24550]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: "The Datadog exporter threats network/connectivity errors (http client doesn't receive a response) as permanent errors, which can lead to log records loss. This change makes these errors retryable."

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
7 changes: 3 additions & 4 deletions exporter/datadogexporter/internal/logs/sender.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (

"github.com/DataDog/datadog-api-client-go/v2/api/datadog"
"github.com/DataDog/datadog-api-client-go/v2/api/datadogV2"
"go.opentelemetry.io/collector/consumer/consumererror"
"go.opentelemetry.io/collector/exporter/exporterhelper"
"go.uber.org/zap"

Expand Down Expand Up @@ -85,9 +84,9 @@ func (s *Sender) handleSubmitLog(ctx context.Context, batch []datadogV2.HTTPLogI
s.logger.Error("Failed to send logs", zap.Error(err), zap.String("msg", string(b[:n])), zap.String("status_code", r.Status))
return err
}
// If response is nil assume permanent error.
// The error will be logged by the exporter helper.
return consumererror.NewPermanent(err)
// If response is nil keep an error retryable
// Assuming this is a transient error (e.g. network/connectivity blip)
return err
}
return nil
}

0 comments on commit 263feaa

Please sign in to comment.