Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix duplicate session recording creation #45877

Merged
merged 3 commits into from
Sep 4, 2024

Conversation

fspmarshall
Copy link
Contributor

@fspmarshall fspmarshall commented Aug 26, 2024

Fixes an issue where duplicate session recordings could end up being created in teleport clusters experiencing interruptions during session recording upload from agent to auth. In all observed cases of this bug, it seems to have been produced by a combination of two factors:

  • Agents weren't syncing their upload status to disk until after the first successful event submission, increasing the likelihood that agents would create orphaned uploads.
  • Auth servers were uploading slices even when empty, increasing the likelihood of orphaned upload attempts resulting in session recordings being created.

Both of the above cases are addressed in this PR. Note, however, that this PR does not address correctly selecting the non-empty upload for existing recordings affected by this issue. That will have to be the subject of future work (see #45917). It also doesn't prevent duplicates due to loss of agent-side cursor state or timeout of a partially completed upload attempt (e.g. due to an agent being offline for a prolonged period of tim), both of which were previously known "expected" cases in which duplicate recordings could be produced.

This issue is loosely related to #45786 as the two issues tend to happen together since they are both triggered when the session recording system is under extremely high load.

changelog: fixed an issue that could result in duplicate session recordings being created

@fspmarshall fspmarshall force-pushed the fspmarshall/fix-duplicate-session-recordings branch from 332bb4e to 3c489ef Compare September 3, 2024 21:43
@rosstimothy
Copy link
Contributor

/excludeflake *

@fspmarshall fspmarshall added this pull request to the merge queue Sep 4, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 4, 2024
@fspmarshall fspmarshall added this pull request to the merge queue Sep 4, 2024
Merged via the queue into master with commit 3558f93 Sep 4, 2024
39 checks passed
@fspmarshall fspmarshall deleted the fspmarshall/fix-duplicate-session-recordings branch September 4, 2024 22:08
@public-teleport-github-review-bot

@fspmarshall See the table below for backport results.

Branch Result
branch/v14 Create PR
branch/v15 Create PR
branch/v16 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants