Skip to content

Commit

Permalink
Users may request output groups' files be inlined in TargetComplete
Browse files Browse the repository at this point in the history
…/`AspectComplete` events in BEP using new flag.

Some services that consume large BEP streams need to be able to eagerly identify critical output files without keeping the entire BEP stream in-memory. Traditionally, these were reported in the `TargetComplete.important_outputs` field, which is deprecated because it can grow without bound without any mechanism to break up the set of files included in that field.

The replacement for `TargetComplete.important_outputs` is the recursive `NamedSetOfFiles` representation which can be broken up arbitrarily and supports efficient serialization directly from `NestedSet` objects inside the build tool. Unfortunately consumers must keep the entire BEP stream in-memory in order to efficiently identify critical files when they are linked from a `TargetComplete.output_groups.file_sets` field.

As a compromise, this change introduces a new field: `OutputGroup.inline_files`. By using a new flag, `--experimental_build_event_output_group_mode`, users can request that specific output groups report their files inline instead of using the recursive `NamedSetOfFiles`. This still suffers from the same failure mode as `TargetComplete.important_outputs` - a single huge output group can produce an extremely large single event. Unlike `TargetComplete.important_outputs`, users may restructure their build rules using additional output groups to shrink the size of their events.

RELNOTES: A new experimental flag, `--experimental_build_event_output_group_mode`, allows users to change how a given output group's files are reported in BEP. The current behavior is `FILESET` which populates `OutputGroup.file_sets`. Users may now specify `INLINE` to instead report files directly in the `TargetComplete`/`AspectComplete` event under `OutputGroup.inline_files`. Users may also specify `BOTH` to populate `OutputGroup.file_sets` and `OutputGroup.inline_files`.
PiperOrigin-RevId: 667601787
  • Loading branch information
Googler authored and copybara-github committed Aug 26, 2024
1 parent 5ea0680 commit ff1f674
Showing 1 changed file with 4 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -611,6 +611,10 @@ message OutputGroup {
// Indicates that one or more of the output group's files were not built
// successfully (the generating action failed).
bool incomplete = 4;

// Inlined files that belong to this output group, requested via
// --build_event_inline_output_groups.
repeated File inline_files = 5;
}

// Payload of the event indicating the completion of a target. The target is
Expand Down

0 comments on commit ff1f674

Please sign in to comment.