Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch ingestion replace #12137

Merged
merged 46 commits into from
Mar 9, 2022
Merged

Batch ingestion replace #12137

merged 46 commits into from
Mar 9, 2022

Commits on Feb 18, 2022

  1. Tombstone support for replace functionality

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    682f957 View commit details
    Browse the repository at this point in the history

  2. A used segment interval is the interval of a current used segment tha… 

    …t overlaps any of the input intervals for the spec
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    2be134d View commit details
    Browse the repository at this point in the history

  3. Update compaction test to match replace behavior

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    26177b6 View commit details
    Browse the repository at this point in the history

  4. Adapt ITAutoCompactionTest to work with tombstones rather than droppi… 

    …ng segments. Add support for tombstones in the broker.
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    3b99a81 View commit details
    Browse the repository at this point in the history

  5. Style plus simple queriableindex test

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    ac44e56 View commit details
    Browse the repository at this point in the history

  6. Add segment cache loader tombstone test

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    86fd028 View commit details
    Browse the repository at this point in the history

  7. Add more tests

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    4855929 View commit details
    Browse the repository at this point in the history

  8. Add a method to the LogicalSegment to test whether it has any data

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    f2e52ad View commit details
    Browse the repository at this point in the history

  9. Test filter with some empty logical segments

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    84ea1a7 View commit details
    Browse the repository at this point in the history

  10. Refactor more compaction/dropexisting tests

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    b66d8a1 View commit details
    Browse the repository at this point in the history

  11. Code coverage

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    4d5df0a View commit details
    Browse the repository at this point in the history

  12. Support for all empty segments

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    b4a935a View commit details
    Browse the repository at this point in the history

  13. Skip tombstones when looking-up broker's timeline. Discard changes ma… 

    …de to tool chest to avoid empty segments since they will no longer have empty segments after lookup because we are skipping over them.
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    6481003 View commit details
    Browse the repository at this point in the history

  14. Fix null ptr when segment does not have a queriable index

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    6141c18 View commit details
    Browse the repository at this point in the history

  15. Add support for empty replace interval (all input data has been filte… 

    …red out)
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    5428675 View commit details
    Browse the repository at this point in the history

  16. Fixed coverage & style

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    3352fb6 View commit details
    Browse the repository at this point in the history

  17. Find tombstone versions from lock versions

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    173c2e1 View commit details
    Browse the repository at this point in the history

  18. Test failures & style

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    d77a0c4 View commit details
    Browse the repository at this point in the history

  19. Interner was making this fail since the two segments were consider eq… 

    …ual due to their id's being equal
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    46cc72e View commit details
    Browse the repository at this point in the history

  20. Cleanup tombstone version code

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    a685cc6 View commit details
    Browse the repository at this point in the history

  21. Force timeChunkLock whenever replace (i.e. dropExisting=true) is bein… 

    …g used
    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    d0fbef9 View commit details
    Browse the repository at this point in the history

  22. Reject replace spec when input intervals are empty

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    a7ec8dc View commit details
    Browse the repository at this point in the history

  23. Documentation

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    faf2720 View commit details
    Browse the repository at this point in the history

  24. Style and unit test

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    4fe9548 View commit details
    Browse the repository at this point in the history

  25. Restore test code deleted by mistake

    Agustin Gonzalez committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    404072c View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2022

  1. Allocate forces TIME_CHUNK locking and uses lock versions. TombstoneS… 

    …hardSpec added.
    Agustin Gonzalez committed Feb 19, 2022
    Configuration menu
    Copy the full SHA
    c6d0271 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2022

  1. Unused imports. Dead code. Test coverage.

    Agustin Gonzalez committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    617eb0b View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2022

  1. Coverage.

    Agustin Gonzalez committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    a977f21 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. Prevent killer from throwing an exception for tombstones. This is the… 

    … killer used in the peon for killing segments.
    Agustin Gonzalez committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    151d674 View commit details
    Browse the repository at this point in the history

  2. Fix OmniKiller + more test coverage.

    Agustin Gonzalez committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    3f5d1dc View commit details
    Browse the repository at this point in the history

  3. Tombstones are now marked using a shard spec

    Agustin Gonzalez committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    c62d36b View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2022

  1. Drop a segment factory.json in the segment cache for tombstones

    Agustin Gonzalez committed Feb 26, 2022
    Configuration menu
    Copy the full SHA
    0ad63a8 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2022

  1. Merge branch 'master' into batch_ingestion_replace

    Agustin Gonzalez committed Feb 27, 2022
    Configuration menu
    Copy the full SHA
    8680fe4 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2022

  1. Style

    Agustin Gonzalez committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    55c0a38 View commit details
    Browse the repository at this point in the history

  2. Style + coverage

    Agustin Gonzalez committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    9f7cc10 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'temp' into batch_ingestion_replace

    Agustin Gonzalez committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    f4c3440 View commit details
    Browse the repository at this point in the history

  4. style

    Agustin Gonzalez committed Feb 28, 2022
    Configuration menu
    Copy the full SHA
    15f9db3 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2022

  1. Add TombstoneLoadSpec.class to mapper in test

    Agustin Gonzalez committed Mar 1, 2022
    Configuration menu
    Copy the full SHA
    1f6a80e View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2022

  1. Merge branch 'master' into batch_ingestion_replace

    Agustin Gonzalez committed Mar 5, 2022
    Configuration menu
    Copy the full SHA
    db30f5f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into batch_ingestion_replace

    Agustin Gonzalez committed Mar 5, 2022
    Configuration menu
    Copy the full SHA
    ca1d5a2 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2022

  1. Merge branch 'master' into batch_ingestion_replace

    Agustin Gonzalez committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    36d638c View commit details
    Browse the repository at this point in the history

  2. Update core/src/main/java/org/apache/druid/segment/loading/TombstoneL… 

    …oadSpec.java


Typo

Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
    @jon-wei
    Agustin Gonzalez and jon-wei authored Mar 8, 2022
    Configuration menu
    Copy the full SHA
    c14cdfc View commit details
    Browse the repository at this point in the history

  3. Update docs/configuration/index.md 

    Missing

Co-authored-by: Jonathan Wei <jon-wei@users.noreply.github.com>
    @jon-wei
    Agustin Gonzalez and jon-wei authored Mar 8, 2022
    Configuration menu
    Copy the full SHA
    fcd2833 View commit details
    Browse the repository at this point in the history

  4. Typo

    Agustin Gonzalez committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    d6a0179 View commit details
    Browse the repository at this point in the history

  5. Integrated replace with an existing test since the replace part was r… 

    …edundant and more importantly, the test file was very close or exceeding the 10 min default "no output" CI Travis threshold.
    Agustin Gonzalez committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    c966bdb View commit details
    Browse the repository at this point in the history

  6. Range does not work with multi-dim

    Agustin Gonzalez committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    933a3fc View commit details
    Browse the repository at this point in the history