Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds trough dedup performance rework #388

Draft
wants to merge 18 commits into
base: master-ait-contrib
Choose a base branch
from
Draft

Adds trough dedup performance rework #388

wants to merge 18 commits into from

Commits on Jan 28, 2021

  1. Fixing synchronization issues in TroughCrawlLogFeed.java to reduce bl… 

    …ocking
    @adam-miller
    adam-miller committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    14be875 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2021

  1. Trying to avoid hard locks on updating url caches

    @adam-miller
    adam-miller committed Feb 2, 2021
    Configuration menu
    Copy the full SHA
    d22ef12 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2021

  1. Fix error log reporting of batch size for trough crawl logs

    @adam-miller
    adam-miller committed Feb 18, 2021
    Configuration menu
    Copy the full SHA
    79e5633 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2021

  1. Create and insert into local sqlite dedup db shards

    @adam-miller
    adam-miller committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    86e238e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-… 

    …trough-dedup-performance-rework
    @adam-miller
    adam-miller committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    09485a5 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-d… 

    …edup-performance-rework
    @adam-miller
    adam-miller committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    e300051 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2021

  1. Fixing sync issues on uncrawled urls batch as well

    @adam-miller
    adam-miller committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    00947b0 View commit details
    Browse the repository at this point in the history

  2. On disk - per-seed sqlite dedup

    @adam-miller
    adam-miller committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    88b7eea View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2021

  1. Switching to in memory sqlite for dedup cache

    @adam-miller
    adam-miller committed Feb 26, 2021
    Configuration menu
    Copy the full SHA
    e700f38 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2021

  1. Replacing some synchronized lists with threadsafe objects and fixing … 

    …discrepency between crawled and uncrawled lists
    @adam-miller
    adam-miller committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    8724b14 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'adds-trough-local-dedup-write-cache' into adds-trough-d… 

    …edup-performance-rework
    @adam-miller
    adam-miller committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    54abea3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fixes-trough-crawl-log-feed-synchronization' into adds-… 

    …trough-dedup-performance-rework
    @adam-miller
    adam-miller committed Mar 10, 2021
    Configuration menu
    Copy the full SHA
    466c679 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2021

  1. Ensure that the final batch is posted during shutdown. Make posting o… 

    …f uncrawled urls optional.
    @adam-miller
    adam-miller committed Apr 5, 2021
    Configuration menu
    Copy the full SHA
    d721756 View commit details
    Browse the repository at this point in the history

  2. Fix inconsistency with cached batch sizes and refine logic for forcin… 

    …g final batch posting at end of crawl
    @adam-miller
    adam-miller committed Apr 5, 2021
    Configuration menu
    Copy the full SHA
    0b09340 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2021

  1. Refactor crawl log batch posting. Add configuration options for troug… 

    …h dedup batches. Fix posting of trough dedup at end of crawl.
    @adam-miller
    adam-miller committed May 13, 2021
    Configuration menu
    Copy the full SHA
    18edb3e View commit details
    Browse the repository at this point in the history

Commits on May 17, 2021

  1. Ensure we set dedup schema when writing dedup shards

    @adam-miller
    adam-miller committed May 17, 2021
    Configuration menu
    Copy the full SHA
    6fad293 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2021

  1. Add dedup load from in-memory dedup cache db

    @adam-miller
    adam-miller committed May 20, 2021
    Configuration menu
    Copy the full SHA
    7fc648c View commit details
    Browse the repository at this point in the history

Commits on May 28, 2021

  1. Refactor Trough client URL cache put. Limit TroughContentDigestHistor… 

    …y query.
    @adam-miller
    adam-miller committed May 28, 2021
    Configuration menu
    Copy the full SHA
    58bf0cf View commit details
    Browse the repository at this point in the history