Skip to content

Commit

Permalink
✨ Better perf using the cursor. (#416)
Browse files Browse the repository at this point in the history
To support pagination, the hub sets the X-Total header to tell the
caller (The UI really) the total number available to be displayed as the
total number of pages. To accomplish this, the hub executes the query 2
times. First to get the count(*), again to fetch the data. This doubles
the latency and overhead in the hub for each request. Unfortunately, the
sq.Rows does not provide the count.

Better performance iterating the _cursor_ (sql.Rows) instead of running
the query twice. This also eliminates the race condition between the
count (query) and fetch query.

Adds a new _cursor_ object used to consistently:
- execute the query
- iterate the sql.Rows
- scan
- paginate
- count

The count is capped at 50,000 because for tables (like incident) the
total can be tens-hundreds of millions. Iterating the rows to determine
the count becomes expensive (cpu/time) with those numbers. The X-Total
header is only used to support/improve pagination. After a certain
point, extremely large numbers become meaningless since users will never
page though them. After a discussion with @mturley, the hub will set
X-Total with a leading `>` when capped. Example: `X-Total: >50000`. The
UI will display accordingly.

50,000 may/may-not be the appropriate number.

PoC using the largest table `Incident` and one of the reports:
RuleReport.

---------

Signed-off-by: Jeff Ortel <jortel@redhat.com>
  • Loading branch information
jortel committed Jun 22, 2023
1 parent e7a5936 commit 908a0a3
Show file tree
Hide file tree
Showing 3 changed files with 321 additions and 243 deletions.
Loading

0 comments on commit 908a0a3

Please sign in to comment.