✨ Better perf using the cursor.

[jortel]

To support pagination, the hub sets the X-Total header to tell the caller (The UI really) the total number available to be displayed as the total number of pages. To accomplish this, the hub executes the query 2 times. First to get the count(*), again to fetch the data. This doubles the latency and overhead in the hub for each request. Unfortunately, the sq.Rows does not provide the count.

Better performance iterating the cursor (sql.Rows) instead of running the query twice. This also eliminates the race condition between the count (query) and fetch query.

Adds a new cursor object used to consistently:

• execute the query
• iterate the sql.Rows
• scan
• paginate
• count

The count is capped at 50,000 because for tables (like incident) the total can be tens-hundreds of millions. Iterating the rows to determine the count becomes expensive (cpu/time) with those numbers. The X-Total header is only used to support/improve pagination. After a certain point, extremely large numbers become meaningless since users will never page though them. After a discussion with @mturley, the hub will set X-Total with a leading > when capped. Example: X-Total: >50000. The UI will display accordingly.

50,000 may/may-not be the appropriate number.

PoC using the largest table Incident and one of the reports: RuleReport.

[mansam]

LGTM