perf: improve GetExperiments showTrialData performance

[NicholasBlaskey]

Description

Use bestTrialID to improve the performance of get experiments.

We already do this for search experiments.

Test Plan

integration test should cover it

Commentary (optional)

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

Ticket

[netlify]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	620d22c
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/65bcf3e10dea200008dda8d7
😎 Deploy Preview	https://deploy-preview-8753--determined-ui.netlify.app/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (71db4e1) 47.72% compared to head (620d22c) 47.72%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8753   +/-   ##
=======================================
  Coverage   47.72%   47.72%           
=======================================
  Files        1049     1049           
  Lines      167250   167238   -12     
  Branches     2241     2243    +2     
=======================================
+ Hits        79816    79818    +2     
+ Misses      87276    87262   -14     
  Partials      158      158           

Flag	Coverage Δ
backend	43.25% <100.00%> (+0.02%)	⬆️
harness	64.32% <ø> (ø)
web	42.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
master/internal/api_experiment.go	54.87% <100.00%> (+0.49%)	⬆️

... and 5 files with indirect coverage changes

[NicholasBlaskey]

original query plan

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.86..25185.26 rows=200 width=788) (actual time=0.268..1054.195 rows=200 loops=1)
   ->  Nested Loop Left Join  (cost=0.86..5364906.34 rows=42605 width=788) (actual time=0.268..1054.166 rows=200 loops=1)
         ->  Nested Loop  (cost=0.43..36612.91 rows=42605 width=400) (actual time=0.033..0.997 rows=200 loops=1)
               Join Filter: (p.workspace_id = w.id)
               Rows Removed by Join Filter: 400
               ->  Nested Loop  (cost=0.43..34886.37 rows=42605 width=367) (actual time=0.027..0.824 rows=200 loops=1)
                     Join Filter: (e.project_id = p.id)
                     ->  Nested Loop  (cost=0.43..32115.98 rows=42605 width=326) (actual time=0.017..0.661 rows=200 loops=1)
                           ->  Index Scan using experiments_pkey on experiments e  (cost=0.29..25151.98 rows=42605 width=298) (actual time=0.012..0.263 rows=200 loops=1)
                           ->  Index Scan using users_pkey on users u  (cost=0.14..0.16 rows=1 width=32) (actual time=0.001..0.001 rows=1 loops=200)
                                 Index Cond: (id = e.owner_id)
                     ->  Materialize  (cost=0.00..1.07 rows=5 width=45) (actual time=0.000..0.000 rows=1 loops=200)
                           ->  Seq Scan on projects p  (cost=0.00..1.05 rows=5 width=45) (actual time=0.005..0.006 rows=1 loops=1)
               ->  Materialize  (cost=0.00..1.04 rows=3 width=37) (actual time=0.000..0.000 rows=3 loops=200)
                     ->  Seq Scan on workspaces w  (cost=0.00..1.03 rows=3 width=37) (actual time=0.003..0.003 rows=3 loops=1)
         ->  Memoize  (cost=0.43..3.76 rows=1 width=36) (actual time=0.000..0.000 rows=0 loops=200)
               Cache Key: e.best_trial_id
               Cache Mode: logical
               Hits: 199  Misses: 1  Evictions: 0  Overflows: 0  Memory Usage: 1kB
               ->  Index Scan using trials_pkey on runs  (cost=0.42..3.75 rows=1 width=36) (actual time=0.001..0.002 rows=0 loops=1)
                     Index Cond: (id = e.best_trial_id)
         SubPlan 1
           ->  Aggregate  (cost=5.02..5.03 rows=1 width=8) (actual time=0.143..0.143 rows=1 loops=200)
                 ->  Index Only Scan using trials_experiment_id_external_trial_id_unique on runs runs_1  (cost=0.42..4.94 rows=30 width=0) (actual time=0.014..0.124 rows=358 loops=200)
                       Index Cond: (experiment_id = e.id)
                       Heap Fetches: 0
         SubPlan 2
           ->  Limit  (cost=118.16..118.16 rows=1 width=16) (actual time=4.938..4.938 rows=1 loops=200)
                 ->  Sort  (cost=118.16..118.24 rows=30 width=16) (actual time=4.938..4.938 rows=1 loops=200)
                       Sort Key: (CASE WHEN COALESCE((((e.config -> 'searcher'::text) ->> 'smaller_is_better'::text))::boolean, true) THEN runs_2.searcher_metric_value ELSE ('-1'::double precision * runs_2.searcher_metric_value) END)
                       Sort Method: top-N heapsort  Memory: 25kB
                       ->  Index Scan using ix_trials_experiment_id on runs runs_2  (cost=0.42..118.01 rows=30 width=16) (actual time=0.095..4.861 rows=358 loops=200)
                             Index Cond: (experiment_id = e.id)
 Planning Time: 1.035 ms
 Execution Time: 1054.508 ms
(35 rows)

updated query plan

                                                                                       QUERY PLAN                                                                                        
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.86..3239.81 rows=200 width=788) (actual time=0.220..16.458 rows=200 loops=1)
   ->  Nested Loop Left Join  (cost=0.86..689976.47 rows=42605 width=788) (actual time=0.219..16.430 rows=200 loops=1)
         ->  Nested Loop  (cost=0.43..36612.91 rows=42605 width=400) (actual time=0.032..0.649 rows=200 loops=1)
               Join Filter: (p.workspace_id = w.id)
               Rows Removed by Join Filter: 400
               ->  Nested Loop  (cost=0.43..34886.37 rows=42605 width=367) (actual time=0.026..0.515 rows=200 loops=1)
                     Join Filter: (e.project_id = p.id)
                     ->  Nested Loop  (cost=0.43..32115.98 rows=42605 width=326) (actual time=0.016..0.401 rows=200 loops=1)
                           ->  Index Scan using experiments_pkey on experiments e  (cost=0.29..25151.98 rows=42605 width=298) (actual time=0.010..0.175 rows=200 loops=1)
                           ->  Index Scan using users_pkey on users u  (cost=0.14..0.16 rows=1 width=32) (actual time=0.001..0.001 rows=1 loops=200)
                                 Index Cond: (id = e.owner_id)
                     ->  Materialize  (cost=0.00..1.07 rows=5 width=45) (actual time=0.000..0.000 rows=1 loops=200)
                           ->  Seq Scan on projects p  (cost=0.00..1.05 rows=5 width=45) (actual time=0.006..0.006 rows=1 loops=1)
               ->  Materialize  (cost=0.00..1.04 rows=3 width=37) (actual time=0.000..0.000 rows=3 loops=200)
                     ->  Seq Scan on workspaces w  (cost=0.00..1.03 rows=3 width=37) (actual time=0.003..0.004 rows=3 loops=1)
         ->  Memoize  (cost=0.43..3.76 rows=1 width=36) (actual time=0.000..0.000 rows=0 loops=200)
               Cache Key: e.best_trial_id
               Cache Mode: logical
               Hits: 199  Misses: 1  Evictions: 0  Overflows: 0  Memory Usage: 1kB
               ->  Index Scan using trials_pkey on runs  (cost=0.42..3.75 rows=1 width=36) (actual time=0.001..0.001 rows=0 loops=1)
                     Index Cond: (id = e.best_trial_id)
         SubPlan 1
           ->  Aggregate  (cost=5.02..5.03 rows=1 width=8) (actual time=0.046..0.046 rows=1 loops=200)
                 ->  Index Only Scan using trials_experiment_id_external_trial_id_unique on runs runs_1  (cost=0.42..4.94 rows=30 width=0) (actual time=0.002..0.029 rows=358 loops=200)
                       Index Cond: (experiment_id = e.id)
                       Heap Fetches: 0
         SubPlan 2
           ->  Index Scan using trials_pkey on runs runs_2  (cost=0.42..8.44 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=200)
                 Index Cond: (id = e.best_trial_id)
 Planning Time: 0.999 ms
 Execution Time: 16.615 ms
(31 rows)