fix: slots being filled returned out of order on k8s [RM-42]

[NicholasBlaskey]

Ticket

Description

Fix issues where slots were being returned out of order. There are two fixes in this PR

1. Make k8s use slot IDs that will return sorted. For more than 10 gpus on an agent the response would be out of order
2. The issue in the ticket, rbac would jumble the order.

Test Plan

Unit tests cover this pretty well

Manual

Run a devcluster with

        security:
          authz:
            type: rbac

and agent

        artificial_slots: 8

run a experiment, create a user with no access det -u admin user create test

check agents det -u test dev curl /api/v1/agents and see that the contaner is on the same slot id every refresh

Checklist

• Changes have been manually QA'd
• User-facing API changes need the "User-facing API Change" label.
• Release notes should be added as a separate file under docs/release-notes/.
  See Release Note for details.
• Licenses should be included for new code which was copied and/or modified from any external code.

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	341ab9c
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/663262020d28e10008b45dcb

[netlify]

✅ Deploy Preview for determined-ui ready!

Name	Link
🔨 Latest commit	aaadd0f
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/6633935b793dcf0009b63e89
😎 Deploy Preview	https://deploy-preview-9276--determined-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

Attention: Patch coverage is 91.30435% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 44.60%. Comparing base (a9c8700) to head (aaadd0f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9276      +/-   ##
==========================================
+ Coverage   44.58%   44.60%   +0.01%     
==========================================
  Files        1275     1275              
  Lines      156214   156226      +12     
  Branches     2451     2450       -1     
==========================================
+ Hits        69655    69679      +24     
+ Misses      86319    86307      -12     
  Partials      240      240              

Flag	Coverage Δ
backend	41.80% <91.30%> (+0.03%)	⬆️
harness	64.09% <ø> (ø)
web	35.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
master/internal/authz/obfuscate.go	54.09% <100.00%> (+33.72%)	⬆️
master/pkg/model/agent.go	1.81% <100.00%> (+1.81%)	⬆️
master/internal/rm/kubernetesrm/pods.go	21.82% <83.33%> (+0.32%)	⬆️

... and 3 files with indirect coverage changes