WaitForPodsReady: a mode where jobs don't block the queue head

[ahg-g]

What would you like to be added:
A mode of operation for WaitForPodsReady where jobs don't block the head of the queue, but still get suspended if they aren't ready after a while.

Why is this needed:
Blocking the queue until a Job is ready guarantees all-or-nothing scheduling, but it is slow at scale. Consider the case where a large number of jobs are awaiting to be scheduled and suddenly lots of resources become available (e.g., a large job finishes, releasing significant amount of resources).

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

We probably would start optimistically admitting every workload and then setting some kind of backoff when resources are unavailable.

Should the backoff be per flavor?

Note: not expecting an answer... just dumping my current open questions :)

[KunWuLuan]

Hi, I think we can do two more things to help solve the problem:

1. continue admitting more workloads until there is no more resources in cohort
2. requeue the current workload if it wait too long for pods ready

How do you think? @alculquicondor @ahg-g

[KunWuLuan]

/assign

[trasc]

Hi @KunWuLuan,

2. requeue the current workload if it wait too long for pods ready

is done in #599 / #689 .

I think, what we need to, is to investigate the effect of dropping the s.cache.WaitForPodsReady(ctx) in

kueue/pkg/scheduler/scheduler.go

Lines 179 to 189 in 9ca57c8

	if !s.cache.PodsReadyForAllAdmittedWorkloads(ctx) {
	log.V(5).Info("Waiting for all admitted workloads to be in the PodsReady condition")
	// Block admission until all currently admitted workloads are in
	// PodsReady condition if the waitForPodsReady is enabled
	if err := workload.UnsetAdmissionWithCondition(ctx, s.client, e.Obj, "Waiting", "waiting for all admitted workloads to be in PodsReady condition"); err != nil {
	log.Error(err, "Could not update Workload status")
	}
	s.cache.WaitForPodsReady(ctx)
	log.V(5).Info("Finished waiting for all admitted workloads to be in the PodsReady condition")
	}
	}

and continue from there.

[KunWuLuan]

Hi, @trasc I think you are right. 💯

Moreover, maybe we can add a switch to let user choose whether to block the admission while still waiting for pods ready.
If false, we just skip all these checking in

kueue/pkg/scheduler/scheduler.go

Lines 178 to 189 in 9ca57c8

	if s.waitForPodsReady {
	if !s.cache.PodsReadyForAllAdmittedWorkloads(ctx) {
	log.V(5).Info("Waiting for all admitted workloads to be in the PodsReady condition")
	// Block admission until all currently admitted workloads are in
	// PodsReady condition if the waitForPodsReady is enabled
	if err := workload.UnsetAdmissionWithCondition(ctx, s.client, e.Obj, "Waiting", "waiting for all admitted workloads to be in PodsReady condition"); err != nil {
	log.Error(err, "Could not update Workload status")
	}
	s.cache.WaitForPodsReady(ctx)
	log.V(5).Info("Finished waiting for all admitted workloads to be in the PodsReady condition")
	}
	}

. Then the other jobs can continue being admitted until resources are exhausted. WDYT?

@alculquicondor If you have time, you can also participate in the discussion, which will be of great help. Thank you very much. 😆 👍

[alculquicondor]

@KunWuLuan thanks for your feedback. I'm currently with limited availability as I'm attending kubecon. I'll get back to this thread next week.
If you have some time, feel free to review the open PRs listed above.

But in general, this feature should be optional.

[alculquicondor]

/close
Fixed in #708

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close
Fixed in #708

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.