FEAT-#7202: Use custom resources for Ray

[YarShev]

What do these changes do?

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
• passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
• signed commit with git commit -s
• Resolves Use custom resources for Ray to schedule a task on a concrete node #7202
• tests passing
• module layout described at docs/development/architecture.rst is up-to-date

[anmyachev]

@YarShev do we need to adjust the procedure for determining the total available number of cores, depending on these custom resources?

[YarShev]

Custom resources have nothing to do with num_cpus so no need for adjustment.

[anmyachev]

Custom resources have nothing to do with num_cpus so no need for adjustment.

Isn’t it possible to use these resources to limit the number of nodes on which calculations will be launched? It turns out that we will be dividing into a much larger number of partitions than can be executed in parallel.

[YarShev]

Your thoughts pushed me to a problem in the current setup. The issue is that if the user sets resources={"special_hardware": 1}, we pass this parameter as is in remote functions. This way we limit the parallelism to only one remote task to be executed. I would think of the following preprocessing resources to further pass those in remote functions. What do you think?

resources_per_task = {}

for k, v in RayCustomResources.get():
    resources_per_task[k] = v / v / num_cpus

[anmyachev]

Can we add a test for this case?

[YarShev]

What exactly would you like to test with this?

[anmyachev]

We do not test the situation when this config is different from None.

[anmyachev]

It's good that there is now an option to limit concurrency, but this only works for Ray. Let's create an issue for the rest of the engines.

[YarShev]

This config is not generic but only specific to Ray. I am not sure if there is a way to limit concurrency for other engines. I think if we will want to have something similar for other engines, we will open an issue and explore options if they are there. Do you still think we should create an issue now?

[anmyachev]

Limiting concurrency in context looks like a good feature for an advanced user. We can create a low priority issue, but now.

[YarShev]

#7211

[anmyachev]

I wonder if we should just call it RayTaskResources? (the same for RayInitResources) Since this config is used to pass values to resources.

[YarShev]

I would prefer to be explicit here as Ray itself calls it as custom resources - https://docs.ray.io/en/latest/ray-core/scheduling/resources.html#custom-resources.