**AI Providers** - easy Queue implementation

[bigcat88]

Regarded yesterday discussion about Async AI providers

From the Nextcloud server side all requests look like this:

https://github.com/nextcloud/server/blob/fdc64ea2f527d25c382901ed906f71fca89fd1b3/lib/public/TextProcessing/Task.php#L98-L107

We can, from our implementation side, without changing the NC Core, make it asynchronous in the implementation of providers, but we will have to add a table with ID queries.

How we can do it changing only AppAPI:

1. We create a separate table for each implemented Provider Type:
   id: bigint | finished: bool | error: string | result: depends on provider

2. We add a route in AppAPI for each provider type to receive the results

3. During "process" we give each request it's ID and send request to ExApp, after that we in cycle check when record for this ID will have "finished" to true

We rewrite this and take results from task from our new DB table instead from getting it from response

https://github.com/cloud-py-api/app_api/blob/f169bae7b4a3088a6991b0482f1a333ade836f33/lib/Service/TextProcessingService.php#L198-L206

4. When ExApp Provider finished processing task he informs AppAPI Provider Implementation (2) and record in DB got filled with the results.
5. Step 3: "process" returns result after that.

This proposed solution will allow ExApps providers to easy implement queue, on getting request it will add request data to queue and return empty HTTP response for example.
After ExApp finish processing request it will store results of it using route added in Point 2.

@APP.post("/stable_diffusion")
async def stable_diffusion(
    task_data: Annotated[...],
    background_tasks: BackgroundTasks,
):
    TASK_LIST.put(task_data, block=True)
    return Response()
    
class BackgroundProcessTask(threading.Thread):
    def run(self, *args, **kwargs)
        while True:
            r = process_task()  # process one entry from TASK_LIST - **blocking**
            inform_about_result(r) # async backround informing without blocking this thread

Any thoughts , criticism?

[andrey18106]

We would need adjustments for OCP providers:

1. Add interface with processBatch method or option (bool value, showing should ExApp Provider unload model immediately after processing task)
2. Add to providers missing interfaces to keep consistency between them to always have userId and custom ID (e.g. ISpeechToTextProviderWithUserId).

cc @marcelklehr @julien-nc

[marcelklehr]

Add interface with processBatch method or option (bool value, showing should ExApp Provider unload model immediately after processing task)

Could we implement some kind of GPU VRAM booking in app_api? Like, an ex-app gets a task, then requests X VRAM from app_api to load the model, once the VRAM is booked, it can load the model, process the task, and perhaps employ a timeout to run more incoming tasks. Once no tasks have reset the timeout anymore, the VRAM is freed and other ex-apps can use it.

[andrey18106]

One more note:

3. The same as for other providers to keep consistency between them - we need IProviderWithUserId for TextToImage OCP as well.

[bigcat88]

Steps for this:

1. Implement ExApp for collecting hardware data, that will be run during daemon setup
2. Implement API in AppAPI for booking RAM/VRAM
3. Implement in AppAPI feature for allowing of resettings(reboot) apps

Probably, this is a hard theme how to correctly do it, can discuss it in Talk :)

P.S. Sami showed me a really nice scheme with idea close to this what you described

[marcelklehr]

Hah, I love it.