Core Arbiter Notes

The core arbiter sits between the Arachne library and the kernel, and decides how to allocate cores amongst multiple applications.

Startup

• The arbiter needs to be told which cores it will control
  • Throw an error if given all cores. At least one needs to be reserved for itself and other processes
• Kick all other processes (including itself) off of those cores. The arbiter runs at high priority (possibly even sched_fifo).
• Later, we can add the functionality to only take cores from the kernel when needed and to give them back when possible.

Communication

Talking to the arbiter

• Use a domain socket to talk to the arbiter
• connect(): Use at startup to establish connection with Arbiter
  • Exchange information about processId, shared memory page
• setNumCores(priorityArray[]): Given an array of the number of cores to allocate at each priority, the arbiter will allocate a certain number and communicate to the application via its shared memory page how many it has.
  • send(priorities)
• addThread(threadId): Called once per kernel thread to inform the arbiter about a processId/threadId relationship
  • send(threadID)
  • connect()
• blockUntilCoreAvailable(): Called by individual threads
  • send(): Tells the arbiter that the given process has a thread that is going to block until given a core.
  • recv(): Blocks the caller until the arbiter decides to give the application a core (it's important that send be called first so the Arbiter is aware that the thread is waiting on it)

Reading from the arbiter

• Use a shared memory page to read from the arbiter (makes polling efficient).
• This page tells the application which cores it has available.
• Use shm_open()

Internals

Arbitration Loop

create a domain socket for communication with applications (appSocket)
initialize set of sockets to epoll on to contain just appSocket's file descriptor
while (true):
    epoll (blocking)
    if new application joining:
        add processId to list of processes
        add new file descriptor to epoll list
    else if new thread is joining the pool:
        add threadId to list of threads for the sending process
        add new file descriptor to epoll list
    else if blockUntilCoreAvailable:
        set thread's state information to sleeping
        if we have other applications waiting for cores:
            putThreadOnCore()
    else if setNumCores:
        decide on how many cores to grant
        for each available core:
            putThreadOnCore()
        for the unavailable cores:
            update shared memory pages of other applications to request they give back cores
            set a timer for each application we're taking cores from
    else if timeout on request to application:
        move as many threads as you need to the general cpuset
        for each thread waiting to be woken up:
            putThreadOnCore()

function putThreadOnCore(processId, coreId): add thread to appropriate cpuset update shared memory page send(coreId) to a blocked thread that belongs to the given process

Arbitration Policy

• If an application does not give up a core in a timely manner when asked to, the arbiter demotes threads by putting them in the cpuset with itself and all the other non-Arachne threads.
• TODO: come up with a more detailed policy

Arachne Side

The Arachne library creates as many threads as there are cores. Each thread has the following main loop:

while (true):
    blockUntilCoreAvailable()
    run until the arbiter tells us via shared memory that we should yield

Questions

• Arbitration policy?
• How are we going to unit test?
• Logging?