Tasks: Add support for task context switching #50

roy-hopkins · 2023-06-19T15:33:51Z

Draft PR for review and comments.

Implements a mechanism to switch the current CPU between different task contexts that include per-task pagetables and will in the future support per-task virtual memory mapping.

The current version creates a global task list and launches an initial task on each virtual CPU. Additional tasks can be created and added to the task list which will then be activated by a subsequent call to schedule(). This has been tested locally but is not currently called within this PR.

The current X86Regs structure is only used by exception handlers and interrupt routines and as such contains the interrupt routine stack frame within the structure. This patch separates that out to break registers and other context into their respective categories to allow the structures to be reused in other situations. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

src/task/tasklist.rs

src/cpu/percpu.rs

roy-hopkins · 2023-07-03T10:25:13Z

I've just pushed V2 of this draft PR which has some significant changes to the previous version. In particular I wanted to address the use of a self-implemented linked list, as commented on above and also add the ability to support more advanced scheduling from the outset. So I've made the following big changes:

Replaced the linked list in task.rs and tasklist.rs with an RB tree implemented using an imported crate in a new file: schedule.rs
Used Box to allocate memory and manage the lifetime of the Task structures.
Let the task tree (in TASKS) own the tasks themselves but use reference counting for extending the lifetime where necessary. Note here though that if an additional reference to a task is held then it is not possible to mutate the task during scheduling and a runtime panic will be generated.
Use the sorting of the RB tree on CPU usage to determine the next task to schedule
Implement two different ways of measuring CPU usage: switch count and rdtsc.

The memory management/borrowing/reference counting in this implementation was particularly tricky to get right. I'd appreciate it if reviewers could pay particular attention to this area of task ownership.

joergroedel

In general this looks good to me. The task switching code is a good start and the beginnings of per-task kernel memory are also there. As one of the next steps we probably need an address allocator to manage the kernel task dynamic memory and the user-space address layout as well. I will look into this going forward.

src/cpu/registers.rs

src/cpu/tsc.rs

src/task/task.rs

roy-hopkins · 2023-07-05T07:08:38Z

As one of the next steps we probably need an address allocator to manage the kernel task dynamic memory and the user-space address layout as well. I will look into this going forward.

I haven't started thinking about user-space address layout at the moment but I have used the current bitmap allocator to implement a per task kernel address allocator in my WIP PR for module loading in my own fork. roy-hopkins#1.

src/task/schedule.rs

00xc · 2023-07-05T10:06:47Z

src/task/task.rs

+        Ok(task)
+    }
+
+    pub fn set_current(&mut self, previous_task: Option<*mut Task>) {


As before, Option<*mut Task> could perhaps be switched to Option<NonNull<Task>>, since we use None to signify null.

Also, does it need to be *mut? Could it be *const?

Yes. In fact, I've just working on this and have changed this to use a raw pointer instead as the null pointer is handled in the assembly code in set_current().

00xc · 2023-07-05T10:19:31Z

src/task/schedule.rs

+    // Restrict the scope of the mutable borrow below otherwise when the task context
+    // is switched via set_current() the borrow remains in scope.
+    let task_ptr = {
+        let mut task = task_node.task_mut();
+        if task.state != TaskState::RUNNING {
+            panic!("Attempt to launch a non-running initial task");
+        }
+        this_cpu_mut().current_task = Some(task_node.clone());
+        task.state = TaskState::SCHEDULED;
+        task.as_mut() as *mut Task
+    };
+    unsafe { (*task_ptr).set_current(None) };


Not a huge fan of this unsafe, I wonder if we could remove it. If Task::set_current() took &self instead of &mut self we could avoid it I think. There are two things that prevent this:

This line in Task::set_current():

let new_task_addr = (self as *mut Task) as u64;

But perhaps this could be *const?

The fact that TaskRuntime::schedule_in() takes &mut self (and therefore CountRuntime::schedule_in(). If we could make the trait signature take &self that would be great for this. For CountRuntime this is trivial, we simply need to use an AtomicU64 instead of an u64:

pub struct CountRuntime { count: AtomicU64, } impl TaskRuntime for CountRuntime { fn schedule_in(&self) { self.count.fetch_add(1, Ordering::SeqCst); } ... }

I'm unsure if we could use a more relaxed ordering though. For TscRuntime we could do the same via AtomicU64::store()

This is a tricky one. Certainly 2. is easy - in fact I've just moved TaskRuntime::schedule_in() from this function in the the scheduler itself, so that solves that.

Is a bit harder as set_current() does indeed mutate the task in that the contents will certainly have changed by the time the function returns, even if it happens in the context of another task.

The bigger problem though is that the task structure itself uses runtime borrow checking. Therefore to call it without a pointer means having an active borrow which remains in scope until the task context is resumed. This prevents the scheduler (and the gdbstub) from being able to access the task.

src/task/task.rs

Signed-off-by: Roy Hopkins <rhopkins@suse.de>

The svsm_paging module was referenced only in svsm.rs. This module will need to be used in the task management code which is about to be introduced so the module reference has been added into lib.rs. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

Adds a new function that can be called to read the 64-bit tsc value from the current CPU. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

src/task/schedule.rs

src/task/tasks.rs

This adds basic support for creating tasks which equate to processes or threads that can be scheduled on a vCPU. Each task is represented by a structure that contains the relevant context, which at the moment consists of a task stack, pagetable and stack register. Tasks are added to a task list in TASKS which is implemented as an RB tree which is keyed on the priority of the task: the lower the key, the more likely the task will be switched to. There are two methods for setting the priority; one based on CPU time using rdtsc and the other (the default) the amount of times a task has been scheduled. This patch adds all the infrastructure to support this but does not yet integrate it into the boot processor or secondary APs, so current functionality is unchanged. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

In order to bootstrap task support on the boot CPU, the entry point is created as the initial task on the CPU and added to the task list with an affinity that ensures it only ever runs on the boot CPU. This patch enables future tasks to be created that can be cooperatively switched using a call to schedule() on the boot CPU. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

This change uses the task list to create a new initial task for each AP that bootstraps task support on vCPUs by launching the entry point rather than jumping directly into the request loop. The intial task has an affinity for the specific AP to prevent it from being scheduled on any other CPU. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

This patch makes the gdbstub aware of the task list in SVSM allowing the developer to inspect the state and call stack of any stopped threads regardless of which vCPU the task is running on. Each task has its own pagetable so the gdbstub now uses its own stack and switches to the task pagetable whenever a task context is queried or modified. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

Signed-off-by: Roy Hopkins <rhopkins@suse.de>

roy-hopkins · 2023-07-10T10:00:32Z

Thanks to the people that looked at and reviewed this. I'll close this draft now and raise new PRs based on this work.

roy-hopkins marked this pull request as draft June 19, 2023 15:34

00xc reviewed Jun 20, 2023

View reviewed changes

src/task/tasklist.rs Outdated Show resolved Hide resolved

00xc reviewed Jun 20, 2023

View reviewed changes

src/task/tasklist.rs Outdated Show resolved Hide resolved

00xc reviewed Jun 20, 2023

View reviewed changes

src/cpu/percpu.rs Outdated Show resolved Hide resolved

roy-hopkins force-pushed the task_switching branch from 2c10720 to f441d84 Compare July 3, 2023 10:14

joergroedel reviewed Jul 4, 2023

View reviewed changes

src/cpu/registers.rs Outdated Show resolved Hide resolved

src/cpu/tsc.rs Show resolved Hide resolved

src/task/task.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 5, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 5, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 5, 2023

View reviewed changes

src/task/task.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 5, 2023

View reviewed changes

src/task/task.rs Outdated Show resolved Hide resolved

roy-hopkins added 3 commits July 6, 2023 15:30

svsm/cpu: Fix typo in log message

2ea32f2

Signed-off-by: Roy Hopkins <rhopkins@suse.de>

svsm: Add svsm_paging module to lib.rs

4aa0241

The svsm_paging module was referenced only in svsm.rs. This module will need to be used in the task management code which is about to be introduced so the module reference has been added into lib.rs. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

svsm/cpu: Add rdtsc() to read tsc from current CPU

1b28bac

Adds a new function that can be called to read the 64-bit tsc value from the current CPU. Signed-off-by: Roy Hopkins <rhopkins@suse.de>

roy-hopkins force-pushed the task_switching branch from f441d84 to ab0f6a6 Compare July 6, 2023 14:36

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/schedule.rs Outdated Show resolved Hide resolved

00xc reviewed Jul 6, 2023

View reviewed changes

src/task/tasks.rs Outdated Show resolved Hide resolved

roy-hopkins added 4 commits July 7, 2023 08:31

roy-hopkins force-pushed the task_switching branch from ab0f6a6 to 8698c17 Compare July 7, 2023 07:32

Merge branch 'main' into task_switching

6a30a57

Signed-off-by: Roy Hopkins <rhopkins@suse.de>

roy-hopkins closed this Jul 10, 2023

roy-hopkins mentioned this pull request Jul 13, 2023

task/schedule: Implement scheduler and per-cpu runqueues #64

Merged

roy-hopkins deleted the task_switching branch August 4, 2023 08:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tasks: Add support for task context switching #50

Tasks: Add support for task context switching #50

roy-hopkins commented Jun 19, 2023

roy-hopkins commented Jul 3, 2023

joergroedel left a comment

roy-hopkins commented Jul 5, 2023

00xc Jul 5, 2023

00xc Jul 5, 2023 •

edited

Loading

roy-hopkins Jul 5, 2023

00xc Jul 5, 2023 •

edited

Loading

roy-hopkins Jul 6, 2023

roy-hopkins commented Jul 10, 2023

Tasks: Add support for task context switching #50

Tasks: Add support for task context switching #50

Conversation

roy-hopkins commented Jun 19, 2023

roy-hopkins commented Jul 3, 2023

joergroedel left a comment

Choose a reason for hiding this comment

roy-hopkins commented Jul 5, 2023

00xc Jul 5, 2023

Choose a reason for hiding this comment

00xc Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

roy-hopkins Jul 5, 2023

Choose a reason for hiding this comment

00xc Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

roy-hopkins Jul 6, 2023

Choose a reason for hiding this comment

roy-hopkins commented Jul 10, 2023

00xc Jul 5, 2023 •

edited

Loading

00xc Jul 5, 2023 •

edited

Loading