Implement Protothreads as Fibers

[salkinium]

Adds a modm:processing:protothread:use_fiber option that makes the modm::Protothread behave like a fiber with a default stack.
The only change required is to replace the main loop calling all the protothread run() functions with the modm::fiber::Scheduler::run(); function.

• Remove modm:__fibers guard option and make the modm:processing:fiber module available by default.
• ReImplement the Protothread class by inheriting from the Fiber class.
• Reimplement the RF_* and PT_* macros.
• Adapt the manual implementations of resumable function (SPI transfer) to use yields.
• Separate Stack and Task and combine into Fiber class.
• Much more Documentation
• Simpler instantiation of Scheduler class. One scheduler class per (hardware) thread, fibers must be added only within the thread context, not from the outside.
• Fibers can be restarted now!
• Simpler unregister scheme that save a bit of stack.
• More accurate minimal stack sizes.
• Stack usage measurement through watermarking!
• Simpler context API.
• Fibers can be placed into .noinit sections, since they are constructed only at runtime.

[chris-durand]

Thanks for starting to work on this.

I have the feeling that the resumable function compatibility layer could be much simpler than the proposed solution. The resumable function mechanism itself shouldn't be required anymore using fibers. Can't we define all the resumable classes as empty no-ops and just use modm::yield() from the WAIT_ macros? Many other macros could be doing nothing. RF_BEGIN/END could be empty, RF_CALL(x()) is x(), RF_RETURN(x) is return x, ResumableResult<T> is an alias for T and so on? Or do we really need the nesting level checking in fiber mode?

I really want to get rid of all the potential undefined behaviour issues with resumable functions (like in #993) when fibers are enabled. EDIT: Your solution should not have that anymore.

A long time ago I have played around with this. That was my first prototype of that simple concept which completely ignored protothreads, but it let's you use resumable drivers from fibers: chris-durand@5f3c900

[salkinium]

Yeah, good point. I wanted to have something completely drop-in replacable, but it makes no sense to only go half-way.

[salkinium]

I made the resumables transparent now, which is nice, because you can now simply call "ex-resumabe" functions without any macros at all. This even works without a scheduler at all (yield() simply returns in-place), which is the case inside the main function or when there is only one fiber running. So wins all around.

[salkinium]

Ok, I've refactored a lot of the implementation to make fiber consume a little bit less stack and be restartable.
Long-term goals to be implemented some time later:

• Support for signalling from interrupt context, ie. wake this fiber up.
• Support for Semaphores and Mutexes, Message Queues between fibers.
• Support for schedulers in FreeRTOS threads, using thread-local storage.
• Perhaps use the FPU lazy-stacking feature to speed up the fiber context switch.
• Use Stack Limit Registers on ARMv8-M devices.
• Refactor more examples and more drivers to use fibers directly.
• Add static stack usage calculator, integrate puncover? watermarking

[salkinium]

Ok, this is now quite polished. I refactored the context API and implementation so that stack watermarking can be used to measure stack usage on all platforms. Fixed some minor things on AVR. I'm very happy with this now.

If you want to review and it's a bit much, then focus more on the usability of the fiber API, rather than the implementation. It's easier to get the implementation right, since it simply crashes if its wrong ;-P

[salkinium]

Man, this crap macOS CI is a pain to maintain without containers.

[salkinium]

I added a bunch more doxygen documentation and moved the stack usage into the fiber context.

[chris-durand]

Very nice!

[salkinium]

Alright, thanks!

My next plans are to implement light-weight mutex, semaphore, signals, and queues, so that we can have a smarter/simpler scheduler than the linked list right now.

My ultimate goal is to have a composable fiber implementation that still works with FreeRTOS above and C++ coroutines below:

• Completely Separate Devices
  • Asynchronous MultiCore Execution
    • Preemptive, Stackful Threads
      • Cooperative, Stackful Fibers
        • Cooperative, Stackless C++ Coroutines or C Protothreads/Resumables
          • Blocking Functions
            • Nested Interrupts

[hshose]

Huge shout-out to everyone who helped implement the Fibers 🥇

I just started using them for the first time in a larger project and about 1h in, it feels so much nicer then the @%$!!@# PROTOTHREADS already 😸

Although, default stack size could be 1k (at leasts thats what I need for even mild use of MODM_LOG_XXX)

Also, my systems default arm gcc 13 can be used again, no more #1012 ❤️

Maybe, this comment should have gone to #743 too