Add multiprocessing to speed up analysis

[jennyfolkesson]

An straightforward division would be to split jobs into individual images.

[mattersoflight]

This will be great feature to have - thanks for proposing. This pipeline will run on ubuntu, mac, jetson nano (initially) and windows (potentially). So let's find a multi-threading library that works across them.

Also, I think we should wait to implement multi-processing until a single pipeline with modules for robust spot detection, background estimation, and OD calculation is established (#23), and those modules are refactored for readability (#29)

[jennyfolkesson]

It was @bryantChhun s initial suggestion hidden as a comment in the code, I just moved it to be an issue instead.

I've used python's concurrent.futures a lot before so I can add that later. Hopefully it will work across platforms if the same versions are used but I can only test on a mac.

[bryantChhun]

i implemented this on a local branch a while ago using multiprocessing.Pool. It's very doable! I'm concerned a jetson nano would break however.

The only technical difficulty is that the OD .xlsx file requires input from every image. My approach was to have each process submit the OD results to a multiprocessing.Queue that is managed by a multiprocessing.Manager. After everything is finished, you can pull from Queue, sort, then write.

[jennyfolkesson]

Yes, I'm not worried about sorting the returned results from each image for the xlsx file.
I'm not familiar at all with Jetsons though, what about them makes you think multiprocessing might break on them?

[bryantChhun]

They're more powerful raspberry pi's that have a GPU.
https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-nano/?nvid=nv-int-mn-78458
But based on these specs, it looks quite strong and will probably deal with our pipeline just fine