Threading and Modules

[n1474335]

This pull request introduces threading for the baking process. All operations are stored in a Web Worker and the recipe is executed there. This means that the UI does not lock up when a bake is taking place. It is also now possible for the user to cancel a bake that is taking too long. This terminates the entire Web Worker and reinitialises it.

There are quite a lot of changes here. Most should be self-evident but just ask about any that don't make sense.

I have added a "staleness indicator" to the output which is displayed when the input or recipe changes but a bake does not take place immediately.

I have also added webpack-dev-server as Web Workers don't work with local files - they require an HTTP server. This should make development a bit smoother. The dev version is hosted on localhost:8080 when you run grunt dev.

I imagine there will be bugs in some operations that don't function as expected in a Web Worker. If you find any, please let me know!

Closes #154.

[mattnotmitt]

Doesn't this defeat the purpose of one of our core features - that we can run it from a single file on the clientside? This means users wanting to use it offline will have to install a local HTTP server.

[n1474335]

@artemisbot, I was playing around with this last night and it turns out you can run an inline web worker locally. Chrome was complaining that it was loading the worker file from a "null origin" (not from a server with a host name). If you inline the worker by creating a URL Blob, it does not complain.

I have an inline version working with full web worker support which I will push soon. I'm currently messing around with dynamic module loading which is more tricky to inline.

I'm still very committed to providing the single-file version as I recognise it is very useful in a number of use cases - we won't release these new features until they work properly.

[tlwr]

Maintaining linkability is very important 👍 for catching @artemisbot

@n1474335 good solution, excited to see it in action. XOR Brute Force could be extended as an example of a long-running operation that is enabled by this change.

[n1474335]

@tlwr, absolutely agree about XOR Brute Force. I'll probably make the key length an editable field so you can choose how long you want to wait for. I'm going to add the ability for the worker to send back status messages as well so that users can get progress updates from operations.

I also have some much more wide-ranging plans for decoding operations that might take a while to run. CyberChef v6 (we'll bump the major version number when we merge this PR) will be all about increasing the scale and scope of the web app. The web worker will remove the limitations on processing time and the module system will remove the limitations on code size (think large libraries).

[n1474335]

This one made me scratch my head a bit but I'm quite happy with the final solution. It certainly makes for a much smoother user experience.

• The initial load time is reduced as a lot of large libraries have been moved into separate modules.
• The module system itself works transparently and each individual module is generally small enough that the initial loading of it isn't very noticeable. Just run a hashing operation to see how quickly the Hashing module loads and executes. (This will obviously be dependent on your network speed.)
• The threading support and ability to cancel long-running bakes is really nice. See this recipe for a good example.
• It's still possible to crash the app by trying to load a large file into the input, or generating a huge amount of output. This is something we'll have to address in future, however this update massively reduces the chances of the app crashing as the vast majority of "dangerous" processing happens in the web worker and can be easily terminated and restarted if anything goes wrong.
• The standalone version (cyberchef.htm) works well. It includes all modules packaged up together so does not rely on any external sources. It uses an inline web worker, so bakes can still be cancelled as with the modular version.

Now that the restrictions on file size and processing time have been largely remediated, we can start adding more bulky operations like PGP (#6), Charts (#143), Shellcode Disassembly (#68) and many more.

The "How to add an operation" guide has been updated to explain how to add an operation to a module or create a new module if necessary.

🍾