Feature: Run a recipe over multiple inputs #566

j433866 · 2019-05-31T09:12:13Z

This PR adds the ability to load multiple inputs into the web app (#505) , and run a recipe over all of them. I have also made some major changes to the input/output/worker waiters to work with multiple inputs, and have moved a lot of the functionality from the input waiter into a new worker called InputWorker. There are also some improvements to the UI (new themes!)

Multiple inputs

The input/output area now has tabs! You can create a new tab in the input, either by loading in multiple files or adding a tab manually. Each tab can be either a text input or loaded file.
When loading multiple files, a pool of LoaderWorkers is created to load them as quickly as possible, and when bake is clicked, a pool of ChefWorkers is used to utilise multi-processor systems. This uses the navigator.hardwareConcurrency value. (Or 4 if this is unavailable)
You can also now download all the outputs in a zip file, which uses a new worker (ZipWorker) to zip up the files in the background before downloading them. When zipping, we can use Magic to automatically detect the file extension for each file. (Defaults to .dat when we can't detect it)
There is a new dialog for searching for input and output tabs, where you can filter by status (pending / loading / loaded for inputs, and pending / baking / baked / stale / errored for outputs), and also regex the contents of the tab
The majority of the logic for the inputWaiter has been moved into a new worker (InputWorker), as the browser would frequently hang when handling a large amount of inputs. Now, it should handle a large number reasonably smoothly. Generally we can't load more than around 500,000 at once, the browsers tend to get quite unhappy after that!
Also, the bake button is now a progress bar!

Other UI Improvements

Added an input debouncer to avoid baking too often
Images loaded into the input can show a preview instead of the file icon (this can be disabled)
Added a new open folder button for opening an entire folder (and its subfolders)
Added new solarized light and dark themes

Add more elements to be controlled by theme css: - Preloader spinner colours - Operation disable / breakpoint icons - Auto bake checkbox - Search highlight colour - Categories header colour

This can be turned off

Change fonts in solarized themes to match classic. Add styling of input fields to modals

Can add, remove and switch tabs

Allow selecting multiple files. Allow drag and dropping multiple files. Clear all IO button will close all tabs. Will now spawn multiple loaderworkers simultaneously. Improve tab logic to break less and work in Firefox.

Don't show lines value if tab contents is a file.

Limits number of workers to number of cpu threads (4 if not supported) Creates output tabs (switching doesn't work yet) Disabled some highlighting for now.

(This needs some work!)

Output tabs are created when input tabs are. Fix initial bake not working

Don't create a DOM element for every tab, just reuse the same ones. Display file information while the files are loading. (Output tabs no longer work)

Automatically disable auto-bake if more than 20 files are input.

Change some of the HTML around the tab bar

- Will run and display outputs in the output area as they're baked - Creates output tabs - Can change output tabs (only the first 4 at the moment)

Add downloading files as an archive (needs work) Add option for keeping the tabs in sync

Adjust worker logic to avoid closing and reopening a worker as often

Move tab buttons onto tab bar. Calculate size of maxTabs automatically on page load. Display total execution time when a bake finishes.

Move addition of close tab button back

Force wheel event to be not passive. Change maxworkers to be navigator.hardwareConcurrency - 1

Run detectFileType on switched output

n1474335

Awesome stuff as always. A few things to review here, then it'll be ready to merge.

src/core/ChefWorker.js

src/core/Dish.mjs

src/web/workers/InputWorker.mjs

Use Dish.STRING instead of "string" for get()

Move stepping logic into App. Change toggleBakeButtonFunction to use a str instead of booleans. Tidy up handleLoaderMessage in InputWaiter.

Add comment to inputworker default values.

Debounce bake button function changes.

Tab shadows are now displayed using css classes

j433866 added 30 commits March 19, 2019 14:37

Add new solarized light and dark themes.

d09ab4a

Add more elements to be controlled by theme css: - Preloader spinner colours - Operation disable / breakpoint icons - Auto bake checkbox - Search highlight colour - Categories header colour

Add image preview to input file overlay.

2049e4e

This can be turned off

Remove old css file

b195fc6

Change file thumb to normal icon when file closed

674f879

Add styling of checkboxes.

e03e51c

Change fonts in solarized themes to match classic. Add styling of input fields to modals

Add css styling option for banner URL

4a84cf3

Adjust css to keep aspect ratio of previewed image

d9fdae9

Fix the aspect ratio fix

7571767

Add UI for input tabs.

37218c1

Can add, remove and switch tabs

Stores inputs in memory instead of in textareas.

c0c83c5

Allow selecting multiple files. Allow drag and dropping multiple files. Clear all IO button will close all tabs. Will now spawn multiple loaderworkers simultaneously. Improve tab logic to break less and work in Firefox.

Fix input highlighter size

9b86ed7

Show tab number on tab title.

fbe1e2c

Don't show lines value if tab contents is a file.

Bake all inputs, not just the active tab.

37428fb

Limits number of workers to number of cpu threads (4 if not supported) Creates output tabs (switching doesn't work yet) Disabled some highlighting for now.

Fix number of running workers not being correct.

61bc543

(This needs some work!)

Add wrapper to output

3597aac

Add css for output tabs

90dd2b5

Sync output tabs to input tabs.

a2cc7a8

Output tabs are created when input tabs are. Fix initial bake not working

Rewrite InputWaiter to be less messy.

c289e1b

Don't create a DOM element for every tab, just reuse the same ones. Display file information while the files are loading. (Output tabs no longer work)

Merge with ui-improvements

0d8584f

Fix file preview not working

f6ab82f

Fix tab info not being updated in refreshTabs()

eb5809f

Add maximum limit to the thumbnail image size

7bf8467

Change "input-tabs" id to be for the actual ul element.

cd094d1

Automatically disable auto-bake if more than 20 files are input.

Remove some inputNum stuff for now

6430129

Add tab buttons (don't work yet).

2aad764

Change some of the HTML around the tab bar

Bring up to date with master

2cb33bf

Improve WorkerWaiter and OutputWaiter.

e0c9aba

- Will run and display outputs in the output area as they're baked - Creates output tabs - Can change output tabs (only the first 4 at the moment)

Output tabs now (mostly) work.

9df26b8

Add downloading files as an archive (needs work) Add option for keeping the tabs in sync

Fix some weirdness with tabs.

1eadc08

Adjust worker logic to avoid closing and reopening a worker as often

Fix more tab weirdness.

b90cca7

Move tab buttons onto tab bar. Calculate size of maxTabs automatically on page load. Display total execution time when a bake finishes.

j433866 added 6 commits June 13, 2019 11:13

Add mime type detection to tab title.

84204c1

Move addition of close tab button back

Improve queueing for getDish... messages.

34d0397

Force wheel event to be not passive. Change maxworkers to be navigator.hardwareConcurrency - 1

Add dishworker

5731f5c

Fix incorrect value being sent to dishworker

370cae3

Merge remote-tracking branch 'upstream/master' into multiple-input-files

05d81eb

Fix switchClick not always being fired.

c529a40

Run detectFileType on switched output

n1474335 changed the base branch from master to v9 June 19, 2019 09:43

Remove loading files modal

906ece0

n1474335 requested changes Jun 27, 2019

View reviewed changes

j433866 added 19 commits July 1, 2019 11:51

Bring up to date with master

dd5038c

Clone dish instead of using the current dish.

86c37ba

Use Dish.STRING instead of "string" for get()

Correct documentation

ba4740a

Update copyright year on solarized themes

3dc57c4

Fix stepping.

a9f8dac

Move stepping logic into App. Change toggleBakeButtonFunction to use a str instead of booleans. Tidy up handleLoaderMessage in InputWaiter.

Remove unused self.port variable

3a19737

Change ZipWorker to use Dish.get()

160e4a0

Update inputWorker description

7a01802

Fix getInputValue not returning inputs that are arraybuffers

cf500b7

Show or hide thumbnail when the option is changed.

4312d39

Add comma to CSS rule

0f87c70

Remove no file extension error

2d95092

Fix whitespace and comment typos.

455b42a

Add comment to inputworker default values.

Merge remote-tracking branch 'upstream/master' into multiple-input-files

a7eefa8

Change tab element height using stylesheet.

6d95cb0

Debounce bake button function changes.

Fix output heights being incorrect

b3fa1ea

Use reduce to find smallest and largest inputNums.

30a66f1

Tab shadows are now displayed using css classes

Use reduce for finding previous and next inputNums

367d30e

Improve layout of addInput parameters.

6964bbc

n1474335 merged commit 6964bbc into gchq:v9 Jul 4, 2019

j433866 deleted the multiple-input-files branch July 24, 2019 08:38

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Run a recipe over multiple inputs #566

Feature: Run a recipe over multiple inputs #566

j433866 commented May 31, 2019

n1474335 left a comment

Feature: Run a recipe over multiple inputs #566

Feature: Run a recipe over multiple inputs #566

Conversation

j433866 commented May 31, 2019

n1474335 left a comment

Choose a reason for hiding this comment