Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'Magic' operation - automatically detect and run operations #239

Merged
merged 20 commits into from
May 20, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
fc2828f
Added Magic operation with the ability to detect language, file type …
n1474335 Jan 14, 2018
a1624a9
Added detection patterns for non-standard Base64 alphabets, Base58 an…
n1474335 Jan 14, 2018
48f8ca6
Added detection patterns for Octal, Binary, Decimal, Hexdumps, HTML E…
n1474335 Jan 14, 2018
615a020
Added detection patterns for UNIX timestamps, Zlib deflate, Gzip, Zip…
n1474335 Jan 22, 2018
b035f6c
Added detection patterns for X.509 certs, Morse Code, Tar, images and…
n1474335 Jan 22, 2018
57314b7
Merge branch 'master' into feature-magic
n1474335 Jan 22, 2018
28abd00
Added speculative execution of recipes to determine the most likely d…
n1474335 Jan 22, 2018
6947d2a
Magic operation now displays an ordered table of the most likely deco…
n1474335 Jan 22, 2018
865ee6a
Magic operation tidying
n1474335 Jan 23, 2018
6624f25
Magic operation now detects UTF8 and gives a probability score for ea…
n1474335 Feb 10, 2018
23bdfd0
Magic operation now shows matching ops even if they are not run.
n1474335 Feb 10, 2018
2bc563b
Added support for 238 languages to the Magic operation.
n1474335 Feb 10, 2018
544d78f
The Magic operation now only checks the most commonly used Internet l…
n1474335 Feb 14, 2018
99ade42
Added 'Intensive mode' to the Magic operation, where it brute-forces …
n1474335 Feb 14, 2018
1760ab2
Recipe errors are now ignored in the Magic operation
n1474335 Feb 14, 2018
27ec4aa
Magic operation now recognises useful operations such as 'Render Imag…
n1474335 Feb 15, 2018
b3c52a8
Magic operation now brute forces character encodings. Linted.
n1474335 Feb 15, 2018
559741f
Fixed a few small bugs
n1474335 Feb 15, 2018
56d33ea
Magic operation now calculates the entropy of each option and display…
n1474335 Feb 19, 2018
ee519c7
Merged esm branch into feature-magic. Ported FileType ops.
n1474335 May 20, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .eslintignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
src/core/vendor/**
src/core/operations/legacy/**
src/core/operations/legacy/**
1 change: 1 addition & 0 deletions Gruntfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,7 @@ module.exports = function (grunt) {
command: [
"echo '\n--- Regenerating config files. ---'",
"node --experimental-modules src/core/config/scripts/generateOpsIndex.mjs",
"echo 'export default {};\n' > src/core/config/modules/OpModules.mjs",
"node --experimental-modules src/core/config/scripts/generateConfig.mjs",
"echo '--- Config scripts finished. ---\n'"
].join(";")
Expand Down
29 changes: 21 additions & 8 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
"bootstrap-colorpicker": "^2.5.2",
"bootstrap-switch": "^3.3.4",
"bson": "^2.0.6",
"chi-squared": "^1.1.0",
"crypto-api": "^0.8.0",
"crypto-js": "^3.1.9-1",
"ctph.js": "0.0.5",
Expand Down
40 changes: 20 additions & 20 deletions src/core/ChefWorker.js
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ self.addEventListener("message", function(e) {
*/
async function bake(data) {
// Ensure the relevant modules are loaded
loadRequiredModules(data.recipeConfig);
self.loadRequiredModules(data.recipeConfig);

try {
const response = await self.chef.bake(
Expand Down Expand Up @@ -144,25 +144,6 @@ async function getDishAs(data) {
}


/**
* Checks that all required modules are loaded and loads them if not.
*
* @param {Object} recipeConfig
*/
function loadRequiredModules(recipeConfig) {
recipeConfig.forEach(op => {
const module = self.OperationConfig[op.op].module;

if (!OpModules.hasOwnProperty(module)) {
log.info(`Loading ${module} module`);
self.sendStatusMessage(`Loading ${module} module`);
self.importScripts(`${self.docURL}/${module}.js`);
self.sendStatusMessage("");
}
});
}


/**
* Calculates highlight offsets if possible.
*
Expand All @@ -182,6 +163,25 @@ function calculateHighlights(recipeConfig, direction, pos) {
}


/**
* Checks that all required modules are loaded and loads them if not.
*
* @param {Object} recipeConfig
*/
self.loadRequiredModules = function(recipeConfig) {
recipeConfig.forEach(op => {
const module = self.OperationConfig[op.op].module;

if (!OpModules.hasOwnProperty(module)) {
log.info(`Loading ${module} module`);
self.sendStatusMessage(`Loading ${module} module`);
self.importScripts(`${self.docURL}/${module}.js`);
self.sendStatusMessage("");
}
});
};


/**
* Send status update to the app.
*
Expand Down
11 changes: 10 additions & 1 deletion src/core/Operation.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ class Operation {
get config() {
return {
"op": this.name,
"args": this._ingList.map(ing => ing.conf)
"args": this._ingList.map(ing => ing.config)
};
}

Expand Down Expand Up @@ -274,6 +274,15 @@ class Operation {
return this._flowControl;
}

/**
* Set whether this Operation is a flowcontrol op.
*
* @param {boolean} value
*/
set flowControl(value) {
this._flowControl = !!value;
}

}

export default Operation;
2 changes: 1 addition & 1 deletion src/core/Recipe.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,9 @@ class Recipe {
numRegisters = state.numRegisters;
} else {
output = await op.run(input, op.ingValues);
this.lastRunOp = op;
dish.set(output, op.outputType);
}
this.lastRunOp = op;
} catch (err) {
// Return expected errors as output
if (err instanceof OperationError ||
Expand Down
2 changes: 1 addition & 1 deletion src/core/Utils.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -720,7 +720,7 @@ class Utils {
* @param {boolean} [newline=false] - whether to add a newline after each operation
* @returns {string}
*/
static generatePrettyRecipe(recipeConfig, newline=false) {
static generatePrettyRecipe(recipeConfig, newline = false) {
let prettyConfig = "",
name = "",
args = "",
Expand Down
1 change: 1 addition & 0 deletions src/core/config/Categories.json
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@
{
"name": "Flow control",
"ops": [
"Magic",
"Fork",
"Merge",
"Register",
Expand Down
4 changes: 4 additions & 0 deletions src/core/config/scripts/generateConfig.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ for (const opObj in Ops) {
args: op.args
};

if (op.hasOwnProperty("patterns")) {
operationConfig[op.name].patterns = op.patterns;
}

if (!modules.hasOwnProperty(op.module))
modules[op.module] = {};
modules[op.module][op.name] = opObj;
Expand Down
Loading