WIP: Charts module

[tlwr]

CyberChef is very useful for slicing and dicing and some analysis, but I often find myself dumping CyberChef output into GNUPlot or a spreadsheet to do some analysis. With a module system in the works I thought charting be a good candidate for an initial module.

Currently I'm using D3 to generate SVGs, this method allows us to just save the generated SVG for embedding elsewhere.

Currently I have implemented:

• Hexagon density charts

I would love to see in the comments what visualisations or charts would be useful: links to blog posts, pictures, and d3 blocks would be highly appreciated.

Here is the Hexagon density chart (heavily influenced by this block):

[sroberts]

Ok, that's just amazing. That looks super super good. This and some time series style charts like these:

• http://charts.animateddata.co.uk/f1/
• https://bl.ocks.org/mbostock/4063423 (By days of the week)
• http://bl.ocks.org/marufbd/7191340

[tlwr]

I've added outlines and colour customization for hex density plots:

Also if anyone knows how to make hexagon plots behave better for disproportional scales please let me know!

--

@sroberts thanks for those suggestions, I've never seen a timeline like that before! The time-series is my next goal.

One constraint I'm applying is SVG only because I think having JS in the output pane would make things very messy and not portable. I would like for a user to be able to copy and paste one of these into a presentation or document. I do think I can figure out how to do these visualizations just with SVG (though but with slightly less interactivity).

This is the native SVG tool-tip (title element) I am currently using:

[mattnotmitt]

Any idea how we can implement tests for this?

…

On Tue, 30 May 2017, 21:25 Toby Lorne, ***@***.***> wrote: I've added outlines and colour customization for hex density plots: [image: image] <https://cloud.githubusercontent.com/assets/1482692/26602997/4e724992-4553-11e7-839d-c03c2b3efcb3.png> Also if anyone knows how to make hexagon plots behave better for disproportional scales please let me know! -- @sroberts <https://github.com/sroberts> thanks for those suggestions, I've never seen a timeline like that before! The time-series is my next goal. One constraint I'm applying is SVG only because I think having JS in the output pane would make things very messy and not portable. I would like for a user to be able to copy and paste one of these into a presentation or document. I do think I can figure out how to do these visualizations just with SVG (though but with slightly less interactivity). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACAWfrGzXgQjgQ2hITgkrc3I2nqvricuks5r_HsygaJpZM4Nqtge> .

[tlwr]

The output is SVG and I plan on ensuring that the charting operations are deterministic (where possible), this will make it easy to write tests, although they probably won't be that elegant.

For non-deterministic charts, presently I have no idea how to effectively test them.

Update: I'm working on allowing dense (rather than sparse) drawing of hexagons:

Same data, different radii:

[n1474335]

This looks pretty cool and will definitely make sense once the module system is in place. Due to d3's size, I doubt we'll be able to merge it before then, but it's good to see the progress.

There are a couple of operations that make use of charts already: 'Entropy' and 'Frequency distribution'. For these I just drew custom diagrams using the HTML5 <canvas>. You can see how it works here, including how to run JS in the output if you want to go down that path. I think the SVG route is a bit cleaner (and less dangerous), but it might be useful if you have no other options.

Nice work.

[tlwr]

Haven't had a ton of time to work on this but here is a quick update.

Series charts with svg title tooltips:

I want to add an option that makes all the Y axes the same.

Heatmap chart:

Scatter chart:

HTML to Text operation

This is so you can easily export a chart, or other HTML tables

@n1474335 thanks for the suggestions, I did look at them before starting this project. I think SVG is more "portable" and as you mentioned safer. I don't think these chart operations should aim to be any more than tools for ad hoc data analysis within CyberChef, or a tool to quickly embed results into a report, rather than fully fledged charting toolkits like Tableau, Plotly, etc.

I'm perfectly happy to wait for the module system to be implemented, however so far the inclusion of d3 and d3-hexbin and all the new operations creates a 3.23MB scripts.js so it may not be necessary to wait.

I need to clean up some of the code and do some refactoring, as well as adding a few more charts:

• Overlap chart: https://bost.ocks.org/mike/miserables/
• Perhaps a graph viewer: http://www.d3noob.org/2013/03/d3js-force-directed-graph-example-basic.html (but static, without forces).
• Calendar chart: https://github.com/DKirwan/calendar-heatmap or http://www.samselikoff.com/writings/intro-to-d3/
• Perhaps the timeline chart from the comments above (http://charts.animateddata.co.uk/f1/)
• Make charts play nicely with dark mode

[n1474335]

These are really great. Any chance of updating them to work with the new operation format?

[tlwr]

over 1y old at this point this PR, i should have another look at these, just busy with work.

the blockers i dealt with at the time (which are not insurmountable) was making it work with themes (if this is a thing we care about).

thanks for jogging my memory (at least PGP operations are in now! 😆 )

[n1474335]

I wouldn't worry too much about themes. I can mess around with that if the operations work.

[shash7]

This would be super useful. Any updates on this pr?

[mattnotmitt]

If @tlwr isn't able to wrap this one up I could take it under my wing once I'm finished with #468?

…

On Fri, 11 Jan 2019, 01:11 Shashwat amin, ***@***.***> wrote: This would be super useful. Any updates on this pr? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACAWfuc-MyTxNL_YIxOQQXxBusG92C3Yks5vB-S9gaJpZM4Nqtge> .

[tlwr]

Closing this in favour of #496 as @artemisbot is taking this on, thanks @artemisbot!