CLI for plotting (sisl.viz)

[pfebrer]

Hi all!

For long I have been thinking that what's missing for sisl.viz is a CLI for plotting. I would like to create it so that sisl.viz becomes more useful to people in a daily basis.

I have been thinking about the design, and there are some things that I'm not sure how to do.

1. Interface

Let's say that the CLI's entry point is splot. What should be the first argument, the file that you want to plot:

splot siesta.bands

where we could use the default plot if no plot class is specified, or the plotting method:

splot bands siesta.bands

I think the latter makes the interface easier to design, but the first might be more convenient for users. Maybe if you pass a file (file) as a first argument it could be equivalent to:

splot file-default-plot file

and then we could have both options coexist?

2. Config files

Some inputs are just a pain to pass through the CLI so I would like to add an extra --config or -c that accepts a yaml file with the inputs. E.g.:

splot bands siesta.bands --draw_gap --config bands_inputs.yaml

bands_inputs.yaml:

bands_style: {color: red, width: 5}
...

3. Using preconfigurations

To make it more convenient, one could have a register of configurations, like:

somewhere/sisl_configs.yaml

red_bands:
  bands_style: {color: red, width: 5}
blue_bands:
  bands_style: {color: blue, width: 5}

And you could use them by name like:

splot siesta.bands --config red_bands

Perhaps you could even use multiple of the configurations (which would merge).

I think this would be very good for convenience of users. What do you think?

What I'm not sure is:

• Where should this register of sisl_configs be stored?
• Should sisl provide already many configs that could be useful to people?
• If so, probably they should not be packaged with sisl because these could change much faster than the release rate. Should we keep a public repo where people can contribute configs and that the CLI would now about?

4. Custom complex workflows

This might be for a second iteration of the CLI.

In the repo maybe we could also have complex plotting workflows that fit specific cases. E.g. you could have a workflow that plots both bands and PDOS and shows them together as subplots, with everything correctly aligned. If that workflow is called bands_pdos, you could use it like:

splot bands_pdos RUN.fdf

But maybe this shouldn't even be something that goes in splot and this is more a thing for sisl's toolbox 🤔

5. Support for multiple subplots directly in the CLI

I.e. merging plots, subplots, animations... I have no idea how this could work. Maybe this is also for a second iteration.

---

I would love your comments (specially about points 1 to 3) so that I can start building it! :)

Thanks!

[pfebrer]

Can the plotting CLI be a typer CLI? 🥺

It looks so beautiful:

[zerothi]

I really don't think we should do typer I have followed some discussions there, and people are genuinely worried it will not be maintained (or left over to another user).

I would still prefer to use click, one can always put rich-click in front (or use it if available, it should be a drop-in replacement).

[pfebrer]

Hmm ok, I will try with rich-click. That means I have to implement some types myself or copy it from Typer (because I want the CLI to be generated from the type hints), but it should be ok.

Any comment on what the CLI should be able to do?

[zerothi]

Could we please try something minimal to begin with?

This is about maintainability... And currently that worries me, generally we want CLI to merge into a single interface. And hence we should choose something stable, something easily extensible, and something that other devs also knows/understands/can tweak.

While I can understand where you are coming from, I am worried your CLI won't so easily be transitioned into the others. Could we arrange a meeting sometime?

[ialcon]

I just would like to mention that I find the CLI idea quite nice. This is essentially the same idea as sgeom, for instance, right? I have the feeling that being able to generate plots for various plottable things from the command line is very powerful. As of now, if I want to do it, I actually have a sisl script somewhere in my docs, and I call it via an alias. However, that is very rigid, and so I have to have a plotting sisl script for every type of thing I want to plot. The idea that you are putting on the table, @pfebrer, would solve this at once - permitting quite professional plots with a great range of possibilities.

Regarding your points 1-3, I kind of agree with them all. Being able to specify plot-type but, at the same time, having some file-formats that the code is able to assign to a particular plot-type, is quite useful. The key specification in input files looks also nice, and having a bunch of pre-configurations somewhere is also a very good idea. I am not sure about having the CLI connected to the cloud and reading stuff from there... Because that would require an internet connection.. Maybe it is not that much of an issue, but I would wait to see the usage of points 1-3 before implementing such a cloud-thing.

[pfebrer]

This is not going to work for now 😅

If one could do:

splot pdos RUN.fdf

That would work, but the problem is that to generate the PDOS data there are different parameters depending on the source. E.g. from a Hamiltonian you need to pass the kgrid which you don't need to from other sources like a .PDOS file.

So if I were to support all the parameters for all the possible sources inside the splot pdos the CLI would become too complex.

What I'm thinking is the best approach is to have a separate CLI to generate the data. There you could support all the sources easily like:

sdata pdos hamiltonian --kgrid 3 3 1

Store the data into some file and use splot pdos to plot it with all the plotting options.

sdata already exists, I don't know if this functionality could fit in there.

But anyway I think having a separate command to generate the data is probably the best idea for usability.

[ialcon]

Sure, I agree that having a separate post-processing-data command is much better, even if only for organization reasons. I have barely used sdata so I cannot really tell if it fits there (though, from the name, it seems it does... 😏). In any case, this I guess is to be decided by @zerothi.

One last question. In your example, if we are dealing with a DFT calculations, I guess that hamiltonian could be replaced by RUN.fdf?

sdata pdos RUN.fdf --kgrid 3 3 1

Right?

[pfebrer]

Sorry, hamiltonian would be like the "source type" . For the PDOS data there are functions to create the data from each source. E.g. for the Hamiltonian:

sisl/src/sisl/viz/data/pdos.py

Line 294 in b0feb0c

def from_hamiltonian(

And then you would have to pass the inputs to that function. The H input could be indeed RUN.fdf

[ialcon]

But then I guess that when you run sdata pdos hamiltonian --kgrid 3 3 1 the from_hamiltonian() function will assume there exists a RUN.fdf file inside the folder, no? I guess you could also specify the input file in the sdatacommand?

[pfebrer]

It would be something like:

sdata pods hamiltonian RUN.fdf --kgrid 3 3 1

But of course we are just speculating so it could be anything else 😅

[zerothi]

I have thought some more about this today.

And to comment on your points:

2), 3) and 4).

I think the ideal thing would be something like:

Currently sisl provides a path for holding configuration files:

export SISL_CONFIGDIR = ~/.config/sisl/

so that would be the place to put stuff.

I would also suggest that we have 2 levels of configurations. Quite often I find the need to change settings locally, and being able to allow more would be natural. So I guess an environment variable like PATH would be more suitable, with precendence of presets etc. of-course.
This can be handled via ChainMap.

So something like:

export SISL_CONFIGDIR=$(pwd)/my_configs:$(pwd)/subset

which would prefer settings in my_configs and then in subset. I am thinking that you could have a preset named foo in subset which is not present in my_configs for instance.

3. Alternatively we could have a specified config file for the current directory (say sisl_config.yaml/toml which would always be read, this could be very useful when doing lots of small variations of calculations in sub-folders. We might even think about searching the 2/3/4? nearest levels to make this simpler.

4. I guess complex workflows should be thought through in terms of their use-case... But I can definitely see the potential.

As for the format of the configuration file, I would assume either toml/yaml files. toml is nice because it is intrinsicly read in python.

@ialcon comments here would be valuable (whether or not both should be allowed for instance)

Here an example:

import yaml, toml
import sys

d = {
    "plots": {
        "red_bands": [
            {"bands_style": {"color": "red", "width": 5}},
            {"legend": "hello"},
        ],
        "blue_bands": {"bands_style": {"color": "blue", "width": 5}},
    },
    "parallel": {"num_procs": 4},
}

from pprint import pprint

print("pprint:")
pprint(d)

print("\nyaml:")
yaml.dump(d, sys.stdout)

print("\ntoml:")
toml.dump(d, sys.stdout)

and the output:

pprint:
{'parallel': {'num_procs': 4},
 'plots': {'blue_bands': {'bands_style': {'color': 'blue', 'width': 5}},
           'red_bands': [{'bands_style': {'color': 'red', 'width': 5}},
                         {'legend': 'hello'}]}}

yaml:
parallel:
  num_procs: 4
plots:
  blue_bands:
    bands_style:
      color: blue
      width: 5
  red_bands:
  - bands_style:
      color: red
      width: 5
  - legend: hello

toml:
[plots]
[[plots.red_bands]]

[plots.red_bands.bands_style]
color = "red"
width = 5
[[plots.red_bands]]
legend = "hello"

[parallel]
num_procs = 4

[plots.blue_bands.bands_style]
color = "blue"
width = 5

A dictionary

[pfebrer]

Thanks for the comments, I was so focused on #801 that I didn't see this 😅

I agree with everything except for automatically looking for a file in the directory. If the user doesn't realize that the file is there, it can create lots of confusion :) I see how that could help with nested folders though, but I don't know if it is worth it to risk the potential confusion.

Regarding yaml vs toml I think yaml feels more natural to write manually.

Anyway probably its best to first try to do it without config files and then add the configs as an extra layer :)

[ialcon]

Everything that I got from @zerothi's appears to make full sense to me. I apologize that I do not have much to add to the yaml vs toml discussion. I still do not fully get the picture of how the configuration files would be edited by the users, if either directly edited (say using vim) or indirectly via sisl? However, from what I saw from @zerothi's example, it seems that yaml provides a more structured dictionary layout, no? However, you mentioned that toml is directly readable from python... so, again, I think I am lacking a bit of technical knowledge here to provide a useful opinion...

As @pfebrer mentioned, probably it is better to go step by step, and implement the basic splot functionality before moving to config files..