How to treat binary data

[edumur]

Hi all,

I am used to get data from databases directly with sqlite3.
However I cam across binary saved data and I failed to understand how to go from binary to numpy array.
Does anyone has an idea?

Thanks

[jenshnielsen]

QCoDeS is not designed for you to access the data directly using sqlite3. We expect you to use the QCoDeS api to load a dataset from the sqlite file and use the functions there to access the data. There are a number of functions to get the data as a dict of numpy arrays (get_parameter_data) or to pandas and xarray datasets.

[astafan8]

we had a lot of crashing happening while we load the data

if you'd like, we can look into more details of how you were doing that (as in, if you're using the correct qcodes api for that, and if there might be some bugs or else). But ideally it would be very helpful to know what specific crashes/exceptions were raised/seen, otherwise it's difficult to help.

---

anyways, the original answer still holds - use qcodes api to read the data, incl. binary. the sqlite database and it's structure and contents are not part of the "public api" (at this point) hence should not be used directly and relied upon. which together should invite for contributions to qcodes api for reading data :) perhaps, indeed, we should look into the specifics of how you're reading the data and then see how to do it using qcodes api and without any overheads.

[edumur]

The software crashed without any message...
At a time the code looked like this

    def openDatabase(self) -> None:
        """
        Open connection to database using qcodes functions.
        Return
        ------
        conn : Connection
            Connection to the db
        cur : Cursor
            Cursor to the db
        """
        
        conn =  qc.dataset.sqlite.database.connect(self.databasePath)
        cur = conn.cursor()

        return conn, cur

    def getParameterData(self, runId : int, paramDependent : str) -> dict:

        conn, cur = self.openDatabase()
        
        ds =  qc.load_by_id(run_id=int(runId), conn=conn)
        d = ds.get_parameter_data(paramDependent)[paramDependent]

        self.closeDatabase(conn, cur)

        return d

    def closeDatabase(self, conn : qc.dataset.sqlite.connection.ConnectionPlus,
                            cur : sqlite3.Cursor) -> None:
        """
        Close the connection to the database.
        Parameters
        ----------
        conn : Connection
            Connection to the db
        cur : Cursor
            Cursor to the db
        """
        
        cur.close()
        conn.close()

You can see the file here: https://github.com/pyplotter/pyplotter/blob/867cd60663a5e2aa6a81d1e79f2aadb312b4affc/sources/qcodesdatabase.py

Again, even simple sql request made the app crashed.
So my conclusion was that qcodes was not responsible of it.
Historically the idea was to put the data loading in another process so that its crash does not make the all software crash.
It just happened that doing 1 data loading in another process is robust (we try 2 loading and sometimes it crashes).

The current way data are loaded on the plotter is:

1. Try simple sql request
2. load qcodes if failed.

It allows "fast" loading for most of our data and used qcodes when something is wrong (usually binary or exotic shape).
Feel free to try the plotter and come back with suggestions.

My goal is really to allow people of my lab to quickly look at their data and interact with them.

[jenshnielsen]

Thanks,

My point is

1. Depending on the schema of the QCoDeS database is a bad idea. This can and will change with new qcodes release so you app will break. If importing QCoDeS is too slow your time would IMHO be much better spent on helping to optimize this as I suggested above.
2. Loading the data in a different process seems to me to be a good idea but I would suggest that you do not restart the process at every load. Even without qcodes being slow to import that is a lot of overhead and sub optimal. Have you looked into using a ProcessPoolExecuter or something similar so there is always a process ready to execute the load see https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor

[edumur]

1. I do agree and the current state of the plotter is just me finding a way to find the plotter users not complaining about the data loading speed, plotter crashing, ... I would gladly help. I tried following qcodes import takes a long time #4543 but was unable to make it work.
   Like typing importtime-waterfall importtime_waterfall gave me the usual result but importtime-waterfall qcodes gave me

Traceback (most recent call last):
  File "***\Anaconda3\envs\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "***\Anaconda3\envs\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "***\Anaconda3\envs\python38\Scripts\importtime-waterfall.exe\__main__.py", line 7, in <module>
    import argparse
  File "***\Anaconda3\envs\python38\lib\site-packages\importtime_waterfall.py", line 140, in main
    res = best_of(args.module)
  File "***\Anaconda3\envs\python38\lib\site-packages\importtime_waterfall.py", line 122, in best_of
    best_time, best = run()
  File "***\Anaconda3\envs\python38\lib\site-packages\importtime_waterfall.py", line 113, in run
    ret = subprocess.run(
  File "***\Anaconda3\envs\python38\lib\subprocess.py", line 495, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "***\Anaconda3\envs\python38\lib\subprocess.py", line 1018, in communicate
    stderr = self.stderr.read()
  File "***\Anaconda3\envs\python38\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 153874: invalid start byte

2. I do not know that that existed. Thanks for the suggesting.

[jenshnielsen]

I have not tried that tool. When I have been doing it I have been using Tuna to render the output of
python -X importtime yourfile.py 2> import.log

where yourfile.py contains the imports I want to profile.