Creating an empty data-frame

[gassechen]

Hi, is it possible to create an empty data frame?

[Symbolics]

It doesn't seem so, at least using external functions. There are checks for empty plists/alists. Still, there's no reason one can't be empty.

If you want to use internal functions, you could use make-data to create an empty data class, the super class of data-frame You'll also want to review define-data-subclass, where data-frame is defined. This macro sets up the convenience functions for, among other things, converting to/from arrays and making slice and other aops functions work with data-frames.

What's the use case for an empty data frame? Maybe there's another way to accomplish what you want.

[gassechen]

Hi thanks. I'm trying to create data-frames from json, not directly, with an intermediary code. I would be interested in being able to create an empty data-frame to be able to add columns dynamically.
From json --> hast-table --> plist
That's what I have for now.
example: '((:a data :b data) (a: data b:data ) ...)
If I try to use (apply #'df plist) it tells me that the key is duplicated.
So my idea is to extract all data with key a: and add a column with (add-column! *d* 'a '(data data)
Well, that's the idea. I accept all kinds of suggestions!!

[Symbolics]

That data schema: '((:a data :b data) (a: data b:data ) ...) is similar to the Vega-Lite data format. If you can format the data as Vega-Lite you could use the functions in the plot system to read in the JSON formatted data. See read-vega. You may also want to investigate the hash table functions in alexandria (conversions to/from plist/alist/hashtables). The 'guaranteed' solution is to loop over the list and collect all the data values into a vector to get a plist in the right format for plist-df.

[gassechen]

hello. Thanks for the information. I was able to solve the problem.
From an api with dexador and yason I get a nested hash-table.
and with this function I transform the hast-tables into a plist

(defun process-hash-table (hash-table &optional key-prefix tmp-list)
 
      (maphash (lambda (key value)
                (let ((full-key
                       (if key-prefix (concatenate 'string key-prefix "_" key) key)))
                  (if (typep value 'hash-table)

(setf tmp-list (process-hash-table value full-key tmp-list)))

(progn
(when (typep value 'list)
(let ((counter-value 0))
(dolist (l value)

(push l tmp-list)
(push (alexandria:make-keyword
(string-upcase
(concatenate 'string full-key "["
(write-to-string (incf counter-value)) "]")))
tmp-list))))
(unless (or
(typep value 'hash-table)
(typep value 'list))
(push value tmp-list)
                        (push (alexandria:make-keyword
(string-upcase full-key))
tmp-list)))))
              hash-table)
     tmp-list)

This function, called process-hash-table, recursively traverses a hash table and generates a list of data that is stored in tmp-list. Here is how it works:

1. hash-table is the hash table to process.
2. key-prefix is an optional prefix that is concatenated to keys before storing them in the list.
3. tmp-list is the list that accumulates the processed data.

The maphash function iterates over each key and value in the hash table. For each key-value:

• full-key is created by concatenating key-prefix (if present) with the current key.
• If the value is a hash table, the process-hash-table function is called recursively with the current value and the full key. The result of this call is stored in tmp-list.
• If the value is a list, its elements are traversed, and for each element it is added to tmp-list and a key of type keyword is generated based on the full key and an incrementing counter.
• If the value is neither a hash table nor a list, it is added to tmp-list and a key of type keyword is generated based on the full key.

In short, the function walks through the hash table and its subhash tables (if any), and generates a list of data. It also generates keywords to identify the data in the list. Each time a list is found, a new keyword key is created for each item in the list using an incrementing counter. The end result is the tmp-list list containing all the processed data and their corresponding keys.

(defun select-column-values (key alist)
  (mapcar (lambda (pair)
            (getf pair key))
          alist))


(defun extract-keys-columns (data)
  (let ((flat-data (alexandria:flatten data)))
    (remove-duplicates(remove-if-not #'symbolp flat-data))))

and make a dataframe

(defun make-data-frame (key-list values)
  (lisp-stat:make-df key-list (mapcar (lambda (key)
                              (coerce (select-column-values key values) 'vector))
                            key-list)))

[snunez1]

Fantastic! As an FYI, yason can also output a/p-lists...

[gassechen]

Yes thanks it was my first option to use a plist/alist from yason, but I didn't know how to solve the nesting to be able to rear a data-frame
this is the example json

{
     "esn":"23015240",
     "payload":"0xee296c38bfdb5a0000",
     position:{
         "type":"Point",
         "coordinates":[90.2013158798218, 29.1253566741943]
     },
     asset: {
         “id”: 126957,
         “name”:“20ft CCU BBB789”,
         “number”: “BBB789”,
         “account:{
             “name”:”ABC Offshore Rentals”
         }
     },
     "status": {
         "speed":"0.0",
         "input1":"false",
         "input2":"false",
         "heading":"0",
         "stopped":"true",
         "ignition":"false",
         "battery_status":"true",
         ...
     },
     "timestamp":"20150604T16:16:26Z",
     "received":"20150604T16:16:56Z"
}