Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nippy freeze / thaw of a certain dataset corrupts it #287

Closed
behrica opened this issue Feb 18, 2022 · 2 comments
Closed

nippy freeze / thaw of a certain dataset corrupts it #287

behrica opened this issue Feb 18, 2022 · 2 comments

Comments

@behrica
Copy link
Contributor

behrica commented Feb 18, 2022

I have a certain dataset ,
which gets corrupted via a freeze / thaw operaton using nippy

Reproducible with this code:

(require '[taoensso.nippy :as nippy])
(require '[tech.v3.dataset :as tds])
(require '[tablecloth.api :as tc])
(def df
  (->
   (tds/->dataset "https://github.com/behrica/kaggleHP/raw/main/train.csv.gz" {:key-fn keyword})
   
   ))
;; working
(tds/rows df)

(nippy/freeze-to-file "/tmp/df.nippy" df)
(def df-thawed (nippy/thaw-from-file "/tmp/df.nippy"))
;;  fails
(tds/rows df-thawed)

 Caused by java.lang.ClassCastException
   class [D cannot be cast to class [Ljava.lang.Object; ([D and
   [Ljava.lang.Object; are in module java.base of loader 'bootstrap')

          array_buffer.clj:  334  tech.v3.datatype.array-buffer/array-buffer/reify
               Buffer.java:   55  tech.v3.datatype.Buffer/get
               readers.clj:   97  tech.v3.dataset.readers/value-reader/reify/reify
               Buffer.java:   55  tech.v3.datatype.Buffer/get
           FastStruct.java:  149  tech.v3.dataset.FastStruct$1/next
                   RT.java:  523  clojure.lang.RT$4/invoke
              LazySeq.java:   42  clojure.lang.LazySeq/sval
              LazySeq.java:   51  clojure.lang.LazySeq/seq
@cnuernber
Copy link
Collaborator

Just for anyone reading this the issue was in dataset->data which the nippy pathway uses to simplify the dataset into base clojure datastructures and primitive arrays.

cnuernber added a commit that referenced this issue Feb 18, 2022
@cnuernber
Copy link
Collaborator

Release 6.065 fixes this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants