Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API method doc/loadjson crashing the server #102

Closed
IgorShadurin opened this issue Aug 13, 2021 · 2 comments
Closed

API method doc/loadjson crashing the server #102

IgorShadurin opened this issue Aug 13, 2021 · 2 comments
Assignees
Labels
bug Something isn't working P1 Priority 1

Comments

@IgorShadurin
Copy link
Collaborator

1 - Create new doc with doc/new and si param first_name=string,age=number
2 - Call doc/loadjson with data [{"first_name": "Hello", "age": 11}]

Get error

time="2021-08-13T08:35:32Z" level=info msg="creeating batch for inserting in document db: my_doc"
time="2021-08-13T08:35:32Z" level=info msg="finding from document db: my_doc1"
time="2021-08-13T08:35:32Z" level=info msg="loading Manifest: 326d58ceafd8my_docid"
time="2021-08-13T08:35:32Z" level=info msg="created simple batch index: id"
time="2021-08-13T08:35:32Z" level=info msg="created simple batch index: first_name"
time="2021-08-13T08:35:32Z" level=info msg="created simple batch index: age"
time="2021-08-13T08:35:32Z" level=info msg="created batch for inserting in document db: my_doc"
2021/08/13 08:35:32 http: panic serving 192.168.100.6:55848: interface conversion: interface {} is []interface {}, not map[string]interface {}
goroutine 603 [running]:
net/http.(*conn).serve.func1(0x206ab40)
        net/http/server.go:1804 +0x104
panic(0x7414e8, 0x1c291b8)
        runtime/panic.go:971 +0x4b4
github.com/fairdatasociety/fairOS-dfs/pkg/collection.(*Document).DocBatchPut(0x1f77d40, 0x1c11970, 0x1e81b30, 0x24, 0x30, 0x0, 0x0, 0x0, 0x0)
        github.com/fairdatasociety/fairOS-dfs/pkg/collection/document.go:1075 +0x1dcc
github.com/fairdatasociety/fairOS-dfs/pkg/dfs.(*DfsAPI).DocBatchPut(0x1f140c0, 0x1d68e10, 0x2c, 0x1c2f2f0, 0xc, 0x1e81b30, 0x24, 0x30, 0x1c11970, 0x0, ...)
        github.com/fairdatasociety/fairOS-dfs/pkg/dfs/doc_api.go:255 +0xe4
github.com/fairdatasociety/fairOS-dfs/pkg/api.(*Handler).DocLoadJsonHandler(0x1f26070, 0xa184e4, 0x1d82dc8, 0x2146b80)
        github.com/fairdatasociety/fairOS-dfs/pkg/api/doc_loadjson.go:112 +0x5ac
net/http.HandlerFunc.ServeHTTP(0x1f089e0, 0xa184e4, 0x1d82dc8, 0x2146b80)
        net/http/server.go:2049 +0x34
github.com/fairdatasociety/fairOS-dfs/pkg/api.(*Handler).LoginMiddleware.func1(0xa184e4, 0x1d82dc8, 0x2146b80)
        github.com/fairdatasociety/fairOS-dfs/pkg/api/login_middleware.go:54 +0x25c
net/http.HandlerFunc.ServeHTTP(0x212b240, 0xa184e4, 0x1d82dc8, 0x2146b80)
        net/http/server.go:2049 +0x34
github.com/fairdatasociety/fairOS-dfs/pkg/api.(*Handler).LogMiddleware.func1(0xa18c00, 0x21ea280, 0x2146b80)
        github.com/fairdatasociety/fairOS-dfs/pkg/api/log_middleware.go:34 +0xac
net/http.HandlerFunc.ServeHTTP(0x212b250, 0xa18c00, 0x21ea280, 0x2146b80)
        net/http/server.go:2049 +0x34
github.com/gorilla/mux.(*Router).ServeHTTP(0x1f22060, 0xa18c00, 0x21ea280, 0x2146a80)
        github.com/gorilla/mux@v1.7.4/mux.go:210 +0x98
github.com/rs/cors.(*Cors).Handler.func1(0xa18c00, 0x21ea280, 0x2146a80)
        github.com/rs/cors@v1.7.0/cors.go:219 +0x170
net/http.HandlerFunc.ServeHTTP(0x1f27420, 0xa18c00, 0x21ea280, 0x2146a80)
        net/http/server.go:2049 +0x34
net/http.serverHandler.ServeHTTP(0x1f4c120, 0xa18c00, 0x21ea280, 0x2146a80)
        net/http/server.go:2867 +0x88
net/http.(*conn).serve(0x206ab40, 0xa195fc, 0x2142ee0)
        net/http/server.go:1932 +0x7f0
created by net/http.(*Server).Serve
        net/http/server.go:2993 +0x318
@IgorShadurin IgorShadurin added bug Something isn't working P1 Priority 1 labels Aug 13, 2021
@asabya asabya self-assigned this Nov 11, 2021
@asabya
Copy link
Collaborator

asabya commented Nov 11, 2021

loadjson only tries to read one json object rather than an array of json objects.

asabya added a commit that referenced this issue Nov 11, 2021
@tfius
Copy link
Contributor

tfius commented Nov 22, 2021

@asabya do we have a test for this ?
This is done with BigQuery in mind (check https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json)
so each line in json is an object that will be inserted into doc db.

asabya added a commit that referenced this issue Nov 24, 2021
asabya added a commit that referenced this issue Dec 13, 2021
* addressing #119, uploading same file multiple times

* addressing #122, file delete Matadata unmarshal

* addressing #123, fix response for non-existing pod

* addressing #127, server requires postageBlockId to run

* addressing #116, non-existing dir response

* addressing #119, backup old file in case same name

* code cleanup

* code cleanup

* code cleanup

* code cleanup

* addressing #117 and #118, issues related to pod deletion (#1)

* addressing #117 and #118, issues related to pod deletion

* fix reploading deleted file

* fix #132, #133

* fixes #141, rmdir updates feed with magicword

* fixes #134

* fixes #135 & #136

* fixing #139

* fix #140, accept mutable bool in request

* minor typo

* #97 handle query param for doc-ls

* #102 handle exception for invalid json

* #106 handled panic

* related to #100, doc new podname missing

* fix #124, handle removed root directory

* handle #96

* handle #24, use shlex to parse prompt args

* fixed #19, pod deletion requires password

* check for root dir presence in mkdir

* mod tidy (#146)

* mod tidy

* manually restore vendor

* manually restore vendor

* upload err fix

* fix #86, shared pod removed from list

* minor changes

* fix #147, check read only feed before access time update in download

* fix #148, close response body in bee client

* minor file deletion test added

* #92, table name added in count response

* fix w/ deepsource recommendation

* fix w/ deepsource recommendation

* #102, add test for invalid json

* 404 on file not present

* add get endpoint for download

* fix #158

* resolve anti-pattern

* bump version
asabya added a commit that referenced this issue Dec 13, 2021
asabya added a commit that referenced this issue Dec 13, 2021
@asabya asabya closed this as completed Jan 13, 2022
asabya added a commit that referenced this issue Jan 25, 2022
* addressing #119, uploading same file multiple times

* addressing #122, file delete Matadata unmarshal

* addressing #123, fix response for non-existing pod

* addressing #127, server requires postageBlockId to run

* addressing #119, backup old file in case same name

* code cleanup

* code cleanup

* code cleanup

* code cleanup

* addressing #117 and #118, issues related to pod deletion (#1)

* addressing #117 and #118, issues related to pod deletion

* fix reploading deleted file

* fixes #141, rmdir updates feed with magicword

* fixing #139

* fix #140, accept mutable bool in request

* #102 handle exception for invalid json

* related to #100, doc new podname missing

* fix #124, handle removed root directory

* handle #96

* handle #24, use shlex to parse prompt args

* fixed #19, pod deletion requires password

* check for root dir presence in mkdir

* mod tidy (#146)

* mod tidy

* manually restore vendor

* manually restore vendor

* fix #86, shared pod removed from list

* minor changes

* #92, table name added in count response

* fix w/ deepsource recommendation

* fix w/ deepsource recommendation

* #102, add test for invalid json

* server running from config vales

* fix lint errors

* fix flag usage

* add default variables

* rebase

* resolve 163

* handle #172

* handle #138, new method for kv/entry/get

* #130 handled, /pod/present api added

* podPresent only for loggedin user

* #82, add /kv/present

* #157, add basic string validation

* update readme

* update openapi (#180)

* update openapi

* update version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P1 Priority 1
Projects
None yet
Development

No branches or pull requests

3 participants