feat: add json streaming to JSONReader (#1119)

.changeset/blue-bears-invite.md

@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+feat: add JSON streaming to JSONReader

apps/docs/docs/modules/data_loaders/json.md

@@ -2,6 +2,7 @@
 
 A simple JSON data loader with various options.
 Either parses the entire string, cleaning it and treat each line as an embedding or performs a recursive depth-first traversal yielding JSON paths.
+Supports streaming of large JSON data using [@discoveryjs/json-ext](https://github.com/discoveryjs/json-ext)
 
 ## Usage
 
@@ -20,12 +21,16 @@ const docsFromContent = reader.loadDataAsContent(content);
 
 Basic:
 
+- `streamingThreshold?`: The threshold for using streaming mode in MB of the JSON Data. CEstimates characters by calculating bytes: `(streamingThreshold * 1024 * 1024) / 2` and comparing against `.length` of the JSON string. Set `undefined` to disable streaming or `0` to always use streaming. Default is `50` MB.
+
 - `ensureAscii?`: Wether to ensure only ASCII characters be present in the output by converting non-ASCII characters to their unicode escape sequence. Default is `false`.
 
-- `isJsonLines?`: Wether the JSON is in JSON Lines format. If true, will split into lines, remove empty one and parse each line as JSON. Default is `false`
+- `isJsonLines?`: Wether the JSON is in JSON Lines format. If true, will split into lines, remove empty one and parse each line as JSON. Note: Uses a custom streaming parser, most likely less robust than json-ext. Default is `false`
 
 - `cleanJson?`: Whether to clean the JSON by filtering out structural characters (`{}, [], and ,`). If set to false, it will just parse the JSON, not removing structural characters. Default is `true`.
 
+- `logger?`: A placeholder for a custom logger function.
+
 Depth-First-Traversal:
 
 - `levelsBack?`: Specifies how many levels up the JSON structure to include in the output. `cleanJson` will be ignored. If set to 0, all levels are included. If undefined, parses the entire JSON, treat each line as an embedding and create a document per top-level array. Default is `undefined`

examples/readers/package.json

@@ -14,7 +14,8 @@
     "start:assemblyai": "node --import tsx ./src/assemblyai.ts",
     "start:llamaparse-dir": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts",
     "start:llamaparse-json": "node --import tsx ./src/llamaparse-json.ts",
-    "start:discord": "node --import tsx ./src/discord.ts"
+    "start:discord": "node --import tsx ./src/discord.ts",
+    "start:json": "node --import tsx ./src/json.ts"
   },
   "dependencies": {
     "llamaindex": "*"

packages/llamaindex/package.json

@@ -25,6 +25,7 @@
     "@azure/identity": "^4.4.1",
     "@datastax/astra-db-ts": "^1.4.1",
     "@discordjs/rest": "^2.3.0",
+    "@discoveryjs/json-ext": "^0.6.1",
     "@google-cloud/vertexai": "1.2.0",
     "@google/generative-ai": "0.12.0",
     "@grpc/grpc-js": "^1.11.1",