Unable to extract and archiving Read Error

[arkajnag23]

Discussed in #1313

^{Originally posted by arkajnag23 August 14, 2024}
I am running CDXGEN in Server mode and using API to generate the SBOMs.
For some multi-module maven projects which contains Java, JS packages, we are seeing errors , which eventually I feel, preventing to generate SBOM completely.

Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete! Unable to extract /tmp/jar-deps-q95rby/webapp.war. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/cde-3.3.3.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/cde-3.2.1.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11) Unable to extract /tmp/jar-deps-q95rby/abc-1.2.3.jar. Skipping. Error: Archive read error at FsRead.readUntilFoundCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:180:46) at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25) at tick (node:fs:653:7) at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

These errors are not providing a lot information.
Require help.

[arkajnag23]

Also, encountered below error stack trace:
Unable to extract /tmp/jar-deps-pWNKTT/731100000-jar-with-dependencies.jar. Skipping. Error: Malicious entry: /
at ZipEntry.validateName (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:898:19)
at FsRead.readEntriesCallback [as callback] (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:349:27)
at FsRead.readCallback (/opt/cdxgen/node_modules/.pnpm/node-stream-zip@1.15.0/node_modules/node-stream-zip/node_stream_zip.js:996:25)
at FSReqCallback.wrapper [as oncomplete] (node:fs:672:5)

[prabhu]

Another valid error. Can you share this problematic jar?

[arkajnag23]

@prabhu Even before trying to fetch the files from temp directories, the files are deleted. Do we have any retention period till how long these files should be available?

[arkajnag23]

Also, there are errors like:
Falling back to parsing pom.xml files. Only direct dependencies would get included!
Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

No further information is provided. Where can we see the error logs?

[arkajnag23]

Another valid error. Can you share this problematic jar?

@prabhu As the jar is internal, it wouldn't be possible to share. What is expected to be verified or debugged here?

[prabhu]

Are you mounting the /tmp directory to the container image using -v /tmp:/tmp? The files must be getting corrupted either during download or during the copy and extract operation. Best to work with a devops person to troubleshoot this further since this is not a cdxgen issue.

[arkajnag23]

@prabhu Let me mount /tmp directory.
Btw, when you say extract, are we decompiling the jar?

[arkajnag23]

Also, there are errors like: Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

No further information is provided. Where can we see the error logs?

@prabhu Do we generate any error logs if not console log out, as the error mentions multiple errors happened, but what are the errors , how can we find them?

[arkajnag23]

/tmp

@prabhu Found out for this one, it is a corrupted jar file.
But we are seeing a lot of
Falling back to parsing pom.xml files. Only direct dependencies would get included! Multiple errors occurred while building this project with maven. The SBOM is therefore incomplete!

And in our CDXGEN SERVER mode API, call we not using requiredOnly or babel parameter.
Falling back to parsing pom.xml files. Only direct dependencies would get included! Executing '/opt/maven/3.9.8/bin/mvn -fn org.cyclonedx:cyclonedx-maven-plugin:2.8.0:makeAggregateBom -DoutputName=bom -q -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-ui-apidoc Executing 'mvn dependency:tree -DoutputFile=/tmp/cdxmvn-dNIoGd/mvn-tree.txt -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-ui-apidoc

[prabhu]

Can you run in cli mode with the environment variable CDXGEN_DEBUG_MODE=debug. Usually these might be related to the version of java needed for the given application.

[arkajnag23]

Can you run in cli mode with the environment variable CDXGEN_DEBUG_MODE=debug. Usually these might be related to the version of java needed for the given application.

@prabhu Actually for our requirement, we can't use the CLI mode, we are using docker run server mode, and I launched container in bash mode, and ran export CDXGEN_DEBUG_MODE=debug;export PREFER_MAVEN_DEPS_TREE=true;

Strangely the console log is saying that its generating maven dependency tree, but when I check the final aggregated SBOM, it doesn't include the information.

$ curl -X POST http://localhost:9090/sbom -H "Content-Type: application/json" -d '{
  "path": "/var/EventHub/event-hub-core/event-hub-core/",
  "type": "maven,js",
  "multiProject": true,
  "resolveTransitive": true,
  "recurse": true
}' 

Falling back to parsing pom.xml files. Only direct dependencies would get included!
Executing '/opt/maven/3.9.8/bin/mvn -fn org.cyclonedx:cyclonedx-maven-plugin:2.8.0:makeAggregateBom -DoutputName=bom -q -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-clients
Executing 'mvn dependency:tree -DoutputFile=/tmp/cdxmvn-cCcF7m/mvn-tree.txt -DincludeCompileScope=true -DincludeProvidedScope=false -DincludeRuntimeScope=true -DincludeTestScope=false' in /var/EventHub/event-hub-core/event-hub-core/event-hub-clients

I ran local mvn dependency:tree as shared above but below dependencies and transitive are not included.

--- maven-dependency-plugin:2.10:tree (default-cli) @ event-hub-clients ---
[INFO] event-hub:event-hub-clients:jar:4.0-SNAPSHOT
[INFO] +- org.apache.kafka:kafka-clients:jar:3.4.0.2:compile
[INFO] |  +- com.github.luben:zstd-jni:jar:1.5.2-1:runtime
[INFO] |  \- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] +- org.apache.zookeeper:zookeeper:jar:3.6.3:compile
[INFO] |  +- org.apache.zookeeper:zookeeper-jute:jar:3.6.3:compile
[INFO] |  \- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] +- org.apache.curator:curator-recipes:jar:5.2.1:compile
[INFO] |  \- org.apache.curator:curator-framework:jar:5.2.1:compile
[INFO] |     \- org.apache.curator:curator-client:jar:5.2.1:compile

[prabhu]

Can you triage further? Looks like you are making good progress. It is difficult to support without access to the source code.

[arkajnag23]

triage

@prabhu Unfortunately can't share the source code. :(
Kind of deadlock situation, logs are not sharing too much information and also the temporary txt file are removed right now, making it more difficult to debug.

[prabhu]

What is resolveTransitive: true? It is not part of cdxgen. Check the base directory since that influences the dependency tree generated by maven. Also try the old cyclonedx maven plugin approach to see if it works better for this app.

[arkajnag23]

@prabhu I saw some where that resolveTransitives is for transitive dependencies, it's my mistake.
I was using cyclonedx maven plugin before and it was good, but went with cdxgen as it supports multiple project types and our project is running multiple types with maven.

[prabhu]

Can you unset PREFER_MAVEN_DEPS_TREE environment variable? cdxgen will then use the cyclonedx maven plugin automatically.

[prabhu]

@arkajnag23 could you kindly retest with the latest 10.9.5. Wondering if this was the fix needed to improve this.

[arkajnag23]

Thanks @prabhu will do and let you know.