feat: Add logging to crawl.log for metadata records created by ExtractorYoutubeDL #593
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ato
reviewed
Jun 18, 2024
Comment on lines
613
to
626
| //Use the recorder object to calculate the content digest and store it on the curi. | ||
| //Must be calculated now, before the warc writer closes the file stream. | ||
| //We don't need an extra copy, so just write to NullOutputStream. | ||
| String recorderBaseName = "ExtractorYoutubeDL-" + nextRecorderId.getAndIncrement(); | ||
| Recorder recorder = new Recorder(new File(controller.getScratchDir().getFile(), recorderBaseName), | ||
| controller.getRecorderOutBufferBytes(), controller.getRecorderInBufferBytes()); | ||
| recorder.getRecordedInput().setDigest("sha1"); | ||
| getLocalTempFile().seek(0); | ||
| recorder.inputWrap(Channels.newInputStream(getLocalTempFile().getChannel())); | ||
| recorder.getRecordedInput().startDigest(); | ||
| recorder.outputWrap(new NullOutputStream()); | ||
| recorder.getRecordedInput().readFully(); | ||
| curi.getData().put(YDL_JSON_FILE_DIGEST,recorder.getRecordedInput().getDigestValue()); | ||
| recorder.getRecordedOutput().close(); |
There was a problem hiding this comment.
Since we don't call recorder.cleanup() would this leak temp files when the resource is larger than the buffer size and Recorder spills to disk?
Are we only using Recorder here for the side-effect of calculating the digest? Since we have DigestUtils from Commons Codec available maybe we could just do:
getLocalTempFile().seek(0);
InputStream inputStream = Channels.newInputStream(getLocalTempFile().getChannel());
curi.getData().put(YDL_JSON_FILE_DIGEST, DigestUtils.sha1(inputStream));There was a problem hiding this comment.
Yeah, my intention was to reuse existing code paths when possible, but this would simplify things, and sha1 should be pretty safe. Thanks for taking a look.
There was a problem hiding this comment.
The hashing portion has been refactored. Much simpler now.
ato
approved these changes
Aug 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.