-
Notifications
You must be signed in to change notification settings - Fork 16
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
#131 - Add the ability to create a ZIP file based on document IDs
* feat: adds documents/compress request handler * feat: adds OpenAPI spec for /documents/compress * refactor: OAPI spec * feat: extends the tests * feat: adds retention policy to tempfiles in STAGING * refactor: prettified the tests * refactor: OAPI spec * feat: DocumentCompressor initial * refactor: used multipart upload only for large files * refactor: handled IOException, provided multipart upload S3 * refactor: refactored * refactor: resolved merge conflicts * refactor: refactored after review * refactor: refactored * refactor: extended the tests * refactor: refactored * Update build.gradle * Update api.yaml * Squashed commit of the following: commit 60eb79e Author: Mike Friesen <mfriesen@gmail.com> Date: Mon Jul 31 23:11:50 2023 -0500 fixed JWT API gateway definition commit acac8d6 Author: Mike Friesen <mfriesen@gmail.com> Date: Mon Jul 31 00:08:30 2023 -0500 tweaked API spec commit 5ec3caf Author: Mike Friesen <mfriesen@gmail.com> Date: Sun Jul 30 23:50:53 2023 -0500 update sharing API commit af9ad6d Author: Mike Friesen <mfriesen@gmail.com> Date: Sun Jul 30 23:28:56 2023 -0500 added PATCH /shares/folders/{documentId} commit 1a245b0 Author: Mike Friesen <mfriesen@gmail.com> Date: Sun Jul 30 22:09:43 2023 -0500 fixed sorting results on getBatch commit e61cf72 Author: Mike Friesen <mfriesen@gmail.com> Date: Sun Jul 30 21:43:02 2023 -0500 added ScanIndexForward to QueryConfig commit 5a590bc Author: Mike Friesen <mfriesen@gmail.com> Date: Sun Jul 30 20:35:42 2023 -0500 added IndexName support to queryBeginsWith() method commit 1f88cd5 Author: Mike Friesen <mfriesen@gmail.com> Date: Sat Jul 29 21:51:19 2023 -0500 removed DocumentEventService commit 54eb09c Author: Mike Friesen <mfriesen@gmail.com> Date: Sat Jul 29 21:40:20 2023 -0500 changed Document Folders to use FolderIndexRecord object * Revert "Squashed commit of the following:" This reverts commit 6803be7. * update securityScheme for IAM / API Key * feat: adds S3MultipartUploader * refactor: refactored * refactor: added logging * refactor: adapted DocumentCompressor to v1.12 changes, added checksum tests * test: skipped ZIP in tempfiles event --------- Co-authored-by: Mike Friesen <mfriesen@gmail.com>
- Loading branch information
Showing
24 changed files
with
1,489 additions
and
48 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,6 @@ | ||
| .gradle | ||
| .idea | ||
| formkiq-core.iml | ||
| build | ||
| bin | ||
| .DS_Store | ||
|
|
||
124 changes: 124 additions & 0 deletions
124
aws-s3/src/main/java/com/formkiq/aws/s3/S3MultipartUploader.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| /** | ||
| * MIT License | ||
| * | ||
| * Copyright (c) 2018 - 2020 FormKiQ | ||
| * | ||
| * Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| * of this software and associated documentation files (the "Software"), to deal | ||
| * in the Software without restriction, including without limitation the rights | ||
| * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| * copies of the Software, and to permit persons to whom the Software is | ||
| * furnished to do so, subject to the following conditions: | ||
| * | ||
| * The above copyright notice and this permission notice shall be included in all | ||
| * copies or substantial portions of the Software. | ||
| * | ||
| * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| * SOFTWARE. | ||
| */ | ||
| package com.formkiq.aws.s3; | ||
|
|
||
| import software.amazon.awssdk.core.sync.RequestBody; | ||
| import software.amazon.awssdk.services.s3.S3Client; | ||
| import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest; | ||
| import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; | ||
| import software.amazon.awssdk.services.s3.model.UploadPartRequest; | ||
| import software.amazon.awssdk.services.s3.model.UploadPartResponse; | ||
| import software.amazon.awssdk.services.s3.model.CompletedPart; | ||
| import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest; | ||
| import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload; | ||
| import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest; | ||
|
|
||
| import java.util.Collection; | ||
| import java.util.HashMap; | ||
| import java.util.ArrayList; | ||
|
|
||
| public class S3MultipartUploader { | ||
| /** | ||
| * S3 client. | ||
| */ | ||
| private final S3Client s3; | ||
| /** | ||
| * Maps uploadId to its uploaded parts. | ||
| */ | ||
| private final HashMap<String, Collection<CompletedPart>> uploadIdCompletedParts; | ||
| /** | ||
| * Maps uploadId to its request metadata (bucket, object key etc.). | ||
| */ | ||
| private final HashMap<String, CreateMultipartUploadResponse> uploadIdMetadata; | ||
| /** | ||
| * Maps uploadId to its recently uploaded part number. | ||
| */ | ||
| private final HashMap<String, Integer> uploadIdPartNumber; | ||
|
|
||
| public S3MultipartUploader(final S3ConnectionBuilder builder) { | ||
| this.s3 = builder.build(); | ||
| this.uploadIdCompletedParts = new HashMap<>(); | ||
| this.uploadIdMetadata = new HashMap<>(); | ||
| this.uploadIdPartNumber = new HashMap<>(); | ||
| } | ||
|
|
||
| public String initializeUpload(final String bucketName, final String objectKey) { | ||
| CreateMultipartUploadRequest uploadRequest = | ||
| CreateMultipartUploadRequest.builder().bucket(bucketName).key(objectKey).build(); | ||
| final CreateMultipartUploadResponse uploadMetadata = | ||
| this.s3.createMultipartUpload(uploadRequest); | ||
| final String uploadId = uploadMetadata.uploadId(); | ||
| this.uploadIdMetadata.put(uploadId, uploadMetadata); | ||
| this.uploadIdCompletedParts.put(uploadId, new ArrayList<>()); | ||
| this.uploadIdPartNumber.put(uploadId, 1); | ||
| return uploadId; | ||
| } | ||
|
|
||
| public void uploadChunk(final String uploadId, final byte[] chunk) { | ||
| final CreateMultipartUploadResponse metadata = this.uploadIdMetadata.get(uploadId); | ||
| final String bucketName = metadata.bucket(); | ||
| final String objectKey = metadata.key(); | ||
|
|
||
| try { | ||
| int partNumber = this.uploadIdPartNumber.get(uploadId); | ||
| UploadPartRequest uploadPartRequest = | ||
| UploadPartRequest.builder().bucket(bucketName).key(objectKey).uploadId(uploadId) | ||
| .partNumber(partNumber).contentLength((long) chunk.length).build(); | ||
|
|
||
| UploadPartResponse uploadPartResponse = | ||
| this.s3.uploadPart(uploadPartRequest, RequestBody.fromBytes(chunk)); | ||
|
|
||
| this.uploadIdCompletedParts.get(uploadId).add( | ||
| CompletedPart.builder().partNumber(partNumber).eTag(uploadPartResponse.eTag()).build()); | ||
| this.uploadIdPartNumber.put(uploadId, ++partNumber); | ||
| } catch (Exception e) { | ||
| this.abortMultipartUpload(uploadId); | ||
| throw e; | ||
| } | ||
| } | ||
|
|
||
| public void completeUpload(final String uploadId) { | ||
| final CreateMultipartUploadResponse metadata = this.uploadIdMetadata.get(uploadId); | ||
| final String bucketName = metadata.bucket(); | ||
| final String objectKey = metadata.key(); | ||
| final Collection<CompletedPart> completedParts = this.uploadIdCompletedParts.get(uploadId); | ||
|
|
||
| CompletedMultipartUpload completedMultipartUpload = | ||
| CompletedMultipartUpload.builder().parts(completedParts).build(); | ||
| CompleteMultipartUploadRequest completeMultipartUploadRequest = | ||
| CompleteMultipartUploadRequest.builder().bucket(bucketName).key(objectKey) | ||
| .uploadId(uploadId).multipartUpload(completedMultipartUpload).build(); | ||
| this.s3.completeMultipartUpload(completeMultipartUploadRequest); | ||
| } | ||
|
|
||
| public void abortMultipartUpload(final String uploadId) { | ||
| final CreateMultipartUploadResponse metadata = this.uploadIdMetadata.get(uploadId); | ||
| final String bucketName = metadata.bucket(); | ||
| final String objectKey = metadata.key(); | ||
|
|
||
| AbortMultipartUploadRequest abortRequest = AbortMultipartUploadRequest.builder() | ||
| .bucket(bucketName).key(objectKey).uploadId(uploadId).build(); | ||
| this.s3.abortMultipartUpload(abortRequest); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.