NIFI-12898 Allow uploading asset and referencing via Parameter

[bbende]

Summary

This pull request adds the ability to upload assets to parameter contexts and then reference them from parameters in the context. This feature is described in NIFI-12898 and the feature proposal.

A new framework extension - AssetManager - is responsible for storing/retrieving assets on the file system.

The following REST end-points are added:

Create/upload an asset:
POST /parameter-contexts/{id}/assets

List assets:
GET /parameter-contexts/{id}/assets

Get asset content:
GET /parameter-contexts/{id}/assets/{assetId}

The last two end-points are primarily intended to be used when a node joins the cluster for it to compare it's current assets with the coordinator and synchronize it's assets to match.

To reference an asset it is done through a normal parameter context update, where a parameter will send no value and instead provide an asset reference:

referencedAssets: [
  {
     "id": "assetId1"
  },
 {
     "id": "assetId2"
  },
]

The back-end will automatically make the value of the parameter become the concatenated paths of the referenced assets.

The following CLI commands were added:

nifi create-asset -pcid <param-context-id> -af <full-path-to-a-file>

nifi list-assets -pcid <param-context-id>

nifi add-asset-reference -pcid <param-context-id> -pn <param-name> -aid <asset-id>

nifi remove-asset-reference -pcid <param-context-id> -pn <param-name> -aid <asset-id>

When a parameter context is updated, any asset that was previously referenced by a parameter and no longer referenced after the update will be deleted.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for putting together this new feature @bbende! I plan to take a closer look soon.

GitHub CodeQL flagged several issues related to using input parameters for file names, so that will require some closer evaluation.

[bbende]

Thanks @exceptionfactory ! I noticed the CodeQL issues as well...

I did sanitize the Filename header value before passing it to any code that would use it to write a File.

If that isn't enough then I suppose a fall back option would be to write the file names with the generated UUID and store some kind of mapping file along side the asset that contains the original filename.

The problem with this is that the value of the parameter that references the asset is going to be the concatenated paths of the assets and it's going to contain file names that the user has no idea what they are. Example, upload PostgresDriver.jar and it generates id 123, then reference this asset in a parameter named db_driver, and the value of the parameter will be '/path/to/assets/123'. So some looking at the parameter has no idea what file that is.

[exceptionfactory]

Thanks for the reply @bbende. Several classes mitigate the path traversal problem using Java Path objects with normalization and checking that the resolved child path starts with the selected parent path.

nifi/nifi-system-tests/nifi-system-test-extensions-bundle/nifi-system-test-extensions/src/main/java/org/apache/nifi/flow/registry/FileSystemFlowRegistryClient.java

Line 434 in 83d2078

private File getChildLocation(final File parentDir, final Path childLocation) {

It appears that the parent Parameter Context ID also needs to be checked. Instead of using the user-provided value, recommend looking up the Parameter Context and using the ID returned.

[bbende]

@exceptionfactory thanks for the pointers! I made a couple of improvements and seems to have resolved the CodeQL issues. I think most of them were actually somewhat false positives because the sanitize method was returning the input variable if null or blank, but that was causing CodeQL to think the unmodified user input was being potentially used later on.

[exceptionfactory]

Great work on this new feature @bbende! The majority of the code changes look good, and the initial changes to address the CodeQL path concerns also look good. I plan to do some additional testing, but on initial review, I just noted a couple minor things.

[bbende]

@exceptionfactory I pushed a commit that addresses the comments you had, and also another commit that adds a delete end-point to be able to clean up assets that were uploaded and maybe never referenced, let me know of anything else

[markobean]

Is this new feature accessible only via the API and not in the GUI?

[bbende]

@markobean at the moment yes, if someone picks up a UI JIRA to add it then it may be in the UI later on

[exceptionfactory]

Thanks for building out these new features @bbende, the latest version looks good! +1 merging