NIFI-3065 Per Process Group logging

[timeabarna]

Summary

Add a toggle in Process Group configuration (default: off), that when turned on would generate a dedicated log file for the process group.

The nifi-app.log file would not be changed and would still contain all of the data.

The nifi_app_pg_.log file would have the same properties as nifi-app.log file (roll over, size, etc) and would contain the logs generated by the components included in that process group.

NIFI-3065

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 11
  • JDK 17

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[timeabarna]

Thanks @turcsanyip for your review, I've corrected SimpleProcessLogger.

[turcsanyip]

Thanks @timeabarna! SimpleProcessLogger now renders the log messages as before.

[ghost]

Hello, I don't know if this is the right place to comment this, but I couldn't find anywhere else to ask but my team and I are currently working on implementing a similar cluster monitoring system within our codebase however we have run into some issues. I understand that you have created this PR recently, but do you have an estimated ETA (Like months or years?) on how long you think this will hit production. Let me know if there's a better place to reach you/ask this. Thank you.

[pvillard31]

Hey @aszkowski - I'm obviously not speaking in the name of the Apache NiFi community, but this pull request should be merged fairly soon as this is under active review. However it's very likely that this change will only be merged on the main branch (which means this change will be for the upcoming NiFi 2.0) and not the support branch for 1.x. In terms of release, the community is making great progress around the NiFi 2.0 goals and I think we can expect a first release of NiFi 2.0 in the next 2-3 months.

[pvillard31]

As an additional note - this change is bringing a very nice feature to NiFi so any help to build the change and give this a try and share feedback is more than appreciated!

[timeabarna]

Hello @markap14,

Thanks for your review, I've modified the PR based on your recommendations.

[markap14]

Thanks @timeabarna I think we're really close! Have been testing this, and everything is working well for the most part. I'll leave a couple of minor comments inline, but also I noticed an issue with regards to the synchronization. I created a group and set the Log File Suffix to "Main" and saved it to the registry a Version 1. I then made some other changes to the flow, and I changed the Log File Suffix to "Main22" and saved that as V2.

Now, I can change the Log File Suffix to "SomeOtherFile" and change version to V1. This changed the Log File Suffix back to "Main". If I then change it again to "SomeOtherFile" and change to V2, it again gets overwritten to "Main22". So in the StandardFlowSynchronizer, we need to ensure that we don't overwrite the value if it's already been set.

[timeabarna]

Thanks @markap14 for your recommendations, I've updated the PR accordingly.

[markap14]

Thanks @timeabarna I did some additional testing, etc. and all looks good to me now. I'm a +1. Looks like Github Actions hasn't yet run. So i kicked that off. Assuming that it's green, I should be able to merge today. Thanks!