One of the most exciting and valuable features (in my opinion) of Cloudera Flow Management 4 (powered by Apache NiFi 2) is the Flow Analysis Rules engine and the various rule implementations you can bring to bear.
The Flow Analysis Rules Engine is a built-in capability to Apache NiFi 2 whose mission is to help Flow Administrators enforce Best Practices in flow design. This includes limiting how many resources (threads, memory, disk, etc.) a component can leverage. For example, a Flow Admin may want to restrict all processors to use a maximum of 10 concurrent tasks. One reason for this (especially in large flows) is to prevent a Flow Designer from "hogging" the threads away from other components / teams, or to maximize parallelization across physical CPU cores rather than concurrency (which can be viewed as "apparently parallel" but not truly parallel). The same goes for disk usage that can be enforced by limiting the size of a connection's backpressure settings to prevent too much data from amassing on disk without being able to be processed by the flow.
I start by showing a Flow Analysis rule and how to configure and enable for reporting warnings or violations. To create a Flow Analysis rule (which is a "management controller service"), go to the hamburger menu and select Controller Services. There is a tab for Flow Analysis Rules. Selecting that then the + icon on the right brings up the creation dialog:
To start, we create a DisallowComponentType rule, then edit the configuration:
Before we get into the properties for this particular, note the Enforcement Policy on the right. The current choices are Warn and Enforce:
If a rule is set to Warn, a component that violates the rule can still be started and run, but the warning will show up on the Rules Violations report which I will show later. If a rule is set to Enforce, even if a component is configured correctly, it will be marked as Invalid and is unable to be started/run. This violation also shows up on the report. In this case I will set it to Enforce and move on to the rule-specific properties:
There is a single property for this rule, namely the type of component that should be disallowed. I have chosen ExecuteScript to prevent any Flow Designer from using ExecuteScript as the script author can perform tasks that may be harmful to the larger CFM/NiFi instance.
I apply this change and return to the canvas, where I create an ExecuteScript processor and configure it so it would normally be ready to run:
Going back to the rule, I will click on the sideways-ellipsis icon on the right and Enable the rule:
Returning to the canvas again, I click on the "first-aid kit" icon on the top-right of the UI. With a Warning violation, the icon will turn orange and with a Enforced violation the icon will be red. Clicking on it brings up the Rules Analysis Report:
Here you can see ExecuteScript has been marked invalid and the violation shows up under Enforced violations. Also, since the Rules Analysis Report is available on the global NiFi canvas, a violation might belong to a component deeply nested in the flow. Clicking on the sideways-ellipsis allows you to navigate in the canvas to the component in violation:
This helps you quickly get to the offending component and take appropriate action.
Here's a list of Flow Analysis Rules available in Apache NiFi 2.6.0:
- DisallowComponentType
- RestrictBackpressureSettings
- RequireServerSSLContextService
- RestrictFlowFileExpiration
The following is the list of Flow Analysis Rules available in Cloudera Flow Management 4.11:
- DisallowComponentType
- RestrictBackpressureSettings
- RequireServerSSLContextService
- RestrictFlowFileExpiration
- RestrictYieldDurationForConsumeKafkaProcessors
- DisallowDeadEnd
- RecommendRecordProcessor
- RequireHandleHttpResponseAfterHandleHttpRequest
- RestrictComponentNaming
- RestrictConcurrentTasksVsThreadPoolSizeInProcessors
- RequireMergeBeforePutIceberg
- DisallowDeprecatedProcessor
- DisallowConsecutiveConnectionsWithRoundRobinLB
- DisallowExtractTextForFullContent
- RestrictSchedulingForListProcessors
- RestrictThreadPoolSize
- RestrictProcessorConcurrency
With more on the way (both in Apache NiFi and Cloudera Flow Management) as more Best Practices are identified and the associated rules are implemented.
In this post I described the Flow Analysis Rules engine and its features and capabilities. Please put any questions or comments (and especially ideas for new rules!) in the comments.
