Member since
07-30-2019
3436
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 143 | 01-27-2026 12:46 PM | |
| 560 | 01-13-2026 11:14 AM | |
| 1231 | 01-09-2026 06:58 AM | |
| 1005 | 12-17-2025 05:55 AM | |
| 498 | 12-17-2025 05:34 AM |
05-28-2021
09:23 AM
@Amn Where did you define the NiFi Attribute "${itemType}" (FlowFile Attribute on a source FlowFile driving execution of GetMongo processor or NiFi variable registry)? NiFi is also case sensitive, so for example "ItemType" and "itemType" would be considered different variables. If you found this helpful to yoru query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
05-28-2021
08:23 AM
@Rupesh_Raghani I just want to make sure that when we are both talking about NiFi "Templates" we are talking about the same thing. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates When you upload a NiFi template (xml file) to NiFi via the UI (Does not matter which node in a NiFi cluster you are accessing), that template will get uploaded and replicated to all nodes in the cluster. So all nodes will have that template in each nodes JVM heap and written to the flow.xml.gz on disk. This statement is not clear to me: "So if I have templates on every node in cluster then will the nifi load templates on each node as per templates available on their node or a parent node will load all templates from other cluster". What do you mean by "if I have templates on every node"? In a NiFi cluster, every node must have the same flow.xml.gz. If the flow loaded in to heap memory does not match between nodes, the nodes no matching the elected cluster flow will be disconnected from the cluster. Each node, while it has its own local copy of the flow, runs the exact same flow. Hope this helps, Matt
... View more
05-24-2021
11:56 AM
Some examples https://community.cloudera.com/t5/Support-Questions/How-Extract-text-from-a-multiline-flow-and-create-only-one/td-p/104706 https://nathanlabadie.com/recombining-multiline-logs-with/ https://github.com/tspannhw/EverythingApacheNiFi/blob/main/README.md
... View more
05-24-2021
07:05 AM
Thanks for taking time to explain, trying to untangle the version numbers etc. are a new challenge. 🙂 I guess I was confused since our cloudera runtime still says cdh7.1.5... I need to get with one of my coworkers on this, because my access still won't let me create a support case. 😞 (which is why I am here. 🙂 )
... View more
05-21-2021
01:46 PM
@SAMSAL Sounds like you are very close... The 403 has nothing to do with authentication, but rather authorization. This eludes to the fact that the client was able to be properly trusted through the mutual TLS handshake, but when that resulting client string was checked against the endpoint policy being checked it was not present, thus resulting in a 403 response. The DN is the DistinquishedName for your certificate. You can use the below command to get the verbose details on the certs added to a keystore: keytool -v -list -keystore <keystore.jks or truststore.jks> You will see that each certificate has either type "PrivateKeyEntry" or "TrustedCertEntry" For each certificate you will see an "Owner" and an "Issuer" For each of those it will show the complete DN which would look something like: CN=<some string>, OU=<some string>, DC=<some string> There are numerous parts that can be part (CN, OU, O, ST, DC, etc.) of a DN and various lengths The "Owner" DN for the PrivateKeyEntry is what is used as the client/user authenticated string after the successful mutual TLS handshake. The nifi.properties file has optional ideNtity.mapping properties that can be configured to trim and manipulated these identity strings (for example pulling our on the string from the CN). With nifi.remote.input.secure= set to false, FlowFiles send over S2S will not be over a TLS encrypted connection. The fact that you say it works when you have this set to false and you can still successfully obtain S2S details from the configured https:// secured NiFi Destination URL tells me that authentication is correct for "Retrieve site-to-site details" NiFi policy. That only leaves having incorrect policy setup for your remote input port. You should open a command prompt on each node and "tail -F /<path to>/nifi-user.log". Then enable your controller services and check the logs being tailed for the authorization exception. It should clearly show you the "string" being checked against the "receive data via site-to-site" endpoint which in the log would look like "/data-transfer/input-ports/<uuid of remote input port>". Also remember that you are NOT authorizing your user to any of these S2S policies, but rather the DN or identity mapped DN string. Really hope this helps you get fully secured here. Matt
... View more
05-21-2021
10:21 AM
@MattWho, Good afternoon. I changed my flow to make a counter by the processor "RouteOnAttribute 1.12.1" so every time I have a record to be sent to the API and have a return to store in Oracle, I do the LOOP through RouteOnAttribut. Thus sending one FlowFile at a time to the PROCEDURE call in Oracle by "ExecuteSQL 1.12.1" But what is happening at the moment is that from the second time it goes through the LOOP, it seems to me that the Flow to get the attributes "CODIGO" and "JSON_ORIGINAL" is not passing the values that I requested in the processor "EvaluateJsonPath 1.12.1" passing the parameters as null. Do you have any idea what may be happening in this second flow passage within the LOOP? This is the JSON ORIGINAL of the second stream that I must send to Oracle = This is OK with the CODE attribute as you can check. This is Split Configure Processor = This is EvaluateJsonPath 1.12.1 Configure Processor = This is MergeContent 1.12.1 Configure Processor = This is ExecuteSQL 1.12.1 Configure Processor =
... View more
05-20-2021
02:48 PM
1 Kudo
@leandrolinof NiFi Expression Language (NEL) [1] does not read content of the FlowFile. The RouteOnAttribute processor never looks at the content of a FlowFile. So verify your source FlowFile has attributes set with valid numeric only values. So your inbound FlowFile would need to have two attributes on it already: 1. cont 2. CONTADOR Note: NiFi is case sensitive as well. And both these attributes need to have assigned values to them. The NEL statement you have will return the value assigned to the FlowFile attribute "cont" and check to see if it is less than the value assigned to the FlowFile attribute "CONTADOR". If that resolves to "True", the FlowFile will be routed to the connection containing the new dynamically created "CONTINUE" relationship. Otherwise, it will route to the "unmatched" relationship which you appear to have auto-terminated. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If you found this addressed your query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-20-2021
02:33 PM
@P_Rat98 Can you share how you have the ListS3 processor configured? What can you tell us about the file being listed? Is it constantly being updated? Thanks, Matt
... View more
05-20-2021
10:56 AM
@Chakkara More detailed would be needed before I would know if I can offer any advice here. Can you share: 1. PutDatabaseRecord processor configuration 2. Assuming you are using the CSVReader, please share its configuration 3. DBCPConnectionPool configuration 4. Complete Exception including any stack trace if exist form nifi-app.log. 5. Any data manipulations made between your GetFile and PutDatabaseRecord processors? 6. Sample input file would also be very helpful. Thanks, Matt
... View more
05-20-2021
06:17 AM
1 Kudo
Not all credit goes to me but I was pointed to a better simpler way to achieve this. There are 2 ways. Solution 1 - and the simplest and elegant Use Nifi JoltTransformJSON Processor. The processor can make use of Nifi expression language and attributes in both left or right hand side of the specification syntax. This allows you to quickly use the JOLT default spec to add new fields (from flow-file attributes) to a new or existing JSON. Ex: {"customer_id": 1234567, "vckey_list": ["test value"]} both of those fields values are stored in flow-file attributes as a result of a EvaluateJSONPath operation. Assume "customer_id_attr" and ""vckey_list_attr". We can simply generate a new JSON from those flow-file attributes with the "default" jolt spec and the right hand syntax. You can even add addition expression language functions to the processing [
{
"operation": "default",
"spec": {
"customer_id": ${customer_id_attr},
"vckey_list": ${vckey_list_attr:toLower()}
}
}
] This worked for me even when storing the entire JSON, path of "$", in a flow-file attribute. Solution 2 - complicated and uglier Use a sequence Nifi ReplaceText Processor. First use a ReplaceText processor to append the desired flow-file attribute to the file-content. If you are generating a totally new JSON, this would do it. If you are trying to modify an existing one, you would need to first append the desired keys, than use ReplaceText again to properly format as a new key in the existing JSON, from {"original_json_key": original_json_obj}{"customer_id": 1234567, "vckey_list": ["test value"]} to {"original_json_key": original_json_obj, "customer_id": 1234567, "vckey_list": ["test value"]} using Then use JOLT to do further processing (that's why Sol 1 always makes sense) Hope this helps, spent about half a day figuring out the 2nd Solution and was pointed to Solution 1 by someone with more experience in Nifi
... View more