Member since
12-02-2020
9
Posts
0
Kudos Received
0
Solutions
09-28-2023
12:54 PM
@sarithe You may also want to take a look at Process Group (PG) FlowFile Concurrency configuration options as a possible design path since there does not appear to be any dependency between task 1 and task 2 in your description. You just want to make sure that not more than 2 tasks are executing concurrently. You move your processors that handle the 2 task executions inside two different child PGs configured with "Single FlowFile per Node" Process Group FlowFile Concurrency. Within the PG you create an input port and output port. Between these two ports you handle your task dataflow. Outside this PG (parent PG level), you handle the triggering FlowFiles. The task PGs will allow 1 FlowFile at a time to enter that PG and because of the FlowFile Concurrency setting, not allow any more FlowFiles to enter this PG until that FlowFile processes out. As you can see from above example, each task PG is only processing a single FlowFile at a time. I built this example so that task 2 always takes longer, so you see that task 1 Pg is outputting more FlowFile processed the Task 2 PG while still making sure that on two tasks are ever being executed concurrently. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-12-2023
04:35 AM
@sarithe, What is the format of the record you are trying to convert? What is the data type of the field you are trying to convert? Are you trying to modify the value of a record or the value of an attribute? If you are trying to update the values from within each record, you should try using an UpdateRecord processor, in which you define a Record Writer and a Record Reader. Next, you can add another property in your processor and define it like: Property: /your_column --> pay attention to the slash in front as it is very important. Value: ${field.value:multiply(1000):toNumber():toDate("yyyyMMdd", "GMT"):format("yyyyMMdd")} --> this is just an example, as I do not know how your data looks like and how you want it displayed. You can use NiFi's EL to define the exact format you required, but make sure you use field.value if you want to modify the value from within that specific column.
... View more
12-08-2020
09:45 AM
@sarithe It is expected behaviour as Ranger has to pull resources from NiFi and as NiFi is secured it will look for authentication token from Ranger but as Ranger is un-secure it can not present the token and get 403 as response. You should enable SSL [or kerberise the cluster] for Ranger so Ranger can authenticate using SSL [or tgt]certificate. Meanwhile you can use https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 as reference to setup policies manually. -Akash
... View more