Member since
07-30-2019
3436
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 129 | 01-27-2026 12:46 PM | |
| 556 | 01-13-2026 11:14 AM | |
| 1194 | 01-09-2026 06:58 AM | |
| 989 | 12-17-2025 05:55 AM | |
| 491 | 12-17-2025 05:34 AM |
10-12-2021
07:24 AM
1 Kudo
@CodeLa Giving detailed responses on such a use case would be very difficult in the community. This would take considerable effort and time and would require you to provide a lot more detail to include sample source json files, schemas, etc... Cloudera offers professional services for its customers to help them with their use case solutions. If you have a support contract with Cloudera, please reach out to your account owner about this service. At a very high level, I would suggest you take a look at the PutDatabaseRecord processor and perhaps configure it to use one of the json readers: JsonPathReader JsonTreeReader The processor would also need a DBCPConnectionPool for connecting to your MySQL DB. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-08-2021
11:30 AM
1 Kudo
@Ankit13 The PutFile is going to execute base on its configured run schedule (Timer Driven Execution) or cron schedule (Cron Driven Execution). If you are adding an attribute to the FlowFile that you are using to evaluate a boolean true or false, the best approach is to add a RouteOnAttribute processor between your FetchFile and PutFile processors to redirect those FlowFiles where your condition does not resolve to "true". In this way you selectively decide which FlowFiles to pass to the putFile to be executed upon. As far as the RouteOnAttribute, it will have an unmatched relationship and you add dynamic properties which become new relationships that can be associated with different connections. You can use NiFi Expression Language (NEL) [1] to construct a boolean statement to evaluate your routing condition. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-07-2021
05:50 PM
Matt, Thanks for your reply. Basically I have two local nifi's: one is that 1.11 and another is 1.14 both of them are setup almost identical in the nifi.properties, however 1.11 is working when passing the azure workspace key and Id and the other 1.14 throwing the error. the log file doesnt have much info besides the message I posted above but here is the stack trace from the log file if that helps in anyway: java.lang.RuntimeException: HTTP/1.1 403 Forbidden at org.apache.nifi.reporting.azure.loganalytics.AbstractAzureLogAnalyticsReportingTask.postRequest(AbstractAzureLogAnalyticsReportingTask.java:164) at org.apache.nifi.reporting.azure.loganalytics.AbstractAzureLogAnalyticsReportingTask.sendToLogAnalytics(AbstractAzureLogAnalyticsReportingTask.java:147) at org.apache.nifi.reporting.azure.loganalytics.AzureLogAnalyticsReportingTask.sendMetrics(AzureLogAnalyticsReportingTask.java:137) at org.apache.nifi.reporting.azure.loganalytics.AzureLogAnalyticsReportingTask.onTrigger(AzureLogAnalyticsReportingTask.java:107) at org.apache.nifi.controller.tasks.ReportingTaskWrapper.run(ReportingTaskWrapper.java:44) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Im hoping someone can try the same on their end to verify if they are getting the same results
... View more
10-06-2021
12:53 PM
Hi Matt, Thank you for your suggestion I've recovered it by using flow.xml.gz file.
... View more
10-06-2021
11:34 AM
@sundaram_idfc Additional information may help here: What versions of NiFi are bing used by each NiFi instance you have installed? What Java version is being used by each of these NiFi instances? What is your configured "Max Timer Driven thread count" setting under global menu --> controller settings? How many cores does your server have that is running NiFi? Any chance you are occasionally sending a larger FlowFile? When in this state, did you collect a series of NiFi thread dumps to see if the thread is progressing or what it is waiting or blocked by (nifi.sh dump <dump filename>) Any WARN or ERROR related log entries in the nifi-app.log related to this port, OutOfMemory, or file handles? Thanks, Matt
... View more
10-06-2021
09:29 AM
Thank you very much for your reply! I followed your advice and managed to get the processor working! Thanks again!
... View more
10-06-2021
06:02 AM
1 Kudo
@CodeLa It is difficult for me to help determine issue in your dataflows without your complete dataflow. My guess is somewhere in the process of splitting your xml files the structure has change in such a way that the Java regex I provided no longer matches.
... View more
10-05-2021
11:33 PM
@TRSS_Cloudera Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
10-05-2021
01:07 PM
Hello @edoS Welcome to the community! NiFi's provides so many option for user authentication and authorization, setting up exactly what you need can be overwhelming at times. This is certainly something the Cloudera support could walk you through if you have a support contract with us that covers the NiFi service. At a high level, here is what you need to understand about the authentication and authorization process in NiFi. Authentication happens first and must be successful before any authorization is verified. NiFi supports numerous ways to authenticate users/clients (TLS, Kerberos, LDAP, openID, etc...). No matter which method is used, the end result of any authentication is a user string that identifies the successfully authenticated user/client. That user string is then evaluated against the identity mappings [1] you may have configured in the nifi.properties file. These identity mappings are used to normalize the user strings. for example: Trim the CN from the full DN in a user/client certificate Trim the user name from a kerberos principal convert the user string to all uppercase or lowercase The resulting user/client string is then passed to the authorizer to verify that user/client is authorized for the NiFi Resource Identifier being requested. NiFi authorizers.xml is where this configuration is setup. This file is easiest to read from the bottom up. At the bottom of the authorizers.xml you will find your authorizer which you have setup as the "Ranger-Provider". It is important to understand how this authorizer works. NiFi runs a background thread that checks in with Ranger to see if there is a new policy definition for the NiFi service. If so, the new definition is downloaded by NiFi. What Ranger provides to NiFi in this downloaded policy definition are all the polices setup in Ranger. For each there will be the "NiFi Resource Identifier(s)" along with the user strings and group strings that have been assigned "Read" and/or "Write" permissions. Now remember up to this point all NiFi knows about the authenticated user is the user string. NiFi has no idea yet what groups that user string may belong to. Within the Ranger-Provider, you will find a property name with "User Group Provider". The value set here tell the authorizer where to check to see if the user string passed from authentication has any known user to group associations. Search your authorizers.xml for configured User Group Provider [2]. There are numerous options that can be configured for determining user to group associations. Some of the available providers allow you to configured multiple providers. While the authorizer "ranger-provider" can only point at 1, it may point at a "composite-configurable-user-group-provider" [3] for example that can be setup to reference multiple user-group-providers. The key here is making sure you have added 1 or more user group providers that will return all the user to group associations you need. Based on the log output you shared from the nifi-user.log. We know that none of the user group providers you may have setup returned any group strings associated to your user string (identity[18330301],groups[] ). This is why "groups [ ]" is empty. The "file-user-group-provider" [4] allows you to create user string to group string associations manually via the NiFi UI directly. The commonly used "ldap-user-group-provider" [5] determines user and group associations via user and/or group syncs with ldap/AD. Now that NiFi knows what groups the authenticated user string is associated with, the user and the groups can be checked against the downloaded policies to see if the user is authorized for the action being performed or the end-point trying to be accessed. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup [3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-implementations [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider [5] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
10-05-2021
12:49 AM
Thanks @MattWho ! It works. The feature "ReportLineageToAtlas" is now enabled in NiFi. Thanks a lot.
... View more