Member since
02-01-2022
245
Posts
76
Kudos Received
54
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
47 | 12-05-2023 06:22 AM | |
89 | 11-28-2023 10:54 AM | |
100 | 11-28-2023 05:55 AM | |
151 | 11-15-2023 05:36 AM | |
153 | 11-13-2023 07:03 AM |
11-28-2023
05:59 AM
@MR_KD You should have a look in the Ranger audit logs for Result: Denied. Then begin to add the appropriate ranger policies. The denial should explicitly show the rule that was violated. Additionally, you may need to create the hdfs location, and or fix permissions if it exists already and has the incorrect permissions.
... View more
11-28-2023
05:55 AM
@Hae I am not exactly sure of your settings or the issue with transactions, but I can share what is working for me. In my CDP Public Cloud nifi flows i use some required settings to PublishKafka. SSL Context Service:Default NiFi SSL Context Service Security Protocol: SASL_SSL SASL Mechanism: PLAIN Username: CDP Workload User Name Password: CDP Workload User Password Make sure you user has access to everything of course and you have set the workload password. Additionally, be patient on first processor run, sometimes takes minute or so of startup time to see SUCCESS. Then give refresh on SMM to make sure the kafka topic is created and has the expected data.
... View more
11-28-2023
05:47 AM
@joseomjr Is on to the right solution here. Your regex statement should match "kafka.topic" not "${kafka.topic}". A quick test in regex101.com confirms "kafka\.topic" should match.
... View more
11-28-2023
05:35 AM
@raj_dev This may be a better question to the OSS NiFi Community. There is a mailing list and Slack (chat) you can find here: https://nifi.apache.org/mailing_lists.html At Cloudera we provide our own branded NIFI UI. You would need to do similar. This is a very complicated task that requires that you build your own binaries w/ the modifications to the UI. Before you go so deep as to build a new UI w/ your logo, be sure to have a look at Nifi of the Future, where you deploy nifi flows as a function in any cloud provider, or nifi on kubernetes. In this future state no one sees or touches the nifi UI. You can find more about Cloudera DataFlow here: https://docs.cloudera.com/dataflow/cloud/index.html
... View more
11-15-2023
05:36 AM
1 Kudo
In regards to your nifi expression language.: Test in a simple flow, and inspect the flowfile's attributes to insure topicName is correct. Then take it to the kafka processor. If you are not seeing the results of the expression, something isnt right, or the value doesnt except Expression Language. Look at the (?) on any property to see what is accepted. I suspect the expression language is invalid as well, so make sure you test and confirm the attribute is as expected before trying to use it deeper in your flow. In regards to mapping. What i suggest could be DistributedMapCache or even a flow that does a lookup against some other service. With this concept you provide a mapping key value pairs that correspond to your consumer and producer topics. When you lookup a key, For Example: "alerts_Consumer_Topic_Name" the value would be "alerts_Producer_Topic_Name". If this is stored outside of nifi, then these values can be managed and changed outside of the scope of the nifi flow. Example flow with DistributeMapCache: https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml
... View more
11-13-2023
07:15 AM
@Arash Did you inspect the flowfile (assure its the expected format, etc) and inspect the flowfile attributes (may be more detail into the conflict). Additionally, you can set the processor log level to DEBUG and/or monitor the nifi-app.log for more details. If this worked before, and now doesnt, i would expect something to have changed in the flowfile.
... View more
11-13-2023
07:03 AM
Sometimes I need to increase the timeouts in the processor configurations. You may also need to reduce the execution time for the processor incase the endpoint cannot handle too many requests at once.
... View more
11-08-2023
07:42 AM
@Arash This error is indicating an issue with your SSL Context Service "SSL Service for elasticsearch cluster". If this was previously working, perhaps the SSL cert has changed? You should check if the elasticsearch cert has been renewed, and if so, update the keystores/truststores accordingly and test again.
... View more
11-08-2023
07:36 AM
@Rohit1997jio If your endpoint test works with Postman, and your invokeHttp is setup similary, the above error suggest the nifi node cannot connect to the endpoint. You would need to ensure you have network connectivity from nifi host to endpoint.
... View more
11-08-2023
07:33 AM
@Rohit1997jio I do not think this is possible. You would need a method outside of the consume/produce that handles logic for which consume topic maps to which produce topic. Then you could use a dynamic topic name in the producer. However, you would still be limited in fact that ConsumeKafka doesnt take upstream connections. In the example above, if customerTopicX is attribute based, you can just use the same attribute logic in topic Name for a single publishKafka versus three seen above. That would atleast clean up your flow.
... View more