About araujo

araujo · ‎02-17-2022

Hi, @hamsterrrrr , Those events represent the point-in-time occurrence when the alert was thrown. They don't tell you the current issues with the cluster, which is what you see in the UI. I don't think the REST API has an endpoint that gives you that list directly, but it's pretty easy to find out. In the results of the /clusters/{cluster}/services endpoint, look for services with the status = BAD or health checks with summary = BAD. For example: { "items": [ { "name": "impala", "type": "IMPALA", "healthSummary": "BAD", <<--- this "healthChecks": [ ... { "name": "IMPALA_IMPALADS_HEALTHY", "summary": "BAD", <<--- or this "suppressed": false }, ... ], ... } ] } Cheers, André

araujo · ‎02-17-2022

Hi, @spserd , Please have a look at this section of the NiFi documentation. It says: Wildcard certificates (i.e. two nodes node1.nifi.apache.org and node2.nifi.apache.org being assigned the same certificate with a CN or SAN entry of *.nifi.apache.org) are not officially supported and not recommended. There are numerous disadvantages to using wildcard certificates, and a cluster working with wildcard certificates has occurred in previous versions out of lucky accidents, not intentional support. Wildcard SAN entries are acceptable if each cert maintains an additional unique SAN entry and CN entry. Even though you are not using an asterisk wildcard your single certificate doesn't meet the requirements of a unique SAN and CN entries and is not recommended/supported. You should have separate certificates for each host. Cheers, André

araujo · ‎02-17-2022

This is the default when you deploy a new insecure cluster. When you implement security you need to review these defaults and tighten security, including employing Ranger to protect, among other things, the HDFS data in external directories. Cheers, André

araujo · ‎02-17-2022

Thanks for the link to the video, @MartinTerreni . In a traditional NiFi flow that read from Kafka and writes to Kafka, the offsets of the source topic are indeed stored by the consumer in Kafka itself. With this, if NiFi crashes or restarts, the flow will continue to read the source topic from the last offset stored in Kafka. The problem, which @mpayne explained in the video, is that there is a decoupling between the consumer and the producer in a traditional flow. And this can cause data loss or data duplication, depending on the scenario. For example, the ConsumeKafka processor commits offsets to the source Kafka cluster in batches at regular intervals. It is possible that some messages read by the consumer are written by the producer to the destination topic before the offsets of those records are committed to the source Kafka cluster. If the flow stops abruptly before the commit happens, when it starts again it will start reading from the previously committed offset and it will write duplicate messages to the target topic. On the other hand, since there's no tight coupling between consumer and producer, the consumer could read some messages and commit their offsets to the source cluster before the ProduceKafka processor is able to deliver those messages to the target cluster. If there's a crash before those messages are sent and some of the flowfiles are lost (e.g. on node of the cluster burned down) that data will never be read again by the consumer and there will be a data gap at the destination. The new feature explained in the video address all of these issues to guarantee Exactly Once Delivery semantics from Kafka to Kafka. I hope this helps clarifying it a bit more 🙂 Cheers, André

araujo · ‎02-16-2022

You're correct, @MartinTerreni . If you set the Consumer Group Id property for the processor that consumes from Kafka the offset will be maintained across NiFi crashes and restarts. Note, though, that the Consumer Group Id in NiFi is specified by processor, not by record reader. Would you please share the link to the said video? Cheers, Andre

araujo · ‎02-15-2022

@wichovalde , The error below is a server-side error that should (hopefully) be logged in the Atlas server log. Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: https://<atlas-server>:31443/api/atlas/v2/entity/uniqueAttribute/type/hive_db?attr%3AqualifiedName=inventarios%40cm&ignoreRelationships=true&minExtInfo=true&user.name=superuser, status: 500, message: Server Error Try to find the entries in the Atlas server log that match this call and it could tell you a bit more about the problem. If the Atlas log doesn't seem to have the corresponding information, try setting its log threshold to DEBUG in Cloudera Manager and restarting the service and repeating the test. André

araujo · ‎02-13-2022

@Chhavi , Please try this: curl -X POST \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'username=bob&password=supersecret1' \ "https://nifi-1.example.org:8443/nifi-api/access/token" Regards, André

araujo · ‎02-13-2022

Nice article, @carrossoni ! I know it's been a while, but just saw this for the first time 🙂

araujo · ‎02-13-2022

@yamaga , does the above help?

araujo · ‎02-12-2022

Some time ago I faced an interesting problem with a cluster failing to start after I replaced an MIT KDC with a FreeIPA KDC. For the replacement, I installed the ipa-client package on the cluster nodes, and then, changed the KDC configuration in Cloudera Manager (CM) (changed realm and KDC details, imported Kerberos user, re-generated credentials, etc..) The cluster refused to start. Besides that, CM's KDC Login monitor kept complaining about the KDC not being healthy. I could manually kinit successfully, though, and there seemed to be no KDC problems at first glance. After enabling debug at different places I saw that there were socket timeouts when processes tried to connect to the KDC and that those processes were actually trying to connect to the KDC over UDP, rather than TCP. The UDP requests explained the problem, since UDP traffic was blocked between the cluster and the KDC. What's strange, though, is that the krb5.conf created by the ipa-client install had the following configuration: udp_preference_limit = 0 According to the MIT documentation, this should force all the communication to be over TCP, instead of UDP. From the MIT website: "When sending a message to the KDC, the library will try using TCP before UDP if the size of the message is above udp_preference_limit. If the message is smaller than udp_preference_limit, then UDP will be tried before TCP. Regardless of the size, both protocols will be tried if the first attempt fails." Even though the "library" above doesn't refer to the Java library, Java does recognize the udp_preference_limit parameter from the krb5.conf, as explained here. So, I'd expect that, with that setting, TCP would be tried first for all requests, but it was not. And after 3 UDP attempts, the connection would actually fail altogether without trying to connect over TCP. I found it interesting, though, that the ipa-client installation set that value to 0. At Cloudera, we have always recommended to customers to set it to 1 instead. So I went ahead and changed the entry in the krb5.conf to: udp_preference_limit = 1 And amazingly everything worked after that!! The debug logs didn't show traces of UDP requests any longer, the cluster came up correctly and the CM alerts went away. Interesting how something really small can badly break things leaving very little vestiges of what's going on... The JDK behavior is coming from this. So, in short, to be on the safe side always set udp_preference_limit to 1 and never to 0.

Online	Offline
Last Visited	‎07-21-2025 10:25 PM

Member Since	‎06-26-2015 11:59 AM
Last Visited	‎07-21-2025 10:25 PM
Posts	515
Kudos received	140

Cloudera Community

Re: Is it possible to use Single User authenticati...

Re: Dynamically Assign an XSD File

Re: "error": "There is no mapped role for the grou...

Re: Read xml file content into an Attribute: How t...

Re: Nifi Lookup CSV values with SQL NULL values

Re: can't get critical alerts from a dashboard

Re: Understanding NiFi Certificates functionality

Re: hive外部表目录权限问题

Re: Is offset of kafka lost on nifi failure?

Re: Is offset of kafka lost on nifi failure?

Re: Authentication Failed in Atlas Hive Import

Re: Nifi API - 403 error

Re: How to create a Centos7 CDP-DC Base VM for san...

Re: Nifi - MergeContent - Multiple CSV files - cou...

Why you should always set udp_preference_limit to ...