Q&A

Hello everyone,I have developed an Apache NiFi flow that ingests streaming data, converts it to JSON, merges the records, and then stores them in HDFS. However, I have observed a couple of issues:File Structure: After the merge operation, many files retain their original structure rather than being merged as expected.Performance: Increasing the number of pins along with the minimum and maximum record counts and bin sizes causes the merge process to become significantly slower. Conversely, reducing these parameters results in files that mostly maintain their original structure.Could anyone provide insights or recommendations on how to resolve these issues?Thank you.My crrunt cnfigration that cause the flow to go the orginalcurrent comigration where it causes go to original  The flowthe flow  The congigration that never showsnmerged records.with this cofigration will take forever to merge I didn't see any output 

Hi,I've got a Ambari 2.7.3 cluster with Ranger 0.7.0 (yes, I know it's old...).I also have usersync integration with FreeIPA. All these works fine, but I have one issue.I want to restrict access to Admin UI to admin only. No matter what I change in configuration, everytime all users are being added to "Resource Based Policies" and "Reports" modules. Is there any option to disable this?Is there any other option to restrict access to GUI?

HiI've got a CDE 1.22.0-h1-b3 cluster that fails to load the Spark UI after the job has completed. I can see the Spark UI during the running of a job but it fails to load/open once the job has completed.I see an http GET to the /jobs endpoint fails with a 404 Not Found.Have you come across this error and resolved it please?CheersCraig

Kafka policies created in Ranger are getting downloaded but not becoming active. Using Apache Ranger 2.6 and Apache Kafka 3.6. Couldn't find any specific errors related to this issue.Ranger and Kafka are configured with LDAP and no kerberos. What could be the possible issue? Any help is appreciated!Ranger policies for HDFS and Hive works fine.Below are the ldap and ranger configs in Kafkaauthorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizersasl.enabled.mechanisms=PLAINlistener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAINlistener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required;listener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandlerldap.java.naming.provider.url=ldap://<ldap_host>:389ldap.java.naming.security.authentication=simpleldap.java.naming.security.principal=CN=<bind_user>,OU=Servic

Hi Team,We are looking for your help to troubleshoot & fix the below issue.Issue description : All yarn jobs are failing to get submitted with below error after enabling Ozone.ERROR :Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1741072771401_0004 to YARN : Failed to renew token: Kind: OzoneToken, Service: <host ip>:9862, Ident: (OzoneToken owner=hdfs/<hostname>@PLATFORMKRB.COM, renewer=yarn, realUser=, issueDate=2025-03-06T06:57:54.532Z, maxDate=2025-03-13T06:57:54.532Z, sequenceNumber=13, masterKeyId=1, strToSign=null, signature=null, awsAccessKeyId=null, omServiceId=<hostname>, omCertSerialId=6)Regards,Sushant

That pretty much sums it up. We had a database server go down over the weekend, and I have a suspicion that things got backed up, but I can't get to the UI to try to unravel it. I have tried setting autoResumeState to false and setting everything in flow.xml.gz to "Disabled", but the UI still won't come up because of this error. Unfortunately, all of this is in an isolated environment, so sharing full logs is not impossible but very, very difficult. This vm has 23GB of RAM and Xms=4096m and Xmx=8192m. niagarafiles-community-11.6.3

Hi Everyone, I am Emmanuel Katto Dubai, United Arab Emirates (UAE) It appears you're encountering an issue when attempting to run an INSERT INTO query on a Hive table with the Tez execution engine. You mentioned that the query works fine when inserting data into integer columns but fails when inserting into string columns. Additionally, the issue seems related to the Kryo serialization in the logs, where Hive is unable to deserialize the required classes. Here's a breakdown and a few suggestions to troubleshoot the issue:   Error Analysis: Error Type: The main error seems to be related to a serialization issue KryoException: Encountered unregistered class ID: 112     This indicates that Hive, during execution, is attempting to deserialize an object that hasn't been registered with the Kryo serializer. This might be related to how the Tez engine handles serialization during the execution of the query. YARN Logs: The error trace shows a failure at the map

How do I get the Client Id to make any processor update using REST PUT call ?

Hello Experts,I have a flow where I use 'SplitRecord' processors to split json records and then in downstream I am merging them back to single file using defragment strategy.Flow is dividing into 2 route downstream so I use merge in 2 places, where merge works fine in one place and does not work in another place.Flow diagram how it is setup - So on the left side MergeContent its merging fine with defragment strategy. But on the right side (red marked) its giving below error (since another api call is in between the flow file arrival to merge record processor might be not in order.)(Example Error : "Cannot defragment flow files with fragment Id XXXXXbecause the expected number of fragment is 5 but found only 3.")I am sending 20 request to end point (HandleHttpRequest) and each request has 5 json records so each request gets split into 5 FF. so it will be come total 100 flow files.(I am sending all 20 request one after other or parallely then this is the issue)Note : If I send

I was able to setup GitLabFlowRegistry in controller settings successfully (which is new feature in Nifi 2.0)From Nifi UI, I am able to import Nifi flow directly from Gitlab repository.  After modifications, I was able to successfully commit into Gitlab repo, from Nifi UI.Do we have any toolkit commands to perform these operations?  Currently we are using toolkit commands to deploy Nifi flow changes via CICD pipeline.  Can I achieve the same with new GitLabFlowRegistry?  Appreciate any help/pointers. Thanks  

Hi,I'm trying to build something that depends on a bunch of ambari packages that are listed as available through the hortonworks maven repo at:https://repo.hortonworks.com/content/repositories/releases/However, when I attempt to make use of the maven repo, I receive a 504 Gateway Time-out, which indicates that something upstream of the awslb is not responding.Based on some notes from previous support questions this appears to have been happening since December of last year.Someone indicated a reference to nexus-private.hortonworks.com URL, which does not work (it's a cname to nexus-private.releng-internal.cloudera.com, which does not have any A/AAAA records)Is there any resolution to this?

Hello, I am working on importing multiple Atlas entities using Atlas API calls. After completing execution of each curl command, the message I get is "connection is left intact". Is there a way to kill / terminate this connection to Atlas server? Thankssnm1523

Hi Team,I am working with Apache Atlas using PyApacheAtles to work on Azure Purview. I already created custom types (table and column) and their relationship definition. In my system there are about 30k entities to upload, so when I try to push all of them in one batch I receive timeout. I tried to apply the logic of upload from Atlas Jira  https://issues.apache.org/jira/browse/ATLAS-4389Firstly upload all parents (tables in my case), then columns (related with tables). After successful upload of tables batch, I received an error, when columns batch upload started"errorCode":"ATLAS-404-00-00A","errorMessage":"Referenced entity -1001 is not found" -1001 is a guid of the table, which already is uploaded. I noticed that in case of upload table and column in one batch everything works fine. It looks like Atlas checks if relationship exists in uploaded batch, not between batch and already uploaded entities.  Is there any way to upload related entities in separate ba

Hi Everyone,Sometimes Beeline command(Hive shell) is freezing without any error code. but impala command is working perfectly on this time.all data loadings are freezing.after resart Cloudera, issue will fix.Can anyone help on that issue?Thank you!

I have configured an InvokeHttp Processor with the 'Retry' relationship enabled to capture HTTP status codes between 500 and 599 and initiate retries. I connected the 'Retry' relationship back to the InvokeHTTP processor to enable automatic reattempts. The number of retry attempts is set to 3. However, after all retry attempts are exhausted, the process does not complete and ends up queuing in the Retry relationship, as shown below:Could you please help how to fix it?

The Nifi version I use is 1.23.2. After restarting, memory drops but increases steadily as time goes by.As a result, memory usage reaches 100% day by day.The MonitorMemory output for now is like this:Jvm memory args is:# JVM memory settingsjava.arg.2=-Xms32Gjava.arg.3=-Xmx32GWhat is the reason of this situation?Shouldn't memory usage reach a certain level and stop? 

Hi All,We have installed Nifi and Nifi Registry 1.28 on an AWS ECS instance running on Fargate. The installation seems to be working fine, and both the applications can communicate with each other. We now wanted to set up Git Integration for Nifi Registry using ssh authentication. Can someone please help with detailed steps, article or content with this. We did try multiple options, but this is not successful.Since our Nifi instance is on ECS Fargate, we tried cloning the repo using Docker Image and SSH key which was successful. However, when we pass the details in the providers.xml file, the instance doesn't come up.Can someone please help with the exact steps and what properties and values to be passed in providers.xml file when using ssh authentication.@steven-matison @MattWho : Your guidance and help will be much appreciated!!!

If my job is running in Yarn and Either Resource Manager or Node Manager goes down. Will it affect my job.

Hello all - I'm trying to run the below spark-shell command from the bin directory of spark 3.4.3 extracted location. I specified the master as my Kubernetes environment as I'd like my executors to run on the k8s environment.I created a service account with all necessary permissions.# kubectl auth can-i create pod --as=system:serviceaccount:my-namespace:spark-sa -n my-namespace                                                         yesCommand:# export K8S_TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='spark-sa')].data.token}"|base64 --decode)# ./bin/spark-shell \--master k8s://https://my-k8s-cluster:6443 \ --deploy-mode client \ --name spark-shell-poc \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=my-docker-hub/spark_poc:v1.4 \ --conf spar

Hi Team,I am getting Authentication Error in ConsumeIMAP 1.23.2and ConsumePOP3 1.23.2 processor. Details are descrideb as below:Hostname : outlook.office365.comPort : 995( For POP3)/993 (For IMAP)Autherization Mode : OAUTH2 (Also Manually verified using the details of autherization serverURL and clientID/Secret, it has all relevant access to generate token)Error Details:ConsumePOP3[id=x3603x64-5f1c-1219-xxxx-xxxx34ffc1] Processing failed: org.apache.nifi.processor.exception.ProcessException: Failed to receive messages from Email server: [javax.mail.AuthenticationFailedException - Read timed out - Caused by: javax.mail.AuthenticationFailedException: Read timed out 

I am trying to retrieve data from a table in HDFS in which a column contains timestamps.I am connected in hdfs using CDSW and  running a python script which opens a spark session and run an sql query to retrieve some rows from the table. Although running the same sql query in HUE imala i get the proper values, in CDSW using the python script i get None values only from the timestamp column. How can i retrieve my data properly. It's a huge table so i cannot just export the csv file from the impala editor. I want to retrieve data for migration to another database. The script i run in CDSW is the following:import pandas as pdimport numpy as npimport sysfrom pyspark import SparkContextfrom pyspark import SparkConffrom pyspark.sql.functions import litfrom pyspark.sql.functions import unix_timestampfrom pyspark.sql import SparkSessionfrom pyspark.sql.functions import to_utc_timestampfrom pyspark.sql import SQLContextos.environ['PYSPARK_PYTHON'] = '/opt/cloudera/parcels/Anaconda-5.1.0.1/

Hello eveyone  we’re experiencing with consuming Kafka from a third-party (Qradar) system outside of CDP. In their configuration, the only SASL mechanisms available for configuration are:PLAINSCRAM-SHA-256SCRAM-SHA-512However, all of these mechanisms are resulting in authentication errors. how we can connect Qradar to kafa to be able to consume the data from kafka Thanks in advance  

 Hi there,I have a minifi flow, designed with Nifi Canvas, and contains sensitive value for certain property (like SSL truststore password).How can I set the sensitive value in the minifi flow before execution?I do not need to encrypt passwords, I need to provide password directly in flow json raw.I am using version 2.2.0 for both nifi and minifi.Thank you.Pasqpasq

knox service after enabling ssl is failing to start and hive beeline string used oot@invgcscdp71901 jks]# beeline -u "jdbc:hive2://invgcscdp71901.informatica.com:8443/;ssl=true;sslTrustStore=/opt/cloudera/security/jks/truststore.jks;trustStorePassword=hadoop;transportMode=http;httpPath=gateway/cdp-proxy-api/hive"SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p1000.55406660/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/

I am trying to find unsync between 2 tables using Apache-Nifi.Any suggestion will be helpful.Both Tables schema is same, and DB is postgres - 17.