Q&A

In my flow iam publishing data to kafka topic , if there is failure , that request should go to failure relation where iam publishing that data to error topic , but in my case when there is not connectivity with kafka or kafka broker value is not correct request is getting queue in PublishKafka processor not moving to failure relation .This queue will effect my other requests any even my error handling logic .Please suggest some solutionAttaching the image of error and my nifi flow- 

I was trying to find some documents about where would the location for HS2 & HMS logs be located for CDP-PC on AWS, but could get none the nearest I could get is https://docs.cloudera.com/data-warehouse/cloud/troubleshooting/topics/impala-logs-s3.html. but this is not what I want. Appreciate any help in this.

Hello Team,I am running a process in which i am trying to run beeline command.While running beeline command with sudo it is working but if i am trying to run beeline in a process which if forked from parent process with the hive user it is failing with error 4 Can you please help me to debug what's going wrong or anything that i need to set.If i do su hiveuser and run the same command t works but while running via process it is failing.Command that i see while running via fork process :bash /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/bin/../lib/hive/bin/beeline -u 'jdbc:hive2://machine1.domain.com:2181/default;password=***;principal=hive/_HOST@DOMAIN.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=*****;user=HiveUser;zooKeeperNamespace=hiveserver2' -n loki -n /tmp/cmdcvhive.4015998.140520474391872 --showHeader=false --outputformat=tsv2 -e 'use underscoretest; show tables;'

Hi,We are currently working on scenario to connect to secured Rabbitmq server for consuming messages. We have used StandardSSLContextService for that purpose and configured both truststore and keystore properties.  These certs are created specific to Rabbitmq connection.Everything is working fine as expected but the only issue we are facing is that when both keystore and truststore are expired and we need to update the certs currently the only way we see is to restart the server or controller service.But we would like to know if there is any other way the NIFI StandardSSLContextService can check if the cert is expired and if we provide a new keystore and truststore then it will reload the latest certs without the need for restarts.Any suggestions on this issue is greatly appreciated.

Hi,I have restarted the cluster and services like hive, yarn, Oozie etc. are not getting started because of following error, we are using CDP 7.1.7Role failed to start due to error java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs.Can anyone please support here as picture is not cleat what to check here.2023-11-28 08:14:06,415 ERROR Staleness-Detector-1:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Failed to check staleness for service client configs: DbService{id=1546335860, name=hive}com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configsat com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:151)at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfig(AbstractClientConfigHandler.java:163)at com.cloudera.cmf.service.AbstractClientConfigHandler.createClientConfig(AbstractClientConfigHandl

Hi ,iam retrying a flow file 3 times before moving moving it to failure relationship  , with each retry i want to have 10 seconds gap in between them , so 2nd retry will happen after 10 seconds .iam not able to understand how to use wait processor for this . What should be the configurations of that processor .iam using older version of nifi so i can not use retryFlowFile processor . 

Hello, We are using AD for Kerberos. When we try to Generate missing credentials, we are getting error "ERROR GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: unable to create credential for role 785 due to:/opt/cloudera/cm/bin/gen_credentials_ad.sh failed with exit code 19 and output" "SASL/GSSAPI authentication startedSASL username: user@domainSASL SSF: 0ldap_add: Constraint violation (19)additional info: 000021C7: AtrErr: DSID-03200E7F, #1:0: 000021C7: DSID-03200E7F, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 90303 (servicePrincipalName)" We got to know that there was a bug in Microsoft AD patch of November 2021 and Cloudera have a KB article with some suggestions to fix this. We have tried all of those options like disabling SPN on AD, Import Credentials with AD administration user account, removing from domain and generating missing credentials...but we get the same error every time. It tried to create HTTP principal everytime and fai

 /opt/cloudera/cm/bin/gen_credentials_ad.sh failed with exit code 19 and output of <<+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/lib/mit/sbin:/usr/sbin:/usr/lib/mit/bin:/usr/bin+ KEYTAB_OUT=/var/run/cloudera-scm-server/cmf2077611128110226334.keytab+ PRINC=HTTP/vsrisdxxxxx.mastnd.cloudera.XXXXX.local@XXXXXXXX.LOCAL+ USER=syzutmdaGr+ PASSWD=REDACTED+ DELETE_ON_REGENERATE=false+ SET_ENCRYPTION_TYPES=false+ ENC_TYPES_MASK=24+ USERACCOUNTCONTROL=66048+ ACCOUNTEXPIRES=0+ OBJECTCLASSES='objectClass: topobjectClass: personobjectClass: organizationalPersonobjectClass: user'+ DIST_NAME='CN=syzutmdaGr,OU=Cloudera-Accounts,OU=Accounts,OU=Teir 2,OU=Admin Teiring,DC=OneBank,DC=Local'+ '[' -z /var/run/cloudera-scm-server/krb56173003423111410977.conf ']'+ echo 'Using custom config path

This is a new setup for CDP 7.1.6. Issue:ResourceManager failed to start. NodeManger and Jobhistory server are running. 8:58:49.871 AM FATAL ResourceManagerError starting ResourceManagerorg.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /confstore/CONF_STOREat org.apache.zookeeper.KeeperException.create(KeeperException.java:120)at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793)at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274)at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268)at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:67)at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:81)at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265)at org.apache.curator.framewo

Hi ,is there any idle connection timeout for ConsumeKafka and PublishKafka processors in NIFI.Do these processors go to idle state if no data in incoming ??if yes then what is the time in which they will go to idle state in case of no data is coming ??? and is there any way to set this time ourself ???

Hi Guys, I would like to know how to set up authentication for spark history server, so that unauthorized users cannot view the spark history server web ui, any help would be appreciated, thanks! 1. I do notice there are below statements in the official spark document:“Enabling authentication for the Web UIs is done using javax servlet filters. You will need a filter that implements the authentication method you want to deploy. Spark does not provide any built-in authentication filters.” 2. There is also a thread on stackoverflow regardig this : “You re-use Hadoop's jetty authentication filter for Kerberos/SPNEGO: spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter and spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params=type=kerberos,kerberos.principal=${spnego_principal_name},kerberos.keytab=${spnego_keytab_path}”. with kerberos authentication enabled in CDH6.3, I followed the instruct

Hello everyone,I'm seeking assistance from those experienced in NiFi for a specific challenge. I use NiFi for internal tools and simple integrations within my company, organizing different functionalities into separate ProcessGroups. These functionalities include converting data types (e.g., from JSON to CSV), generating dummy data, and sending data to a partner through an API.To make all ProcessGroups accessible through an endpoint, I've implemented a setup with a "HandleHttpRequest" processor listening on a specific port in the main canvas. Subsequently, a "RouteOnAttribute" processor is used to route requests to the appropriate ProcessGroup based on the URL path.However, this approach has its drawbacks. The canvas becomes cluttered, requiring encapsulation of flows within more general ProcessGroups. Additionally, every time I add a new flow, I have to stop the "RouteOnAttribute," update it, link the new relationship, and then restart it (I know I can use NiFi API to accelerate this

I am attempting to use the NiFi StandardOauth2AccessTokenProvider 1.23.2 with the ConsumeIMAP processor in order to retrieve emails from Microsoft Outlook.When I configure using a grant type of either Client Credentials or Refresh Token, the "Can acquire token" validation always fails with a 400 response code.Upon examining the logs, the error reads: ERROR [Verify Controller Service Config Thread-1] o.a.n.o.StandardOauth2AccessTokenProvider StandardOauth2AccessTokenProvider[id=ba2358f5-018b-1000-e706-d1406dd00474] OAuth2 access token request failed [HTTP 400], response:{"error":"invalid_request","error_description":"AADSTS900144: The request body must contain the following parameter: 'grant_type'. Trace ID: 42b51bc0-3865-441e-abb2-200be69b6a00 Correlation ID: 1d0d67de-c1c3-4f0c-9f82-528b027ba18b Timestamp: 2023-11-13 14:41:12Z","error_codes":[900144],"timestamp":"2023-11-13 14:41:12Z","trace_id":"42b51bc0-3865-441e-abb2-200be69b6a00","correlation_id":"1d0d67de-c1c3-4f0c-9f82-528b0

Hi guys,I have some problems with transform date from string to long in NIFI. I have date like this: "02-01-1968|" and I need to transform this to long. i am using UpdateAttribte Processor to format it to number like this: ${my_date:toDate("dd-MM-yyyy","GMT"):toNumber()})} but I have result like this: "my_date" : -63072000000 .Does anyone know how to solve this better? 

Hi guys! i have some problems with JoltTransformation in NIFI. I have plain json like that:{"id": "c47f79f5-77b1-45d1-91df-968c2fa3def8","system" : "OP","sourceSystem" : "DKG","name" : "Agnes","lastName" : "Smith","mail" : "a.smith@gmail.com","country" : "Australia","city" : "Sydney","street" : "Colorado","building" : "23","local" : "1',"paycheck" : "3456,98',"joiningDate" : "12-09-2023","isOfAge": "true",}And i need to have output like this:{"paycheck":3456.98,"joiningDate":1702048486451,"person":{"name":"Agnes","lastName":"Smith","mail":"a.smith@gmail.com","isOfAge":true,"adress":{"country":"Australia","city":"Sydney","street":"Colorado","building":"23","local":"1"}},"system_data":{"id":"c47f79f5-77b1-45d1-91df-968c2fa3def8","system":"OP","sourceSystem":"DKG"}}In addition to nesting, I need to convert date to long, paycheck to double and isOfAge to boolean.I tried use JoltTransformation and Jolt Specification like this:[{"operation":"shift","spec":{"paycheck":"paycheck","joiningDate"

Hi,we would like to activate HA for Cloudera CDP MySQL databases (hive, ranger, etc..).We have seen that there are several ways to achieve all this, mysql replication active-slave or active-active, mysql + DRDB, mysql Innodb Cluster, etc..Our goal is to achieve automatic failover of mysql, with functionality similar to those of a cluster with an active node and a standy node.From your experience, is there a recommended architecture for creating HA for a Mysql server compatible with Cloudera services? I read that some solutions could not be compatibile with Hive metastore for example.Thanks in advance.Ivo

Hi, Guys. I am new to use NiFi flow. I am kind of confused about Query database table processor. So now, I am try to get all data from Postgres database and convert them into json and then convert them into xml.But sometimes, the Query database table processor will have a lot of Flow file and then the queue just mess up. I try to set the art_id to be maximum-value column but it doesn't work well because art_id is string.(not sure)This is a screenshot from a sample table called title and it has about 500 rows. How to query them just one time and split them into 500 flowfile and each of flowfile has only one row from table title without any duplicates. Can someone help me?Thank you so much. Following screenshots are my progress so far.     

Hii Everyone, I've been struggling to achieve the desired result with a Jolt transformation for the past few days and could really use some help from the community. input.json: { "email": "abcmail@gmail.com", "dates": [ { "date": "2023-12-01", "type": "full" }, { "date": "2023-11-03", "type": "full" }, { "date": "2023-11-02", "type": "full" }, { "date": "2023-10-01", "type": "half" } ]} output.json [ { "emailId": "abcmail@gmail.com", "date": "2023-12-01", "type": "full" }, { "emailId": "abcmail@gmail.com", "date": "2023-11-03", "type": "full" }, { "emailId": "abcmail@gmail.com", "date": "2023-11-02", "type": "full" }, { "emailId": "abcmail@gmail.com", "date": "2023-10-01", "type": "half" }] spec file for the transformation: [ { "operation": "shift", "spec": { "*": { "email": "[].emailId", "dates": { "*": {

Hello guys,I'm struggling with JsonRecordSetWriter 1.17.0, we have valid nested JSON data, but for some reason whenever this data passes through JsonRecordSetWriter, on nested arrays it adds the following :   "orderItemsCustomizes" : [ "MapRecord[{quantity=1, itemCurrency=AED, totalPrice=0, orderItemId=2767, itemCustomizeGroup=MapRecord[{itemModifierGroupAttributes=[Ljava.lang.Object;@4658e00e, customizeGroupName=Burger Selection}], itemCustomize=MapRecord[{itemCustomizeGroupId=2308, itemId=18622, itemName=Chicken Burger, isAvailabel=true, updated_at=2023-01-27 15:59:31, itemCurrency=AED, itemCost=0.00, created_at=2023-01-27 15:59:31, itemModifierAttributes=[Ljava.lang.Object;@ee87ab1, id=7539, itemImage=https://d2cvcbugmdflrn.cloudfront.net/e777bb72-a445-4162-9844-f9bda5e7a3d0.jpg}], itemId=18622, itemCustomizeGroupId=2308, createdAt=2023-01-27 16:48:02, itemCustomizeId=7539, price=0.00, ID=1153, updatedAt=2023-01-27 16:48:02}]", "MapRecord[{quantity=1, itemCurrency=AED

Hi ,In Apache NIFI  I am  consuming data from one Kafka topic and publishing that data to customers Kafka topic . For this i am using ConsumeKafka and PublishKafka processor respectively.If outgoing dataflow is idle for more than 5 minutes, the Kafka connection with customer should be terminated and  reconnected when there is  any new  messages consumed .how to do this with publish kafka processor ?? how to terminate and start the connection??

I am using ExecuteSQL to pull a colb column from DB. ExecuteSQL provides the result in the avro format. Then I am converting that to Json using AvroToJsonProcessor but it is adding escape characters. I want the value that is being pulled from DB as a Json but I am getting it as a string with escape characters. [    {        "SEARCH_RESULT": "{ \"requestId\": \"203680\", \"responseData\": [ { \"searchKey\": \"cardNumber\", \"data\": [ { \"firstName\": \"Martin\", \"lastName\": \"Garry\" }, { \"firstName\": \"Martin\", \"lastName\": \"Garry\"}, { \"firstName\": \"Martin\", \"lastName\": \"Garry\",  } ] } ] }"    }]I want that value of Search result to be a Json instead of a String so that I can use JOLT to transform it as needed. Can you someone pls suggest how to do it.

Himy backup is failing with following errors :23/11/30 10:08:58 ERROR backup.KuduBackup$: Failed to back up table impala::PPR_CURATED.ACTIVITEorg.apache.kudu.client.NonRecoverableException: Unauthorized actionat org.apache.kudu.client.KuduException.transformException(KuduException.java:110)at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:470)at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:288)at org.apache.kudu.backup.KuduBackup$.org$apache$kudu$backup$KuduBackup$$doBackup(KuduBackup.scala:50)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply$mcV$sp(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144)at scala.util.Try$.apply(Try.scala:192)at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:143)at sca

HiWe've done a recent upgrade from Hive 1.1.0-cdh5.16.2 to Hive 3.1.3 and have noticed the difference in behaviour when no rows have been returned and file generation.On the older version (Hive 1.1.0-cdh5.16.2) it use to create empty part files (000000_0) whereas since the upgrade this isn't the case.The command to run this is beeline and the sql statement starts like:Insert overwrite directory 'some/path'ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'I cannot find any definitive documentation to saw this behaviour does not exist in later versions like Hive 3.1.3Thanks

Hello,I'm running a sqoop job through Hue. This imports data from Oracle, but the value of --query is being divided by whitespace and produces sqoop flag error because it does not recognize these parametersimport --connect jdbc:oracle:thin:@server/service --query 'select * from mytable'The logs show somethng like this:Sqoop command arguments : import --connect jdbc:oracle:thin:@server/service --query 'SELECT * from mytable'-------------------------------------------------------------------------------------------------------17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing arguments for import:17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: select17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: *17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: from&nb

hi everyone ! hoping everybody has an awsome day.I am trying to create a process Group with a flow definition json i downloaded.when i upload a file.json manually and checked the nifi-request.log and the only i see is this pettiotion " http://localhost:8081/process-groups/{id}/process-groups/upload" but i don't see the body in sent request, i want to use the api rest to upload this file but documentation is not clear in this aspect pettition #1:curl --location "http://localhost:8082/nifi-api/process-groups/{id}/process-groups/upload" \--form "file=@/path/to/my/flowDefinition.json"Response: Process Group name is required pettition #2:curl --location "http://localhost:8082/nifi-api/process-groups/{id}/process-groups/upload" \--header 'Content-Type: multipart/form-data' \-F 'id=root' \-F 'name=QA_FLOW' \-F 'position[x]=577.0' \-F 'position[y]=11.0' \-F 'clientId=27bf6df2-018c-1000-362e-61166913eee4' \-F 'disconnectNode=true'-F "file=@/path/to/my/flowDefinition.jso