Q&A

Hi Team, We are facing Nifi performance issues form a while as we increased no of processors and processor groups. We have increased thread count but still facing the issue. CPU utilization is also not reaching minimum level it is utilizing less than 50%. How to increase the performance in Nifi. We have been using 13.2 version till now and migrated to 18.0 to see if there is performance improvement but still facing the same. Could anyone suggest how to improve performance.

Hello team, I have tried to change the cluster certificates following the indications from the manual, plus internal company documents. I am stucked when starting the cluster, as when I try to start the INFR Solr instance, it returns a "no valid keystore" message. I have tried with the passwords that are configured in the config files for the service, and it works smoothly. I have done manual comparisson to working environments and everything seems to be in order. Any idea where to hit? Thanks in advance 🙂

Hello, The duplicate data from 2023-03-26 to 2023-07-10 must be removed.I'm trying to remove duplicates from the table using this command, but I'm getting an error.Commands:set hive.exec.dynamic.partition.mode=nonstrict;INSERT OVERWRITE TABLE db.table_name PARTITION(dt) select distinct * from db.table_name where dt >= '2023-03-26' AND dt >= '2023-07-10';Error: 23/07/26 16:07:46 [LocalJobRunner Map Task Executor #0]: WARN io.CombineHiveRecordReader: Multiple partitions found; not going to pass a part spec to LLAP IO: {{dt=2023-07-10}} and {{dt=2023-07-11}} 2023-07-26 16:07:47,952 Stage-1 map = 0%, reduce = 0% 23/07/26 16:07:47 [aabca681-0714-44f6-bc8d-9be6d7fca9fc main]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead.Note: The partition of this table is just the dates.Example: show partitions db.table_name;dt=2023-07-04dt=2023-07-05dt=2023-07-06dt=2023-07-07dt=2023-07-08dt=2023-

Hi, I got an error when creating external table with multiple delimiter such as "||".It says "Cannot validate serde: org.apache.hadoop.hive.serde2.MultiDelimitSerDe". I have tried add jar hive-contribute.3.1.3.jar, but still not working and also using regex.My Cloudera Version is CDH 6.1.1.How to solve this problem so i can achieve to set multiple delimiter in hive create external table?Please help. Sincerely, Gideon MaruliBank Mayapada

I am a newbie to Nifi Jolt transform.  I've been looking through the document, but still cannot figure out how to do mutiple headings for one array.input json{"dataPackage": {"datatest": {"field": [{"Name": "test1","content": 45},{"Name": "test2","content": 20},{"Name": "test3","content": 23}]}}}Jolt Spec[{"operation": "shift","spec": {"dataPackage": {"datatest": {"field": {"*": {"Name": {"test1": {"#test1": "field[&3].Name","@(2,content)": "field[&3].content"},"test2": {"#test2": "field[&3].Name","@(2,content)": "field[&3].content"},"test3": {"#test3": "field[&3].Name","@(2,content)": "field[&3].content"}}}}}}}}, {"operation": "shift","spec": {"field": {"*": {"@(0,content)": "firstDetails.@(1,Name)"}}}}]current output{"firstDetails" : {"test1" : 45,"test2" : 20,"test3" : 23}}Expected output{"firstDetails" : {"test1" : 45,"test2" : 20},"secondDetails" : {"test3" : 23}}can you please help

hello. I am trying to CopyTable 5 minutes or 1 hour of data from a 15TB HBase Table to another cluster.But then I get hbase.ipc.CallTimeoutException and the transfer keeps failing. #1688313600000 ('2023-07-03 01:00:00') - 1688313900000 ('2023-07-03 01:05:00') : 5 minuteshbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --endtime=1688313900000 'BIG_EMP'  #1688313600000 ('2023-07-03 01:00:00') - 1688317200000 ('2023-07-03 02:00:00')hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --

hello. After registering the target cluster as a peer in the replication source clusterEnabling replication on a table results in "ERROR: No meta znode available" error. >list_peers PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false hbase:032:0> enable_table_replication 'emp'ERROR: No meta znode availableFor usage try 'help "enable_table_replication"'Took 73.6962 seconds ....2023-07-20 15:25:44,586 INFO  [ReadOnlyZKClient-A-hadoo

Hi, My Nifi Registry is hosted on a server and is configured using client certificates and I have my Nifi instance hosted locally on my machine also using client certificates.I am encountering this issues when I try to access buckets created in Nifi Registry from my Nifi instance: More details of Nifi client certificates:Owner: CN=localhost, OU=NIFIIssuer: CN=localhost, OU=NIFIMore details of Nifi Registry client certificates: Owner: CN=<dns>, OU=NIFIIssuer: CN=localhost, OU=NIFI I read somewhere that the issuer of NiFi Registry cert, should be the owner of the trusted cert in NiFI's truststore and same is the case in my issue but even then, I cant access Nifi registry from my NIfi instance.Can anyone point out any technical issues or suggest where I am going wrong?Thanks

I need to expose data through APIs to 3rd party Apps. Those API endpoints are needed to developed By Nifi itself. is that possible in Nifi? does Nifi act as an API manager?

I've developed a flow that depends on importing an Oracle database dump via data pump (i.e., the impdp command). To run this command, you have to pass in the username and password as part of the parameters of the impdp command to connect to the Oracle database. In early development, I simply put the password in a non-sensitive parameter, and at the appropriate time, I'm using a ReplaceText processor to wipe out the current contents of the flowfile and write in the contents of the shell script used to run the impdp command. Of course, the text I'm writing in to the flowfile includes #{oracle_pw}. I then PutFile to write the script to disk and ExecuteStreamCommand to run the script. It works like a charm. Now that development is over and we're making plans to move this flow into production, I need a way to make this password parameter sensitive and still be able to pass it in to the script. I've given this a lot of thought and done some research, but it seems like I might need to re

command rebalance is not currently available for execution because role balancer must be in state N/A, but it is in state Busy I cant start rebalance on cdh 5.16.2. 

Hi Team,I create a dataflow try to insert data into uppercase table in postgesql.But it does not work.Appreciate any help on this issue, I am new to Nifi.  Here are my setting for QueryDatabaseTable and PutDatabaseRecord   The error message showorg.apache.nifi.processors.standard.db.TableNotFoundException: Table "ACPMA" not found, ensure the Catalog, Schema, and/or Table Names match those in the database exactly  Here is a uppercase table ACPMA in postgesql. 

Installed Nifi (v1.20) on RHEL7, java version is openjdk version 1.8.0_181. After starting nifi I can see it reports back that is it running and listening for Bootstrap requests, however, a few seconds later it shuts down. Appreciate any help on this, I new to Apache Nifi INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 33666 2023-07-20 10:43:25,365 ERROR [NiFi logging handler] org.apache.nifi.StdErr Defaulting Uptime to NOIMPL due to (java.lang.UnsupportedOperationException) Implementation not available in this environment2023-07-20 10:43:25,504 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: javax/crypto/SecretKey2023-07-20 10:43:25,504 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...2023-07-20 10:43:25,570 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi

Redhat 7 OS.I try to do a in-place upgrade Cloudera Manager from 7.4.4 to 7.7.1 following the docs: https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdp/topics/ug_cm_upgrade_after-cdp.html I do anything, and after the upgrade, I found the cloudera-scm-agent is not stable. the cloudera-scm-agent log is: [26/Jul/2023 15:11:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:12:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:13:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed with error :- rpcinfo: RPC: Port mapper failure - Timed outprogram 100003 is not available[26/Jul/2023 15:14:31 +0000] 1956 MainThread filesystem_map ERROR NFS server binding info failed

I have set up Quickstart VM on my local machine for learning purposes. Was working on Hive and Sqoop when I imported a File into HDFS. The activity was successful. For some reason, I am unable to go inside the folder using Hue. Nothing happens when I click on the folder. Have restarted the Browser. It didn't make any difference. Can you please let me know how to handle this? 

Hi,I am a newbiee to nifi and its processors.  Want to understand what would be the jolt specification for the following input/output? Or can anyone suggest any other processor . Input JSON:{"cells": {"deviceindicators:1234_32456_789023":"0", "deviceindicators:5678_89213_875943":"110"}} Output JSON:{"cells": {"1234_32456_789023":"0", "5678_89213_875943":"110"}} Want to remove "deviceindicators:" from the key using JoltJSONtransformation.. Please advice

Hello, I have a simple process getting data from sql server and send them by email. Short emails, nothing heavy and not that many during the day. But few days ago, it suddenly started to be frozen and it happened again today. It does not show any error and I do not see any error in logs either. I checked this thread, maybe it's the same, but there is no real solution and in my logs I can't see anything with SMTP or that it's "waiting", so would like to ask you for help or to point me to the right direction.Maybe I'm not checking everything...  ThanksRgdsLudvik 

I need to load the data from Azure Cosmos DB. I am using Nifi 1.21.0 Version. what is the way to do that?

Hi.I'm using NiFi 1.11.2. I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content).  Decoding the field works as expected and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field with the decoded value in stead of writing the decoded value to file. And after replacing the value write the updated Json-file to a new S3 bucket.I'm thinking taking the output from RouteOnAttribute and ConvertRecord in the figure below and perhaps use UpdateRecord or ReplaceText. But I'm not sure how to do this.What is the best way to "re-connect" the data flow here from these to objects in the process?Any tips? Thanksbjornmy     

When creating a container using hue, I mount volume with the hive configuration. The configuration is taken from the cluster host. In the hive-site.xml has the hive.server 2.authentication parameter, which has the ldap value. With this value, hue does not connect to hive. If I replace it with a KERBEROS value, then the value will change on the host (the host is part of a hadoop cluster). If you copy the hive configurations to the container, they will not be updated when updating in the cluster. When trying to define a variable for the container HIVE_OPTS='--hiveconf hive.server2.authentication=KERBEROS' nothing happens. Is it possible to somehow redefine the variable without changing the original configurations

Hi, I have a nifi flow which has 1000's of flow files. Each of the flow file has an impala/hive statement and each of them have different execution times. How do i use control rate processor to trigger next flow file only if the previous flow file has been processed (success/failure of the flow file doesn't matter)?

Hi,I am unable to create a support case for this, as my support license does not appear to cover this ML Experiments.I am evaluating Cloudera CML on my company's CDP instance. I created several Experiments on a project to test out the feature (it was a simple script generating some random numbers for metrics) and the experiment created and ran ok. However, I tried to create the same experiment later and the UI just timed out. I used the developer's tools in the browser to see the request and "runs" failed with 504 timeout error. I am unable to troubleshoot the issue further in the UI, and am uncertain if ECS would have any useful information. I could get it to run successfully a few hours later. To my knowledge, nobody touched the CDP cluster during in between this period. During the period I cannot create experiments, I looked at my kubernetes namespace <CML workspace name - userXX> inside ECS, and did not see a pod starting. There was also no events being created. We would like

Hi all, I have a Hadoop cluster built on Cloudera. Now I am trying to increase concurrent with impala and I have no ideal for change the config files. Do you have experiences to how to resolve this, please help me. Thank you all,

Hello @All, I am totally new in apache NiFi i am going to integrate Polarion ALM to NiFi using Rest Apis. But i am not able to understand where i need to give API link and where i need to give logIn credential i am using invokeHTTP process so anyone having any  idea on this please help me. Thanks and Regards,Digambar Dhere 

Hi Team , As we have more then 14 million small file as per Cloudera Navigator as below : 14.8M Small files created in the last 30 days  14.8M / 21.4M 69.3% small files We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem. Thank You in Advance