Q&A

Hello! So I can't access NiFi UI because the server if probably down exactly because of amount of files that have been stuck at queue. So the question is how can I find out the size of the queue for the moment using may be logs or maybe there are some other ways to do so? Thank you beforehand!

Hello, On our use case, we're receiving data in JSON format and it must be converted into Avro using a schema that has X and Y includes mandatory fields and everything else (Z and W) should go into a Custom map. Here is our Avro schema:"fields": [{"name": "X", "type": "long"},{"name": "Y", "type": "string"},{"name": "Custom", "type": { "type": "map", "values": "string" } }] And an example of data coming in:{"X":123,"Y":"ABC","Z":"zzz","W":"www"} We have implemented a JoltTransform to do the needed transformation to get Z and W into a Custom array.Now the issue is on doing the conversion from JSON into Avro since we get an error stating it cannot convert an java.lang.Object into a Map. How can we properly process this JSON and put that array into a Map so it can be properly stored using that Avro schema? Thank you.Best regards,Miguel

HI , I have a requirement where I need to do case-insensitive joins across the system and I don't wish to apply upper/lower functions. I tried setting TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci') at Table level but still the comparison is happening considering case sensitivity.  PSB - drop table test.caseI;create table test.caseI(name string, id int)TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci');insert into test.caseI values ('hj',1); drop table test.caseI_2;create table test.caseI_2(name string, id int)TBLPROPERTIES('serialization.encoding'='utf8mb4_unicode_ci');insert into test.caseI_2 values ('HJ',1); select * from test.caseI iinner join test.caseI_2 i2 on i.name=i2.name;--No ResultTried with encoding 'SQL_Latin1_General_CP1_CI_AI' but got same result as above.Any help would be appreciated, thanks!

Hi,I want to send specific flowfiles to a specific remote MiNiFi Java.Testing with 1 MiNiFi works great, out from NiFi, into MiNiFi for processing.I'm prepping the MiNiFi flow on NiFi, templating, exporting, converting to yaml, modifying any security settings i need and restarting MiNiFi to accept the updated config.yml. Now i'm adding additional MiNiFis.On NiFi I've setup a PG with a RouteOnAttribute (Attribute is the specific MiNiFi name) to send to a specific S2S Output Port for that MiNiFi. Up to here, all seems fine. The problem i have is in the MiNiFi template.I have the RPG going into a processor, and that's when i specify the From Output Port which is specific to the MiNiFi.How can i template/generalize/variabilize/other the From Output so that i don't need to make separate template for each and every MiNiFi instance?I could theoretically move the RouteOnAttribute from the NiFi to the MiNiFi side but at that point it means i already sent a ton of data to all the MiNiF

Hello everyonei have a content in Json (an array of id)[{"id":7532233},{"id":755163}] and i need to convert this whole array into one attribute so that i can after that refer to it in an ExecuteSql processor with statement :update column 1  where id exists(${attributeName})

I am using the utility of wait and notify and for that I have used Distributed map cache client service and Distributed map cache server, but somehow I am getting an error in service "too many open files", I have even increased the "nofile" in node. In the same instance but in another nifi flow I am able to use this functionality but if I do configure this service in another parallel flow I face issues. Below I am attaching the screenshot of conf and error. any help would be appreciated.

I am using the latest version of impala JDBC Driver 2.6.27.1032, when my sql is  "upsert into my.testtable(id,name,insert_time) values(?,?,?)" and using preparestatement, then the column "insert_time" will be replaced to "upsert_time", so my sql be changed to "upsert into my.testtable(id,name,upsert_time) values(?,?,?)"  and i get a error "errorMessage:AnalysisException: Unknown column 'UPSERTby' in column permutation",。How can I solve this problem？please!

cdp7.1.6SQL_STREAM_BUILDER-1.12-csa1.4.1.0-cdh7.1.6.0-297-15479260   

Hi I have created a new entity and I would like to show all the attributes by default while doing the 'Create new entity' process. At present, the UI mask shows only the Required attributes. How can I do this?

Hi,I want to insert data to Ms. Sql's server from nifi flow.Where can I find a sample or a guide?Also, where have I must install sqljdbc-4.0.jar?

We recently added a two nodes to our cluster through ambari wizard, we installed datanode, nodemanager, Metrics Monitor, LogFeederThe datanode/nodemanager are starting correctly by not livetopology_mappings.data was updated in both mnode and cnodes cat /etc/hadoop/conf/topology_mappings.data[network_topology]cnode2.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.172=/default-rackcnode5.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.169=/default-rackcnode4.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.175=/default-rackcnode3.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.67=/default-rackcnode1.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.188=/default-rackcnode6.2b87d4bc-6cf3-4350-aaf7-eff7227d1aef.datalake.com=/default-rack10.1.2.9=/default-rack  datanodes have 2 external disks to store hdfs data  [root@node6 ~]# df -hFilesystem Size Used Av

Hi Team,From last few days, data03 node was not working, so we moved it to decomissioned state and restarted all our services. Now i need to recomission and start the services here as well.I selected the host and under actions clicked on End maintenace, but it is failing.Please help me out with what shall i do to resolve this?

Hi everyone, I am having some troubles with Nifi Cluster configuraition (version 1.16.2). Actually, the nodes are up and I can access them via https. But then: When i checked the logs, the error was:WARN [main] o.a.nifi.controller.StandardFlowService There is currently no Cluster Coordinator. This often happens upon restart of NiFi when running an embedded ZooKeeper. Will register this node to become the active Cluster Coordinator and will attempt to connect to cluster again INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=false] Attempted to register Leader Election for role 'Cluster Coordinator' but this role is already registered WARN [Heartbeat Monitor Thread-1] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine leader for role 'Cluster Coordinator'; returning null I tried a lot of possible solutions, but i still got the problem My nifi.properties configuration:nifi.cluster.is.node=truenifi.cluster.nod

Can someone point me how to get it configured?  What's the reason HDP/Cloudera dropped the embedded view from Ambari to begin with?  It was a useful tool to see the details of a hive/tez job, good for troubleshooting.

We are planning a production cluster and right now I already have the Mysql being replicated but only have one Cloudera Manager . However we need to patch etc and sometimes have to take downtime of the CM . Is there a way to add a 2nd cloudera manager to the same database ? 

Hi Team,Can anyone please help me with the below file:           cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpmi am unable to find it as the below site is giving 404 error:https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/Regards,Jessica Jindal

Hi,after having deleted tera bytes of data from HDFS (1/4 of the total capacity), the block count among data nodes did not decrease as expected. It is still over the critical threshold.How could it be solved?Thank you 

22/05/25 01:43:01 INFO utils.HiveUtils: load data into hive tableHive Session ID = fc864c41-2236-472d-ac57-96c6df6d8e4822/05/25 01:43:01 INFO SessionState: Hive Session ID = fc864c41-2236-472d-ac57-96c6df6d8e4822/05/25 01:43:01 INFO session.SessionState: Created HDFS directory: /tmp/hive/hive/fc864c41-2236-472d-ac57-96c6df6d8e4822/05/25 01:43:01 INFO session.SessionState: Created local directory: /tmp/root/fc864c41-2236-472d-ac57-96c6df6d8e4822/05/25 01:43:01 INFO session.SessionState: Created HDFS directory: /tmp/hive/hive/fc864c41-2236-472d-ac57-96c6df6d8e48/_tmp_space.db22/05/25 01:43:01 INFO tez.TezSessionState: User of session id fc864c41-2236-472d-ac57-96c6df6d8e48 is hive22/05/25 01:43:01 INFO tez.TezSessionState: Created new resources: null22/05/25 01:43:01 INFO tez.DagUtils: Jar dir is null / directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/hive/.hiveJars22/05/25 01:43:01 INFO tez.TezSessionState: Computed sha: c765910321290465015ccaaeae5dde480fd29e141a4c11594756ecfe

I was able to replicate my problem using the following configuration:FlowThe MergeContent configuration is as followsMergeContent PropertiesMergeContent SchedulingAlthough the conditions of the MergeContent processor are met, it does not concatenate the incoming flowfiles.The last version in which the above flow worked was version 1.15.3.Any suggestions on how to solve this? Is this a bug?P.S.: The flow is so designed due to a dependency on only writing one file per day.

Hi all,i'm searching for a safe procedure to add a new journal node to a Cloudera CDP cluster. Now we have only two master node and so we would like to add the third masternode and deploy the third journal node on it. Is it enough to deploy the role using the cloudera manager add role function? Or is it necessary to take further action? Thank you.

Hello,due to a problem with a script, we have almost saturated the HDFS available space.I suppose this is caused by temporary Hive files that have not been cleaned up due to the abnormal termination of the script.I would like to check the /tmp/hive folder on HDFS but my user, that has administrative privileges, cannot access the /tmp/hive folder.Is there a way to check and clean such folder? Any help would be appreciated. KRs,Andrea

hello, please help with below questions: 1. I have nifi cluster with 2 nodes and running ExecuteGroovyScript processor to create text formatted file generated from flow data content and save it on local filesystem but it is saving file only on one nifi node, is there any way to replicate this file on all nodes at a time? 2. Above file formatting by ExecuteGroovyScript processor as text in Prometheus format and using such flow http_request-> fetchfile-> http_response processors to create custom Prometheus healthcheckmetric web page, it worked for single node but with 2 nodes all data balanced randomly between 2 nodes so I have different content in 2 web pages ( http://node1/healthcheck/ and http://node2/healthcheck/ ) so monitoring not working properly, could you help how to add custom Prometheus metric so it will be available like regular nifi prometheus web page /metrics/ with same content on both nodes? may be other solutions?&n

Hi All, We are facing an issue while adding new host to Cloudera manger. During host installation, Cloudera manager is replacing the /etc/yum.repos.d/cloudera-manager.repo file.  The original repo file on server has the required Cloudera credentials to downlaod packages from Cloudera archive. -------- $ cat cloudera-manager.repo [cloudera-manager] name = Cloudera Manager, Version 5.13.1 #baseurl = https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.13.1/ #gpgkey = https://archive.cloudera.com/redhat/cdh/RPM-GPG-KEY-cloudera baseurl = https://7bf0xxxx-xxxx-xxxxxxb236:15xxxxxxxxxc9@archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.13.1/ gpgkey = https://7bf0xxxx-xxxx-xxxxxxb236:15xxxxxxxxxc9@archive.cloudera.com/redhat/cdh/RPM-GPG-KEY-cloudera gpgcheck = 1 -------- Now when we try to add this host in Cloudera manager, during installation phase the file Cloudera-manager.repo is replaced.  Due to this we are observing authentication failures.

Ranger is not updating the access audits in ranger after a certa date solr logs are giving the below error:ERROR (qtp246273275-120) [c:ranger_audits s:shard1 r:core_node2 x:ranger_audits_shard1_replica_n1] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Exception writing document id 8cb43886-8056-46b2-8b24-0ead48c8280b-1 to the index; possible analysis error: number of documents in the index cannot exceed 2147483519 how can this be troubleshooted?

Hi Community, We are working on a new requirement where I need to disable table-level TTL and rather use cell-level TTL for one of our HBase tables. I can disable the table-level TTL by altering the table property and set cell-level TTL for new incoming data. However, I can't find anything on how can I set cell-level TTL for the data / rows that are already existing in the table. Any suggestions will be very helpful. Thank you,Snehasish