Q&A

Hi, I wanted to install Apache Nifi 1.16.3 or 1.17.0 on my Ubuntu system. However, I am not able to find the required tar files from the Apache Downloads page.   Can someone please drop links where I can see the nifi-1.17.0-bin.tar.gz/nifi-1.16.3-bin.tar.gz   to install in ubuntu 18.04?

Hello Experts, We are using HDP 3.1.5 and since short time we have one specific table with several partitions created in it, where the compaction for one of these partitions fails repeatly. I'm not able to find the exact error why it is failing. from yarn I see the following error in the application log:org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1657267798389_0085_m_000000Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 and in Metastore Log I only find this:ERROR [thread-71]: compactor.Worker (Worker.java:run(198)) - Caught exception while trying to compactid:697908,dbname:standard,tableName:<table-name>,partName:<partitionname>,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestWriteId:0. Marking failed to avoid repeated failures, java.io.IOException: Major compactor job failed for thread-71-compactor-<dbname>.<tablename>.<partit

Hello, I would like to create Hive tables  using Zeppeling and I found the the below document that use HWC:https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/integrating-hive-and-bi/topics/hive-hwc-reading.htmlI'm not sure what means "livy configuration file"? Is "%livy2" a new Interpreter? May I create a new Interpreter with only indicated configurations? Thanks!  

Hello, please kindly help. i am trying to convert a datetime(attribute) from one format to the other but it keeps giving me empty value on the attribute:SCENERIO:Flowfile is json----->extract date from Json flowfile using Evaluatejsonpath---->format date using UpdateattributeActual Date: "2022-08-01 00:00:00.0"Desired date: "01/08/2022 00:00:00.000000"I have tried to use this on the updateattribute, but it is not working:${DATE_TIME:toNumber():toDate('yyyy-MM-dd hh:mm:ss.S'):format('dd/MM/yyyy HH:mm:ss.SSSSSS')} Thank you.

Is there any way to update the flow description in the NiFi Registry after the initial creation of the flow that does not require restarting version control?  I checked the REST API endpoints as well and didn't see anything. I would like to be able to clean up descriptions (or even just correct a typo) without going through a cumbersome export versions, import flow, import versions, delete and re-import PGs process. Thanks.

Hello Lovely community, I am new to NiFi, I am able to connect to mysql dummy database from Nifi and everything works fine.But I want to understand how can we connect to databases that use SSH tunneling..Because I have not seen any properties related to SSH in the database processors.My processors being used is "capturechangeMysql" This source DB server is secured and I need SSH tunnel to make the connection.I have 100 such sources. Should I add my nifi server ssh keys in all the source servers ? and even then where can i give the SSH tunnel information? and the private key..Because while using any SQL client we can use the tunnel along with main DB username and password as shown in attached screen, but for Nifi I am not able to see a way..Please help me with your suggestions.. /\SSH tunnel

good morning cloudera community, we are using apache ambari in version 2.6.2.2 and in this version we are using HDP 2.6.5, with that, we need to configure the parameter below in HDFS, but unfortunately we are not finding this option in the HDFS configuration, so we need to know how configure this parameter in HDFS? parameter: dfs.client.block.write.replace-datanode-on-failure.policy=ALWAYS PS: would it be in the "custom hdfs-site" option by clicking on "add property" and adding the parameter in the "properties" box? 

Hi Team, I am new to Apache Nifi, can you please help me to implement CDC for Postgresql one table to log table in Nifi. I am using Nifi - 1.16.x version.

if I have 5 zookeeper nodes and from that 5 zookeeper nodes one 1 zookeeper nodes goes down which was leader node then how zookeeper will elect the new leader from the remaining 4 zookeeper node?

Hi,  I am new to nifi and stuck with two issues for which i need help:- The task is to migrate a set of tables from source to target database. The data will have to be filtered as well based on a field before migrating. The pipeline has to migrate historical data as well as perform the incremental fetch.  1. For incremental fetch, I used the GenerateTableFetch processor and set the max-value column to a date field in the table. I also used the partitioning feature to get the data in chunks by setting the partitioning size and column. It all works well. But considering it is a stateful processor, i have received feedback to go stateless as we can loose the state in case of node crash. How can i achieve the incremental fetch and partitioning feature in a stateless manner? 2. The set of tables to be migrated have a parent child relationship. The incremental fetch is required for all the tables. However child tables don't have any such column that can be used

Hi Team, We have CDH - 5.16.2 in our environment, recently renewed Cloudera -license.But still since today we are not able to create a case on support portal.Please find the attached snapshot. 

Hello All, I am using the mysql CDC processor in Nifi, to get incremental changes into DWH, in this processor fo the username field, which permissions should be granted to the user so that it can access cdc files ?should be the one with highest privileges/ root user ?because it is not mentioned anywhere in the help section (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.5.0/org.apach... ) And when i provide a user credentials who has GRANT ALL PRIVILEGES ON *.*  then it is able to access CDC bin logsBut getting permission denied when trying to access with a user who has only read permissions on *.* Since the processor has nothing to do with write, i thought only read permissions were enough, but looks like its not the case..  Because I need to create new users in the source to access these CDC files.. and I dont want to give them Grant ALL, as it seems riskier.. Please help with your suggestions and th

Hi, We have got this issue post java upgradation. Kindly let us know whether this error occurred because of java upgrade or else please suggest a solution.  spark-shellSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).22/08/03 15:34:49 ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)at org.apache.spark.sql.SparkSession$

当前Hive版本：3.1.3000.7.1.7.0-551 创建hive外部表： CREATE TABLE IF NOT EXISTS `alter_test` (a string,b string) STORED AS ORC TBLPROPERTIES("transactional"="false");转hive内部表：ALTER TABLE alter_test SET TBLPROPERTIES('external'='false');重命名：ALTER TABLE alter_test RENAME TO alter_test_bak;注：external.table.purge 这个配置是一直为true的此时，表alter_test_bak的location目录名称依然是alter_test。同样的操作步骤，我在Hive版本 3.1.3000.7.1.1.0-565是可以同步更改location目录名的，请问这是怎么回事呢

I have 3 nodes NiFi and 1 node Registry instances running. Registry - SSL enabled and AD-integrated  NIfi  - SSL enabled and AD-integrated    how can I integrate NiFI with Registry, so I can start using Registry.   @MattWho please advice for any recommendations     

Hi,I am using mirrormaker 2 to copy data of one topic from source cluster to destination cluster. data is getting copied but I see 3 hours of lag. and I am not able to find the exact cause of it (whether the delay is at consumer side or producer ) as it is using kafka connect framework.Below are the config detailstest topic having 2 partition and replication factor as 3number of messages coming into the source topic is 8K per second.

Hi ,I have two kafka cluster where I need to copy only the topic configuration from one cluster to other without copying any data. How can we do that. I know using replicator/mirrormaker it can be done but I am not interested in data here. 

Hello All, I am novice to Kafka. I am receiving SNMP traps in a particular server 'X', but my Kafka is installed in a different server 'Y'. How can I publish the traps using the kafka-console-producer.sh script to load to topic ?. Any help is appreciated

Hello,is it possible to alter compression.type on a kafka topic while having applications on the topic running? What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip. In CM, I see a global setting. Does it apply to KafkaMirrorMaker only? Are producer and consumer applications somehow affected, when the global parameter changes.Best regardsJaro

Hi All, I have come across a issue where my disk is filling up because of topic __consumer_offsets as i see the size as below.24G __consumer_offsets-4429G __consumer_offsets-3342G __consumer_offsets-13How do i reduce the size of these partitions, from the logs below is the exception what i am seeing.2022-04-05 03:03:30,041 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0]: Error due tojava.lang.IllegalArgumentException: inconsistent rangeCDH version is 5.16 and kafka is KAFKA-3.1.1-1.3.1.1.p0.2Assuming that log cleaner can be crashed i tried referring below link but that did not help.https://my.cloudera.com/knowledge/Disk-utilisation-growing-because-Kafka-LogCleaner-stopped?id=86234 Need your help and advise to fix the issue.Thanking in advance

Hi, I am getting error when I am querying my table in impala editor.The simple query:select * from table GeoIP2_ISP_Blocks_IPv4;The error:The table creation query is give below:CREATE External TABLE GeoIP2_ISP_Blocks_IPv4 (Client string,ID int,Number int,Name string,)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','WITH SERDEPROPERTIES ("separatorChar" = "\,","quoteChar" = "\"")LOCATION 'hdfs:///user/hive/warehouse/tips/'tblproperties ("skip.header.line.count"="1"); The same error is coming when I am creating external tables from tab delimited files. All this is happening after latest CDP Update. Please someone help me it would be really helpful.

Hi,is there a way to find and remove duplicate entries in two flowfiles?I have one flowfile generated with SQL Processor with entries from my database. The other one contains new and "old" entries. So in order to only write the new entries in the database, I have to detect and remove the entries that already exist in the other flowfile.I already tried the HashContentProcessor but it hashes the content of the whole file. I would need a processor that hashes line for line (and then compares all hashes with each other).Thanks for your help!

We are trying to determine the correct latest version of Cloudera Runtime to use, CM is offering 2 possibilities for CDH 7.1.7-1: 7.1.7-1.cdh7.1.7.p74.210577657.1.7-1.cdh7.1.7.p1000.24102687 What is the best way to determine the latest version? Thanks in advance.    

When using the XMLReader service, how do you retain the outermost parent element of an XML document so when I convert it to JSON I have the outermost element in the JSON version too? For example, given the following XML document how do I capture the "<a>" element rather than just the "<b>" element?   <a> <b>45</b></a> What I want:{"a": {"b":45}} What I get:{"b":45} 

Hi, I'm new to jolt and am trying to remove null values from a nested json using jolt.  Example JSON:{"field1": {"subfield1": null},"field2": {"subfield21": "528010000072312","subfield22": null},"field3": {"field31": null,"field32": null,"field33": {"field331": null,"field332": null}},"field4": {"field41": null,"field42": "UNKNOWN","field43": "NONE","field44": {"field441": {"field4411": null,"field4412": null}}},"field5": {"field51": null,"field52": {"field521": null,"field522": null},"field53": {"field531": null,"field532": {"field5321": null},"field533": {"field5331": "0"}}}  JOLT SPEC: [{"operation": "shift","spec": {"field1": {"*": {"*": {"@1": "&3.&2"}}},"field2": {"*": {"*": {"@1": "&3.&2"}}},"field3": {"*": {"*": {"@1": "&3.&2"},"field33": {"*": {"@1": "&3.&2"}}}},"field4": {"*": {"*": {"@1": "&3.&2"},"field44": {"field441": {"*": {"@1": "&3.&2"}}}}},"field5": {"*": {"*": {"@1": "&3.&2"},"field53":