Q&A

Hello everyone,  I'm new with Apache Nifi, i'm trying to filter on a date column with the format yyyy-mm-dd using the queryRecord processor. Data come from CSV file. What i want to do is to filter and keep all records that have the column date greater or equal than the current date, i tried:  select * from flowfile where date >= currentDate(); But that does not work if it is not possible with the queryRecord processor, which one can achieve that ? Does anyone have an idea ? Thanks in advance 

Hey, we are running a Nifi cluster that connects to a Hive database to pull some data. After a period of time we get the below error.       : java.net.SocketException: Broken pipe (Write failed).: java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)       Our current solution is to restart the Hive Controller services and this refresh's the connection and the process works again, this is obviously not ideal for a production implementation. We have also set the Kerberos user's timeout to be a large number of days in the hive-site.xml but the issue still persists. We are running Nifi version 1.12.1. Any help is very much appreciated.

Hey,  I am using nifi to load data from hdfs to impala table. I am using ExecuteSQL with the following command: LOAD DATA INPATH '/user/nifi/20211115_nifi_test' INTO TABLE arcsight.`20211130_nifi_test` PARTITION (year=${year:toString()}, month=${month:toString()}, day=${day:toString()}) And I am getting the following error:AnalysisException: Value of partition spec (column=year) has incompatible type: 'SMALLINT'. Expected type: 'STRING' Anyone can help?  

I see some query submitted to Impala with "LIMIT 0' at the end which obviously do not return any rows. It seems like these queries are coming from IBM Datastage. I am from Production support team and I monitor such queries. i know we can use limit 10 or so to limit the number of rows returned but why on the earth someone wants to limit to 0? Is it just to check the syntax of query or connection or something? Before I question the team who submits such query, wanted to double check if there is any reason one would like to say limit 0 at the end.  official documentation at official impala document  does not say any thing about it. 

 Is there any way to cast string to map in Impala?e.g: select cast("A:1,B:1,C:1") as map<string, int> 

Hi every one, I encountered the following error when I ran a cde cli command launching spark or airflow job.  ERROR Start TGT gen failed for user: my_user, error: rpc error: code = Internal desc = java.lang.RuntimeException: com.cloudera.thunderhead.service.tgtgenerator.services.KeytabException: unable to get keytab fileName  Have you ever seen this before? To resolve this issue, I just ran my command again and it passed after some tries. Does someone have a solution to pass this command at the first time?My command seems:  cde job run --name my_job_name --variable var1=$(date +'%Y%m%d%H%M%S') --variable var2=$(date +'%Y%m%d%H%M%S') --variable var3=100 --variable var4=yyyyMMddHHmmss  Thank you

keyidtest161289462valid50578283validUGM49O3validI am using below query but still getting wrong results.Select keyid,CASEWHEN keyid like '^[A-Za-z]*%' THEN 'invalid'else 'valid'end as testfrom tableI am expecting belowkeyidtest161289462valid50578283validUGM49O3InvalidAppreciate your help in advance.Thanks

Hi All ,  How can i fund the number of messages published to a kafka topic per hour ?When i got to grafana dashboard from ambari , i could see messages in/s ,does this metric give the number of messages at the time frame in kafka topic ? 

Hi, I'm trying to develop a Jolt Spec for the following transform:JSON Input:  [ { "randomkeyname1": "randomvalue1" }, { "randomkeyname11": "randomvalue11", "randomkeyname22": "randomvalue22" }, { "randomkeyname111": "randomvalue111", "randomkeyname222": "randomvalue222", "randomkeyname333": "randomvalue333" }]  JSON Output:  [ { "data": [ { "key": "randomkeyname1", "value": "randomkeyvalue1" } ] }, { "data": [ { "key": "randomkeyname11", "value": "randomkeyvalue11" }, { "key": "randomkeyname22", "value": "randomkeyvalue22" } ] }, { "data": [ { "key": "randomkeyname111", "value": "randomkeyvalue111" }, { "key": "random

Hello.I'm trying to add users to Ranger via RESTAPI, but I could only add one singular user at a time.This is the command I'm using with a json file curl -u admin:$PASSWORD -i -X POST -H "Accept: application/json" -H "Content-Type: application/json" https://$RANGER_URL:6182/service/xusers/secure/users -d @users_RESTAPI.json -vvv And the json file as the following{ "name":"user_1",  "firstName":"",  "lastName": "",  "loginId": "user_1",  "emailAddress" : "",  "description" : "",  "password" : "pass123",  "groupIdList":[3],  "status":1,  "isVisible":1,  "userRoleList": [ "ROLE_USER" ],  "userSource": 0},{ "name":"user_2",  "firstName":"",  "lastName": "",  "loginId": "user_1",  "emailAddress" : "",  "description" : "",  "password" : "pass123",  "groupIdList":[3],  "status":1,  "isVisible":1,  "userRoleList": [ "ROLE_USER" ],  "userSource": 0} Only the first us

I get following health issue.   The host's NTP service did not respond to a request for the clock offset.   How do I resolve it?   Thanks, Kewal

I have a hive insert statement which by default will use all available resources in YARN as it is reading a large volume of data.   I am happy for the query to take longer and use less resources so that other users can also have access to compute resources.   I don't want to set up YARN queues as this is an unusual query and so don't want to permanently restrict the cluster.   If I was using Spark can do this quite easily with setting a number of executors. Is there a hive config that allows me to do this at a query level.   I have looked at various other posts such as those below, but nothing seems to allow this.  https://community.cloudera.com/t5/Support-Questions/How-to-control-number-of-containers-in-a-hive-query/td-p/297734  https://community.cloudera.com/t5/Support-Questions/Is-there-a-way-to-set-minimum-maximum-number-of-containers/m-p/190660#M152749  https://community.cloudera.com/t5/Support-Questions/Can-I-limit-the-number-of-containers-alloc

Hi, I'm having trouble refreshing a materialized view as it doesn't update properly. I'm going to show an example. Base table "test_table" with 2 rows:  create table test_tableasselect 1 as cunionselect 2 as c   Then I create a materialized view from "test_table":  create materialized view test_mvasselect *from test_table   If I query test_mv, it show 2 rows, it's ok. But if I truncate the original table, and then rebuild the materialized view, the query over the materialized view keeps showing 2 rows!  truncate table test_table;alter materialized view test_mv rebuild;select * from test_mv; -- <-- This query returns 2 rows!!   Any ideas on what's happening? I'm running hive version 3.1.0.3.1.0.0-78  Thanks!

안녕하세요 Streams Replication Manager 를 통해서 아래와 같이 미러링을 하고 있습니다.- Topic 갯수 : 95- AVG Throghput : 319B/s- AVG Replication Latency : 19.6sec- Status : WARNING 위와 같이 계속 WARNING을 표시가 되는데 원인과 조치방법을 문의 드립니다. 

안녕하세요. 아래와 같이 Stream Replication Mananger에서 Error 로그를 발생하고 있습니다.max.request.size 를 어떻게 설정해야 하는지 답변 요청드립니다. ERROR WorkerSourceTaskWorkerSourceTask{id=MirrorSourceConnector-3} failed to send record to m16.ftl.adapter.ic.m16.edp.edes.hub.cmndcol:org.apache.kafka.common.errors.RecordTooLargeException: The message is 1270478 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.ERROR WorkerTaskWorkerSourceTask{id=MirrorSourceConnector-3} Task threw an uncaught and unrecoverable exceptionorg.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callbackat org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:263)at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)at ja

Hello everyone! I need a help in Jolt transform.I have array, that consist from many json objects. (https://jolt-demo.appspot.com/ ) [  {     "table_name": "table_vd",     "scratch_name": "l1_sample_1_1",     "scratch_order": 1  },  {     "table_name": "table_vd",     "scratch_name": "l1_sample_1_1",     "scratch_order": 34  },  {     "table_name": "table_vd",     "scratch_name": "l1_sample_2",     "scratch_order": 3  }]  What transformation of Jolt needs to be done to get the following result? Now i have next Jolt construction. [  {     "operation": "shift",     "spec": {        "*": {           "table_name": "table_name",           "*": { "@0": "data[0].&1" }    &nb

NIFI WEBUI ISN'T STARTING + exec /var/run/cloudera-scm-agent/process/587-nifi-NIFI_NODE/scripts/control.shlog4j:ERROR setFile(null,true) call failed.java.io.FileNotFoundException: /opt/cloudera/parcels/CDH-7.1.4-1.cdh7.1.4.p0.6300266/lib/zookeeper/bin/../logs/zookeeper--cli-linuxcm.log (No such file or directory)at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270)at java.io.FileOutputStream.<init>(FileOutputStream.java:213)at java.io.FileOutputStream.<init>(FileOutputStream.java:133)at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104

Hi list, I’m facing a weird problem I can’t resolve or even understand with my secured nifi cluster. Below is the situation.We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3.For each of theses nodes, we’ve manually created a SSL certificate signing request (CSR) (using a password protected private key) to be signed by our internal CA.Once we’ve get the certificates signed, I’ve installed each node certificates following this procedure: Add the full certificate chain (root + intermediate certificates) into the signed certificate.cat nifi-nodeX.pem cert_chain.pem  > full-nifi-nodeX.pemCreate a PKCS12 certificate using private key (.key) and full signed certificate (.pem)openssl pkcs12 -export  -in full-nifi-nodeX.pem  -inkey nifi-nodeX.key  -out nifi-nodeX.p12 \                    -name nifi-nodeX -passin pass:"XXXXXX" -passout pass:YY

Hello everyone!I have many json files like this:{"table_name" : "train_vd","data" : [ {"battery_power" : 1954,"clock_speed" : 0.5} ]} {"table_name" : "train_vd","data" : [ {"battery_power" : 842,"clock_speed" : 2.2} ]}... I used the MergeContent and MergeRecord processors and used the table_name field as the Correlation Attribute Name (i have ${table_name} attribute). However, this does not work and the result is as follows:[{"table_name" : "train_vd","data" : [ {"battery_power" : 509,"clock_speed" : 0.6} ]}{"table_name" : "train_vd","data" : [ {"battery_power" : 842,"clock_speed" : 2.2} ]}]... However, I want to get the following result:[{"table_name" : "train_vd","data" : [ {"battery_power" : 509,"clock_speed" : 0.6},{"battery_power" : 842,"clock_speed" : 2.2}]}]May you tell me how to solve this problem? Need i use a complex Jolt transformation or to configure the incoming Avro schema in the MergeRecord processor, so that then everything is combined using a s

Hi Team, I've a MergeContent processor  to process the two files (file1, file2) which are coming from two FlowFiles. But here the File2 comes sometimes not regularly , whereas File1 comes on daily basis.so this process two Files successfully. But How we can process the File1 whenever File2 is not present? is it possible through MergeContent Processor or is there any different approach (routing FileFlow) we need to use?2 FlowFiles Routingmerging two FlowFiles   I've tried processing File1 alone but the flow got stuck in queue in between UpdateRecord & MergeContent.   

Hi, We are on nifi-1.11.4, on a 3 node cluster, and we are observing frequent node disconnections. ..Disconnect Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from node in 42 seconds, updateId=62] While we investigate the issue for lack of heartbeat, we were trying to increase the disconnection threshold from the default 40 seconds [nifi.cluster.protocol.heartbeat.missable.max=8 X nifi.cluster.protocol.heartbeat.interval=5 sec] to 300 seconds. However, even when we applied the nifi.cluster.protocol.heartbeat.missable.max=60 setting change on all nodes, we still observe the node disconnections occurring after 40 seconds. Any thoughts on why this setting is not getting picked up? nifi.cluster.protocol.heartbeat.interval=5 secnifi.cluster.protocol.heartbeat.missable.max=60 Thank you-MT13

Hi, We have recently upgraded Nifi to 1.14 and are now receiving this error when trying to write to HDFS."Failed to write to HDFS due to java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonIncludeProperties: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonIncludeProperties↳ causes: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonIncludeProperties"I have checked and all the jackson libraries are there. If anyone has any ideas it would be very much appreciated .

How can we make the PutFile Nifi processor execute only at some specified date and time?

I am trying to upload some files having special characters in filename and getting below error. case:1======Command: hdfs dfs -put 900-0314-Slide#2.vsi /user/xyz12345error: put: `900-0314-Slide': No such file or directory It might be due to # character. case: 2======Command: hdfs dfs -put 'abc.html?C=S;O=A'  /user/xyz12345error:  No such file or directory

Hello ! I am performing extensive experiments over my 3-nodes (VMs) cluster. my VMs have a disk space of 50GB each, and checking the space available (on localhost:9870 (namenode's UI)) after 10 spark-submit application submissions reveal that the hard disks are almost plenty. How to delete that created data without restarting and reformatting the hdfs ? I was thinking of a datanode clean up command to use here. Thanks.