Q&A

I trying to ingest a SQL Table to Hive using NiFi. This is easily done by using a QueryRecordTable. However, the records in the SQL table can be modified by hand, so they are not static.Is there any way to update the data in HDFS according to the changes made in the SQL table?

I'm checking whether there are any effects to spark2 by CVE-2022-33891 or not.Is there anyone who can explanin it to me?

Hi, few hours ago zeppelin stopped to run queries and begins to throw errors like WARN org.apache.zeppelin.socket.NotebookServer: Job 20230217-161657_1725299508 is finished, status: ERROR, exception: null, result: %text javax.security.auth.login.LoginException: Unable to obtain password from userIts kerberized AD KDC cluster.ldapsearch and kinit with right principal works fine./etc/hosts are fine too.The rest services on the cluster are green.http://hostname:port/api/credential/ returns {"status":"OK","body":{"userCredentials":{}}}What else can I check?

Guys, have any of you ever had this problem when using the QueryDatabaseTable?I have a datetime field in sql server but when it passes through the QueryDatabaseTable it becomes a varchar and ends up generating an error in Nifi.

Please assist me to find the list of unaccessed or last accessed hive tables/partitions using cloudera navigator, using CDH 6.3.4 version.    

Hi,I have query on "Delivery guarantee" parameter in publishKafka processor in NIFI. I have query on below two option supported for the "Delivery guarantee"  1. Best effort2. Guarantee single node delivery I know that the "best effort" does not wait for the ack/response from kafka, but in case of publish failure, when either of the above two option is set, will NIFI try to publish among the available nodes? Like in my setup its the cluster of three kafka nodes. If one of the node is down or if we observe connection timeout issue with one of the node, will NIFI tries to publish with other available nodes? Thanks

Hi ,Am using MSSQL version 12+ , I have connected to DB successfully . I could able to fetch the data eg : Select * from dbname.sys.tables But If i try to query fetch the table records using below processor   Getting below error  :   

I use Impala JDBC to query, I find  close() and execute() with synchronized. So if I invoke close() in other thread will wait execute() ，because if execute() is running，close() will blocked waiting for lock. So I should invoke cancal() before close() to avoid  block? 

I am new to Apache NiFi and need to know how to find a processor or processor group by its integration name, folder, or event name, as well as how to find the specific source and destination file details within any given processor or processor group.

I have step up 3 node (node1, node2, node3) nifi cluster and I am able to access it with http protocol, When I try to make those 3 node secure(https). I am facing "insufficient permissions untrusted proxy" in web UI.After setup 3 node cluster in node1 i followed the steps to make secure connection. Step:1 By using below step I have Generate Keys and Certificates/opt/nifi-toolkit/bin/tls-toolkit.sh standalone -n 'node1' -B ClientPassword -C 'CN=admin,OU=NIFI' -O -o /opt/nifi-1.19.1/conf/certsafter run this I got below files/opt/nifi-1.19.1/conf/certs/├── CN=admin_OU=NIFI.p12├── CN=admin_OU=NIFI.password├── node1│ ├── keystore.jks│ ├── nifi.properties│ └── truststore.jks├── nifi-cert.pem└── nifi-key.keyStep:2I have over write the /opt/nifi-1.19.1/conf/certs/node1/nifi.properties to conf/nifi.propertiesThen Edit the nifi.propertiesnifi.security.user.authorizer=managed-authorizerStep:3Updated Security Properties Key/opt/nifi-1.19.1/bin/nifi.sh set-sensitive-properties-key xyz12345Step

Hello Team, I am trying to send logs to the gaylog server from Nifi for the transfer of each file log.could you please help me with the stepwise procedure to do so?Also i have been trying to copy few files from remote server 1 to 2 and then the same files from 1 will be moved to 3 with the logs of each files in getting pushed to graylog. can you please help me with the step wise procedure to perform this action. i have been struggling from weeks to do above two task. 

Hello We have a HDFS cluster configure to work with Kerberos and we want to use nifi processor PutParquet to write file into hadoop cluster.do we need to configure nifi to work with kerberos too in order to connect to Hadoop?what are the steps i need to take in order to do the connection between the two clusters (nifi --> HDFS) Thanks

Hi All, Greetings! Assume that I have a data flow, and the final result is inserting to the SQL database (Microsoft), but the step before! I have the following JSON format for each flow file content as following:  { "sbrActivityType" : { "1" : "ACT+A", "3" : "ACT+A", "2" : "ACT+X", "5" : "ACT+X", "4" : "ACT+X" }, "sbrDateTime" : "Sun Mar 19 13:45:15 AST 2023"} Then, my desired output format look like the following:INSERT INTO sbrTestNifiData (sbrActivityType, sbrDateTime) VALUES (?, ?) So, it does not insert correctly to my database table which is looks like the following: CREATE TABLE [dbo].[sbrTestNifiData]( [sbrId] [int] IDENTITY(1,1) NOT NULL primary key, [sbrActivityType] [varchar](50) NULL, [sbrDateTime] [varchar](50) NULL);  So, it should inserted to my database table as a five records:sbrIdsbrActivityTypesbrDateTime1ACT+ASun Mar 19 11:56:40 AST 20232ACT+ASun Mar 19 11:56:40 AST 20233ACT+XSun Mar 19 11:56:40 AST 2023

Hi everyone, I'm working to a new feature with an existing nifi cluster to provide a new service to add an interface with serveral kerberized HDP Cluster. I would like to know if a single Nifi cluster can use several realms in the same krb5 file.Reading official documentation, nifi can do it (https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#kerberos_properties) : " If necessary the krb5 file can support multiple realms." It seems ok but because at this time I have no way to have several hadoop cluster for testing (to notice : my cluster is working already with one kerberized cluster hadoop ), is anybody can confirm or reject this design: one cluster Nifi with different realms to communicate with multiple kerberized hdp cluster. Thanks for your help and as soon as I have several kerberized cluster hadoop for testing, I will update this article. 

Hi, All!I have a problem with NIFI ListSFTP processor. After started processor errored with Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty! ListSFTP processor Version 1.12.0 and the Properties as follows:Error screenshot from UI:

Dear all, I want to run COMPUTE INCREMENTAL STATS with PARTITION option on some partitions of a partitioned table.It is unclear to me from the documentation if DROP INCREMENTAL STATS must be run on those partitions. For my understanding of how INCREMENTAL STATS with the PARTITION option work, it should not be required, but I've found below sentence on this page https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_perf_stats.html#perf_stats "In cases where new files are added to an existing partition, issue a REFRESH statement for the table, followed by a DROP INCREMENTAL STATS and COMPUTE INCREMENTAL STATS sequence for the changed partition.". I'd prefer to avoid running DROP INCREMENTAL STATS if not required. Does anyone know? Many thanks in advance.

Hello,I am trying to solve a strange behavior of NiFi but can't really figure it out.My flow consists of a GenerateTableFetch (which gets executed once per day), a ExecuteSQLRecord (which gets executed once it receives the files from GenerateTableFetch) and finally an PutS3Bucket.Now, both the GenerateTableFetch and the ExecuteSQLRecord are using the same Controller Service, a DBConnectionPool which connects to an Oracle Database.In the previous days I have noticed that I get plenty of sessions from NiFi (which is ok) but these sessions never get killed/terminated and they remain inactive. I am using the following configurations within my DBConnectionPool Service:Max Wait Time: 500 millisMax Total Connections: 51Minimum Idle Connections: 0Max Idle Connections: 5Max Connection Lifetime: -1Time Between Eviction Runs: -1Minimum Evictable Idle Time: 30 minsSoft Minimum Evictable Idle Time: -1Unfortunately I was not able to find any information in the documentation (or maybe I did not fully

error " Cannot find CDH's bigtop-detect-javahome" when add any service in cluster please we need a helpTue Mar 7 14:18:59 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:03 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:08 EET 2023Cannot find CDH's bigtop-detect-javahome.Tue Mar 7 14:19:13 EET 2023Cannot find CDH's bigtop-detect-javahome.----------------------and stderr log   --------------------------- from cryptography import x509[07/Mar/2023 14:19:12 +0000] 20862 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hdfs/hdfs.sh datanode[07/Mar/2023 14:19:12 +0000] 20862 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 20870[07/Mar/2023 14:19:12 +0000] 20871 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5/opt/cloudera/cm-agent/lib/python2.7/site-packages/OpenSSL/crypto.py:12: CryptographyDeprecationWarning: Python 2 is no longer suppor

Failing at 38%  NAMENODE : AMBARI server  NAMENODE : AMBARI server stderr:   errors-440.txt stdout:   output-440.txt    2021-07-22 09:41:36,191 - Processing identities...2021-07-22 09:41:36,222 - Creating keytab file for hdpcluster-072221@ on host myserver.com2021-07-22 09:41:36,230 - Processing identities completed.      DATANODE : HDP CLUSTER   DATANODE : HDP CLUSTER stdout:   /var/lib/ambari-agent/data/output-438.txt   2021-07-22 09:41:34,474 - Missing keytabs:Keytab: /etc/security/keytabs/kerberos.service_check.072221.keytab Principal: hdpcluster-072221Command completed successfully! This is the krb5.conf file placed in ambari server (name node) and hdpcluster 3.1.5 (datanode) # Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/# output settings[logging]default = FILE:/tmp/krb5libs.logkdc =

Dear Members, I have accidently created a partition having characters %01 in the name. %01 is start of header `as per the notepad++`.When i am trying to delete the partition using ALTER statement, there is no error but partition also not getting deleted. can anyone suggest how to delete such partitions with special characters. ALTER TABLE <table_name> DROP IF EXISTS PARTITION (code='YATHAH%0188QW'); 

when we rename partition of a column, would underlying data of the column would also change as per our rename. this is on ORC file format. for ex. I've partition named activeFlag='N', when I rename this partition to activeFlag='Y', would those activeFlag column value change to 'Y' (OR) data would be intact as 'N' only?

Hi,I have two NIFI instance running and i am sending data. Both the nodes receives the data in random order. when i kill one of NIFI instances(first instance), all the data is processed by other NIFI instance(Second instance). And again when i bring back the other NIFI instance which was killed , the data is distributed among them in random order. Here there is no data loss observed, but we are observing some log messages are missing in the nifi-app.log as part of this restart.For example, during this testing if i have sent 100 messages, i see only log messages for 97 messages. Is this expected in NIFI or is there a way we can fix this issue? Is it related to gracefully shutting down the NIFI? Any help would be appreciated. Thanks

I have a custom processor which takes in a flowfile and based on the given attributes it creates new attributes and returns. However, I keep getting this error and I've checked the forums and I can't see where the problem is. Any help is appreciated.  o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in Component [LocationParserProcessor[id=630e7e24-0183-1000-905c-8f31dc8a1e22]]org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=82eb8ad5-a9e3-473a-991b-0c4a3ab2bf36,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663856909589-1, container=default, section=1], offset=698, length=188],offset=0,name=file.csv,size=96] transfer relationship not specified. This FlowFile was not created in this session and was not transferred to any Relationship via ProcessSession.transfer() at org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:259) at org.ap

HI i am using cdp 6.3i am trying to connect impala shell it is giving me this errorimpala-shell. i am using 3 nodes , and i tried with all nodes . impala service is working fine and even i can run queries from Hue dashboard but self impala shell is not connecting   Running Command : impala-shell Result :Starting Impala Shell without Kerberos authenticationError connecting: TTransportException, Could not connect to mprl508:21000***********************************************************************************Welcome to the Impala shell.(Impala Shell v3.4.0-SNAPSHOT (27b919f) built on Tue Aug 3 21:32:01 UTC 2021)When you set a query option it lasts for the duration of the Impala shell session.***********************************************************************************[Not connected] > 

Hello Team, I have a pipeline that starts with an ad-hoc flow file processor as the starting processor. During cluster maintenance, the processors often go to a stopped state. Is there a way, I can get an alert with the note this processor is in a stopped state? Basically, the ask is to send alerts based on the state of the processor. Thanks in advance.