Solved

Hi,After tried to login, it is taking me back to reenter the credentials.I have tried all the passwords.Pls help.... 

Nifi registry key was expired, I have regenerated key for nifi registry ans also nifi nodes.After setting up new key ssl error resolved but now i still can't commit to nifi registry from nifi node.Error log from nifi registry log :2023-09-15 12:18:56,526 ERROR [NiFi Registry Web Server-25] o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: org.apache.nifi.registry.flow.FlowPersistenceException: Git directory ./nifi01_flow is not clean or has uncommitted changes, resolve those changes first to save flow contents.. Returning Internal Server Error response.org.apache.nifi.registry.flow.FlowPersistenceException: Git directory ./nifi01_flow is not clean or has uncommitted changes, resolve those changes first to save flow contents.Note: I did not made any config change other then adding new ketstore and truststore to nifi registry and nodes.

Here is my  login-identity-providers.xml <provider><identifier>ldap-provider</identifier><class>org.apache.nifi.ldap.LdapProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">cn=admin,dc=example,dc=com</property><property name="Manager Password">secret</property><property name="TLS - Keystore"></property><property name="TLS - Keystore Password"></property><property name="TLS - Keystore Type"></property><property name="TLS - Truststore"></property><property name="TLS - Truststore Password"></property><property name="TLS - Truststore Type"></property><property name="TLS - Client Auth"></property><property name="TLS - Protocol"></property><property name="TLS - Shutdown Gracefully"></property><property name="Referral Strategy">FOLLOW</property><pr

Is it possible, or does it make sense to use AWS Load Balancers in front of a NiFi Cluster?We are running a 3 node nifi cluster using v 1.19.1 on aws and I wanted to see if it is possible, or makes sense to use an AWS Load Balancer(s) in front of the nifi cluster.Any guidance, suggestions would be tremendously appreciated.

Hi All ,I planning to upgrade NiFi 1.9.2 to 1.23.0. I read some articles about upgrading.https://community.cloudera.com/t5/Support-Questions/Nifi-Upgrade/m-p/346309If I follow the take special note above , can I directly migrate 1.9.2 to 1.23.0 ? Will there be any problem ?

I'm trying to use a "GenerateTableFetch" process, but it always returns empty flow files, with no error.When I turn on trace, it runs the followingGenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Setting CLUSTER State to {}GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Executing SELECT COUNT(*) FROM TESTTRANSFERGenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]GenerateTableFetch[id=9adb6bf3-018a-1000-517b-b3f88a43e39e] Returning CLUSTER State: StandardStateMap[version=663, values={}]So it just runs a COUNT(*) and then exits it seems?When I run the same query against the ORACLE 19c database I get a count of 500kI can't find any info that suggests Oracle isn't supported, but I can't figure out a reason it would just return nothing. 

On a processor, right click -> view status history....there is a data available by name 'Records Processed' in dropdown.For one of the processor in my flow, this is not available.What can be the reason for this? Is it possible that value was so high that system lost track somewhere?

I have a nifi flow, working fine.  It picks file from a source directory, processes it, and file is deleted from source directory once picked up.This all works fine first time. But later on, if I copy another file in source directory, that file is not picked up. I even stop all processors, clean any queue if not empty, and restart processors, but still file is not picked from input directory.What causes this behaviour?As a result, I have to stop-restart nifi, making it a bit cumbersome.PS: Based on some google suggestions, I tried to delete flowfile_repository, but it doesn't allow so with message 'Device or resource busy'.

Does HDP and Ambari install on Rocky 8?

Hello everybody,One of my disks on a datanode brake down few months ago. I changed it to a new one but my HDFS won't balance datas between disks. So, i have 7 disks with 80-90% of their capacity and 1 disk with only 15-20% of his capacity.I tried couple things, such as balancing HDFS or delete every small files.Furthermore, this difference between disks created a huge amount of blocks in my cluster. (The datanode has 1,043,627 blocks. Warning threshold : 0)Do you have any ideas to fix this ? Thanks,Oscar

Hi , Kindly for your support, I'm trying to create a kudu table from impala shell after enabling kerberos but i got the following error:  ERROR: AnalysisException: Cannot analyze Kudu table 'courses_kudu': Error determining if Kudu's integration with the Hive Metastore is enabled: Couldn't find a valid master in (cloudera.ngdw.com:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)] Kindly find attached steps and error returned.  

Hi everyone,I use NiFi to frequently check if a website is up or down. I get the following response as text in a single Flowfile:HTTP/1.1: 200 Connection establishedProxy-Agent: Fortinet-Proxy/1.0Content-Type: image/jpegConnection: keep-aliveDate: Tue, 12 Sep 2023 14:05:43 GMTServer: ApacheStrict-Transport-Security: max-age=31536000URL: https://test.com How can I turn these lines into Key/Value-Pairs or Attributes, to use them with further processors? My goal is to save these Informations in an mySQL Database where the following fields are given:HTTP/1.1Proxy-AgentContent-TypeConnectionDate...Best regardsMaik

Hello everyone, I am new to docker and I have been researching and working with nifi recently. I have one query for my use case where I need to use code repositories that are outside the docker container (I have already bind them while docker run command) and I can access the folder via exec but I need to access these scripts in the executestreamcommand processor's working directory and command path fields. Even after putting the destination directories into the fields I am getting no such file or directory exists. So, please help me out.

I have some basic questions about nifi cluster.I understand that nifi makes use of multiple nodes to speed up the processing of data.But how do we ensure durability of data processed? Suppose we distribute n flowfiles in a queue to several nodes, with each node getting a subset of n files. Does nifi assign same subset to more than one nodes, so that if one of those processing node goes down, we still have others working on same subset?Second, how is result of processing  done on different nodes aggregated? Is it done on primary node OR there is some other mechanism?

I am working in NiFi 1.16 and trying to use ExecuteSQL processor to write out a Parquet file.  My challenge is the decimal format is being changed to "LogicalType(STRING: StringType())".The data is coming from an MS SQL server.  We have tried to set the precision and scale, but still continues to come out in a string instead of a decimal.  

What does this message mean on an invalid processor:I see it for processor FetchXMLFile.On scheduling tab, execution dropdown has 'Primary Node' selected. On changing it to 'All Nodes', error goes away.I am on 1.22.0(with some nars from 1.10.0 also added to lib folder).

I am configuring a new 3 node NiFi cluster on three AWS EC2 instances.  I'm attempting to use the tls-toolkit.sh utility to generate the SSL keystore.jks and truststore.jks for each of the 3 nodes.  It generates them and I can start the cluster but when I load the UI it seems like it's working but then I get this message.An unexpected error has occurred (log out) (home)javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: certificate: sha256/LhH2H...hkIE= DN: CN=node1.server.name, OU=NIFI subjectAltNames: [node1.server.name]The nifi-app.log has basically the same to say about it.WARN [Replicate Request Thread-7] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to my.elb.name:port due to javax.net.ssl.SSLPeerUnverifiedException: Hostname my.elb.name not verified: ... subjectAltNames: [node1.server.name]I would have expected the subjectAltNames to include my.elb.name because I included it in my tls-tool

I'm having an issue with a Spark-Hive application running on a Kerberos cluster. I receive a javax.security.sasl.SaslException: GSS initiate failed error, which appears to be caused by not finding any Kerberos tgt. Here's the error log:     23/08/04 22:56:55 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.    23/08/04 22:56:55 INFO HiveClientImpl: Attempting to login to Kerberos using principal: hdfs01@HDP.COM and keytab: hdfs01.keytab-2ca1f730-bef7-4166-90ce-67317c75c793    23/08/04 22:56:55 INFO UserGroupInformation: Login successful for user hdfs01@HDP.COM using keytab file hdfs01.keytab-2ca1f730-bef7-4166-90ce-67317c75c793    23/08/04 22:56:55 INFO metastore: Trying to connect to metastore with URI thrift://master3.abc.xyz.com:9083"    23/08/04 22:56:55 ERROR TSaslTransport: SASL negotiation failure    javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExce

For my work, I had to copy nifi 1.10.0 nars into lib folder of nifi 1.22.0. I do this as part of docker image build process.This work seems to support this.First, container launch has issues with multiple jetty nars, so I dont't copy them, and jetty issue is gone. But container launch still fails with:java.lang.IllegalStateException: Multiple framework NARs discovered. Only one framework is permittedIs this due to nar versions(1.10.0 and 1.22.0) or do I need some more configuration to make it work?  

Hi Team,I am facing following issue(error) to start region and master servers. Anyone is having any idea on this.023-09-01 10:51:24,807 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Failed construction RegionServerjava.lang.UnsatisfiedLinkError: failed to load the required native libraryat org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:54)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:186)at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:37)at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)at

Hi, I have just started using Apache NiFi, so things are pretty new to me. What I want is to migrate our existing NARs(created for nifi 1.10) to version 1.22.We actually tried it with 1.15.1 too but we found that there were new processor properties which were required, and so things didnt go well. So I want to know if there is some well documented way of achieving it?

when I am using PutFile Processor and ExecuteScript Processor,the object in the flow between them stuck for longer time than expected and processed many times.this case happend only between these 2 processors only.even if I set the expiration time for the flow and set it to 1 second, and scheduled the run of ExecuteScript to 4 sec, still processing the flow at least 2 times.which cause sending the file to the API many times.the version of the NIFI I am usnig is: NIFI v1.18, also I tried to upgrade and use version 1.22, but this issue still as is. The script I am using in ExecuteScript is:import org.apache.commons.io.IOUtilsimport org.apache.nifi.processor.io.OutputStreamCallbackdef flowFile = session.create()def filePath = 'E:/tmp/tmp_file.txt'def file = new File(filePath)if (file.exists()) {def fileContent = IOUtils.toByteArray(new FileInputStream(file))flowFile = session.write(flowFile, { outputStream ->outputStream.write(fileContent)} as OutputStreamCallback)session.transfer

I have a secure hadoop cluster with HDP3.1，I recently tried to interconnect this cluster with the Knox component to implement a secure proxy. This cluster has kerberos, Ldap, and https enabled, I create one config like this: <topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>main.ldapRealm</name> <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value> </param> <param> <name>main.ldapContextFactory</name> <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value> </param> <param> <name>main.ldapRealm.contextFactory</name> <value>$ldapContextFactory</val

Problem Summary---------------------------------------------------Is Cloudera have estimation time for jobs completion.Problem Description---------------------------------------------------Hi Support,I need to understand and have information on any time estimation of completion of jobs running in Hadoop respective of map reduce or spark jobs?

如何下载CDH6.3.2？（免费帐户）谢谢