Solved

I am trying to connect to impala using jdbc from one of the BI tools however ,The connection being successful sometimes disconnects for a while with below errorCaused by: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Connection or outbound has closed. ExecuteStatment for query "SELECT a,b from test limit 101".at com.cloudera.impala.hivecommon.api.HS2Client.handleTTransportException(Unknown Source)at com.cloudera.impala.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source)at com.cloudera.impala.hivecommon.api.HS2Client.executeStatement(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeNonRowCountQueryHelper(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeQuery(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.<init>(Unknown Source)at com.

Hi,I have a json data as follow.INPUT :{  "vlineid" : "9PIA",  "type" : "M",  "catg" : "C",  "nhhigh" : 100.0,  "nhigh" : 50.0,  "nlow" : null,  "nllow" : null,  "uom" : "sec",  "vsectionid" : "PEPO60",  "vparamdesc" : "CYCLE TIME",  "vpartid" : "-",  "vflgipc" : "IPC02",  "vwctid" : "P9PIA0",  "vmachineid" : "IMM04",  "vparam" : "CYCLE-TIME",  "nvalue" : 0.0,  "t" : 1722833150160,  "vcrea" : "NIFI",  "vmodi" : "NIFI",  "dmodi" : "2024-08-05 11:49:17",  "dcrea" : "2024-08-05 11:49:17",  "t_epoch" : 1}I tried to remove the null field with this JOLT SpecificationJOLT :[{    "operation": "shift",    "spec": {      "*": "temp.&"    }  },  {    "operation": "modify-overwrite-beta",    "spec": {      "*": "=recursivelySquashNulls"  &nbsp

We want to setup some Impala monitoring and look specifically for queries that have a "Waiting for client" state.  I am pretty familiar with the Cloudera Manager API calls to get Impala query information, but can't seem to figure out how to get this particular state.  Can someone shed light on how Cloudera Manager itself does it, since I see that status in the Cloudera Manager UI when looking at Impala queries?  Thanks, I appreciate any help in advance.Mark

I need to inject data read from elastic in Json format to two different tables having parent child relationship.i.eUser tableName, Age, RoleDavid,35,1Tay,23,2John,27,2Role table:1,admin2,userMy requirement is to insert in user table first get the Role ID and then include that id in next insert to dump data in Role tableHow can i achieve this thing Apache Nifi?@mburgess

In Apache NiFi, We are interacting to SOAP api through powershell. As API expect attachment with MTOM.Now if we load 10000 request we can see only one server is acting on those requeset while other remainnig 2 server are sitting idle. Is there way to share the load acrooss all three servers?

Hi,I am new to NIFI, and I was wondering if I could change the specific column in the csv file.for example, if the column has numerical values it is supposed to be converted to string type like if the column had a numerical value 1 it should be converted to apple in the same columnI tried doing it by referring to websites but still I was not successful. Looking forward for some response for this problem. Thanks in advance   

Hi All,I need to assess a Cloudera platform for a particular application of a Bank and create a utilization report. How to approach this task. I couldn't find any sample reports for reference. I found steps to download utilization report through Cloudera manager, but how to interpret it. What does each metric mean ? -- Does the  CPU / memory utilization percentage we get in the report are w.r.t the total resources that is allocated to that particular application or the overall available resources. What is the difference between pool vs users... etc.I haven't worked on this before. Any help is much appreciated.Thank You.

My application is streaming data to NIFI continuosly. During that time If NIFI Restartsthen the data flow in progress between any processors lost it's content with below error.Note: This is not related to processor, it happens for any processor.Before Restart:List queue->View-> I can able to see my streaming record.Now After doing NIFI RestartList Queue ->View or Download->My flowfile atrributes are there and other details are available but the flowfile content is not there.Kindly help me to solve this issue. All relevant idea's are welcome.

Hello there..  I am new to Nifi and have started using QueryDatabaseTable for incremental fetch. I want to know how to handle a situation as below. For exampleQDT processor fetched new data based on id column - id 50 to id 55State of QDT processor (max value QDT processor processed/View State) - 55Next Steps - ConvertAvrotoJson----->ConvertJsontoSQL ---->PutSQLNow what if the PutSQL processor failed for some reason ? Should not the QDT processor roll back the state of max value from 55 to 49 so that next time QDT processor executes it again starts fetching id from 50  ?  When I checked the state of QDT processor it showed me 55 although the new records were not inserted in final table and when i execute the QDT processor again it fetched data from 56 causing data loss at the destination side.. Is there any way to let the QDT processor know that the downstream processors failed and it should now roll back the state  to previous value before the current run ?

Hi everyone,I'm setting up nifi to run in cluster, I have 3 instances configured, but when I start them I get this error.2024-04-10 20:42:59,080 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at myip3:11443; will use this address for sending heartbeat messages2024-04-10 20:42:59,081 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at myip3/<unresolved>:11443. Will send Cluster Connection Request to this address2024-04-10 20:42:59,165 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors  Has anyone already ever faced this issue?

how are you? I need assistance with converting a string to an integer or numeric value using Apache NiFi’s UpdateRecord processor. I’ve tried using toNumber and toInt, but neither worked. Can you provide guidance on how to achieve this conversion?tHANK  

hi community cloudera, we are verifying a strange behavior of an Impala connection controller on nifi, when the processor starts the connection, it displays the error below:[Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Connection or outbound has closedcommunication between hosts is ok, ports communicating without problem.when we start and stop the processor the connection occurs without problem, this shows that the error is intermittent.has anyone experienced this type of problem?

After I deleted the old cluster and then recreated a new one. I had a new error never got before at the Command Details step. I don't know exactly what is the reason, can someone help me fix this issue? here is the Error Log of Cloudera scm server:2024-07-31 15:56:09,184 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546343795(Format) has completed. finalstate:FINISHED, success:false, msg:Failed to format NameNode.2024-07-31 15:56:09,188 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546343686(First Run) has completed. finalstate:FINISHED, success:false, msg:Completed only 3/4 steps. First failure: Command (Format (1546343795)) has failedThank you.   

Hello everyone,Our team wants to migrate the services running on CM Server i.e node03 to other nodes in the cluster except the Cloudera Manager Services. This is to test high availability so that developer won't get any issues if there is any issue with the node03 of the cluster.Please note: We want to migrate the services individually or to find a better alternative so that we can follow the same procedure in the event of an issue. Need your expertise to discuss further on this.CDH6.3.4

Hello, I have a problem with execute script processor. I have an incoming flowfile with json content like this:{ "setName": "name", "optionalDict": {}, "directories": [ { "path":"path/to/file", "fileName":"filename", "id":1 } ], "setId":12}  My script is the following: import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackfrom org.apache.nifi.processor.io import InputStreamCallbackflowFile = session.get() if (flowFile != None): stream_content = session.read(flowFile) text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8) IOUtils.closeQuietly(stream_content) json_data = json.loads(text_content) session.remove(flowFile) d = {} for x in range(len(json_data['directories'])): d["flowfile{0}".format(x)] = None for i, file in enumerate(d): file = session.create() file = session.putAttribute(file, "setId", str(json_data['setId'])) file = sessi

I want to utilise the ConvertRecord 1.23.2 processor to convert text based CEF into JSON using the CEFReader to JsonRecordSetWriter, but somehow keep the original text CEF log as it appears in its text formate as a field called event.original. Is this at all possible within Nifi?

I have two hive tables and I want to manually create a lineage relationship between them using the Atlas API.I'm trying to run this POST request:```curl --location 'https://[url]/atlas/api/atlas/v2/entity' \--header 'Content-Type: application/json' \--header 'Authorization: ••••••' \--data-raw '{"entity": {"typeName": "hive_process","qualifiedName": "my_etl_process@cluster","name": "my_etl_process","description": "ETL process from input_table to output_table","attributes": {"inputs": [{"guid": "9768381b-8783-49c3-850d-39bf1f14b73f"}],"outputs": [{"guid": "998de7ba-2254-418d-b405-656eba428643"}]}}}'```I'm getting this 404 response:```{"errorCode": "ATLAS-404-00-007","errorMessage": "Invalid instance creation/updation parameters passed : hive_process.qualifiedName: mandatory attribute value missing in type Referenceable"}``` Any suggestions for how to manually create lineage using the API would be appreciated.

Is it a best practice the team make use of the nifi to build Rest API endpoints ?How nifi handle multiples http connections ?I see nifi as a data transform pipeline tool instead of a drag and drop webui to build Rest API endpoint.Best Regards.

Templates are deprecated from 2.XI have existing nifi process groups in the form of exported .json filesBelow is the requirement for REST API:1. Create process group using exported .json file if not exist2. Update existing process groupI am able to uplod to existing process group using below curl:curl --location 'https://localhost:8443/nifi-api/process-groups/1e373804-0191-1000-e950-856de82e267d/process-groups/upload' \--header 'Cookie: __Secure-Request-Token=2dae7036-f4c3-4115-9865-6c9d7984dfa8' \--header 'Authorization: Bearer esometoken3M' \--form 'positionX="11.0"' \--form 'clientId="11"' \--form 'disconnectNode="false"' \--form 'groupName="empty"' \--form 'positionY="557.0"' \--form 'file=@"/home/gwl/Downloads/existing-flow.json"' But when i try to upload completely new process group using below curl:curl --location 'https://localhost:8443/nifi-api/process-groups/01911000-3804-1e37-10a0-827d946513c6/process-groups/upload' \--header 'Cookie: __Secure-Request-Token=2dae7036-f4c

Hello,I wanted to understand what happens to the process and flow files which is currently running when the nifi gets restarted? Please share the details for the better understanding of nifi.

Hi everyone, I would like some clarification. Thank you.When we enabled Kerberos, we encountered some errors during the "start clusters" phase in the command details, specifically when starting HBase.Subsequently, we examined the HBase log: cat /var/log/hbase/hbase-cmf-hbase-REGIONSERVER-cdp717w2.am.ocp.poc.log.out2024-05-02 23:14:41,118 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:16020-0x30046d0bb0d003e, quorum=cdp717m1.am.ocp.poc:2181,cdp717mo.am.ocp.poc:2181,cdp717w0.am.ocp.poc:2181, baseZNode=/hbase Unable to get data of znode /hbase/runningorg.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:120) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2131) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:35

I used NiFi from HDF sandbox (PC A) to copy files to HDFS of HDP sandbox (PC B). NiFi can read files list from HDFS without any error. The flowchart is shown below:However, NiFi can not get files or put files on HDFS, the errors are shown below:GetHDFS processor error:PutHDFS processor error:Can anyone help me? Thank you very much.

Error while trying to upgrade nifi-registry from 1.25.0 to 2.0.0M1-------------------------------------------------------------------------------------------------------------------Error : ERROR [main] o.springframework.boot.SpringApplication Application run failedorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessResource' defined in file [/opt/nifi-registry/nifi-registry-current/work/jetty/nifi-registry-web-api-2.0.0-M1.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessResource.class]: Unsatisfied dependency expressed through constructor parameter 5: Error creating bean with name 'getIdentityProvider' defined in class path resource [org/apache/nifi/registry/security/authentication/IdentityProviderFactory.class]: Failed to instantiate [org.apache.nifi.registry.security.authentication.IdentityProvider]: Factory method 'getIdentityProvider' threw exception with message: Identity Provider configuration XML was prote

I'm working on a data processing flow using Apache NiFi, and I have a scenario that I'd like some guidance on.I have an Execute SQL script that retrieves data from a table in a SQL Server. Among the data returned, I have source and target folder paths.My goal is to loop through each of these source folder paths. For each folder, if there are any files present, I need to:Insert a record into the database to log the file's details.Move the file from the source folder to the corresponding target folder.I'm currently using NiFi, and I'm wondering what processors and strategies I should consider to achieve this workflow efficiently. Any insights, recommendations, or examples of similar workflows would be greatly appreciated.Thank you in advance for your help!

We have a requirement to collect all the data that we receive for one day in one text file. We get the data at different time intervals, We need to collect all day on a text file with the filename as currentdate.txt.Let's say, on 24th July 2024 we received some data like "This is data 1" in the morning then we need to create a new file with the name "24072024.txt" and add the data to the file. Later in the afternoon, we received data like "This is data2".  then this data needs to be added to the text file - "24072024.txt".  Also if the file does not exist then we need to create the new file with that date. So please help us with the NiFi flow how to append the text in the file if it exists and then how to handle the condition to create a new text file if it does not exist.