Q&A

I had an authorization error after running successfully NiFi service on the Cloudera Private Cloud Base Cluster. Here is the error log in nifi-user.log file:2024-08-07 17:57:56,775 INFO [main] o.a.n.a.FileAccessPolicyProvider Authorizations file loaded at Wed Aug 07 17:57:56 ICT 20242024-08-07 17:58:48,621 INFO [NiFi Web Server-30] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 10.62.142.124 [CN=node2, ST=CA, C=US] GET https://node2:8443/nifi-api/flow/metrics/json2024-08-07 17:58:48,627 INFO [NiFi Web Server-30] o.a.n.w.s.NiFiAuthenticationFilter Authentication Success [node2] 10.62.142.124 GET https://node2:8443/nifi-api/flow/metrics/json2024-08-07 17:58:48,747 INFO [NiFi Web Server-30] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[node2], groups[nifi] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden response.Here are all the services I am using:I am using Java 11, I changed the NiFi directory from /var

Below is the query causing row locks in Mariadb Galera Cluster 10.2.32 (3Node Cluster). select MT_COMMENT from AUX_TABLE where MT_KEY1='Cleaner' and MT_KEY2=0 for update > Hive is a framework, we couldn't get the flow to understand also can't alter the logic from app end. > Looking for solutions like any version that works in Mysql/Postgresql or any Metastore DBs that can scale for Hive Metastore without lockign issues. > Also looking for any work around solution from DB end for this Mysql Row Locking.

Hello there..  I am new to Nifi and have started using QueryDatabaseTable for incremental fetch. I want to know how to handle a situation as below. For exampleQDT processor fetched new data based on id column - id 50 to id 55State of QDT processor (max value QDT processor processed/View State) - 55Next Steps - ConvertAvrotoJson----->ConvertJsontoSQL ---->PutSQLNow what if the PutSQL processor failed for some reason ? Should not the QDT processor roll back the state of max value from 55 to 49 so that next time QDT processor executes it again starts fetching id from 50  ?  When I checked the state of QDT processor it showed me 55 although the new records were not inserted in final table and when i execute the QDT processor again it fetched data from 56 causing data loss at the destination side.. Is there any way to let the QDT processor know that the downstream processors failed and it should now roll back the state  to previous value before the current run ?

[10/Aug/2024 14:57:40 +0800] 3876691 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh setup[10/Aug/2024 14:57:40 +0800] 3876691 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 3876714[10/Aug/2024 14:57:40 +0800] 3876715 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5[10/Aug/2024 14:57:40 +0800] 3876715 MainThread redactor INFO Started redactorSat Aug 10 14:57:41 CST 2024+ locate_java_home+ locate_java_home_no_verify+ JAVA17_HOME_CANDIDATES=('/usr/java/jdk1.17' '/usr/lib/jvm/jdk-17' '/usr/lib/jvm/java-17-oracle')+ local JAVA17_HOME_CANDIDATES+ OPENJAVA17_HOME_CANDIDATES=('/usr/lib/jvm/java-17' '/usr/lib/jvm/jdk-17' '/usr/lib/jvm/jdk1.17' '/usr/lib/jvm/zulu-17' '/usr/lib/jvm/zulu17' '/usr/lib64/jvm/java-17' '/usr/lib64/jvm/jdk1.17')+ local OPENJAVA17_HOME_CANDIDATES+ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/

Hi everyone,I'm setting up nifi to run in cluster, I have 3 instances configured, but when I start them I get this error.2024-04-10 20:42:59,080 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at myip3:11443; will use this address for sending heartbeat messages2024-04-10 20:42:59,081 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at myip3/<unresolved>:11443. Will send Cluster Connection Request to this address2024-04-10 20:42:59,165 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors  Has anyone already ever faced this issue?

We are trying to set up a 3 node nifi cluster on GCP virtual machine (Ubuntu). Have used a CA signed certificate for creating truststore and keystore (Followed this link for creation). Have attached them through nifi.properties file. Still getting below exception on tailing logs:2022-05-26 18:14:26,544 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at hostname:7474. Will send Cluster Connection Request to this address2022-05-26 18:14:26,780 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target2022-05-26 18:14:31,783 INFO [main] o.a.n.c.c.n.LeaderElectionNodeProtocolSender Determined that Cluster Coordinator is located at ho

how are you? I need assistance with converting a string to an integer or numeric value using Apache NiFi’s UpdateRecord processor. I’ve tried using toNumber and toInt, but neither worked. Can you provide guidance on how to achieve this conversion?tHANK  

We are experiencing an issue with converting data from string to timestamp in Hive, where the timestamps undergo unexpected hour changes during conversion and processing. We would like to verify if this could be a bug in the tool or if there are additional configurations required. Process and Steps to Reproduce the Issue: Loading Data into HDFS:1. We use the hdfs put command to load a positional file into HDFS and store it in a directory accessible via an external table ex_sa_beneficios_mantidos where the data is in string format. 2. We create an external table in Hive to access the data as strings. 3. Converting String to Timestamp and Writing to ORC Table: When converting the string to timestamp, we notice an alteration in the hour of the dt_afast_trabalho field. 4. Verifying the Conversion and Data Reading: When reading the data from the ORC table, the hours in the dt_afast_trabalho field were different from expected. 5. Counting Timestamps by Hour: We performed a count by hour of t

My requirement is to retrieve the total number of files in a given HDFS directory and based on the number of files proceed with the downstream flowI cannot use the ListHDFS processor as it does not allow inbound connections. The GetHDFSFileInfo processor generates flowfiles for each HDFS file, causing all downstream processors to execute the same number of times.I have observed that we can use ExecuteStreamCommand to invoke a script and execute HDFS commands to get the number of files. I would like to know if we can obtain the count without using a script? Or if there is any other option available besides the above.

Support QuestionHow to solve the following issue?We run CDP 7.1.7 SP2 CHF 18.We followed Cloudera's description "CDP 7.1.7 / Authenticating Hue users with Knox SSO".- The details given below are redacted -IssueWith a newly started browser Knox was called, having no Cookies cached.When connecting the service Hue a second time by clicking the appropriate icon in Knox, it works fine. The URL https://knox.mydomain.com/gateway/Business/hue/hue/editor/?type=impala is used.The first connection to Hue via Knox fails, and the browser calls the URL https://knox.mydomain.com/gateway/Business/hue/%3FdoAs%3Duserid.We get an HTTP error 404 (Page not found) with following details:Request Method:GETRequest URL:https://hue_loadbalancer.mydomain.com:8888/%3FdoAs%3Duserid?doAs=useridfollowed by the messageUsing the URLconf defined in desktop.urls, Django tried these URL patterns, in this order:  1. ^hue/accounts/login [name='desktop_auth_views_dt_login']  2. ^accounts/l

When accessing the project page in cdsw, we can' t find that page will display an error page. The project has been deleted, but is now displayed in the project list. Any idea how to erase this?

Hi We are configuring ConsumeJMS Processor using JNDI Controller service and ConsumeJMS Properties. as Shown here.and the controller service configurations are as follow.This works perfectly in test pod, but when we deploy same changes in other server different environment pod we get following error.o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS[id=xxxxxxxxxxxx] Failed to initialize JMS Connection Factory: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target↳ causes: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target↳ causes: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target↳ causes: javax.naming.CommunicationException: WFNAM00018: Failed to connec

Hello Team,I have scenario, where whenever there is a failure in the ETL process with NiFi at the SOURCE layer, would like to rerun the failed ones !! I agree to the fact that Apache NiFI does not support directly the option of rerun/reprocessing of the failed files, still some kind of a hack exists and below is my approach.Configure a path to move all the failed file(s) for reprocessingclone the flowfile (from the original/failed ones) and make use of the UpdateAttribute optionTrigger a new run for the failed onesNow, will the above approach works in case if NiFi is deployed in distributed mode?Kindly help me to get a feasible solution for this problem statement

Hello All, Just checking if anyone is aware or can guide me to a documentation that would provide details of comparison between CDP Private Cloud vs HPE Ezmeral Data Fabric Platform. I have read somewhere that HP took over MapR and then formed HPE Ezmeral. However, can't find what are the differences. Please advise. Thankssnm1523

hi community cloudera, we are verifying a strange behavior of an Impala connection controller on nifi, when the processor starts the connection, it displays the error below:[Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Connection or outbound has closedcommunication between hosts is ok, ports communicating without problem.when we start and stop the processor the connection occurs without problem, this shows that the error is intermittent.has anyone experienced this type of problem?

Hi, we need to upgrade an old Cloudera CDP 7.1.1 installation to Cloudera CDP 7.1.9 version. The components we use are the following: Apache Hadoop (Includes YARN and HDFS) ,Apache Hive, Apache Impala, Apache MapReduce, Apache ORC, Apache Parquet, Apache Spark 2, Apache Sqoop, Apache Tez, Hive on Tez, Apache Zookeeper.For your knowledge, is the cdp 7.1.9 version backwards compatible with the cdp 7.1.1? Thanks in advance.

Hi Team,I am facing the issue, while trying to fetch the value from jsonpathreader with avroschema attributes as below.json reading from avro schema with jsonpathreader, and want to fetch home number with expression $.phoneNumbers[?(@.type=="home")].number. However, it is getting failed.Please help on this. { "firstName": "John", "lastName": "Smith", "isAlive": true, "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021-3100" }, "phoneNumbers": [ { "type": "home", "number": "212 555-1234" }, { "type": "office", "number": "646 555-4567" } ], "children": [], "spouse": null} 

After I deleted the old cluster and then recreated a new one. I had a new error never got before at the Command Details step. I don't know exactly what is the reason, can someone help me fix this issue? here is the Error Log of Cloudera scm server:2024-07-31 15:56:09,184 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546343795(Format) has completed. finalstate:FINISHED, success:false, msg:Failed to format NameNode.2024-07-31 15:56:09,188 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546343686(First Run) has completed. finalstate:FINISHED, success:false, msg:Completed only 3/4 steps. First failure: Command (Format (1546343795)) has failedThank you.   

While working with JsonRecordSetWriter 1.22.0 on an ExecuteSQLRecord 1.22.0 processor, we have noticed that when selecting columns with arrays such as the below example:the writer has the below output:However, if I select a smaller number in the array:I seem to get  the correct result:This seems to be an overflow of some kind that did not seem to exist on our previous version of nifi when working with nifi 1.11.4I am providing below both the settings of the ExecuteSQLRecord processors and the JsonRecordSetWriter. Please advise if this is a bug or If there might be something I am overlooking. Thank you. 

Hello everyone,Our team wants to migrate the services running on CM Server i.e node03 to other nodes in the cluster except the Cloudera Manager Services. This is to test high availability so that developer won't get any issues if there is any issue with the node03 of the cluster.Please note: We want to migrate the services individually or to find a better alternative so that we can follow the same procedure in the event of an issue. Need your expertise to discuss further on this.CDH6.3.4

Hey all,So I'm trying to check some columns together to ensure that the sums of A & B equal to row C, for each given currency. Table Sales;CurrencyColAColBColCAUD50,00050,000100,000CAD{null}20,00020,000MZN30,000{null}30,000YER10,00010,00020,000Query that I can't seem to get working properly:SELECT Currency,  CAST((SUM)CASE WHEN((coalesce(ColA,0)) + (coalesce(ColB,0))) = (coalesce(ColC,0)) THEN 0 ELSE SUM((coalesce(ColA,0)) + (coalesce(ColB,0))) - (coalesce(ColC,0) END AS sales_check)The output that I'd like to get:Currencysales_checkAUD0CAD0MZN0YER0Thanks,

Hello, I have a problem with execute script processor. I have an incoming flowfile with json content like this:{ "setName": "name", "optionalDict": {}, "directories": [ { "path":"path/to/file", "fileName":"filename", "id":1 } ], "setId":12}  My script is the following: import jsonimport java.iofrom org.apache.commons.io import IOUtilsfrom java.nio.charset import StandardCharsetsfrom org.apache.nifi.processor.io import StreamCallbackfrom org.apache.nifi.processor.io import InputStreamCallbackflowFile = session.get() if (flowFile != None): stream_content = session.read(flowFile) text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8) IOUtils.closeQuietly(stream_content) json_data = json.loads(text_content) session.remove(flowFile) d = {} for x in range(len(json_data['directories'])): d["flowfile{0}".format(x)] = None for i, file in enumerate(d): file = session.create() file = session.putAttribute(file, "setId", str(json_data['setId'])) file = sessi

We have a CDH 5.16 cluster with:- two way Active Directory kerberos & sentry enabled, also Hue backend authentication enabled with AD... We have a "user1" added in AD.... user1 can do kinit & get the tgt. & "user1" is not added in linux user /etc/passwd Problem statment:- user1 logs in to Hue, it clicks to the database icon & since no privileges are given to user1, so it can not see any hive tables...that is expected with sentry authorization..... But...the problem is:- if the user1 clicks HDFS icon in Hue, then user1 is able to see all the data present in hdfs including /user/hive/warehouse/ Why in Hue GUI, user1 is able to read all the data from hdfs including hive warehouse directory...even though sentry is enabled...Isn't the purpose of authorization defeated the moment the "user1" is able to access the data using Hue GUI by clicking HDFS icon, even though we have not provided any privileges to it & also user1 is not added in Linux user in /e

I want to utilise the ConvertRecord 1.23.2 processor to convert text based CEF into JSON using the CEFReader to JsonRecordSetWriter, but somehow keep the original text CEF log as it appears in its text formate as a field called event.original. Is this at all possible within Nifi?

So I have 2 values from different tables that i'm trying to join and aggregate. The problem is that i get results where the math doesn't add up at times- and have tried changing the join\inner join, group by, select distinct, cast- it doesn't seem to work. I need coalesce, as there are null values.Dataset:Table Sales:CurrencyColAColBColCDateFieldAUD208,883{null}{null}'2024-06-30'CAD215,913{null}{null}'2024-06-30'MZN00{null}'2024-06-30'YER0{null}10'2024-06-30'Table OTHER_Sales:CurrencyCol1Col2Col3DateFieldAUD{null}208,88320'2024-06-30'CAD{null}215,9130'2024-06-30'MZN{null}013'2024-06-30'YER{null}00'2024-06-30'Query:SELECT a.currency as currency, SUM((coalesce(a.ColA,0)) + (coalesce(b.Col2))) as sales_checkFROMdb.sales a inner join db.OTHER_sales bON a.currency = b.currencyWHERE a.DateField = '2024-06-30'GROUP BY a.currencyThe incorrect results that I currently get:currencysales_checkAUD-1645761638.790000CAD5855379123.770950MZN0YER0The correct results that I need to get:currenc