Solved

Hi ,In my profile, I could see company as "London Stock Exchange Group PLC" and it needs to be change to "Refinitiv US LLC". Can you please refer my teammate details (Srikanth Macha) who is in same company and do the needful.Thanks, Lakshmi

Hello, I want to migrate data from MsSQL Server to Oracle Database. I added ConnectionPools for both of. I am trying to export data andimport data ,but i cant. Which processor i need to use and how ? I tried to fecth data with ExecuteSQL but it is fetching as Avro. Than i wanted to convert it to json but i didn't. Because i cant configure the ConvertRecord processor.

hello cloudera community, could you help us with a problem regarding the /var/log directory at certain times (we couldn't identify the right moment) the /var/log directory is changing the permissions to 750 and with that the CDH 6.3.x cluster services is failing because it can't write the logs. Have you ever had a problem like this? did you manage to solve it? we had to leave a crontab running every 5 minutes to always change it to 755. ps: selinux is disabledps: firewall is disabled

Hi,  i want to ask about nifi login, i can login but cant acces nifi ui in nifi-user log success. this my nifi-user.log and error in nifi ui  thank you 

Hi,I want to put the alert name in the subject of the email alert I followed the example in https://github.com/apache/ambari/blob/branch-2.1/ambari-server/src/main/resources/alert-templates.xml but the following does not fill in the  variables and I just the the variable names in the email.<![CDATA[Ambari: [$alert.getAlertState()]-[$alert.getAlertName()]-[$alert.getAlertText()]]]>If I use the template as it is, the email populates with the summary variables, ie:<![CDATA[Alert Summary: OK[$summary.getOkCount()], Warning[$summary.getWarningCount()], Critical[$summary.getCriticalCount()], Unknown[$summary.getUnknownCount()]]]>Is there a trick to get the subject to show the $alert variables to the subject?Thanks in advance! Eric 

We are facing no applicable policies issue on nifi ui ,then we renew Kerberos tickit it works ,but after that we got know that cluster status is disconnectedAnd git below popup.       Please help in this, nifi-app.log not showing any errorexcept one warn, failed to send heartbeat to nifi.cluster.protocal.protocalexception, PKIX validation failed 

Hi,I am in need for some sort of alerting mechanism to monitor long running CDE jobs, basically we use Airflow to call /run Spark DBT jobs I am looking if there is a way to monitor / alert if any jobs cross certain threshold time limit.Would appreciate if anyone has implemented it / guide in this direction.Thanks Wert

While executing the spark application in yarn i was facing the following issue twice or thrice a day these days. ERROR MicroBatchExecution:91 - Query [id = 0ec965b7-5dda-43be-940c-3ec8672bcd5c, runId = 17c15719-ab20-4488-ba37-ccf6a6ca27e1] terminated with errororg.apache.spark.SparkException: Job aborted due to stage failure: ResultStage 185575 (start at DeviceLocationDataListener.scala:148) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: Stream is corruptedat org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:554)at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:470)at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:64)at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)at sc

Hello,Complete noob here.I have a file, and have created an iceberg table over it (via Hue). However, I can't select any data. How do I set the schema up correctly?CREATE TABLE iceberg_test_1 (month STRING, day STRING, time STRING, host STRING, source STRING, body STRING) PARTITIONED BY(event_day INT, event_month INT, event_year INT) STORED BY ICEBERG LOCATION 'ofs://ozone/user/logs/test';The file is:Jun 4 08:26:57 host smartd[2657]: Device: /dev/bus/0 [megaraid_disk_05], SMART Failure: DATA CHANNEL IMPENDING FAILURE GENERAL HARD DRIVE FAILURE

Hello,When creating my Cloudera account, I found multiple names for my company, and I unfortunately selected the wrong one. It displays now as non-editable. Please let me know how I can correct it. Thank you.

 HiInitially I thought this was fixed in M3 and felt quite the relief for not having to manually rename the scripts folder to bin and restarting nifi twice for that: once to  create the initial extension folders under the work folder, and second after renaming the scripts to bin. It turns out that it works for the very first time however upon subsequent restarts the python extension processors start going into invalid state. When I check the log I found that its still looking for the bin folder:2024-06-12 15:58:22,105 ERROR [Initialize ExcelTableToJson] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#118,Initialize ExcelTableToJson]/runnable@ForkJoinPool-1-worker-2: java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT] at org.apache.nifi.py4j.StandardPythonBridge.getProcessF

 Hi @MattWho , @pvillard I just found another bug in the 2.0.0 M3 release when I was trying to view the content of flowfile where it shows the following screen instead:When I check the log I found the following: 2024-06-13 15:10:13,718 ERROR [NiFi Web Server-536] o.a.nifi.web.ContentViewerController Content preparation failed for Content Viewer [/nifi-standard-content-viewer-2.0.0-M3]com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'This': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 6] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2567) at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2593) at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2601) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.j

I have a NiFi cluster consisting of 3 nodes, and I secured the cluster using a single signed certificate for all nodes. However, I am encountering an error that I suspect might be due to using just one certificate.Error Details: - Logs: [Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to  nifi01:8443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:    certificate: sha256/*********/GessD8=    DN: CN=nifi01    subjectAltNames: [nifi03,nifi02]2024-06-13 17:34:07,555 WARN [Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicatorjavax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:    certificate: sha256/************/GessD8=    DN: CN=nifi01    subjectAltNames: [nifi03,nifi02]        at okht

I'm using "PutDatabaseRecord (NiFi 1.25.0)" for data insert/update purposes in MySql.Imagine, I have a table called "Customer" which has running primary key called "Id" and the another field call "id_from_core". The only mentioned "Id" field as auto generated primary key in "Customer" table. But when data insert through the NiFi flow, I mentioned "id_from_core" field as "Update Key" property in "PutDatabaseRecord".Here is the "Customer" table,CREATE TABLE `Customer` (`id` bigint(20) NOT NULL AUTO_INCREMENT,`created_by` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,`created_date` datetime(0) NULL DEFAULT NULL,`record_status` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,`surname` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,`id_from_core` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULLPRIMARY KEY (`id`) USING BTREE) ENGINE = InnoDB AUTO_INCREMENT = 175

Hi ,I need  cloudera's\community's help with this because I have been struggling with it for some time. we got new VMs that going to be dedicated to Nifi.  We have been using Nifi 2.0.0 M1 on an old physical servers with custom developed  python custom extensions and we did not see any issues. When we moved to the new VM servers ( See spec below) at that time M2 was released and we decided to go with the latest, however once its installed and we got it working we started noticing the CPU utilization Peaks to almost 99-100% ! When I check the Processes under task manager I notice this utilization is taken by Java that is running Nifi:This happens even without running anything on Nifi. I reverted back to M1 release thinking maybe its an M2 issue since M1 did not have this issue on the old physical servers, however the problem persisted even when using the same Java & Python versions. After some troubleshooting and going through process of elimination, I found that

Hello there! I am currently running NiFi 2.0.0 M1 on my host machine and I want to use the NiFi REST API to start and stop processors for my Airflow tasks. Airflow is running inside a Docker container. However, I am facing issues when trying to access the NiFi REST API from within the container.I attempted to access the NiFi API from the Airflow container using https://host.docker.internal:8443/nifi-api but I received an HTTP ERROR 400 with the message "Invalid SNI".I also tried containerizing both NiFi 2.0.0 M1 and Airflow using Docker Compose and access the API endpoint using https://<nifi-container-name>:8443/nifi-api. This also resulted in an HTTP ERROR 400 with the "Invalid SNI" message.I understand this is related to the SSL/TSL handshake and that from NiFi2.x default behaviour does not support accessing NiFi using IP addresses. However, I'm not sure how to resolve this. Please suggest a way to connect to NiFi from inside a container?

  I am running NiFi 1.24 in my local as a docker image in WSL as a standalone mode, trying to implement SiteToSiteProvenanceReportingTask.I was referring to Extracting NiFi Provenance Data using SiteToSitePr... - Cloudera Community - 248469 to set it up. docker has volume mounted the conf directory so that i can change properties.Every time I set the nifi.remote.input.secure to false and rerun the docker, it reset the property file to true. I have tried changing the other properties, they don't reset back to default.I have tried setting the env variable using -e SITE_TO_SITE_SECURE=false and -e NIFI_REMOTE_INPUT_SECURE=false, both have unfortunately not taken effect.docker command  docker run -d --name nifi24 -p 8443:8443 -e SITE_TO_SITE_SECURE=false -v ~/tools/nifi24_conf/conf:/opt/nifi/nifi-current/conf -v ~/tools/nifi24_conf/lib:/opt/nifi/nifi-current/lib -v ~/tools/nifi24_conf/nar_extensions:/opt/nifi/nifi-current/extensions apache/nifi:1.24.0  &n

This is my input JSON,[{"CustomFields": [{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD2","FieldName": "PaymentMode","DataType": "nvarchar","Value": "Cash","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "PaymentMode","Type": "dropdown","Id": 118},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD3","FieldName": "test","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "test","Type": "inputfield","Id": 119},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD4","FieldName": "Reason for Transfer","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "ReasonforTransfer","Type": "dropdown","Id": 120},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD5","FieldName": "Remption Mode","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "RemptionMode","Type": "dropdown","Id": 128},{"PrimaryId": 598870,"ColumnName": "TRX_OR

Does Cloudera CDP still support the Sqoop service?

Hi ClouderaBackground: We run nifi as a standalone instance in a docker container in on all our stages, the statemanagement provider used by our instance is the local WAL backed provider.Description: We observed that the flowfiles in front of one of our processor groups stopped to be ingested once in a while and it happens on all our stages without noticeable pattern. The flowfile concurrency policy of the processor group that stopped ingesting data is set to SINGLE_BATCH_PER_NODE and the outbound policy is set to BATCH_OUTPUT. The ingestion should continue since the processor group had already send out every flowfile in it, but it stopped.  There is only one brutal solution from our side(We need to delete the processor group and restore it from nifi registry again) and the occurrence of this issue had impacted our data ingestion.      As you can see in the screenshot that the processor group 'Delete before Insert' has no more flowfile to output but s

In an UpdateAttribute processor I am trying to get the date of yesterday, at midnight in the CET timezone then have it transformed to a specific format in UTC.What I did was : ${now():minus(86400000):format('yyyy-MM-dd\'T\'00:00:00\'Z\'' , 'CET')} but this still gave me the time at midnight in UTC and not CET.What I want basically is if today is 2024-06-11 I want to get : 2024-06-10T22:00:00Zalso, there is this issue with the CEST , If I specify CET is CEST not considered ? I just don't want to have to change my processor every 6 months a year and have something that works no matter what

The new cumulative hotfix 7 for Cloudera Runtimes 7.1.9 points to CHF6 instead 😉https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/runtime-release-notes/topics/chf7-pvcb-overview.html(probably a copy/paste issue 😉 )

Hello,We have a 3 node cluster where one of the nodes is only 64 cores 128 GB RAM and other two machines are each 128 cores 128 GB RAM. I am using Round Robin load balancing in the flow as Listfile --> LB --> FetchFile, as a result of which load is getting divided (almost) uniformly among all the machines, which is expected behavior. However, the load average on smaller machine is exceeding 64 (this machine is sweating!) whereas the load average on other two machines is almost 50.So my question is there a way in NiFi to distribute load in such a way that the load on the smaller machine can be maintained to stay less than 64, and give the other two machines some more work to do?I tried using DistributeLoad processor, but not sure which strategy to use because Round robin would distribute load equally, next available would distribute when the next node is available. However, how do I configure DistributeLoad such that it divides load equally among two bigger machines but lesser loa

We have 2 instances of nifi-registry with git as flowProvider, when we make some changes and commit from nifi instances, sometimes appears everything is ok, sometimes appears "there is a new version of the flow". I think one of the nifi-registry instances it's not synced.¿How can we force the sync between multiple nifi-registry instances?

Example:Flowfile databranches,name,id,acc,serialNoparent1,abc,123,1234567788,1child1,eee,345child2,rrr,678parent2,mno,786,97865662762,2child1,ttt,764child2,ryy,456output:parent1,abc,123,1234567788,1child1,eee,345,1234567788,1child2,rrr,678,1234567788,1parent2,mno,786,97865662762,2child1,ttt,764,97865662762,2child2,ryy,456,97865662762,2I have to add the parent last two columns details to Childs of the same parent like parent1 last column details to child1, child2 rows in the flow file.Please suggest a possible way to use the processor.