Q&A

Hello,When creating my Cloudera account, I found multiple names for my company, and I unfortunately selected the wrong one. It displays now as non-editable. Please let me know how I can correct it. Thank you.

Hi,Currently we have nifi logrotation in such a way that rotation is happening per day for 100mb of size. So, we have approximately 700 to 800 log files per day. So, my question is if I change the size from 100mb to 500mb /1GB will it affect the performance of Nifi. Or should I rotate logs per hour with 50mb of files size for better performance.Current log rotation is as follows: <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> 

 HiInitially I thought this was fixed in M3 and felt quite the relief for not having to manually rename the scripts folder to bin and restarting nifi twice for that: once to  create the initial extension folders under the work folder, and second after renaming the scripts to bin. It turns out that it works for the very first time however upon subsequent restarts the python extension processors start going into invalid state. When I check the log I found that its still looking for the bin folder:2024-06-12 15:58:22,105 ERROR [Initialize ExcelTableToJson] org.apache.nifi.NiFi An Unknown Error Occurred in Thread VirtualThread[#118,Initialize ExcelTableToJson]/runnable@ForkJoinPool-1-worker-2: java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT]java.lang.RuntimeException: Failed to launch Process for Python Processor [ExcelTableToJson] Version [2.0.0-SNAPSHOT] at org.apache.nifi.py4j.StandardPythonBridge.getProcessF

 Hi @MattWho , @pvillard I just found another bug in the 2.0.0 M3 release when I was trying to view the content of flowfile where it shows the following screen instead:When I check the log I found the following: 2024-06-13 15:10:13,718 ERROR [NiFi Web Server-536] o.a.nifi.web.ContentViewerController Content preparation failed for Content Viewer [/nifi-standard-content-viewer-2.0.0-M3]com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'This': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 6] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2567) at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2593) at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2601) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.j

NiFi canvas (Version 1.14) is  failing to load when launched with the loading indicator being indefinitely shown on the screen . The console has the following two  errors:Uncaught Error: SlickGrid requires a valid container, #controlle-service-types-table does not exist in the DOM.Uncaught TypeError : can not read properties of undefined (reading 'get')Uncaught TypeError : can not read properties of undefined (reading 'top') This issue is observed intermittently  on a particular environment while it works fine in other environments. Thanks,Pratyush   

I have a NiFi cluster consisting of 3 nodes, and I secured the cluster using a single signed certificate for all nodes. However, I am encountering an error that I suspect might be due to using just one certificate.Error Details: - Logs: [Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET /nifi-api/flow/current-user to  nifi01:8443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:    certificate: sha256/*********/GessD8=    DN: CN=nifi01    subjectAltNames: [nifi03,nifi02]2024-06-13 17:34:07,555 WARN [Replicate Request Thread-25] o.a.n.c.c.h.r.ThreadPoolRequestReplicatorjavax.net.ssl.SSLPeerUnverifiedException: Hostname nifi01 not verified:    certificate: sha256/************/GessD8=    DN: CN=nifi01    subjectAltNames: [nifi03,nifi02]        at okht

I'm using "PutDatabaseRecord (NiFi 1.25.0)" for data insert/update purposes in MySql.Imagine, I have a table called "Customer" which has running primary key called "Id" and the another field call "id_from_core". The only mentioned "Id" field as auto generated primary key in "Customer" table. But when data insert through the NiFi flow, I mentioned "id_from_core" field as "Update Key" property in "PutDatabaseRecord".Here is the "Customer" table,CREATE TABLE `Customer` (`id` bigint(20) NOT NULL AUTO_INCREMENT,`created_by` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,`created_date` datetime(0) NULL DEFAULT NULL,`record_status` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,`surname` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,`id_from_core` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULLPRIMARY KEY (`id`) USING BTREE) ENGINE = InnoDB AUTO_INCREMENT = 175

Hi Everyone,I have ranger 1.1 installed.On the ranger ui, upon clicking on edit button for any policy, it is getting stuck on Loading...On creating a new policy and then editing it, it is working fine.Upon debugging a little, found out at that there is some error in the console:This was happening because it was not getting any resource for this policy.I checked the x_policy_resource table, and for the policy id it was returning empty set. But whereas when i created a new policy the x_policy_resource was showing some value.This policy earlier used to open up but now it is getting stuck whenever clicking edit for this policy.Can you help me with why the table is not showing any value or why it is getting stuck on loading.Any idea how can I resolve this?

Hi anyone know where i can download complete documentation for cloudera quick start vm 5.13 ?

Hello,I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

Hi ,I need  cloudera's\community's help with this because I have been struggling with it for some time. we got new VMs that going to be dedicated to Nifi.  We have been using Nifi 2.0.0 M1 on an old physical servers with custom developed  python custom extensions and we did not see any issues. When we moved to the new VM servers ( See spec below) at that time M2 was released and we decided to go with the latest, however once its installed and we got it working we started noticing the CPU utilization Peaks to almost 99-100% ! When I check the Processes under task manager I notice this utilization is taken by Java that is running Nifi:This happens even without running anything on Nifi. I reverted back to M1 release thinking maybe its an M2 issue since M1 did not have this issue on the old physical servers, however the problem persisted even when using the same Java & Python versions. After some troubleshooting and going through process of elimination, I found that

Hello there! I am currently running NiFi 2.0.0 M1 on my host machine and I want to use the NiFi REST API to start and stop processors for my Airflow tasks. Airflow is running inside a Docker container. However, I am facing issues when trying to access the NiFi REST API from within the container.I attempted to access the NiFi API from the Airflow container using https://host.docker.internal:8443/nifi-api but I received an HTTP ERROR 400 with the message "Invalid SNI".I also tried containerizing both NiFi 2.0.0 M1 and Airflow using Docker Compose and access the API endpoint using https://<nifi-container-name>:8443/nifi-api. This also resulted in an HTTP ERROR 400 with the "Invalid SNI" message.I understand this is related to the SSL/TSL handshake and that from NiFi2.x default behaviour does not support accessing NiFi using IP addresses. However, I'm not sure how to resolve this. Please suggest a way to connect to NiFi from inside a container?

  I am running NiFi 1.24 in my local as a docker image in WSL as a standalone mode, trying to implement SiteToSiteProvenanceReportingTask.I was referring to Extracting NiFi Provenance Data using SiteToSitePr... - Cloudera Community - 248469 to set it up. docker has volume mounted the conf directory so that i can change properties.Every time I set the nifi.remote.input.secure to false and rerun the docker, it reset the property file to true. I have tried changing the other properties, they don't reset back to default.I have tried setting the env variable using -e SITE_TO_SITE_SECURE=false and -e NIFI_REMOTE_INPUT_SECURE=false, both have unfortunately not taken effect.docker command  docker run -d --name nifi24 -p 8443:8443 -e SITE_TO_SITE_SECURE=false -v ~/tools/nifi24_conf/conf:/opt/nifi/nifi-current/conf -v ~/tools/nifi24_conf/lib:/opt/nifi/nifi-current/lib -v ~/tools/nifi24_conf/nar_extensions:/opt/nifi/nifi-current/extensions apache/nifi:1.24.0  &n

Hi Everyone can help me please, I started NodeManager via ambari but showing an error like belowSkipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari 

Hi Experts,I need to connect my Impala data source to Metabase (existing metabase impala driver) but still got error like this in the log.  [a75a5199-ad1e-4c89-b92b-2e9b6079df7b] 2024-06-11T19:49:04+07:00 ERROR metabase.api.database Cannot connect to Database,clojure.lang.ExceptionInfo: No method in multimethod 'connection-details->spec' for dispatch value: :impala {:message "No method in multimethod 'connection-details->spec' for dispatch value: :impala"}, at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:165), at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:136), at clojure.lang.RestFn.invoke(RestFn.java:442), at metabase.api.database$test_database_connection.invokeStatic(database.clj:706), at metabase.api.database$test_database_connection.doInvoke(database.clj:696), at clojure.lang.RestFn.invoke(RestFn.java:425), at metabase.api.database$test_connection_details.invokeStatic(database.clj:760), at metabase.api.datab

This is my input JSON,[{"CustomFields": [{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD2","FieldName": "PaymentMode","DataType": "nvarchar","Value": "Cash","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "PaymentMode","Type": "dropdown","Id": 118},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD3","FieldName": "test","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 0,"ViewName": "test","Type": "inputfield","Id": 119},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD4","FieldName": "Reason for Transfer","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "ReasonforTransfer","Type": "dropdown","Id": 120},{"PrimaryId": 598870,"ColumnName": "TRX_ORDER_CUSTOMFIELD5","FieldName": "Remption Mode","DataType": "nvarchar","Value": "","HideFromExternalDistributor": 0,"IsRequired": 1,"ViewName": "RemptionMode","Type": "dropdown","Id": 128},{"PrimaryId": 598870,"ColumnName": "TRX_OR

Hello,We have cloudera 7.4.4 installed on customer network with 8 Kafka broker node. When the application get used heavily the memory of all Kafka node get exhausted(more that 98%) because of the messages get stuck.  We have heap size of 4GB, 8 CPU, 32GB memory on all Kafka nodes.$ ps -ef| grep kafkakafka 508659 16233 0 May29 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm proc_watcher 508665kafka 508665 508659 15 May29 ? 1-09:22:48 /usr/lib/jvm/jre-openjdk/bin/java -Xmx4096M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/amdocs/external/cloudera/kafka/heap/kafka_kafka-KAFKA_BROKER-6bc6dfe85482761059e0f8085a79733f_pid508665.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dcom.sun

Hi, we would like to have a setup in which we manage two separate CDP base clusters (for dev and test) with one Cloudera Manager instance and also share some services between these two clusters.For example, we would like to have one instance of Solr, but have two instances of Ranger (one installed on each cluster) that share the same Solr service (indexes are logically separated like dev_ranger_audits and test_ranger_audits).But it seems that Cloudera Manager doesn't allow installation of Ranger service if Solr is not installed on the same cluster.Does anyone know if this is possible to accomplish somehow? Thanks

Everyone, can help meHow to create keytab krb5.keytab in kerberos ?when I list keytab use " klist -k" show error like belowroot@master1:~# klist -kKeytab name: FILE:/etc/krb5.keytabklist: Key table file '/etc/krb5.keytab' not found while starting keytab scan 

Does Cloudera CDP still support the Sqoop service?

Hi ClouderaBackground: We run nifi as a standalone instance in a docker container in on all our stages, the statemanagement provider used by our instance is the local WAL backed provider.Description: We observed that the flowfiles in front of one of our processor groups stopped to be ingested once in a while and it happens on all our stages without noticeable pattern. The flowfile concurrency policy of the processor group that stopped ingesting data is set to SINGLE_BATCH_PER_NODE and the outbound policy is set to BATCH_OUTPUT. The ingestion should continue since the processor group had already send out every flowfile in it, but it stopped.  There is only one brutal solution from our side(We need to delete the processor group and restore it from nifi registry again) and the occurrence of this issue had impacted our data ingestion.      As you can see in the screenshot that the processor group 'Delete before Insert' has no more flowfile to output but s

In an UpdateAttribute processor I am trying to get the date of yesterday, at midnight in the CET timezone then have it transformed to a specific format in UTC.What I did was : ${now():minus(86400000):format('yyyy-MM-dd\'T\'00:00:00\'Z\'' , 'CET')} but this still gave me the time at midnight in UTC and not CET.What I want basically is if today is 2024-06-11 I want to get : 2024-06-10T22:00:00Zalso, there is this issue with the CEST , If I specify CET is CEST not considered ? I just don't want to have to change my processor every 6 months a year and have something that works no matter what

Hi Everyone can help me,I'm strat NodeManger in ambari but show error "failure to login: for principal: nm/slave1.hadoop.com@HADOOP.COM from keytab /etc/security/keytabs/nm.service.keytab"  for detail like below 2024-06-11 09:30:28,202 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(611)) - NodeManager metrics system shutdown complete.2024-06-11 09:30:28,202 ERROR nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(965)) - Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed NodeManager login at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:488) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:962) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1042)Caused by: org.apache.hadoop.security.KerberosAuthException: failure to lo

Hi,I m trying to put files into AWS S3 browser from Nifi using PutS3Object processor and getting below error.Bucket Name ,  AccessKeyID and SecretAccessKey are used correctly.I m stuck here. Please help.

Hello, I'm quite new to Nifi. I'm using v. 1.24.0I'm working on a project which requires fetching blobs from Azure Blob Storage and putting them into SFTP.  To achieve the desired result, there are two major steps : Get Token Bearer from Authentification Server​Calling a webservice to get SAS key with Token Bearer gotten from previous call.There are two limitations : The SAS Key expires every 1440 minutes. The ListAzureBlobStorage processor doesn't accept any incoming flowfiles. Therefore, I thought about dynamically updating Parameter Context and using the parameter as property value in the ListAzureBlobStorage. Following is the high-level approach of my nifi job : GenerateFlowFile : scheduled to trigger at every  1430 minutes + contains the body of the InvokeHTTP requestInvokeHTTP (Get Bearer Token) : processor to request a bearer token from the authentication