Solved

Hello Exeprts, I am using hive-jdbc jar in my application to connect Hive and Delta (via Thrift server)  from Nifi flow using a custom processor - https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc Now the problem is this has not released any major version in recent times and all the available older versions have velnurables in it.(Even I have tried '4.0.0-beta-1' , I don't see any velnurables with this, but looks like they have removed some internal dependencies, so it did not work for me) So to get rid of velnurbles I am checking to see if there is any alternate option for me to use. Thanks Mahendra

We have the below requirements in the Nifi flowa) Fetch a file from HDFS and transfer it to a Linux Server. I have achieved this using FetchHDFS and PutSFTP NiFi Processors, and it is working fine as expected for a single file.Now, we have a requirement that this HDFS directory may contain multiple files, and all of them need to be transferred to the SFTP server. I tried passing the HDFS directory path in the "HDFS File Name" Property of the FetchHDFS processor; however, it seems to accept only file names. Could you please suggest an alternative processor/method to achieve this?I tried to use ListHDFS -> FetchHDFS -> PutSFTP Actually the HDFS files are output of sqoop jobs. I tried connecting RouterAtribute to ListHDFS, but that connection is not allowed.b) The second requirement is to combine all the files in a specific HDFS directory. Can we achieve this using the MergeContent processor?Could you please provide some inputs?

Hello,My team is about to deploy a clustered scaling NiFi, along with NiFi Registry and Zookeeper. We are deploying on Openshift Kubernetes.We are determining which directories to mount our persistent storage to on both NiFi and the Registry.1. Since we are using Registry, do we need to back anything up on the NiFi pod?2. What are all of the locations we would want mount persistent storage for both NiFi and Registry? 3. What are all of the locations we would want to establish a backup procedure for on NiFi and Registry if we are using local filesystem persistence?4. Do we still need to backup any locations if we use the Git and S3 persistence providers?5. What does the restoration process look like for bothA. local persistence providers onlyB. Git + S3 persistence6. Do we need to worry about Zookeeper at all for backups?Thanks ahead of time!

Installed CFM (Nifi) on Cloudera 6.2 using parcels. I see default 512 M being allocated for Heap size and want to increase it. I did not see nifi.properties file where I could do the same...    Any guidance would be appreciated.. 

I'm new to NiFi. I have a flow that's consuming messages from Kafka and producing them to Redis. The issue I'm experiencing is the MergedContent queue is filling up within seconds, which is slowing down the process of getting the records into Redis. I'm wondering what the recommended approach is to improve things (being new to Redis, I'm aware that I'm most likely missing some obvious things). My flow looks like this: 

Hi there,Maybe some one can help us out. Searching on the web for similar error's doesn't give us a lead on what is going on.Let me sketch out what is happening. Last week we were cleaning up some old buckets with flows from the nifi-registry manually thru the web gui. At some point we received an error that pointed us the registry logs. Actually didn't see anything there. To test if everything is still working, we tried to commit a change to a versioned flow. Which failed. Unfortunately I don't have the error anymore. So decided to restart the registry hoping that this will help. And now it has gotten worse, the registry isn't starting anymore and is showing the error below.2024-03-26 09:40:28,551 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getFlowPersistenceProvider' defined in class path resource [org/apache/nifi/registry/provider/StandardProviderFactory.class]: Bean in

I'm trying to use the cdp cli (v. 0.9.106 on my local machine) to get KPI metrics from CDF. I'm using the command `cdp df list-deployment-system-metrics --deployment-crn my-deployment-crn` but struggling to correctly format the required `--metrics-time-period` arg. The documentation isn't helpful here:I've tried a few things like:`--metrics-time-period "2024-03-24T12:00:00Z:2024-03-24T16:00:00Z"``--metrics-time-period LAST_24_HOURS``--metrics-time-period "yesterday"`But they all scream back:An error occurred: No enum constant com.cloudera.dfx.metrics.TimeSpan.TimePeriod.yesterday (Status Code: 400; Error Code: INVALID_ARGUMENT; Service: df; Operation: listDeploymentSystemMetrics;If someone could elaborate on what this command is expecting, or better yet, offer up some examples of successful executions, that'd be great!

I am running a CDH 5.5 cluster, and my NameNode edit logs are not getting automatically deleted. How can this issue be fixed?

I come from a communication equipment company and we are using data services from cloud vendors. We mainly use services such as HDFS, Yarn, Hive, Spark, etc. Due to certain specific reasons, some cloud vendors do not provide EMR services in some regions, so we are now planning to deploy CDH6 cluster with K8S.  As we know Hadoop and K8S are two different things, I want to know is it possible to deploy CDH on K8S? Is there any tutorials or best practices? Thanks.

What mount points we need to create before Cloudera installation?

Hello everyone,I have an issue with PutSplunkHTTP processor trying to push some Sysmon logs to my Splunk Indexer.Both NiFi and Splunk are in the same system (Windows).I keep receiving an error:"Error 404 - The requested URL was not found on this server"I use curl from a Linux box and event is added:curl -k http://ipaddress:8088/services/collector/raw?=a98d4602-3a79-4dbc-b733-0977d2aaaec7 -H "Authorization: Splunk <token>" -d '{"event": "Hello, world!", "sourcetype": "manual"}'Using latest NiFi 2.0.0-M2 Release and SplunkAny help would be appreciated.Thank you.

After disabling a service using both the CDP Web UI and the CDPCLI tool, when I list all of the services:$  cdp de list-servicesThe disabled services still appear with  "status": "ClusterDeletionCompleted".Is it possible to completely remove these services from that list so they dissappear after being removed?As I understand, once disabled they cannot be restored, so whats the point of keeping that data?

We are running a HDP Cloudera Cluster version 3.1, kerberized.All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service. Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB" In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden.Herewith the error stack trace (or part of it): 2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/app

Hi,We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.We've tried curl with webhdfs, but that is a bit limited.This almost seems to work:$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_testBut it seems to get stuck here:> GET /s3v/bucket/curl_test HTTP/1.1> User-Agent: curl/7.29.0> Host: host73.fqdn:9879> Accept: */*> Authorization:AWS user@FQDN:aws_key>It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

I have a flow usecase wherein I use RouteOnAttribute to route to FetchFile.FetchFile processor is then supposed to read a file from a directory and send it to PutSFTP processor.But I observe that file routed from RouteOnAttribute to FetchFile is what is forwarded to PutSFTP.What I want is that FetchFile forwards that file to PutSFTP that it reads from directory.Is it possible?

FetchFile processor has property 'Move Conflict Strategy':Basically its about file conflict in destination folder.But it doesn't have any property for destination folder, and correctly so as this processor is only about pulling a file in nifi system.So why is this property needed?

I have to create a connection from RouteOnAttribute  processor to GetFile processor based on value of an attribute which I have set in processor preceding RouteOnAttribute .But I am unable to do so on NiFi ui. Connection dialogbox jut doesn't appear.What am I missing?

Hello Team, We are on Apache nifi 1.16.0 and openjdk 11.0.20, using parameter contexts to store sensitive parameters for the components of the flow. We have 3 issues when using parameter contexts 1. Issue when updating parameter contexts, it fails to update parameter context and gives below error after I click on Apply. However, when I click on OK and refer back to parameters they are updated sometimes or depends on the stage it fails parameters not updated. I thought its worth mentioning that updating parameter contexts fails at different stages i.e  disabling controller services/Re-enabling effected controller services...2. Inheritance for parameter contexts misses after Nifi restart. Some process groups couldn't read parameters after restart. When checked inheritance for the parameter context, observed few parameters contexts are not inherited. So, we had to manually map the available parameter contexts to be inherited every time after restart. 3. Sometimes

Hello , I am facing issues while executing some queries , I am getting below error for time being and the issue is being resolved autoamtically . ERROR : File 'xxx/yyy/zzz/part-00000-c25df0cd-b7fa-4432-b416-16411dc390d4.c000.snappy.parquet' has an incompatible Parquet schema for column 'proc_mws.mws_xml.zeits'. Column type: TIMESTAMP, Parquet schema: optional byte_array charge [i:12 d:1 r:0] I checked the files but there is no schema mismatch . Can you please suggest me how i can fix the issue. 

Good afternoon,I'm about to deploy CDP 7.1.8 and CP 7.7.1 but seems the licensing is going to take some time, my question is:is the Trial version same as licensed one? can I install Trial version and then update the cluster with the license?Question key: I'm about to build an HA environment, that means to deploy external Postgresql database, can I install Trial but to avoid using the embedded one?Many thanks in advance.

hello cloudera community, I would like to know if the cdh 5.16.2 express version does not have a warning about "Memory Overcommit Validation Threshold"? I did some example tests, configuring more memory than the host has in the services, but we did not receive the alert about "memory overcommitted"

Hello All,I am using Nifi 2.0I want to update existing Parameter Context using Nifi API.  For that , following API is used https://localhost:8443/nifi-api/parameter-contexts/2ddc0cea-018e-1000-3852-3611d730f705/update-requestsWhere  "2ddc0cea-018e-1000-3852-3611d730f705" is the Id of parameter context. Issue : I tried to update the ParameterContext using https://localhost:8443/nifi-api/parameter-contexts/2ddc0cea-018e-1000-3852-3611d730f705/update-requestsbut its throwing 403 error .I updated the Request with  token (Bearer) but still getting the same error. Any Pointers..??Thanks, Riyaz   

Can I know the working table property for splitting the records as shown below.Input field - 123,"456","INDIA","INDIA",789,"DELHI INDIA, PIN. North INDIA","101","NEW Delhi ","LOCATION"Expected hive output("|" indicates split) - 123 |  "456" |  "INDIA" |  "INDIA" |  789 |  "DELHI INDIA, PIN. North INDIA" |  "101" |  "NEW Delhi " |  "LOCATION" 

Rename of the HADOOP/HUE table nameHi everyone,I need to rename the table name in HUE. i used following command.ALTER TABLE db.table1 RENAME TO db.table2;Please let me know the correct way to rename the table without any issue.Thank you.

Apache NiFi 1.25.0I'm trying to connect SQL Server 2008 R2 using ConvertJSONToSQL processor but got this error :Using this driver :jar : mssql-jdbc-12.6.0.jre11.jarclass : com.microsoft.sqlserver.jdbc.SQLServerDriverI have no problem connect SQL Server 2008 R2 using SSMS. Am I missing some configuration? I did using default https configuration on nifi.properties provided by Apache NiFi when installed.