Q&A

I'm trying to use the cdp cli (v. 0.9.106 on my local machine) to get KPI metrics from CDF. I'm using the command `cdp df list-deployment-system-metrics --deployment-crn my-deployment-crn` but struggling to correctly format the required `--metrics-time-period` arg. The documentation isn't helpful here:I've tried a few things like:`--metrics-time-period "2024-03-24T12:00:00Z:2024-03-24T16:00:00Z"``--metrics-time-period LAST_24_HOURS``--metrics-time-period "yesterday"`But they all scream back:An error occurred: No enum constant com.cloudera.dfx.metrics.TimeSpan.TimePeriod.yesterday (Status Code: 400; Error Code: INVALID_ARGUMENT; Service: df; Operation: listDeploymentSystemMetrics;If someone could elaborate on what this command is expecting, or better yet, offer up some examples of successful executions, that'd be great!

We are working on deploying the NiFi Registry platform on kubernetes, however, we are hitting some issues. I've added a diagram of our deployment to help give some context as well. Test Case 1: When proxying NiFi registry with settings as unsecure, the proxying works fine. We are able to connect to https://nifi-registry.company.domain.com and view the registry as needed. Note the HTTPS in the URL still - we are using SSL offloading in our nginx's.Test Case 2: When proxying NiFi registry with settings as secure, we get ERR_TOO_MANY_REDIRECTS. I have attached our nifi-registry.properties file as well. One thing that we found interesting, was that when we tried the below curl request from within the pod we get back a normal responsecurl -k --cacert /opt/nifi-registry/nifi-registry-current/tls/cert-manager/ca.crt --key /opt/nifi-registry/nifi-registry-current/tls/cert-manager/tls.key --cert /opt/nifi-registry/nifi-registry-current/tls/cert-manager/tls.crt

hello everyone. is cloudera shop a product of this company? do you know this?

I am running a CDH 5.5 cluster, and my NameNode edit logs are not getting automatically deleted. How can this issue be fixed?

I come from a communication equipment company and we are using data services from cloud vendors. We mainly use services such as HDFS, Yarn, Hive, Spark, etc. Due to certain specific reasons, some cloud vendors do not provide EMR services in some regions, so we are now planning to deploy CDH6 cluster with K8S.  As we know Hadoop and K8S are two different things, I want to know is it possible to deploy CDH on K8S? Is there any tutorials or best practices? Thanks.

What mount points we need to create before Cloudera installation?

Hello everyone,I have an issue with PutSplunkHTTP processor trying to push some Sysmon logs to my Splunk Indexer.Both NiFi and Splunk are in the same system (Windows).I keep receiving an error:"Error 404 - The requested URL was not found on this server"I use curl from a Linux box and event is added:curl -k http://ipaddress:8088/services/collector/raw?=a98d4602-3a79-4dbc-b733-0977d2aaaec7 -H "Authorization: Splunk <token>" -d '{"event": "Hello, world!", "sourcetype": "manual"}'Using latest NiFi 2.0.0-M2 Release and SplunkAny help would be appreciated.Thank you.

After disabling a service using both the CDP Web UI and the CDPCLI tool, when I list all of the services:$  cdp de list-servicesThe disabled services still appear with  "status": "ClusterDeletionCompleted".Is it possible to completely remove these services from that list so they dissappear after being removed?As I understand, once disabled they cannot be restored, so whats the point of keeping that data?

We have Hadoop cluster with `487` data-nodes machines ( each data-node machine include also the Service node-manager ) , all machines are physical machines ( DELL ) , and OS is RHEL 7.9 version.Each data-node machine have 12 disks, each disk is with size of 12THadoop cluster type installed from HDP packages ( previously was under Horton-works and now under Cloudera )Users are complain about slowness from spark applications that run on data-nodes machinesAnd after investigation we seen the following warning from data-node logs2024-03-18 17:41:30,230 WARN datanode.DataNode (BlockReceiver.java:receivePacket(567)) - Slow BlockReceiver write packet to mirror took 401ms (threshold=300ms), downstream DNs=[172.87.171.24:50010, 172.87.171.23:50010]2024-03-18 17:41:49,795 WARN datanode.DataNode (BlockReceiver.java:receivePacket(567)) - Slow BlockReceiver write packet to mirror took 410ms (threshold=300ms), downstream DNs=[172.87.171.26:50010, 172.87.171.31:50010]2024-03-18 18:06:29,585 WARN data

We are running a HDP Cloudera Cluster version 3.1, kerberized.All the services but HBase are starting with no issue. For HBase, when the service starts, both nodes are starting as Master, then switch to Standby, and finally goes down. telling in the logs that the service doesn´t have owner permissions in an specific HDFS filesystem path, for an user that is not the one starting the service. Even further, eventhough the HDFS before I start the service, at the mentioned filesystem path has everything owned by an user with a name "userA" and a group "groupA", all of the sudden when starting, it changes, for instance, the .tmp directory to be owned by a different user, let´s call it "userB" In Ranger the rules are properly defined, Kerberos is working fine (tickets are valid), and it worth to mention that it started all of the sudden.Herewith the error stack trace (or part of it): 2024-01-11 13:01:05,695 DEBUG [Thread-18] util.FSTableDescriptors: Current path=hdfs://env1/app

Hi,We can access/read/write Ozone etc using aws s3api, but would like to see if we can do the same using curl. We may not have the option of using the awscli package in RHEL.We've tried curl with webhdfs, but that is a bit limited.This almost seems to work:$ curl -vvv -H "Authorization:AWS user@FQDN:aws_key" https://host73.fqdn:9879/s3v/bucket/curl_testBut it seems to get stuck here:> GET /s3v/bucket/curl_test HTTP/1.1> User-Agent: curl/7.29.0> Host: host73.fqdn:9879> Accept: */*> Authorization:AWS user@FQDN:aws_key>It makes the connection to the Ozone S3 gateway, but then just hangs. Does anyone know why it's getting stuck? Log messages seems to be inconsistent, and there's nothing in Ranger. Everything we've tried hasn't worked yet.

I have a flow usecase wherein I use RouteOnAttribute to route to FetchFile.FetchFile processor is then supposed to read a file from a directory and send it to PutSFTP processor.But I observe that file routed from RouteOnAttribute to FetchFile is what is forwarded to PutSFTP.What I want is that FetchFile forwards that file to PutSFTP that it reads from directory.Is it possible?

Hi there ,We are looking for a way to list all keys of redis cache thru nifi. The best solution we found was a groovy script from this article Get list of keys in NiFi DistributedMapCache The script looks good and exactly what we are looking for. But when we try to implement it, we get the following error:We are using the invokescriptprocessor and configure like thisWhat are we missing here 😉 Or another solutions is also more than welcome.Thank you in advanced.Kind regards,Dave

I am trying to kill an existing coordinator job and re-deploy it on oozie. I am running the command -oozie job -run --config siftcuration.properties. Here is my oozie properties file - nameNode=hdfs://ip-<ip>:8020jobTracker=ip-<ip>:8032master=yarnmode=clientqueueName=default#oozie.libpath=${nameNode}/user/oozie/share/liboozie.use.system.libpath=trueoozie.wf.rerun.failnodes=trueruntimeEnvironment=aws_prodappName=SiftCurationappBaseDir=${nameNode}/user/${user.name}/oozieJobs/${appName}oozie.coord.application.path=${appBaseDir}/workflow/coordinator.xmletl.OverwriteOrAppend=Overwritespark.driver.memory=4gspark.executor.memory=4gspark.num.executors=3spark.executor.cores=1etl.default.date.format=yyyy-MM-ddetl.run.date=yesterdaycoordinatorKickDate=2024-03-14 When I run this command, it gives me a coordinator job ID. But when I check the coordinator status, I see that is is still in PREP state. It has been in PREP state for over 4 hours.0000002-240312195311040-oozie-

FetchFile processor has property 'Move Conflict Strategy':Basically its about file conflict in destination folder.But it doesn't have any property for destination folder, and correctly so as this processor is only about pulling a file in nifi system.So why is this property needed?

I have to create a connection from RouteOnAttribute  processor to GetFile processor based on value of an attribute which I have set in processor preceding RouteOnAttribute .But I am unable to do so on NiFi ui. Connection dialogbox jut doesn't appear.What am I missing?

Hello Team, We are on Apache nifi 1.16.0 and openjdk 11.0.20, using parameter contexts to store sensitive parameters for the components of the flow. We have 3 issues when using parameter contexts 1. Issue when updating parameter contexts, it fails to update parameter context and gives below error after I click on Apply. However, when I click on OK and refer back to parameters they are updated sometimes or depends on the stage it fails parameters not updated. I thought its worth mentioning that updating parameter contexts fails at different stages i.e  disabling controller services/Re-enabling effected controller services...2. Inheritance for parameter contexts misses after Nifi restart. Some process groups couldn't read parameters after restart. When checked inheritance for the parameter context, observed few parameters contexts are not inherited. So, we had to manually map the available parameter contexts to be inherited every time after restart. 3. Sometimes

Hello , I am facing issues while executing some queries , I am getting below error for time being and the issue is being resolved autoamtically . ERROR : File 'xxx/yyy/zzz/part-00000-c25df0cd-b7fa-4432-b416-16411dc390d4.c000.snappy.parquet' has an incompatible Parquet schema for column 'proc_mws.mws_xml.zeits'. Column type: TIMESTAMP, Parquet schema: optional byte_array charge [i:12 d:1 r:0] I checked the files but there is no schema mismatch . Can you please suggest me how i can fix the issue. 

Good afternoon,I'm about to deploy CDP 7.1.8 and CP 7.7.1 but seems the licensing is going to take some time, my question is:is the Trial version same as licensed one? can I install Trial version and then update the cluster with the license?Question key: I'm about to build an HA environment, that means to deploy external Postgresql database, can I install Trial but to avoid using the embedded one?Many thanks in advance.

hello cloudera community, I would like to know if the cdh 5.16.2 express version does not have a warning about "Memory Overcommit Validation Threshold"? I did some example tests, configuring more memory than the host has in the services, but we did not receive the alert about "memory overcommitted"

I have a usecase wherein in case of an error I need to make a copy of input file and send it to some FTP location so that it can be assessed.What is idiomatic way of achieving this? 

Hello All,I am using Nifi 2.0I want to update existing Parameter Context using Nifi API.  For that , following API is used https://localhost:8443/nifi-api/parameter-contexts/2ddc0cea-018e-1000-3852-3611d730f705/update-requestsWhere  "2ddc0cea-018e-1000-3852-3611d730f705" is the Id of parameter context. Issue : I tried to update the ParameterContext using https://localhost:8443/nifi-api/parameter-contexts/2ddc0cea-018e-1000-3852-3611d730f705/update-requestsbut its throwing 403 error .I updated the Request with  token (Bearer) but still getting the same error. Any Pointers..??Thanks, Riyaz   

Can I know the working table property for splitting the records as shown below.Input field - 123,"456","INDIA","INDIA",789,"DELHI INDIA, PIN. North INDIA","101","NEW Delhi ","LOCATION"Expected hive output("|" indicates split) - 123 |  "456" |  "INDIA" |  "INDIA" |  789 |  "DELHI INDIA, PIN. North INDIA" |  "101" |  "NEW Delhi " |  "LOCATION" 

Rename of the HADOOP/HUE table nameHi everyone,I need to rename the table name in HUE. i used following command.ALTER TABLE db.table1 RENAME TO db.table2;Please let me know the correct way to rename the table without any issue.Thank you.

Hi all,Nifi is not starting up and below given is the log2024-02-22 23:51:14,673 ERROR [pool-2-thread-2] org.apache.nifi.BootstrapListener Failed to process request from Bootstrap due to java.io.IOException: Received invalid Secret Key for request type GETjava.io.IOException: Received invalid Secret Key for request type GETat org.apache.nifi.BootstrapListener.readRequest(BootstrapListener.java:357)at org.apache.nifi.BootstrapListener$Listener$1.run(BootstrapListener.java:184)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)at java.base/java.lang.Thread.run(Thread.java:1583)