Q&A

I am fetching files from a particular HDFS path and using a MergeContent processor to merge all the fetched files. Then, I transfer them to an SFTP server using a PutSFTP Processor. There are currently 20 files present in the path, with a total file size of 1.2 GB (This may vary in my production environment, ranging around ~300GB).Initially, my MergeContent processor handled 1GB of file size (14 out of 20 files), merging and transferring to the SFTP server. Later, it picked up the remaining 0.2GB of files(the remaining 6 files) and transferred another file to my SFTP server.I updated the queue limit size to 2GB for the MergeContent processor incoming connection, and then it merged all 20 files and copy a 1.2GB file at once.In another flow, I have a FetchHDFS-->PutSFTP processor, which copies a single file with a size exceeding 100GB to the SFTP server. The Back Pressure Size Threshold is set to 1GB, and it's working. I am wondering why it is not working in the MergeContent Processor

When I try to connect with mssql and kerberos I get the message "java.lang.NoClassDefFoundError: Could not initialize class com.microsoft.sqlserver.jdbc.KerbAuthentication"When I'm trying to set false credentials I get another message so the kerberos itself seems to work.I set the driver location, classname and in the connectionstring itself I use authenticationScheme=JavaKerberos. Jaas Config and kerberos config are also there. How I can get more Information for my problem? What I am missing?

HI,Does Apache storm still have support in cloudera CDP?if it is supported,  please mention methods to install it in CDP Public cloud.I tried installing form external source but facing kerberos issue on connecting HDFS Bolt with secure HDFSplease help on this 

My Nifi server uses PDT timings and i want to list files from a source server which uses UTC timing. While using ListSFTP processor, only files available till current PDT timings are getting listed. (since UTC timing is ahead, many latest files from source end are not getting listed )I don't want to change my Nifi server timezone to UTC as it would effect other existing flows.i dont have an option to change timezone directly in my ListSFTP processor either 😞 How i can fix this and list all the files directly using ListSFTP processor? 

I'm trying to build a Docker container from Cloudera's repo but am getting an authentication error. This is the beginning of my .docker file: FROM docker.repository.cloudera.com/cdsw/engine:13 I have the credentials from the generator and can easily pull-down things via wget but how do I tell the docker engine to use the credentials when it goes to retrieve the engine from the cloudera repo?

Hello,  My ConsumeKafka_2_6 processor reports "Failed to commit ProcessSession after consuming records from Kafka. Will rollback Kafka Offsets".How can I get more details for this error, as to why it is failing to commit ProcessSession?Thanks.

Hi,I have seen an error in the processor PutS3Object, Even though it shows the error, it is successfully putting the object in AWS S3. Looks like the below line in the onTrigger method is causing this. If I comment on this and build, it does not throw any error.ageoffS3Uploads(context, s3, now, bucket);this is the error it is throwing in the UI.else { getLogger().info("Error checking S3 Multipart Upload list for {}: {}", new Object[]{bucket, e.getMessage()});}My Processor config looks like this, I have tried both ways for ObjectKey and Bucket attributes and still see the unnecessary errors. And this as well.Error's in both cases.  

Hello.We want to start the Apache Atlas in our Cloudera Private Cloud but we use in the cluster Python3.9 as a default one and Atlas only support Python2, we want to configure Atlas to use Python2 on our host because we have Python2 also installed but how to configure path to this python2 on Atlas?

I have installed Livy service in Cloudera CDP (7.1.7) cluster, accessing the admin web ui throwing 403 error. See the details below:   

Hey community, is it possible to connect to cloudera data warehouse using trino driver? 

My data flow starts from a single FlowFile produced by a Sqoop job, which then expands into multiple FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). To capture all failure scenarios, I have created a Child processor group with the following processorsInput Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output PortMain Processor Group--------------------RouteOnAttribute --> Above mentioned Child Processor Group --> MergeContent --> downstream processorsThe child processor group is configured with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed (written to the SFTP server).My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file tra

Hi experts,I was wondering if it is possible to see based on the application logs alone that it is trying to find available resources from the cluster, assuming that there are currently other running jobs that are utilizing the cluster resources? If so, how would it look like in the application logs or what do I look for? Thanks,

I encountered an error with the following description after installing YARN Queue Manager: Caused by: org.h2.jdbc.JdbcSQLNonTransientException: Log file error: "/var/lib/hadoop-yarn/config-service.trace.db", cause: "java.io.FileNotFoundException: /var/lib/hadoop-yarn/config-service.trace.db (Permission denied)" [90034-198] at org.h2.message.DbException.getJdbcSQLException(DbException.java:502) at org.h2.message.DbException.getJdbcSQLException(DbException.java:427) ... 38 more Caused by: java.io.FileNotFoundException: /var/lib/hadoop-yarn/config-service.trace.db (Permission denied) I have searched, but I couldn't find anyone experiencing the same error as me. Could you please assist me?😥

 The above file is the Hadoop-policy.xml in that file we have given the properties for enabling ACL for users when I am adding users like yarn mapred hdfs etc so it throws anERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Application application_1699592375401_1753 failed 2 times due to Error launching appattempt_1699592375401_1753_000002. Got exception: org.apache.hadoop.security.authorize.AuthorizationException org. apache. Hadoop.security.authorize.AuthorizationException: User appattempt_1699592375401_1753_000002 (auth: TOKEN) is not authorized for protocol interface org. apache. Hadoop.yarn.api.ContainerManagementProtocolPB: denied by configured ACLNot able to identify which user is running the container, due to which ACL blocking When put * it is working finePlease help me with this

I faced failed Initialize embedded Vault shile installing DataServices.It happened everytime evenif I try install again from start if it happened once. I tried install on system below- Red Hat Enterprise Linux release 8.4 (Ootpa)- Cloudera Manager 7.11.3 (#50275000 built by jenkins on 20240213-1404 git: 14e82e253ab970bfd576e4f80d297769a527df18)- 1.5.2-b886-ecs-1.5.2-b886.p0.46792599 / 1.5.3-b297-ecs-1.5.3-b297.p0.50802651 both I tried stdoutFri Apr 12 11:36:52 KST 2024Running on: cdppvc1.hostname.com (192.168.10.10)JAVA_HOME=/usr/lib/jvm/java-openjdkusing /usr/lib/jvm/java-openjdk as JAVA_HOMEnamespace/vault-system createdhelmchart.helm.cattle.io/vault createdcertificatesigningrequest.certificates.k8s.io/vault-csr createdcertificatesigningrequest.certificates.k8s.io/vault-csr approvedsecret/vault-server-tls createdsecret/ingress-cert createdhelmchart.helm.cattle.io/vault unchangedWait 30 seconds for startup...Timed out waiting for vault to come up stderr++ kube

Hello Experts,I am using Apache Nifi 1.25 now and have a requirement to consume data from a specific offset of a topic partition.I see in Nifi ConsumeKafka processor to configure only "earliest" or "latest" but not specific offset number, is there any way to achieve this ? Any info/suggestion would be appreciated. Thanks,Mahendra

Hi everyone, I would like some clarification. Thank you.CDP recommends disabling the 'tuned' service for performance optimization.Disable the tuned ServiceThe suggestion to disable 'tuned' stems from its capability to transition the CPU into c-state idle mode, potentially leading to decreased performance.Disable the tuned Service on RHEL 8 We would like some clarification about the explanation regarding 'tuned', such as its potential impact on Hadoop. Thank you. For example, THP:Disabling Transparent Hugepages (THP) Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance.For example, swappiness:Setting the vm.swappiness Linux Kernel ParameterOn most systems, vm.swappiness is set to 60 by default. This is not suitable for Hadoop clusters because processes are sometimes swapped even when enough memory is available. Regards

 What is correct?

Hi team!We are currently running a standalone nifi 1.13.2.Our upgrade strategy is 1.13.2 (outside docker) -> 1.25.0 (dockerized) -> 2.0.0-M2 (dockerized)Reference: https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance#MigrationGuidance-Migratingto2.0.0-M2When we upgrade to 1.25.0 (dockerized) all states are still there, but after updating to 2.0.0-M2 (from 1.25.0) we no longer have the processor's states.Our docker volumes are these- /opt/nifi/nifi-current/database_repository:/opt/nifi/nifi-current/database_repository- /opt/nifi/nifi-current/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository- /opt/nifi/nifi-current/content_repository:/opt/nifi/nifi-current/content_repository- /opt/nifi/nifi-current/provenance_repository:/opt/nifi/nifi-current/provenance_repository- /opt/nifi/nifi-current/state:/opt/nifi/nifi-current/state- /opt/nifi/nifi-current/logs:/opt/nifi/nifi-current/logs- /opt/nifi/nifi-current/conf:/opt/nifi/nifi-current/conf- /opt/nifi/nifi

I am new to this platform, and I need your help to get a complete case study (that includes spark MLlib and other tools to classify the data for example in Twitter) to apply in my environment.

hello cloudera community,today the cluster already has thousands of blocks in hdfs, more than 23 million blocks. after configuring the rack in the cluster, the hdfs will recognize the racks and will start moving the blocks to the racks to increase the availability of the blocks or will i have to rebalance the hdfs?

Hi.First of all, I'm not good at English.So please understand my English 🙂  Now, I have no any application on YARN.But, "Used Application Master Resources" isn't empty!!! Especially, when actual applicaion is running, a lot of job become pending state... Is there great guy to help me? ㅠ_ㅠ  Thank you to read my question.

hello cloudera community, we have a problem we are not able to view the execution logs of the jobs that were executed in yarn by the resourcemanager of yarn when we click on logs it presents the error: "Error getting logs at hostname:8041" how can we solve this problem? we are using cdh express 5.16.2

I am observing following error for one of my nifi data flow:o.a.nifi.processors.kite.InferAvroSchema InferAvroSchema[id=9fa25adc-91da-31e3-97ff-d5012c68588c] Failed to infer Avro schema for StandardFlowFileRecord[uuid=a8cf241e-8804-4177-837f-f832657sa596,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1713770110449-11, container=default, section=11], offset=0, length=44],offset=0,name=iSAM_ontEthPort.csv,size=44] due to java.lang.NullPointerException: No content to infer schema: java.lang.NullPointerException: No content to infer schemajava.lang.NullPointerException: No content to infer schema What to make out of it?

I'm using CDH 6.2. I add and change some Yarn config inside Cloudera Manager Web UI (related to yarn shuffle service), something like "NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml"I preview the result and Cloudera shows that some lines will be added to yarn-site.xml. But after restarting everything, I check yarn-site.xml at "/etc/hadoop/conf.cloudera.yarn/yarn-site.xml", I can't find any setting above, even the word "shuffle" doesn't appear anywhere at all? I also check for other files like marped-site.xml or core-site.xml, no luck so far  So where the heck do all the settings above add to? I'm confused. I need the proper yarn-site.xml file with all the settings just like in the Cloudera web UI