Q&A

I have successfully deployed one node quickstart on AWS but are there any docs for a decent sized 3-5 node cluster on AWS to learn cloudera ? I am new to Cloudera and wanted to know what gets installed first etc ..If you have a doc or steps to install Cloudera ...say 3  or 5 node cluster in AWS it would be ideal ...thanksSam

Hi All I am working for a customer to set up a DR environment.  Customer has created a DR as a Separate instance of Cloudera and PROD is a different instance.  The replication is schedule-based as of now.  Can we use Kafka production offset location in DR instance? the customer is working on replication of Hive data and Kafka topics. ? will this work out ?  What is the ideal way to restore a streaming job in DR when PROD is down? Any best practices?  Let me know if you need more details.  Thanks in advanceMuthu   

I'd like to update stats (compute stats) for every table I have.  I have a ton of queries that are running without them.  I personally have access only to Hue for Impala, no console kind of access to server or anything.  I'm able to see status of stats for a table with describe formatted <table>, but it's a major pain to look through, and I have a ton of tables.   Is there any way to get a list of all tables and the status of their stats from Hue?   Thanks!

Hi,  is there some way that nifi could remove the attributes inside the XML tag name? Please advise with some example to help me. Appreciate. Thanks  Example: <particular>   <name>Nicholas</name>   <height>183</height>   <weight>75</weight></particular><particular>   <name>Debbie</name>   <height163</height>   <weight>45</weight></particular> to something like: <particular>   <name>Nicholas</name>   <height unit = "meter">183</height>   <weight unit = "kilogram" unit = "lb">75</weight></particular><particular>   <name>Debbie</name>   <height unit = "meter">163</height>   <weight unit = "kilogram" unit = "lb">45</weight></particular>

Hi all, I have a NiFi flow using DistributedMapCacheServer 1.11.3. I've noticed that this controller service is actually excluded when I create a template from the process group. I see other controller services that are being referenced from the flow in the template, but not this one, presumably because it isn't referenced. How exactly do I get this exported to a template? Any ideas? Regards

I am trying to load a process template in NIFI but it gives below error: org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_0_10.Below is the log from nifi-app.log file:2021-07-26 17:51:01,017 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository2021-07-26 17:51:01,017 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds2021-07-26 17:51:01,379 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@4a6d6308 checkpointed with 0 Records and 0 Swap Files in 86 milliseconds (Stop-the-world time = 39 milliseconds, Clear Edit Logs time = 33 millis), max Transaction ID -12021-07-26 17:51:02,179 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from archive2021-07-26 17:51:21,019 INFO [pool-10-thread-1] o.a.n.c.r.WriteA

Is it possible to mask or greyed out the Token passed in Custom Property (Authentication )? I have one Token received from one API and have to pass it to another request.Issue: If I pass the token in custom property, it will be visible to all users who can access the flowfile. I need to mask these token as these value are displayed in Attribute and Content. Thanks for early response.   

Please help me out in this scenario:Token is returned from InvokeHttp processor. The value of Token is displayed in Attribute and Content.is there any mechanism by which token value can be masked, so that anonymous user can not see the value. Thanks in advance.

I have the above flow and I am successfully listing the files in a given blob storage container. When I feed that to the FetchAzureBlobStorage processor however, the queue starts to fill up but not a single task in the queue ever gets processed. Any ideas regarding why this happens?

Every now and then, one of my processors fills the bulletin board with connection error messages from a FetchSentinelData 1.5 processor. Is there a way to suppress only those messges or route them to an output file to prevent them from filling the bulletin board each time?

Hello - Cloudera Manager is returning the following warning when connecting to MySql hosted in AWS RDS. How can this warning be resolved / suppressed? Log Warning: May 17 12:15:48 ip-10-179-61-80 cm-server[23717]: Mon May 17 12:15:48 BST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL SSby setting useSSL=false, or set useSSL=true and providetruststore for server certificate verification OS Type: [root@ip-10-179-61-233 ~]# cat /etc/redhat-releaseRed Hat Enterprise Linux Server release 7.9 (Maipo) Cloudera Binaries: [root@ip-10-179-61-233 ~]# rpm -qa | grep clouderacloudera-manager-daemons-6.3.1-1466458.el7.x86_64cloudera-ma

Hello Experts, I have configured Git as FlowPersisteceProvider for nifi registry.Tested with test repo and worked fine, but now pointed to my company repo and getting error in push/commit. The error is due to the policy is set in my org git repo, it accepts push/commit from the accout with email as *@myorg.com only.But Even though I configured my credentials (user & PAT) which belongs to my org for Git access from registry, NiFi registry pushes the code to Git with email as "nifiregistry@<hostname>". As its org level git repo ops team cant change this policy, so is there any way I can override this default "nifiregistry@<hostname>" to "my-user@myorg.com" so that git push will not be rejected ? This is how nifi reg is pushing and comitting, Authored and Committed Identity should be changed from default   Tried this : https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.2.0/administering-nifi-registry/content/git-user-configuratio

Use Case: - Delete the partitions from HDFS by Manual - Run MSCK repair- HDFS and partition is in metadata -Not getting sync. Tried multiple times and Not getting sync after upgrading  CDH 6.x to CDH 7.x Any solutions please 

Hi.I am trying to install Ambari server on a three node cluster with one namenode and other two as data nodes.Ambari Version: 2.7.3.0HDP Version: 3.1.0Linux Version: RedHat 7  I am able to connect to the internet through each server and hence using public repository to install ambari server. I have also the necessary authentications from cloudera for ambari and hdp repos. I have followed instructions as mentioned in the following urlhttps://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html. My repository base urls are as followsAmbari: https:///LOGIN:PASSWORD@https://archive.cloudera.com/p/ambari/2.x/2.7.3.0/centos7/HDP: http:///LOGIN:PASSWORD@archive.cloudera.com/p/HDP/3.x/3.1.0.0/centos7HDP Utils: http://LOGIN:PASSWORD@archive.cloudera.com/p/HDP-UTILS/1.1.0.22/repos/centos7I am using the same above urls in Select Version step of installaton wizard and checked on Public Repository as below

I installed NIFI in my system but when I tried to check the status using nifi.sh status under my bin directory it gives the below error:Request you to please suggest a solution for this.Thanks!

There are user directories under hdfs:///user/ When a user `foo` retires, I need to delete a root directory for the user: hdfs:///user/foo However, it occasionally fails because of the snapshottable directories under the user root directory. Checking all the subdirectories whether it's snapshottable, disallowing snapshottable subdirectories , and then deleting the user's root directory doesn't seem to be the best way.(Or if this is the best way, I cannot come up with a simple code..) Is there a command to disallow snapshots for all the subdirectories? How can I effectively delete a directory that might have snapshots?  

From ambari webui (Admin -> Kerberos -> Regenerate Keytabs) when i try to regenerate keytabs it fails on Create Principals step with the following error message2021-07-22 17:39:06,690 - Failed to create principal, HTTP/cnode28.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake@26F5DE01-5E40-4D8A-98BD-A4353B7BF5E3.DATALAKE - Failed to create service principal for HTTP/cnode28.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake@26F5DE01-5E40-4D8A-98BD-A4353B7BF5E3.DATALAKESTDOUT: STDERR: ipa: ERROR: service with name "HTTP/cnode28.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake@26F5DE01-5E40-4D8A-98BD-A4353B7BF5E3.DATALAKE" already existsBellow ambari kerberos config:authentication.kerberos.auth_to_local.rules=DEFAULTauthentication.kerberos.enabled=trueauthentication.kerberos.spnego.keytab.file=/etc/security/keytabs/spnego.service.keytabauthentication.kerberos.spnego.principal=HTTP/enode6.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalakeauthentication.kerberos.user.types=LDAPThanks in advance for y

Is there a way to test MIT kerberos high availability functionality?Any approaches?thanks in advance.

Hello, I have been tasked with POSTing a .rpm file (RedHat package manager) to a server location, this is to automate a manual process that we perform on a weekly basis.   I am able to download the file into NiFi using InvokeHTTP GET. At this stage the 'Content-Type' attribute shows as 'audio/x-pn-realaudio-plugin' , I have read online that this is a common mistake when dealing with .rpm files. I have then used a IdentifyMimeType processor which updates mime type to: application/x-rpm.   In the API documentation is says I require POST /rest/v1/updates as location 'file' as a parameter, see below: I have been given the following python script that achieves what I am trying to do in Python, so far I have not been able to replicate in Nifi. I have tried numerous different configs of the processor, manually changed attributes: file, filename, mime.extension, mime.type, Content-Type, Accept-Encoding. I have tried manually adding attributes from the Python script and referencing

When running sqlline.py, I get the following error:Also the file "/usr/lib/phoenix/phoenix-server.jar" exists on all the hbase master and regionserver in the cluster.  [root@test01 bin]# ./sqlline.pyPicked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueOpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0Setting property: [incremental, false]Setting property: [isolation, TRANSACTION_READ_COMMITTED]issuing: !connect jdbc:phoenix: none none org.apache.phoenix.jdbc.PhoenixDriverConnecting to jdbc:phoenix:SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/lib/phoenix/phoenix-4.7.0.2.6.1.0-129-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.21/05/26 17:18:52 WARN util.NativeCo

I would like to run Hbase major compaction on particular yarn queue. so that way the compaction activity will not impact the other resources in the cluster. Do we have option to so.     

Facing these errors:  PLEASE SOMEONE HELP IT OUT.THANK YOU

I am having a CDH 6.3.2 cluster and would like to backup Hbase table in an incremental fashion.Also need the backup to be done on AWS S3. Please suggest. Thanks in advance. -Aman

hello. I'm using on apache ambari 2.7.5.0 and hdp 3.1.4.0. After installing kerberos on ambari, hdfs, hive, and hbase were successfully authenticated using keytab and then used services. However, if I use yarn to run wordcount, I get an error. In the case of the file permission executing wordcount, the same was confirmed, and in the case of the user name authenticated using keytab, the same was confirmed. When running wordcount using yarn, the same error message as when yarn node -list -all is output is displayed. Below is the error message printed by 'yarn node -list -all.'   21/07/21 14:24:03 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort hadoop1:8050 , LocalHost:localPort hadoop1/192.168.22.168:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ApplicationClientProtocolPBClientImpl.getClusterNodes over null after 24 failover attempts. Tr

I have to create one attribute with the following details:1. There are two attributes are flowing from EvaluateJson Proecessor :   Location1 = $.place1 and Location2 = $.place22. There is one string like folder1 is to be append in these attributes as mentioned below:Place : /folder1/Location1/Location2 I am trying with the below expression:path: ${path: append("folder1/"):append(${Location1}):append('/'):append(${Location2})}Please let me know where I am doing wrong. Early reply will be highly appreciated. thanks & regards