Q&A

kindly anyone answer this question i have been asked in an interview i am preparing for interviews

After upgrading from Ambari 2.5.0.3 to 2.6.2.2, ambari-server.log file suddenly start recording following entries: 28 Nov 2020 16:48:52,372 WARN [ambari-action-scheduler] ActionScheduler:316 - Exception receivedjava.lang.RuntimeException: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat7 could not be found at org.apache.ambari.server.actionmanager.ExecutionCommandWrapper.getExecutionCommand(ExecutionCommandWrapper.java:253) at org.apache.ambari.server.actionmanager.ActionScheduler.processInProgressStage(ActionScheduler.java:704) at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:417) at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:310) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.ambari.server.controller.spi.SystemException: Operating System matching redhat7 could not be found at org.apache.ambari.server.state.st

Hello everyone, I am trying to authenticate with  InvokeHttp NiFi Processor to Salesforce by using Bulk API v2 and with OAuth 2.0 : ck_sadchaouche_6-1606816870985.png The InvokeHttp Processor is configured as follow : ck_sadchaouche_7-1606817038511.png The Remote URL is made by using this syntax:   https://< mydomain >.salesforce.com/services/oauth2/token?grant_type=password&client_id=<client_id>&client_secret=<client_secret>&username=<username>&password=<password + key> As we are using the HTTPS protocol, we add a “SSL Context Service” by setting a new “SSL Context Controller Service”: ck_sadchaouche_8-1606817459585.png And the Post request is already tested and worked with Postman tool, by responding with an authentication token. The blocking point for this use case is that, with InvokeHTTP NIFI Processor we have not been able to get a response from the Bulk API ( the first image show that we never receive any response from Sale

I have 20 CSV files and I need to convert these files into parquet format before loading them to S3. I prepared schemas for 20 files(different schema for each file). For this to work I need to route all these files using route on attribute and process. This works but it create flow with 20 convert record processors. I want to know is there a way that I can store all these 20 schemas and I can use them according to filename with single convert record processor. Thanks.

https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ch04s03s05.htmlI read above documentaion and I don't understand how to input username and password.I tried to input my cloudera username and password but it resulted 403 forbiddenhttps://username:password@archive.cloudera.com/p/ambari/ubuntu18/2.x/updates/2.7.5.0/ambari.list Please give me username and password.If cannot, please tell me how can I contact cloudera using e-mail.

Hi, I was trying to install Cloudera Manager 6 with CDH 6.3.0 for a 3 node cluster. I have downloaded and installed cloudera-manager-installer.bin in CentOS6 and installed cloudera server and agent. Then I have logged into CM using admin:admin to configure cluster and I stuck here at Add Cluster Configuration step as Upload Oozie ShareLib step failed with no log4j2 file found. PFB logs. Upload Oozie ShareLib Also, I have checked in /var/log/cloudera-scm-server/cloudera-scm-server.log and found the same error message there as well. Is this log4j2 something that I need to install manually before configuring cluster ? If so, please suggest me on how to do that. stdout and stderr for reference. PFB!! stdout:Mon Nov 30 08:10:40 UTC 2020JAVA_HOME=/usr/java/jdk1.8.0_181-clouderausing 6 as CDH_VERSIONCONF_DIR=/var/run/cloudera-scm-agent/process/61-oozie-OOZIE-SERVER-upload-sharelibCMF_CONF_DIR=Found Hadoop that supports Erasure Coding. Trying to disable

Hi everyone, Ambari :- 2.7.4HDP :- 3.1.4.0 I had done the ambari installation using local repositories with no Internet access for Ambari and HDP as mentioned in the official docs by creating directory structure /var/www/html and, untarred all the HDP and Ambari files into /var/www/html/hdp and /var/www/html respectively, as mentioned in the docs. :- [ Ref :- https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/getting_started_setting_up_a_local_repository.html https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/setting_up_a_local_repository_with_no_internet_access.html ] Then, I used python's Simple HTTP Server to host my local repository by running the command `python -m SimpleHTTPServer 2000` from inside /var/www/ instead of using Apache HTTP server as mentioned in the docs, to host my repositories in html directory on port 2000. Installation was done and services are running fin

Hi,    I have one question about the impact of change exist rack id.   we have a CDH 5 cluster with 6 data nodes(replication factor is 3), they are belong one rack rack0. and we add 1 data node to this cluster, but it is added rack rack1 by mistake.    now we met issue "Under-Replicated Blocks" due to only 1 node in rack1, it is need replicate 1 of 3 to node in rack1. but the disk capacity is not enough to storage 1 replica of all files.   Now, we have decomissioned the node in rack1, and want to change the node back to rack0. my question is what is the steps to change the node back to rack0 safely? any suggestions?   Thanks!  

reasons behind name node failure?

Hello, I would like to know if it's possible to revert or reinstall an older version of Cloudera Manager. I have the following case: the current CDH and Cloudera Manager are both running on version 5.16.2. I have a request to upgrade only Cloudera Manager to 6.3.3. Now, I would like to know if it's possible to revert/reinstall to CM 5.16.2 in case the CM 6.3.3 version will not work as expected on CDH 5.16.2. Thank you,  

Hi All,I'm using Cloudera Express 6.3.0. I'm trying to configure alert emails to be received by a list of users. While configuring properties, it is asking for mail server user name & password. We have SMTP enabled and we are triggering emails with sendmail command on the nodes where Cloudera is installed. But, here why we need to provide the mail server user name & password for receiving the alert emails on our mail id. I have found a solution for this in Cloudera community (https://community.cloudera.com/t5/Support-Questions/Why-does-Alert-Publisher-require-mail-server-login-amp/m-p/41003#M36254), but it is not working for my version of Cloudera. Does any know a turnaround for this.Please refer to the link above for the solution provided & please also see the picture attached for better understanding. When i click on Send Test Alert, i see this pop up & the email is not triggering Mail Server UN & PWD

Hi,   When i try to 'Generate Missing Credentials' in Cloudera Manager:  CM >Administration>Secuity>Kerberos Credentials , I get the following error 2020-11-20 00:13:31,793 INFO GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: Generating kerberos credentials using custom script /etc/hadoop/conf/keytabgen.sh2020-11-20 00:13:31,794 INFO GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: Creating credentials for roleId 1322020-11-20 00:13:31,794 INFO GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: Generating new keytab for spark/clouderadp02.ibm.cloud@IBM.CLOUD2020-11-20 00:13:31,805 ERROR GenerateCredentials-0:com.cloudera.cmf.security.GenerateCredentialsCommand: unable to create credential for role 132 due to:/etc/hadoop/conf/keytabgen.sh failed with exit code 1 and output of <<cp: cannot stat ‘/etc/hadoop/conf/keytabs/spark_clouderadp02.ibm.cloud@IBM.CLOUD.keytab’: No such file or d

Hi,I would like to run the oozie job only on the weekends.Can someone help me on how to configure this in the coordinator.xml?Thanks in advance

Hi,I am trying to register an environment using AWS but I get this error:  Data Lake creation failed: Operation timed out. Template install timed out with this command id: 1546333784  When I log into AWS I can see that three instances were provisioned1. cdp-env7-freeipa-10778-master02. datalake7-79244-master3. datalake7-79244-idbroker Where can I find more logs? What am I doing wrong? Thanks, Marcin

I have Nifi flow created (ListSFTP-->FetchSFTP-->PutHDFS) and have parameters like sourcedir,hostname,HDFS path ,etc..I would like to pass the values dynamically instead of maintaining the parameter values in parameter context so that if there is any value changes , i can do in properties file itself. I have tried to use lookup attribute processor but we do not have CouchDB in the cluster . Please suggest option to pass dynamic values to this parameters. Like in Java program , we pass properties files to maintain default values.@Nifi @NifiDev 

I have to retrieve information from all columns of a row based on a given value.the data isfname - Johnlname - Andrewssnumber - 008degree - MIT I have this query that returns only the degree column:scan 'CWRK', {COLUMNS => 'STUDENT', FILTER => "ValueFilter ( =, 'binaryprefix:MIT')"} this gives the result:student:008             column=STUDENT:degree, timestamp = ......... , value=MITI want to run a query that returns values for all 4 attributes (fname, lname, snumber, degree) for the row-number student:008 as that is the row that has the MIT value for degree key.Can this be done?

Hello, I am a newbie to Cloudera, so give me some slackI  assigned 4 cores and 8GB to my Cloudera VM in VMware players. It seemed to be working fine. After several suspends, restarts, and queries, the system becomes really slow and I cannot pinpoint why. I am not doing anything special, simple queries to small 100k tables. I even reinstalled the VM, restoring its speed, but with later degradation.  I was never able to get Cloudera Manager working, it just stays there.How can I diagnose what is making my system so slow? I am running this in an i7 8th gen with 16GB RAMNote: I usually get a message of an unresponsive script like , "Script: http://quickstart.cloudera:888…ery.nicescroll.7958d4c833ae.js:1990"  Thank you for your inputs.Cheers.

Dear community,We have a NiFi cluster installation of 3 nodes with 30GB of RAM and 15 CPUs each. The NiFi version that we are using is the 1.7.1 and we are running 43.000 NiFi processors in the cluster.The question is if there is a known limitation of the amount of processors that we can run in the same cluster considering that we can scale horizontally and also add more CPU and memory.Also if somebody is running more processors in the same cluster or have experience in this topic that would be a great piece of information.Thanks a lot for any advice on thisBest Regards!Juan Pablo

Hello, I am unable to know the size of the hive table. Actually it is a view specifically which we wanted to know the size of. I have tried by running commands such as hdfs dfs -du -s -h /path/to/table, but the view can't be found in the location. beeline> describe formatted/extended table name; Also does not give the output  I also tried with ANALYZE but no output as well. Can some one tell me how to know the size of the view table. Since we have created view of the different table in database.  And we want to know the size of the particular table.

Have a list of about 100+ SQL  Count Queries to run against a Hive Data Table, Looking for the most efficient way to run these queries. Queries are accessed at runtime as a list of queries stored in another Hive Table, generated by a different process.     Queries like these each with a different where clause, where clauses are complex: 1. Select count(1) as count1 from MyTable where (... complex where clause here...) 2. Select Count(1) as Count1 from MyTable where (... where clause here ...) 3. etc..   Environment: Cloudera CDH 6.2  

I've enabled Auto-TLS on an existing CDP 7.1.3 cluster as instructed in https://blog.cloudera.com/auto-tls-in-cloudera-data-platform-data-center/ option 2b.Initially after completing the wizard and restarting CM agent and servers agent heartbeat was lost due to missing agent certificates from truststore. I was able to fix this by adding hosts to trustore "Cloudera Manager TLS/SSL Client Trust Store File" /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks with Java Keytool. However now I'm observing some oddities in services. - YARN, Kafka seem to report in their Role Log at server startup (Error 403 Broken pipe (Write failed) the server declined access to the page or resource.) - If I try to access any logs in the cluster via CM GUI (Clusters -> Service -> Instances -> Any role -> Log files -> Any log -> Download full log ) I get the following message "HTTP ERROR 403 Problem accessing /cmf/process/all/logs/download. Reason: Receiv

Hi,I have a cluster running Hbase with 1 Master and 6 RS. Recently I noticed that Hbase commands are sort of queuing up and possibly hung(RUNNABLE state) for one table.When checking the state of the table, I an see that the table is disabled. The command executed to enable the table failed when executing from the Hbase shell stating that process id xxx(previous truncate command) is already running. In the HBase UI, I can see both the truncate and enable commands(shown under Procedures) is in RUNNABLE state.I tried the kill procedure command on the truncate command but it returns false, indicating the process cannot be killedI have data on other tables and scan commands on these tables work fine.What might be the issue here, how can I kill the command running at Hbase and get the table back to working?Any Help is much appreciated.Regards,ThomasOn checking the Master log, I can see the following Warnings:2020-01-08 11:03:28,168 WARN  [RegionOpenAndInitThread-buckets_2-10] ipc.Client

Does Major compaction in Hbase lead to regions merging or the regions are merged only manually by applying merge command?Please help @Manoj69 @jsensharma 

Hello!The maven artifacts list for 7.1.3 is available at :https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdppvc-runtime-maven-7.1.3.html Is the same list for 7.1.4 available?  Is it the same list?  I am unable to find this list for CDP 7.1.4. Thanks.

Hello,On our data node, we are increasing getting alerts related to Data Node Pause Duration. So far, this is happening on a single data node out of nine data nodes.Following is the error captured from DN logs 2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms) Current Java Heap Size of Data Node in Bytes is at 6GB CM / CDH – 5.16.x Any help is appreciated. RegardsAmn