Solved

Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:   If I try and run any of the examples ...they fail. [hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4WARNING: Use "yarn jar" to launch YARN applications.Number of Maps = 4Samples per Map example.com= 4Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Starting Job20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_000920/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 420/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:420/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address

Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//__ / .__/\_,_/_/ /_/\_\ version 2.3.0.2.6.5.0-292/_/Using Python version 2.7.14 (default, Dec 7 2017 17:05:42)SparkSession available as 'spark'.>>>>>> df=spark.sql('select * from sws_dev.vw_dlx_rpr_ordr_dtl_base limit 1').show()[Stage 0:=====================> (18 + 28) / 46]20/03/03 07:01:08 ERROR DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file /tmp/blockmgr-c5bcbbe3-8da0-44a0-8025-1b183c81d532/03/temp_shuffle_280c5065-f954-4ec8-b3d0-7c1f5c18b581java.io.FileNotFoundException: /tmp/blockmgr-c5bcbbe3-8da0-44a0-8025-1b183c81d532/03/temp_shuffle_280c5065-f954-4ec8-b3d0-7c1f5c18b581 (Too many open files)at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270)at java.io.FileOutputStream.<init>(FileOutputStream.java:213)at org.apache.spark.storage.DiskBlockObjectWriter$$anonfun$revertPartialWritesAndClose$2.apply$mcV$sp(D

I have triggered dataflow and its running everyday. For 1st run, ListFile is able to list files and pass it on to next starge FetchFile. But when i touch file in my input directory, ListFile is unable to list those files. 

Hi,   Which property in Cloudera manager should i update to toggle between the cluster and standalone mode for the nifi cluster? Also, the location of the nifi.properties in the CFM.... Its not under /var/lib/nifi/

We are loading data from oracle table to Hive using NiFi. At the time of loading we will need to do some reference table lookup.(e.g Customer data has Country_code = IN in Oracle, while loading Customer data in HDFS,lookup using Country_code=In will be looked up in reference table and new column with value Country_detail = INDIA will be populated in Customer data).   Could you please guide me how it can be done?

Dear all.   I created one hive temp tables as below - but how can we identified that tables is temp tables or not.       CREATE temporary TABLE IF NOT EXISTS employee ( eid int, name String,salary String, destination String)COMMENT ‘Employee details’ROW FORMAT DELIMITEDFIELDS TERMINATED BY ‘\t’LINES TERMINATED BY ‘\n’STORED AS TEXTFILE;     i used below command to described but not getting any info-     describe formatted employee;col_name data_type comment1 # col_name data_type comment 2 NULL NULL3 eid int 4 name string 5 salary string 6 destination string 7 NULL NULL8 # Detailed Table Information NULL NULL9 Database: h7 NULL10 OwnerType: USER NULL11 Owner: **** NULL12 CreateTime: Tue Mar 03 08:50:28 EST 2020 NULL13 LastAccessTime: UNKNOWN NULL14 Retention: 0 NULL15 Locati

Hi,I'm kinda a beginner in using REST API and i want to create multiple entities at the same time.Watching documentation, you can do it with v2/ENTITY/BULK, however i don't understand how the json file have to be formatted.When I create ONE entity, I do it with v2/ENTITY i just do it like this:curl -s -i -X POST -H 'Content-Type: application/json' 'Accept: application/json' -u admin:$MDP_ATLAS "$ATLAS_BASE_URL/v2/entity" -d '{"entities":{ "typeName": "Referenceable", "attributes": {"qualifiedName":"some.qualified.name1"}}}' But with BULK the doc is saying to format the file like this:{ "entities" : [ { "classifications" : [ { "entityGuid" : "...", "entityStatus" : "ACTIVE", "propagate" : true, "removePropagationsOnEntityDelete" : true, "validityPeriods" : [ { }, { } ], "attributes" : { "property1" : { }, "property2" : { } }, "typeName" : "..." }, { "entityGuid" : "...", "entityStatus" : "ACTIVE", "propagat

Hi, we have a couple of brokers that need to be life-cycled out.  We have reassigned all topics to the other brokers in the cluster and Stopped the Kafka role on that host.  To totally remove it, is it as simple as choosing the Instance and then from the Action Menu -> Delete?  CM -> Kafka -> Instances -> (choose instance) -> Actions Menu -> Delete I don't see a "Decommission" option like with a Datanode.  Anything need to be done in ZooKeeper or does the "Delete" option handle that?  Thank you!

Hi,    I'm planning to bring the Kafka as a Service. I would need any client or command-line tools to add a cluster. Right now, I'm able to add from Cloudera Manager, but I would want to do it from the CLI. Do we have any API or any leads to get this done?

Hello, After 13th January, all my store streamsets stopped storing data with an error similar like this below. We are collecting data from ETW and IIS log data, that is first collect, then transform, and finally store. This last step is falling. Can someone help?   KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string environment="prod", string groupid="iis")' not found ----------------------------------- Error Stack Trace KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string environment="prod", string groupid="iis")' not found com.streamsets.pipeline.api.base.OnRecordErrorException: KUDU_15 - Row key '(string id="3d755d2d-949e-4fba-9925-63f658b44773", int32 year=2020, int32 month=2, int32 day=21, string host="vipe-exch03", string type="etw", string en

Hi All, I would like know regarding how can i verify or validate the SSL certificates expiry for all the hadoop components installed on my HDP cluster. Would really appreciate getting any help on methods to check the SSL certificates. Thanks

Hi everyone So I'm an intern and it's my first time using apache NiFi. I wonder if there is any way to create a whole apache NiFi project using java: i.e: I have a button in java I need a code that able me to when I press this button a processor will be dragged and dropped in apache NiFi UI !!Is it possible ?? if it is, how can we make it?

Hello,  I'm trying to connect from beeline/jdbc to ssl enabled Hive (3.1.0) via ssl enabled Knox ( 1.0.0 ) but without success.   1) The same connection works perfectly fine with ssl disabled on Hive side ( Ambari -> Hive -> Configs -> Settings -> Use SSL = false )     beeline -u "jdbc:hive2://knox_gateway:8443/;ssl=true;sslTrustStore=user_keystore.jks;trustStorePassword=password;transportMode=http;httpPath=gateway/default/hive" -n valid_knox_user -p valid_knox_passwordSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See <a href="<a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a>" target=

hi all   in our cluster HDP - 2.6.4 , we have kafka cluster with 11 nodes ( 11 machines )   restart the kafka service , from AMBARI REST API , will stop/start is brutal    when we do it from rest API then its cause corrupted indexes files in kafka topic partitions   what we search is , ambari rest API that do the restart in parallel   for example   first stop kafka01   then   start kafka01    wait until alert gone (from ambari)   then stop kafka02 , and start kafka02   etc....     is this process could be done by ambari rest API?    

Hello, I would like to understand the internal in / outs of Cloudera flow. How CM interacts  with other components in the Hadoop stack ( HDFS / YARN/ IMPALA etc.), would like to know if there are any references material / documents which I can go through to enhance my knowledge in this area. Appreciate any information in this regard. Regards

hi all   in ambari we have kafka service , and this service include 7 kafka brokers   how to stop/start the kafka broker on specific host?   in this example kafka service contain the following kafka broker hosts   kafka01 kafka02 kafka03 kafka04 kafka05 kafka06 kafka07   other repleted post - https://community.cloudera.com/t5/Support-Questions/how-to-identify-alert-from-kafka-service-in-ambari-by-REST/m-p/284675#M211372  

Hi,   I have large HBase table and would like to add a new column family with 1 column. I know about "alter" which adds the to schema to all regions. But it really means adding schema to region ? Is it only change schema or rewrite data ?      Regards, Cezary

Hi Team, @jsensharma @MattWho @Shelton @venkatsambath  I have installed CDH 6.2.0 on CM 6.2.0 with the help of local repository parcel file created by me following the Cloudera documents ( https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html ). I have successfully installed the CDH. however, CDH 6.2 comes with in-built parcels for Spark-2. which I am unable to find in the parcels.  Please assist me on how do I add Spark-2 as a service in my Cloudera 6.2.0 now. Appreciate your assistance.  

Hi All,    Kindly help me in adding services to the cluster. I have been struggling to add Hive and other services as the Test connection is getting failed with the error. Able to find server and databases, and logon failed.    The error in the scm logs state no pg_hba.con available for the host and user. However, have checked them all.  Also, attached are the pictures as I work on a remote desktop where I cannot login to this platform.  Kindly assist.    Thanks, Ansar

move a file from: EX - http://77.17.6.166/tls/file/nn.csvto: hostname:77.17.6.122    remote directory - /tls/fileand rename as hello.csv guide me with processor an there respective configuration 

I plan to get a Cloudera certification and need help on following questions:Question #1. I reviewed the page https://www.cloudera.com/about/training/certification.html, It looks like that CCP Data Engineer is the only certification that has not been suspended or retired.  Am I right on this?Question #2. To prepare DE575, the only recommended Cloudera course is "Spark and Hadoop Developer" training course. according to this page. https://www.cloudera.com/about/training/certification/ccp-data-engineer.html. Should I consider other courses?Questions #3. My workplace uses HDP. Do I need to get familiar with products like CDH before taking the exam? 

Hi allI have the following Nifi template which ingests data from a csv file and put them in a database.   The template is available herehttps://github.com/cetic/fadi/blob/master/examples/basic/basic_example_final_template.xml The template is composed of two processes InvokeHTTP and PutDatabaseRecord   InvokeHTTPPutDatabase It has also two controller services : CSV Reader and DBCPCOnnectionPool   The complete explanation of this small example is available herehttps://github.com/cetic/fadi/blob/master/USERGUIDE.md#3-ingest-measurements The csv file is herehttps://raw.githubusercontent.com/cetic/fadi/master/examples/basic/sample_data.csv   The table creation is: CREATE TABLE example_basic (measure_ts TIMESTAMP NOT NULL,temperature FLOAT (50));   Until Apache Nifi 1.9.2 everything works are expected!Since Apache Nifi 10.0.0, I have an Error in the PutDatabaseRecord processor "org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert

Hi all, tried to use the new parameter thing on PutEmail processor in 1.10.0.   Checked spelling and value a thousand times. Doing the same thing with variables works fine. Any help is appreciated!

  the following API example , will stop the kafka service in ambari export service=kafkacurl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.STOP.$service","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME","service_name":"$service"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://$HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/$service the problem is about the syntax - _PARSE_.STOP.$service and we see that actually service value - kafka not set in - PARSE.STOP.$service so ambri saw the name as - PARSE.STOP.$service and not PARSE.STOP.kafka any idea how we can set the value kafka inside the json syntax?

Hi Folks ,I am facing one issue , i have tried all the methods but its not working can you please help on this .  one column have data like below its there row wise (Original column) IT Strategy& Architecture BDC India [MITX 999],Corporate & IC Solution Delivery [SVII],Operations Solution Delivery [SVIA],Mainframe Service [MLEM],Strategy & Architecture [MLEL] the above 3 rows i need to split into two columns like below 1st columnIT Strategy& Architecture BDC India,Corporate & IC Solution Delivery,Operations Solution Delivery,Mainframe Service,Strategy & Architecture  2nd columns [MITX 999],[SVII],[SVIA],[MLEM], [MLEL] please help , working new for impala data lake . Thank You ..!