Member since
01-24-2016
47
Posts
11
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1546 | 07-02-2018 01:22 PM | |
7610 | 07-22-2016 06:04 PM |
07-02-2018
01:22 PM
I resolved this by installing an earlier version of Spark2 1. Deleted Spark2 Service 2. Deactivated, and removed distributed parcel 2.3.0.cloudera2-1.cdh5.13.3.p0.316101 3. Got earlier parcel 2.2.0.cloudera2-1.cdh5.12.0.p0.232957 4. Downloaded and distributed and activated this parcel 5. Got the CSD for this and put into /opt/cloudera/csd 6. Installed Spark2 from CM. Spark2 up and running on my cluster !
... View more
06-10-2018
09:49 PM
sudo grep ERROR /var/log/cloudera-scm-agent/cloudera-scm-agent.log [10/Jun/2018 21:46:46 +0000] 32685 MainThread util ERROR Lineage file not found, skipping Spark plugin creation: /etc/spark2/conf.cloudera.spark2_on_yarn/navigator.lineage.client.properties
... View more
06-10-2018
09:44 PM
Hi everyone I just upgraded by 3 node research cluster to 5.14.2-1.cdh5.14.2.p0.3. The HDFS, YARN, ZOOKEEPER, HIVE, IMPALA services got successfully installed and upgraded. I followed the steps in https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html to install 2.3.0.cloudera2-1.cdh5.13.3.p0.316101 install on the cluster Get the following errors. I am not sure where do I go to see any detail error logs. Nothing is shown in CM Deploying Client Configuration Failed to deploy client configuration to the cluster. Execute DeployClusterClientConfig for {hdfs,yarn,spark_on_yarn,hive,spark2_on_yarn} in parallel. Completed only 4/5 steps. First failure: Failed to execute command Deploy Client Configuration on service Spark 2 Execute command Deploy Client Configuration on service Spark 2 Deploy Client Configuration failed. Generate and deploy client configuration. Completed only 0/3 steps. First failure: Client configuration (id=12) on host hp8300one (id=1) exited with 1 and expected 0.
... View more
Labels:
05-06-2018
02:41 PM
3 Kudos
Dear Cloudera CDH folks, First of, a big thank you. I have been a cloudera CDH user (starving developer version) since 2011 and since around that time, I have been actively pursuing the research of adverse events (side effects) of medicines. All my hadoop, hive and impala and lucene jobs were written and run on my home 3 node CDH cluster (refurbished repurposed hardware with 8 cores and 32GB RAM per node). I released the iPhone App in 2015 called MedicalSideFx. Free to download. Lifetime free updates of data. And now I am happy to announce the release of the digital first edition of my book MedicalSideFx - A Technical Guide to Adverse Events Analytics using Apache Hive. https://github.com/sanjaysubramanian/msfx_book/blob/master/msfx_tech_book_20180506.pdf Neither the App nor the Book are financially motivated, rather it is a first step towards hopefully collaborative work happening among like minded folks to push this ahead and throw more light on complex drug interactions. I want to take the opportunity to say a big thank you the Cloudera team since its on your platform I did hundreds of hours of research ! Thanks Warmly sanjay
... View more
Labels:
02-19-2017
10:26 PM
I would still love to know the solution. Decided to go for the nuclear option 😞 Walked on fire for 8 hours today but I am back again ! 1. Stopped all cron jobs 2. Backed up the 1TB of HDFS data 3. Backed up Hive metastore MySQL 4. Uninstalled CDH and CM 5. upgraded all my nodes to 14.04 LTS 6. Fresh installed CM and CDH 5.10.1 warmly sanjay
... View more
02-19-2017
11:39 AM
Facing same issues here Ubuntu 12.04.5 LTS 3 node hadoop cluster want to ugrade Cloudera Manager from 5.8.0 to 5.9.x thanks warmly sanjay
... View more
01-31-2017
08:16 PM
Awesome Thanks Tim I did the mem checks specifically "memz detailed=true" And realized the mem_limit was somehow 6GB for node 1 and 2 but 256MB on node 3 😞 I changed all three to 6GB each and the query works now. Really appreciate your help and my belief in Cloudera only becomes 10 fold stronger ! warmly and appreciatively sanjay
... View more
01-31-2017
08:15 PM
1 Kudo
Awesome Thanks Tim I did the mem checks specifically http://hostname:25000/memz?detailed=true And realized the mem_limit was somehow 6GB for node 1 and 2 but 256MB on node 3 😞 I changed all three to 6GB each and the query works now. Really appreciate your help and my belief in Cloudera only becomes 10 fold stronger ! warmly and appreciatively sanjay
... View more
01-30-2017
08:02 AM
I am having same issues. I use CDH 5.8.0 CM 5.8.1 WARNINGS: Memory limit exceeded The memory limit is set too low to initialize spilling operator (id=3). The minimum required memory to spill this operator is 264.00 MB. Memory Limit Exceeded Query(60409f68f36d7b3d:301437049bd7bba0) Limit: Consumption=160.58 MB Fragment 60409f68f36d7b3d:301437049bd7bba2: Consumption=123.18 MB AGGREGATION_NODE (id=3): Consumption=122.02 MB EXCHANGE_NODE (id=2): Consumption=0 DataStreamRecvr: Consumption=1.16 MB Fragment 60409f68f36d7b3d:301437049bd7bba5: Consumption=37.40 MB AGGREGATION_NODE (id=1): Consumption=11.03 MB HDFS_SCAN_NODE (id=0): Consumption=26.23 MB DataStreamSender: Consumption=80.00 KB Block Manager: Limit=156.00 MB Consumption=114.00 MB Could not execute command: select isr, count(isr) as counts from aers.demo_drug_reac_combo_clean group by isr having counts > 1 Impala 2.6.0+cdh5.8.0+0 My query is ultra simple select isr, count(isr) as counts from aers.demo_drug_reac_combo_clean group by isr having counts > 1 aers.demo_drug_reac_combo_clean contains only 10 million records and 9 cols Metadata is as follows | isr | drugname | pt | year | age | age_cod | age_norm | age_group | | 3175747 | troglitazone | hepatotoxicity nos | 1999 | 68 | YR | 68 | 65-69 | Hadoop Cluster Setup ==================== 3 nodes (HP8300 Elite Desktops) , 32GB RAM each node
... View more
12-02-2016
02:26 PM
Hi guys I face this problem every day now on at least two of the hosts "This host has been out of contact with the Cloudera Manager Server for too long. This host is in contact with the Host Monitor. The host's Cloudera Manager Agent version matches the Host Monitor version (5.8.1)." Support admin Inspector Results Validations Inspector failed on the following hosts... ip-10-231-19-[23, 196]: Can only run host inspector when host is healthy. Inspector ran on 6 hosts. Individual hosts resolved their own hostnames correctly. No errors were found while looking for conflicting init scripts. No errors were found while checking /etc/hosts. All hosts resolved localhost to 127.0.0.1. All hosts checked resolved each other's hostnames correctly and in a timely manner. Host clocks are approximately in sync (within ten minutes). Host time zones are consistent across the cluster. No users or groups are missing. No conflicts detected between packages and parcels. No kernel versions that are known to be bad are running. No problems were found with /proc/sys/vm/swappiness on any of the hosts. No performance concerns with Transparent Huge Pages settings. CDH 5 Hue Python version dependency is satisfied. 0 hosts are running CDH 4 and 8 hosts are running CDH 5. There are mismatched versions across the system, which will cause failures. See below for details on which hosts are running what versions of components. Java versions are inconsistent across the managed hosts. Check the component version below to identify hosts with inconsistent Java versions. All checked Cloudera Management Daemons versions are consistent with the server. All checked Cloudera Management Agents versions are consistent with the server. Version Summary HostsComponent Version Release CDH Version ml-nonprod — Group 1 (CDH 5) ip-10-231-18-[68, 203]; ip-10-231-19-[23, 86-87, 196] Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable HostsComponent Version Release CDH Version ml-nonprod — Group 2 (CDH 5) ip-10-231-19-197 Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Java 8 JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable HostsComponent Version Release CDH Version ml-nonprod — Group 3 (CDH 5) ip-10-231-18-181 Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Java 8 JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable
... View more
- Tags:
- cloudera manager
Labels:
11-15-2016
09:21 PM
Hi guys We have a cluster on AWS with EC2 instances 1 NN (r3.4xlarge) DN1 DN2 DN3 DN4 (r3.8xlarge) Ubuntu 12.04.4 LTS Cloudera Manager CM installation CDH 5.8.0 We were facing a funny problem since lastweek. We would start a Spark Scala job and within 10 minutes DN2 would be be reachable via SSH and the job would eventually hang. I upgraded DN2 (just one node) to Ubuntu 14.04.5 LTS. After that when I start the cluster , then no CDH components start on this node - cloudera agent - datanode - node manager - region server The log file is below. I do see "Connection refused" CLOUDERA-SCM-AGENT ==================== [16/Nov/2016 02:17:45 +0000] 2207 Dummy-1 agent INFO Cleaning up daemon [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO SCM Agent Version: 5.8.1 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent Protocol Version: 4 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using Host ID: 832b2fcf-426f-4fba-a8fe-0cae3e239957 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using directory: /run/cloudera-scm-agent [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/build/env/bin/supervisord [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certifica tes in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation. [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent Logging Level: INFO [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO No command line vars [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type) [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type) [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent starting as pid 3573 user root(0) group root(0). [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: cpu [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO cgroup pseudofile /tmp/tmpBDmF82/cpu.rt_runtime_us does not exist, skipping [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: cpuacct [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: memory [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: blkio [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft _limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpu acct': True, 'has_blkio': True} [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Setting up supervisord event monitor. [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using timeout of 2.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using join timeout of 0.100000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using tolerance of 60.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] [16/Nov/2016 04:52:01 +0000] 3573 MainThread kt_renewer INFO Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/tasktracker/schema.json [16/Nov/2016 04:52:01 +0000] 3573 MainThread ntp_monitor INFO Using timeout of 2.000000 [16/Nov/2016 04:52:02 +0000] 3573 MainThread dns_names INFO Using timeout of 30.000000 [16/Nov/2016 04:52:02 +0000] 3573 MainThread __init__ INFO Created DNS monitor. [16/Nov/2016 04:52:02 +0000] 3573 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 [16/Nov/2016 04:52:02 +0000] 3573 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/schema.json [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent ERROR Failed to connect to previous supervisor. Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2039, in find_or_start_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:02 +0000] 3573 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process [16/Nov/2016 04:52:03 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1) [16/Nov/2016 04:52:03 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:04 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 2) [16/Nov/2016 04:52:04 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:05 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 3) [16/Nov/2016 04:52:05 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:06 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 4) [16/Nov/2016 04:52:06 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 5) [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent ERROR Failed to connect to newly launched supervisor. Agent will exit [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Stopping agent... [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO No extant cgroups; unmounting any cgroup roots [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Cleaning up daemon [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO Stopping agent... [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent ERROR Shutdown callback failed. Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2777, in stop f() File "/usr/lib/python2.7/asyncore.py", line 409, in close self.socket.close() File "/usr/lib/python2.7/asyncore.py", line 636, in close os.close(self.fd) OSError: [Errno 9] Bad file descriptor [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO Cleaning up daemon
... View more
Labels:
11-10-2016
03:05 PM
Hi guys
Yesterday I completed upgrading our 3 node dev cluster from 5.8.0 to 5.9.0 CDH parcels using CM. Impala is now at 2.7.0, which is cool.
I am very thankful as ever to the Cloudera team for striving to keep the starving developer version alive for data wranglers like me 🙂
Some quick stats of query timings. Impala versus HiveMR versus HiveSpark
Not sure why Impala is slower 😞
Machine 1 = NN + DN1
Machine 2 = DN2
Machine 3 = DN3
Each Machine = 8 cores 32GB RAM
CDH 5.9.0 Impala 2.7.0 impala-shell -q "select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Fetched 1281 row(s) in 256.58s
hive -e "hive.execution.engine=mr;select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Time taken: 181.024 seconds, Fetched: 1281 row(s)
hive -e "set hive.execution.engine=spark;select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Time taken: 360.214 seconds, Fetched: 1281 row(s)
Thanks
Warmly
sanjay
... View more
Labels:
10-11-2016
07:31 PM
Hi guys I have 4 datanodes in my cluster that have 6 X 1TB each. I wanted to downsize each datanode to 3 X 1TB. So essentially move 3 X 1TB per datanode. This is the process I folowed. Please tell me if its correct or not. 1. On a running cluster , go to DN1 2. Edit /etc/fstab . Remove the disk6 mountpoint and save. 3. Reboot the DN1 4. Login back to DN1 and do "hdfs fsck /" 5. Make sure of the following Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Corrupt blocks: 0 6. Repeat process 1.2.3.4.5 on DN2 7. Then remove disk5 from DN1, DN2, DN3, DN4 by following 1.2.3.4.5 for each datanode 7. Then remove disk4 from DN1, DN2, DN3, DN4 by following 1.2.3.4.5 for each datanode 8. All good till now 9. Go to cloudera manager and change dfs.datanode.failed.volumes.tolerated = 1 (from 3) 10. Modify dfs.data.dir, dfs.datanode.data.dir (remove the three disks you removed) 11. Restart Hadoop cluster 12. This is where I observed 24 blocks corrups or missing ? Why is this happening ? Please advise a better process that will result in 0 corrupt/missing blocks warmly sanjay
... View more
08-02-2016
10:33 AM
Thats awesome thanks a ton Mike. Early this morning before your mails came in - I grew impatient 🙂 as is my nature - and did give it a shot to the Cloudera director as-is-wher-is scripts. 1. Used the Cloud Formation template here https://s3.amazonaws.com/quickstart-reference/cloudera/latest/templates/Template2-Cloudera-AWS-ExistingVPC.template 2. Created a "ClusterLauncher Instance" on AWS 3. SSH to "C lusterLauncher Instance" cloudera-director bootstrap cloudera/setup-default/aws.reference.conf Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-plugins OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 Cloudera Director 2.1.0 initializing ... The configuration file aws.reference.conf is not present or cannot be read. [ec2-user@ip-10-219-178-74 ~]$ cloudera-director bootstrap cloudera/setup-default/aws.reference.conf Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-plugins OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 Cloudera Director 2.1.0 initializing ... Installing Cloudera Manager ... * Starting ..... done * Requesting an instance for Cloudera Manager .......................... done * Installing screen package (1/1) ....... done * Running custom bootstrap script on [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com] .......... done * Waiting for SSH access to [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com], default port 22 ..... done * Inspecting capabilities of 10.219.177.189 .......... done * Normalizing a3508870-3de7-4dc0-84a5-f69c77610c89 ..... done * Installing ntp package (1/4) ..... done * Installing curl package (2/4) ..... done * Installing nscd package (3/4) ..... done * Installing gdisk package (4/4) ..................... done * Resizing instance root partition ......... done * Mounting all instance disk drives ............ done * Waiting for new external database servers to start running ........ done * Installing repositories for Cloudera Manager ....... done * Installing oracle-j2sdk1.7 package (1/3) ..... done * Installing cloudera-manager-daemons package (2/3) ..... done * Installing cloudera-manager-server package (3/3) ...... done * Setting up embedded PostgreSQL database for Cloudera Manager ...... done * Installing cloudera-manager-server-db-2 package (1/1) ..... done * Starting embedded PostgreSQL database ...... done * Starting Cloudera Manager server ... done * Waiting for Cloudera Manager server to start ..... done * Setting Cloudera Manager License ... done * Enabling Enterprise Trial ... done * Configuring Cloudera Manager ..... done * Deploying Cloudera Manager agent ...... done * Waiting for Cloudera Manager to deploy agent on 10.219.177.189 ... done * Setting up Cloudera Management Services ............ done * Backing up Cloudera Manager Server configuration ...... done * Inspecting capabilities of 10.219.177.189 ...... done * Done ... Cloudera Manager ready. Creating cluster C5-Reference-AWS ... * Starting ..... done * Requesting 11 instance(s) in 3 group(s) ....................................... done * Preparing instances in parallel (20 at a time) .............................................................. done * Waiting for Cloudera Manager installation to complete ... done * Installing Cloudera Manager agents on all instances in parallel (20 at a time) ........ done * Waiting for new external database servers to start running ... done * Creating CDH5 cluster using the new instances ... done * Creating cluster: C5-Reference-AWS .... done * Downloading parcels: CDH-5.7.2-1.cdh5.7.2.p0.18,KAFKA-2.0.2-1.2.0.2.p0.5 ... done * Distributing parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ... done * Activating parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ...... done * Configuring Hive to use Sentry ... done * Creating Sentry Database ... done * Calling firstRun on cluster C5-Reference-AWS ... done * Waiting for firstRun on cluster C5-Reference-AWS .... done * Running cluster post creation scripts ...... done * Adjusting health thresholds to take into account optional instances. ... done * Done ...
... View more
08-01-2016
01:49 PM
http://docs.aws.amazon.com/quickstart/latest/cloudera/welcome.html Hey guys Will this work for the starving developers version as well ? I am using CDH 5.8.0 thanks sanjay
... View more
07-27-2016
05:56 PM
1 Kudo
Hi guys This is not an installation issue for me. This cluster I setup here is running 24X7 for 2 years ! In my CM managed CDH 5.8.0 hadoop cluster I am getting this error on one datanode and this error I used to get in 5.6.0 and now in 5.8.0 ( I thought this error may go away after I move to 5.8.0) 1. From this datanode if I telnet, its actually successful telnet xx.xxx.xx.xxx 7182 Trying xx . xxx . xx . xxx ... Connected to xx . xxx . xx . xxx . Escape character is '^]'. 2. Also if I go to http://namenode_ip:50070 then I can see this datanode up and ready. None of the services on this node are really disrupted. But somehow cm-agent is not able to talk to cm-server ? [28/Jul/2016 00:38:41 +0000] 1200 MainThread agent ERROR Heartbeating to xx.xxx.xx.xxx:7182 failed. Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 1211, in _send_heartbeat response = self.requestor.request('heartbeat', dict(request=heartbeat)) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 136, in request self.write_call_request(message_name, request_datum, buffer_encoder) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 178, in write_call_request self.write_request(message.request, request_datum, encoder) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 182, in write_request datum_writer.write(request_datum, encoder) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/io.py", line 768, in write
... View more
07-25-2016
10:25 PM
2 Kudos
Greetings my beloved friends at Cloudera and all CDH users, I am so excited to announce that I just finished upgrading our 9 node non-prod cluster (on AWS) to CM 5.8.1 and CDH parcels 5.8.0. Went off without an issue at all. As I have always proudly announced , I have been a starving developer version user of CDH for the past 5 years ! We are heavy users of Hive on Spark , HBase and Impala for creating curated datasets for our machine learning models and we can truly say we do not know where we will be without Cloudera ! So a big thank you to all Cloudera team with our folded hands and heads bent with respect and gratitude. I want to say a big thank you to the ever responsive cloudera community for answering my questions and clarifying problems. We would not have come so far without your die-hard positive attitude and hard-core community support. Warmly and appreciatively, sanjay
... View more
07-22-2016
06:04 PM
So here is how I solved this issue. - Installed 5.8.0 Cloudera Manager and corresponding 5.8.0 parcels for the cluster - Added Spark (on Yarn) as a service - no issues faced Maybe something was not right in CM 5.7.1 ?
... View more
07-20-2016
12:32 PM
These are errors from /var/run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/logs/stderr.log + perl -pi -e 's#{{HIVE_HBASE_JAR}}#/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.7.1.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-hadoop-compat.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-common.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-protocol.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core-3.2.0-incubating.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/lib/htrace-core4-4.0.1-incubating.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-server.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-hadoop2-compat.jar,/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hbase/hbase-client.jar#g' /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/spark-conf/hive-env.sh Can't open /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/spark-conf/hive-env.sh: No such file or directory. + /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/scripts/control.sh client /usr/lib/cmf/service/client/deploy-cc.sh: line 190: /run/cloudera-scm-agent/process/ccdeploy_spark-conf_etcsparkconf.cloudera.spark_on_yarn_-8951873637091761528/scripts/control.sh: Permission denied
... View more
07-20-2016
12:17 PM
Add a Spark Service to Cluster 1 First Run Command Status: Failed Start Time: Jul 20, 12:15:05 PM Duration: 15.37s Retry Failed to perform First Run of services. All Failed Only Running Only Details Completed 1 of 5 step(s). Step Context Start Time Duration Actions Deploy Client Configuration Failed to deploy client configuration to the cluster. Cluster 1 Jul 20, 12:15:05 PM 15.36s Execute DeployClusterClientConfig for {yarn,hdfs,hive,spark_on_yarn} in parallel. Completed only 3/4 steps. First failure: Failed to execute command Deploy Client Configuration on service Spark Jul 20, 12:15:05 PM 15.36s Deploy Client Configuration Successfully deployed client configuration. YARN (MR2 Included) Jul 20, 12:15:05 PM 242ms Deploy Client Configuration Successfully deployed client configuration. HDFS Jul 20, 12:15:05 PM 212ms Deploy Client Configuration Successfully deployed client configuration. Hive Jul 20, 12:15:05 PM 170ms Deploy Client Configuration Deploy Client Configuration failed. Spark Jul 20, 12:15:05 PM 15.21s Generate and deploy client configuration. Completed only 2/3 steps. First failure: Client configuration (id=25) on host n1-3hadoop-dev01 (id=1) exited with 1 and expected 0. Jul 20, 12:15:05 PM 15.21s Execute command Create Spark User Dir on service Spark Execute command Create Spark History Log Dir on service Spark Execute command Install Spark JAR on service Spark Start Spark
... View more
07-20-2016
10:33 AM
Hi guys First of, my Iove for CDH and Cloudera Manager will never decrease ! I know I use the starving developers version of Cloudera and possibly have lesser or no rights to complain 🙂 ; however the support you guys have demonstrated over the past 4 years has kept my faith and confidence and I feel like a privileged Cloudera user and continue to report problems and issues with the hope that it helps other users and the feedback makes the product even more bulletproof ! However yesterday while I was installing CM 5.7.1 on a new 3 node Ubntu 12.04 cluster , I faced the maximum issues ever 1. After selecting services to install (ZK, HDFS, Yarn, Hive, Impala, Spark on Yarn) , during the first services start phase of the CM install, spark failed to start 2. I aborted installation and started manually adding service from CM - first ZK, then HDFS(had to format HDFS) , YARN, Spark - Spark failed...I copied spark-assemply jar from the /opt/cloudera/parcels/CDH//lib/spark/assembly/lib/spark-assembly.jar directory to /user/spark/share/lib/spark-assembly.jar on HDFS and put the appropriate chown and chmod settings 3. Spark on Yarn just fails to start... Any ideas ? Recommendations ? Thoughts ? Would love to learn and implement warmly sanjay
... View more
07-19-2016
01:39 PM
Facing this same issue with 5.7.1 cloudera installation on Ubuntu 12.04 (3 node installation) "While the screen shows that the downloading of the parcels is done, it does not progress to distributing or activating from there."
... View more
07-19-2016
10:10 AM
Hi guys I have a 3 node cluster (static IPs). h1, h2, h3 (all Ubuntu 12.04 LTS Trusty) FIRST TRY OF INSTALL =================== - On machine h1, I did a fresh install of Cloudera manager 5.7 using cloudera-manager-installer.bin - Thru the h1:7180 I tried installing parcels on h1 , h2 , h3 - hi and h2 succeed. - h3 reports following error Installation failed. Failed to receive heartbeat from agent. Ensure that the host's hostname is configured properly. Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules). Ensure that ports 9000 and 9001 are not in use on the host being added. Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details). If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here. SECOND TRY OF INSTALL ===================== - I uninstalled all CDH components on h1,h2,h3 - Now On machine h3, I did a fresh install of Cloudera manager 5.7 using cloudera-manager-installer.bin - Thru the h3:7180 I tried installing parcels on h1 , h2 , h3 - Now h3 and h2 succeed and h1errors out with the same error as above Any help , suggestions ? PING, SSH work between the machines. the user on all three machines has passwordless sudo..... I have installed clusters for 4 years now 🙂 but this one is beating me ! Please put forth your ideas and recommendations...I would be super grateful warmly sanjay
... View more
07-19-2016
12:56 AM
@Deepesh Following query returns values in both TEZ and MR mode select split(ln,',')[0]as cid, split(ln,',')[1]as zip from utils.file1 where fn='foo1.csv' limit 1000;
... View more
07-13-2016
10:01 PM
- Setup 3 node hadoop cluster on Amazon using Ambari. Each instance is r3.xlarge (30GB RAM) - I adjusted the YARN cluster params per this link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_installing_manually_book/content/determine-hdp-memory-config.html The query is set hive.execution.engine=tez; (I tried "mr" as well ) select zip, count(cid) as ginti from (select split(ln,',')[0] as cid, split(ln,',')[1] as zip from utils.file1 where fn='foo1.csv') dd group by zip order by ginti desc The CSV is a 2 column data with 18 million rows RESULT The query just seems to hang and does not return with results !
... View more
Labels:
05-11-2016
08:33 AM
As a comparison I wrote Java MR code that does exactly what the query does and that ran in 1m 30s ! So something does not seem all right with Hive
... View more
05-10-2016
04:47 PM
Hi guys I have a 3 node development cluster CDH 5.6.0 (managed by cloudera manager). This time I configured MySQL in AWS RDS as the Hive Metastore. 1NN + 1 DN 1 DN 1 DN each node is 32GB RAM and 2X2TB 7200 rpm disks Like always I have tuned the memory params in yarn. A simple SELECT query on 1 Hive tablewith 22 million records with one WHERE clause takes 15 minutes ! Can it be that the AWS RDS Hive Metastore is slowing the Hive query down ? thanks sanjay
... View more
04-20-2016
02:21 PM
1 Kudo
Hi guys I just setup Phoenix 4.5.2-1.clabs_phoenix1.2.0.p0.774 through Cloudera Manager on CDH 5.6.0. My dev cluster is 3 boxes Each is HP 8300 8 core, 32GB RAM 1NN and 3DN DDL (this table is created in Phoenix on HBase as well as in Hive) ==== CREATE TABLE IF NOT EXISTS resume_dates (resid VARCHAR, cd VARCHAR, uts BIGINT CONSTRAINT pk PRIMARY KEY (resid)); Sample Data ========== 14008_1_1000522248_0_1108045212,2014-01-30,1391093927 14025_1_1010236513_0_1107883638,2014-01-30,391093930 Num of records ============ 23,748,651 Query ===== select substr(cd, 1,4) as yyyy, count(resid) from RESUME_DATES group by substr(cd, 1,4) order by yyyy asc Comparison of Timings ================== Hive on MR = 81.829 seconds Hive on Spark = 32.78 seconds Phoenix = 12.234 seconds Impala = 0.99 seconds Thanks sanjay
... View more
04-07-2016
09:38 AM
Hey guys On the CDH 5.6.0 (I have been happily and successfully using the CDH starving developers version since 2012) this is how you can create and use tables with data location pointing to S3 on AWS. I am sure there are possibly better and more elegant ways to do this (and guys please educate if so) - but this is one way that works successfully...so here goes [1] In the HDFS Configuration in Cloudera Manager : ===================================== SECTION = "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" Add the following <property> <name>fs.s3.awsAccessKeyId</name> <value>YOUR_AWS_ACCESS_KEY</value> </property> <property> <name>fs.s3.awsSecretAccessKey</name> <value>YOUR_AWS_SECRET_ACCESSKEY</value> </property> <property> <name>fs.s3a.awsAccessKeyId</name> <value> YOUR_AWS_ACCESS_KEY </value> </property> <property> <name>fs.s3a.awsSecretAccessKey</name> <value> YOUR_AWS_SECRET_ACCESSKEY </value> </property> <property> <name>fs.s3n.awsAccessKeyId</name> <value> YOUR_AWS_ACCESS_KEY </value> </property> <property> <name>fs.s3n.awsSecretAccessKey</name> <value> YOUR_AWS_SECRET_ACCESSKEY </value> </property> [2] Create Table in Hive ================== set hive.execution.engine=mr ; use openpv ; CREATE EXTERNAL TABLE IF NOT EXISTS solar_installs( zipcode STRING, state STRING, sizekw DOUBLE, cost DOUBLE, date_ STRING, lat DOUBLE, lon DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ; [3] Set Data Location (note that I am pointing to the AWS S3 bucket and sub folder and not to a specific file) ================================================================================== NOTE: If you want the data you can go here https://openpv.nrel.gov/search and hit search with no criteria defined and then download the CSV set hive.execution.engine=mr ; use openpv ; ALTER TABLE solar_installs SET LOCATION 's3a://some-aws-bucket-name/openpv' ; [4] Run a query in Hive using MR as execution engine ========================================= set hive.execution.engine=mr ; use openpv ; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc ; [5] Run a query in Hive using Spark as execution engine =========================================== use openpv ; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc ; [6] Run a query in Impala =================== impala-shell -q "invalidate metadata" impala-shell -q "use openpv; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc " [7] Results Comparison ================== These were run on a 3 node cluster runing under my cube, 1NN +1DN on one node , DN2 on node 2 and DN3 on node 3. Each node is 8 core HP 8300 32GB RAM Impala = 3.81s Hive-on-Spark = 27.582 seconds Hive-on-MR = 46.774 seconds
... View more
03-24-2016
11:35 AM
ok guys something is wrong and maybe there is a clue here Query: select count(*) from sansub01.benji_bc_q1 +------------+ | count(*) | +------------+ | 2344418707 | +------------+ Fetched 1 row(s) in 716.47s The tables I INNER JOINED have 9 million and 7 million rows....so an inner join should have less than 9 or 7 million !
... View more