Dear Cloudera CDH folks,   First of,  a big thank you. I have been a cloudera CDH user (starving developer version) since 2011 and since around that time, I have been actively pursuing the research of adverse events (side effects) of medicines. All my hadoop, hive and impala and lucene jobs were written and run on my home 3 node CDH cluster (refurbished repurposed hardware with 8 cores and 32GB RAM per node).   I released the iPhone App in 2015 called MedicalSideFx. Free to download. Lifetime free updates of data.   And now I am happy to announce the release of the digital first edition of my book MedicalSideFx - A Technical Guide to Adverse Events Analytics using Apache Hive. https://github.com/sanjaysubramanian/msfx_book/blob/master/msfx_tech_book_20180506.pdf   Neither the App nor the Book are financially motivated, rather it is a first step towards hopefully collaborative work happening among like minded folks to push this ahead and throw more light on complex drug interactions.      I want to take the opportunity to say a big thank you the Cloudera team since its on your platform I did hundreds of hours of research !   Thanks Warmly   sanjay ... View more

Hi guys    I face this problem every day now on at least two of the hosts      "This host has been out of contact with the Cloudera Manager Server for too long. This host is in contact with the Host Monitor. The host's Cloudera Manager Agent version matches the Host Monitor version (5.8.1)."   Support admin Inspector Results Validations   Inspector failed on the following hosts... ip-10-231-19-[23, 196]: Can only run host inspector when host is healthy. Inspector ran on 6 hosts.   Individual hosts resolved their own hostnames correctly.   No errors were found while looking for conflicting init scripts.   No errors were found while checking /etc/hosts.   All hosts resolved localhost to 127.0.0.1.   All hosts checked resolved each other's hostnames correctly and in a timely manner.   Host clocks are approximately in sync (within ten minutes).   Host time zones are consistent across the cluster.   No users or groups are missing.   No conflicts detected between packages and parcels.   No kernel versions that are known to be bad are running.   No problems were found with /proc/sys/vm/swappiness on any of the hosts.   No performance concerns with Transparent Huge Pages settings.   CDH 5 Hue Python version dependency is satisfied.   0 hosts are running CDH 4 and 8 hosts are running CDH 5.   There are mismatched versions across the system, which will cause failures. See below for details on which hosts are running what versions of components.   Java versions are inconsistent across the managed hosts. Check the component version below to identify hosts with inconsistent Java versions.   All checked Cloudera Management Daemons versions are consistent with the server.   All checked Cloudera Management Agents versions are consistent with the server. Version Summary HostsComponent Version Release CDH Version ml-nonprod — Group 1 (CDH 5) ip-10-231-18-[68, 203]; ip-10-231-19-[23, 86-87, 196] Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable HostsComponent Version Release CDH Version ml-nonprod — Group 2 (CDH 5) ip-10-231-19-197 Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Java 8 JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable HostsComponent Version Release CDH Version ml-nonprod — Group 3 (CDH 5) ip-10-231-18-181 Bigtop-Tomcat (CDH 5 only) 0.7.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Crunch (CDH 5 only) 0.11.0+cdh5.8.0+91 1.cdh5.8.0.p0.77 CDH 5 Flume NG 1.6.0+cdh5.8.0+50 1.cdh5.8.0.p0.75 CDH 5 MapReduce 1 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 Hadoop 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HDFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HttpFS 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 hadoop-kms 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 MapReduce 2 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 YARN 2.6.0+cdh5.8.0+1601 1.cdh5.8.0.p0.93 CDH 5 HBase 1.2.0+cdh5.8.0+160 1.cdh5.8.0.p0.80 CDH 5 Lily HBase Indexer 1.5+cdh5.8.0+64 1.cdh5.8.0.p0.75 CDH 5 Hive 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 HCatalog 1.1.0+cdh5.8.0+610 1.cdh5.8.0.p0.77 CDH 5 Hue 3.9.0+cdh5.8.0+2512 1.cdh5.8.0.p0.88 CDH 5 Impala 2.6.0+cdh5.8.0+0 1.cdh5.8.0.p0.111 CDH 5 Kite (CDH 5 only) 1.0.0+cdh5.8.0+136 1.cdh5.8.0.p0.73 CDH 5 Llama (CDH 5 only) 1.0.0+cdh5.8.0+0 1.cdh5.8.0.p0.73 CDH 5 Mahout 0.9+cdh5.8.0+27 1.cdh5.8.0.p0.71 CDH 5 Oozie 4.1.0+cdh5.8.0+291 1.cdh5.8.0.p0.83 CDH 5 Parquet 1.5.0+cdh5.8.0+174 1.cdh5.8.0.p0.71 CDH 5 Pig 0.12.0+cdh5.8.0+83 1.cdh5.8.0.p0.71 CDH 5 sentry 1.5.1+cdh5.8.0+244 1.cdh5.8.0.p0.83 CDH 5 Solr 4.10.3+cdh5.8.0+423 1.cdh5.8.0.p0.79 CDH 5 spark 1.6.0+cdh5.8.0+205 1.cdh5.8.0.p0.74 CDH 5 Sqoop2 1.99.5+cdh5.8.0+38 1.cdh5.8.0.p0.72 CDH 5 Sqoop 1.4.6+cdh5.8.0+65 1.cdh5.8.0.p0.69 CDH 5 Whirr 0.9.0+cdh5.8.0+17 1.cdh5.8.0.p0.68 CDH 5 Zookeeper 3.4.5+cdh5.8.0+94 1.cdh5.8.0.p0.76 CDH 5 Cloudera Manager Management Daemons 5.8.1 1.cm581.p0.7 Not applicable Supervisord 3.0-cm5.8.1 Unavailable Not applicable Java 7 JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) Unavailable Not applicable Java 6 JAVA_HOME=/usr/lib/jvm/j2sdk1.6-oracle java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Unavailable Not applicable Java 8 JAVA_HOME=/usr/lib/jvm/java-8-oracle java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) Unavailable Not applicable Cloudera Manager Agent 5.8.1 1.cm581.p0.7 Not applicable ... View more

Hi guys We have a cluster on AWS with EC2 instances  1 NN (r3.4xlarge) DN1 DN2 DN3 DN4  (r3.8xlarge) Ubuntu 12.04.4 LTS Cloudera Manager CM installation  CDH 5.8.0   We were facing a funny problem since lastweek. We would start a Spark Scala job and within 10 minutes DN2 would be be reachable via SSH and the job would eventually hang.   I upgraded DN2 (just one node) to Ubuntu 14.04.5 LTS. After that when I start the cluster , then no CDH components start on this node - cloudera agent - datanode - node manager - region server The log file is below. I do see "Connection refused"   CLOUDERA-SCM-AGENT ==================== [16/Nov/2016 02:17:45 +0000] 2207 Dummy-1 agent INFO Cleaning up daemon [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO SCM Agent Version: 5.8.1 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent Protocol Version: 4 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using Host ID: 832b2fcf-426f-4fba-a8fe-0cae3e239957 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using directory: /run/cloudera-scm-agent [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using supervisor binary path: /usr/lib/cmf/agent/build/env/bin/supervisord [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Neither verify_cert_file nor verify_cert_dir are configured. Not performing validation of server certifica tes in HTTPS communication. These options can be configured in this agent's config.ini file to enable certificate validation. [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent Logging Level: INFO [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO No command line vars [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Missing database jar: /usr/share/java/mysql-connector-java.jar (normal, if you're not using this database type) [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type) [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Agent starting as pid 3573 user root(0) group root(0). [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/cgroups [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: cpu [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO cgroup pseudofile /tmp/tmpBDmF82/cpu.rt_runtime_us does not exist, skipping [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: cpuacct [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: memory [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Found cgroups subsystem: blkio [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/memory [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpu [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/cpuacct [16/Nov/2016 04:52:01 +0000] 3573 MainThread cgroups INFO Reusing /run/cloudera-scm-agent/cgroups/blkio [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': -1, 'default_memory_soft _limit_in_bytes': -1, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': -1, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpu acct': True, 'has_blkio': True} [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Setting up supervisord event monitor. [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs'] [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using timeout of 2.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using join timeout of 0.100000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Using tolerance of 60.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread filesystem_map INFO Local filesystem types whitelist: ['ext2', 'ext3', 'ext4'] [16/Nov/2016 04:52:01 +0000] 3573 MainThread kt_renewer INFO Agent wide credential cache set to /run/cloudera-scm-agent/krb5cc_cm_agent_0 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using metrics_url_timeout_seconds of 30.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using task_metrics_timeout_seconds of 5.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread agent INFO Using max_collection_wait_seconds of 10.000000 [16/Nov/2016 04:52:01 +0000] 3573 MainThread metrics INFO Importing tasktracker metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/tasktracker/schema.json [16/Nov/2016 04:52:01 +0000] 3573 MainThread ntp_monitor INFO Using timeout of 2.000000 [16/Nov/2016 04:52:02 +0000] 3573 MainThread dns_names INFO Using timeout of 30.000000 [16/Nov/2016 04:52:02 +0000] 3573 MainThread __init__ INFO Created DNS monitor. [16/Nov/2016 04:52:02 +0000] 3573 MainThread stacks_collection_manager INFO Using max_uncompressed_file_size_bytes: 5242880 [16/Nov/2016 04:52:02 +0000] 3573 MainThread __init__ INFO Importing metric schema from file /usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/monitor/schema.json [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_PARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'} [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels. [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/process [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/flood [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent/supervisor/include [16/Nov/2016 04:52:02 +0000] 3573 MainThread agent ERROR Failed to connect to previous supervisor. Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2039, in find_or_start_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:02 +0000] 3573 MainThread tmpfs INFO Reusing mounted tmpfs at /run/cloudera-scm-agent/process [16/Nov/2016 04:52:03 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 1) [16/Nov/2016 04:52:03 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:04 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 2) [16/Nov/2016 04:52:04 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:05 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 3) [16/Nov/2016 04:52:05 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:06 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 4) [16/Nov/2016 04:52:06 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Trying to connect to newly launched supervisor (Attempt 5) [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent ERROR Failed! trying again in 1 second(s) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2163, in connect_to_new_supervisor self.get_supervisor_process_info() File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2185, in get_supervisor_process_info self.identifier = self.supervisor_client.supervisor.getIdentification() File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request verbose=self.__verbose File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/supervisor-3.0-py2.7.egg/supervisor/xmlrpc.py", line 460, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent ERROR Failed to connect to newly launched supervisor. Agent will exit [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Stopping agent... [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO No extant cgroups; unmounting any cgroup roots [16/Nov/2016 04:52:07 +0000] 3573 MainThread agent INFO Cleaning up daemon [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO Stopping agent... [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO No extant cgroups; unmounting any cgroup roots [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent ERROR Shutdown callback failed. Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 2777, in stop f() File "/usr/lib/python2.7/asyncore.py", line 409, in close self.socket.close() File "/usr/lib/python2.7/asyncore.py", line 636, in close os.close(self.fd) OSError: [Errno 9] Bad file descriptor [16/Nov/2016 04:52:07 +0000] 3573 Dummy-1 agent INFO Cleaning up daemon ... View more

Hi guys   Yesterday I completed upgrading our 3 node dev cluster from 5.8.0 to 5.9.0 CDH parcels using CM. Impala is now at 2.7.0, which is cool.    I am very thankful as ever to the Cloudera team for striving to keep the starving developer version alive for data wranglers like me 🙂   Some quick stats of query timings. Impala versus HiveMR versus HiveSpark Not sure why Impala is slower 😞     Machine 1 = NN + DN1 Machine 2 = DN2 Machine 3 = DN3 Each Machine = 8 cores 32GB RAM   CDH 5.9.0 Impala 2.7.0 impala-shell -q "select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Fetched 1281 row(s) in 256.58s   hive -e "hive.execution.engine=mr;select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Time taken: 181.024 seconds, Fetched: 1281 row(s)   hive -e "set hive.execution.engine=spark;select last_name, first_name from cdr.cdr_mjp_people where lower(last_name) like '%subramanian%'" Time taken: 360.214 seconds, Fetched: 1281 row(s)   Thanks Warmly   sanjay ... View more

Hi guys   I have 4 datanodes in my cluster that have 6 X 1TB each. I wanted to downsize each datanode to 3 X 1TB. So essentially move 3 X 1TB per datanode. This is the process I folowed. Please tell me if its correct or not.   1. On a running cluster , go to DN1 2. Edit /etc/fstab . Remove the disk6 mountpoint and save. 3. Reboot the DN1 4. Login back to DN1 and do "hdfs fsck /" 5. Make sure of the following     Over-replicated blocks: 0 (0.0 %)     Under-replicated blocks: 0 (0.0 %)     Mis-replicated blocks: 0 (0.0 %)       Corrupt blocks: 0 6. Repeat process 1.2.3.4.5 on DN2 7. Then remove disk5 from DN1, DN2, DN3, DN4 by following 1.2.3.4.5 for each datanode 7. Then remove disk4 from DN1, DN2, DN3, DN4 by following 1.2.3.4.5 for each datanode 8. All good till now  9. Go to cloudera manager and change      dfs.datanode.failed.volumes.tolerated = 1 (from 3) 10. Modify dfs.data.dir, dfs.datanode.data.dir       (remove the three disks you removed)  11. Restart Hadoop cluster  12. This is where I observed 24 blocks corrups or missing ? Why is this happening ?    Please advise a better process that will result in 0 corrupt/missing blocks   warmly   sanjay   ... View more

Hi guys   This is not an installation issue for me. This cluster I setup here is running 24X7 for 2 years !    In my CM managed CDH 5.8.0 hadoop cluster I am getting this error on one datanode and this error I used to get in 5.6.0 and now in 5.8.0 ( I thought this error may go away after I move to 5.8.0)   1. From this datanode if I telnet, its actually successful     telnet xx.xxx.xx.xxx 7182     Trying xx . xxx . xx . xxx ...     Connected to xx . xxx . xx . xxx .      Escape character is '^]'.   2. Also if I go to http://namenode_ip:50070 then I can see this datanode up and ready. None of the services on this node are really disrupted. But somehow cm-agent is not able to talk to cm-server ?        [28/Jul/2016 00:38:41 +0000] 1200 MainThread agent         ERROR     Heartbeating to xx.xxx.xx.xxx:7182 failed. Traceback (most recent call last):   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.1-py2.7.egg/cmf/agent.py", line 1211, in _send_heartbeat     response = self.requestor.request('heartbeat', dict(request=heartbeat))   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 136, in request     self.write_call_request(message_name, request_datum, buffer_encoder)   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 178, in write_call_request     self.write_request(message.request, request_datum, encoder)   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 182, in write_request     datum_writer.write(request_datum, encoder)   File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/io.py", line 768, in write ... View more

Greetings my beloved friends at Cloudera and all CDH users,   I am so excited to announce that I just finished upgrading our 9 node non-prod cluster (on AWS) to CM 5.8.1 and CDH parcels 5.8.0. Went off without an issue at all.   As I have always proudly announced , I have been a starving developer version user of CDH for the past 5 years !   We are heavy users of Hive on Spark , HBase and Impala for creating curated datasets for our machine learning models and we can truly say we do not know where we will be without Cloudera !  So a big thank you to all Cloudera team with our folded hands and heads bent with respect and gratitude.   I want to say a big thank you to the ever responsive cloudera community for answering my questions and clarifying problems. We would not have come so far without your die-hard positive attitude and hard-core community support.   Warmly and appreciatively,   sanjay ... View more

Hi guys   I have a 3 node cluster (static IPs).  h1, h2, h3 (all Ubuntu 12.04 LTS Trusty)   FIRST TRY OF INSTALL =================== - On machine h1, I did a fresh install of Cloudera manager 5.7 using  cloudera-manager-installer.bin - Thru the h1:7180 I tried installing parcels on h1 , h2 , h3 - hi and h2 succeed.  - h3 reports following error Installation failed. Failed to receive heartbeat from agent. Ensure that the host's hostname is configured properly. Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules). Ensure that ports 9000 and 9001 are not in use on the host being added. Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details). If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.   SECOND TRY OF INSTALL ===================== - I uninstalled all CDH components on h1,h2,h3 - Now On machine h3, I did a fresh install of Cloudera manager 5.7 using  cloudera-manager-installer.bin - Thru the h3:7180 I tried installing parcels on h1 , h2 , h3 - Now h3 and h2 succeed and h1errors out with the same error as above   Any help , suggestions ? PING, SSH work between the machines. the user on all three machines has passwordless sudo.....   I have installed clusters for 4 years now 🙂 but this one is beating me !   Please put forth your ideas and recommendations...I would be super grateful    warmly   sanjay         ... View more

- Setup 3 node hadoop cluster on Amazon using Ambari. Each instance is r3.xlarge (30GB RAM) - I adjusted the YARN cluster params per this link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_installing_manually_book/content/determine-hdp-memory-config.html The query is set hive.execution.engine=tez; (I tried "mr" as well ) select zip, count(cid) as ginti from (select split(ln,',')[0] as cid, split(ln,',')[1] as zip from utils.file1 where fn='foo1.csv') dd group by zip order by ginti desc The CSV is a 2 column data with 18 million rows RESULT The query just seems to hang and does not return with results ! ... View more

Hi guys   I have a 3 node development cluster CDH 5.6.0 (managed by cloudera manager). This time I configured MySQL in AWS RDS as the Hive Metastore.   1NN + 1 DN 1 DN 1 DN each node is 32GB RAM and 2X2TB 7200 rpm disks    Like always I have tuned the memory params in yarn. A simple SELECT query on 1 Hive tablewith 22 million records  with one WHERE clause takes 15 minutes !       Can it be that the AWS RDS Hive Metastore is slowing the Hive query down ?   thanks   sanjay   ... View more

Hi guys   I just setup Phoenix  4.5.2-1.clabs_phoenix1.2.0.p0.774 through Cloudera Manager on CDH 5.6.0. My dev cluster is  3 boxes Each is HP 8300 8 core, 32GB RAM 1NN and 3DN   DDL (this table is created in Phoenix on HBase as well as in Hive)  ==== CREATE TABLE IF NOT EXISTS resume_dates (resid VARCHAR, cd VARCHAR, uts BIGINT CONSTRAINT pk PRIMARY KEY (resid));   Sample Data ========== 14008_1_1000522248_0_1108045212,2014-01-30,1391093927 14025_1_1010236513_0_1107883638,2014-01-30,391093930     Num of records  ============ 23,748,651   Query ===== select substr(cd, 1,4) as yyyy, count(resid) from RESUME_DATES group by substr(cd, 1,4) order by yyyy asc   Comparison of Timings ================== Hive on MR      = 81.829 seconds Hive on Spark  = 32.78 seconds Phoenix            = 12.234 seconds Impala              = 0.99 seconds   Thanks   sanjay ... View more

Hey guys   On the CDH 5.6.0 (I have been happily and successfully using the CDH starving developers version since 2012) this is how you can create and use tables with data location pointing to S3 on AWS.  I am sure there are possibly better and more elegant ways to do this (and guys please educate if so)  - but this is one way that works successfully...so here goes   [1] In the HDFS Configuration in Cloudera Manager : ===================================== SECTION = "HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" Add the following  <property> <name>fs.s3.awsAccessKeyId</name> <value>YOUR_AWS_ACCESS_KEY</value> </property> <property> <name>fs.s3.awsSecretAccessKey</name> <value>YOUR_AWS_SECRET_ACCESSKEY</value> </property> <property> <name>fs.s3a.awsAccessKeyId</name> <value> YOUR_AWS_ACCESS_KEY </value> </property> <property> <name>fs.s3a.awsSecretAccessKey</name> <value> YOUR_AWS_SECRET_ACCESSKEY </value> </property> <property> <name>fs.s3n.awsAccessKeyId</name> <value> YOUR_AWS_ACCESS_KEY </value> </property> <property> <name>fs.s3n.awsSecretAccessKey</name> <value> YOUR_AWS_SECRET_ACCESSKEY </value> </property>   [2] Create Table in Hive ================== set hive.execution.engine=mr ; use openpv ; CREATE EXTERNAL TABLE IF NOT EXISTS solar_installs( zipcode STRING, state STRING, sizekw DOUBLE, cost DOUBLE, date_ STRING, lat DOUBLE, lon DOUBLE) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ;     [3] Set Data Location (note that I am pointing to the AWS S3 bucket and sub folder and not to a specific file) ================================================================================== NOTE: If you want the data you can go here https://openpv.nrel.gov/search and hit search with no criteria defined and then download the CSV   set hive.execution.engine=mr ; use openpv ; ALTER TABLE solar_installs SET LOCATION 's3a://some-aws-bucket-name/openpv' ;   [4] Run a query in Hive using MR as execution engine  ========================================= set hive.execution.engine=mr ; use openpv ; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc ;     [5] Run a query in Hive using Spark as execution engine  =========================================== use openpv ; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc ;       [6] Run a query in Impala    =================== impala-shell -q "invalidate metadata" impala-shell -q "use openpv; select zipcode, count(*) from solar_installs group by zipcode order by zipcode asc "     [7] Results Comparison ================== These were run on a 3 node cluster runing under my cube, 1NN +1DN on one node , DN2 on node 2 and DN3 on node 3. Each node is 8 core HP 8300 32GB RAM   Impala = 3.81s Hive-on-Spark  =  27.582 seconds Hive-on-MR =  46.774 seconds ... View more