About cchahadoop

cchahadoop · ‎03-08-2017

I added hive user to the below two properties in the kms-acls.xml to make it work. hadoop.kms.acl.GET_METADATA whitelist.key.acl.READ Also, I noticed that this happens only when we are doing a JOIN on two tables. When we are querying a single table, it is fine. Are the GET_METADATA and READ operations happening on HDFS on the parquet metadata OR on the Hive Warehouse?

cchahadoop · ‎09-14-2016

I looked in a little more depth on this issue. Here are the findings: 1. When I use 1 HS2 behind the LoadBalancer, it succeeds without any error 2. When I use 2 HS2 behind the LoadBalancer, I see one error but it retries and succeeds 3. When I use 3 or more than 3 HS2 behind the LoadBalancer, it errors out. I have tried turning the sticky connections for TCP inside the LoadBalancer but to no avail. Looks like there's some sort of Kerberos handshake for which it comes back to the LoadBalancer. Here, it'll then hit another server and then fail.

cchahadoop · ‎08-23-2016

I was able to resolve this issue by using HS2 on DataNodes. I don't have HMS HA but I do have multiple Hive Servers behind a load balancer. The problem arose when I moved the HS2 roles to application nodes (non-Datanodes) but when I moved them back to the DataNodes, these errors swiftly disappeared.

cchahadoop · ‎08-19-2016

I am facing the same issue as well. How are you passing the JDBC URL to Oozie?

cchahadoop · ‎07-13-2016

Suppose we have 8 DataNodes with 128 GB RAM each of which 64 GB is allocated to Impala and we add 8 more nodes of 256 GB and we intend to allocated 128 GB to Impala. My concern is, will the coordinator be smart enough to know the mem_limit of each node it sends fragments to? Are there any other known issues that come up with such a configuration? NOTE: Current CDH version is CDH 5.4.4 (Impala v2.2)

cchahadoop · ‎04-05-2016

We have recently enabled Sentry in our cluster. When I try to do a SHOW TABLE STATS on a HBase Table through the Impala shell, it runs into the following permission error: 0: jdbc:hive2://LB> show table stats database.table_summary; Error: RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=impala, access=EXECUTE, inode="/hbase":hbase:hbase:drwx------ I have tried adding the impala user to the hbase.superuser but still see the same issues. I have also granted impala and hive RWACX through the hbase shell. Is there something else that needs to be done? Thanks

cchahadoop · ‎01-13-2016

Were you able to find a way past this?

cchahadoop · ‎12-15-2015

Hi Dice, Have you had the chance to look at those logs?

cchahadoop · ‎12-04-2015

You can find the Reports Manager logs here. Changed the hostnames for security reasons 🙂

cchahadoop · ‎12-03-2015

This is the CM host. We do not have hadoop installed on this node. I ran the Host Inspector which alerts me that hdfs,hadoop and other such users are not present on this node and also this returns CDH version as "None". Could this be causing the Reports Manager to fail? We have a similar cluster (where CM host reports CDH version as NONE) but we do not see the same issue there. That cluster runs CM v5.4.7 and CDH 5.4.3.

cchahadoop · ‎12-02-2015

Can you please confirm/deny the following conclusions I have made by reading the new 5.5 docs, your above post and testing "Static Service Pools" in a test environment: 1. Impala's integration with YARN through Llama is being discontinued from CDH 5.5/Impala 2.3 onwards? 2. Configuring "Static Service Pools" means that both Impala and YARN (and other services too) are managed by cgroups rather than Impala being managed by YARN? If point # 2 is true, it basically means that Impala can no longer be managed by YARN CDH 5.5 onwards. I tried enabling the "Static Service Pools" but I do not see the Impala queries I ran in YARN Applications.

cchahadoop · ‎12-02-2015

Yes, this issue is still prevalent for me. [user@cmhost ~]$ rpm -qa | grep cloudera cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64 cloudera-manager-server-db-2-5.4.3-1.cm543.p0.258.el6.x86_64 cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64 cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64

cchahadoop · ‎12-01-2015

CM Version: Cloudera Enterprise 5.4.3 CDH Version: 5.4.4

cchahadoop · ‎11-05-2015

I was trying to access some of the HDFS reports (through the Reports tab) and I ran into the following error The Reports Manager (reportsmanager (cmhost)) could not complete the request: Aggregate data has not been initialized. Please retry later. If problem persists for several hours, please check the Reports Manager log for errors. When I check for the Report Manager's logs I see the following: 2015-11-05 18:01:27,865 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build 2015-11-05 18:01:27,877 ERROR com.cloudera.headlamp.AbstractIndexBuilder: Failed to build index from FSImage at /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage.tmp java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58) at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76) at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69) ... 9 more Caused by: java.io.IOException: Unsupported layout version -60 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106) at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69) ... 10 more 2015-11-05 18:01:27,880 ERROR com.cloudera.headlamp.HeadlampIndex: Failed to build index java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58) at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76) at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69) ... 9 more Caused by: java.io.IOException: Unsupported layout version -60 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106) at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69) ... 10 more 2015-11-05 18:01:27,882 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58) at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60 at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76) at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69) ... 9 more Caused by: java.io.IOException: Unsupported layout version -60 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106) at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69) ... 10 more Seems that it is failing to build index from FSImage? Can anyone highlight what this is?

cchahadoop · ‎11-04-2015

The asterisk (*) basically means that the NTP Daemon is in sync with the particular Remote Server (getting it's time from this server) Were you able to move past this issue? I'm facing a similar issue where the clocks randomly go out of sync on/off. I checked with my Network team for any firewall changes but there haven't been any.

cchahadoop · ‎10-30-2015

You can either allocate more space to it or lower the monitoring thresholds through CM. Generally, you need 10+ GB free space for the logs. You can lower the monitoring thresholds by going to each service in the CM, and searching for "log space" in the Configuration To allocate more space, you can move the logs to another folder and create a symlink from the existing folder. This way, you don't need to change the log directory location in CM. Note that, you would need to stop each Service before moving the logs.

cchahadoop · ‎10-27-2015

Through CM, Go to HBase >> Configuration and search for hbase-site. You should see a HBase Advanced Configuration Snippet (Safety Valve). Enter the xml in these depending on wherever you want (RegionServer, Master, REST etc). Through CM, all the non-defaults that you want to add to the hbase-site.xml must be done through the "Safety Valve".

cchahadoop · ‎10-23-2015

One of my nodes is over-loaded and I needed to move some of the services on it to a new node. I wanted to move the Oozier Server to another node but, since this is not a documented process and the cluster is in production right now, I wanted to know if there are some special cases to be taken care of? I understand that I need to do the following: 1. Stop Oozie Server 2. Add Oozie Server role on another host while deleting on the old one 3. Copy and paste the contents under /var/lib/oozie from the old host to the new host 4. Move the MySQL Connector Jar to under /usr/share/java 5. Start the new Oozie Server Am I missing something here?

cchahadoop · ‎10-22-2015

Are you using Cloudera Manager? If so, these properties can be found in the "Configuration" page of the HDFS Service. Just search for these properties in the tab on the side. Let me know which folder are you looking at? The current configurations for each service are usually stored in /var/run/cloudera-scm-agent/process/XYZ-NameOfTheService. But, it can be only changed through the CM UI.

cchahadoop · ‎10-22-2015

I have a cluster with 8 worker nodes (DN,NM and RS). The dev team are running a MapReduce program using an Oozie workflow. This step in the workflow is a MR job to enter the data into HBase tables. There are basically two things that happen 1. Heavy load on had01 causes the Region Server to shut down. The other Region Servers are working fine but the issue only seems to be on this one. I see a lot of JVM pauses in the log (Non GC) and it loses connection to the ZooKeepers before shutting down. 2. In the case the RS doesn't shut down, I still see heavy load on this node (121.2, 97.3, 87.3) and the map tasks that run on this node take much much more longer than on other nodes. Others nodes -> less than 2 mins Had01 -> 7 + mins Other Observations: 1. Heavy I/O (700 MB/s - 2 GB/s) 2. Number of blocks on this node is twice when compared to the other 7 nodes. 3. HBase Web UI shows the Write Request Count for this node as 0 Can someone point me where I can troubleshoot more? It only seems to happen when this step of the workflow is running. It comes back up anad is stable after this.

cchahadoop · ‎10-19-2015

2015-10-19 00:15:11,477 INFO [main] org.apache.zookeeper.ZooKeeper: Client environment:host.name=host1 2015-10-19 00:15:11,478 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x57101ba40x0, quorum=localhost:2181, baseZNode=/hbase 2015-10-19 00:15:11,490 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-10-19 00:15:11,491 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused When I run through Oozie, I see a lot of above warnings of it using localhost. When I run it through the Shell, I see that it's using the correct hostnames from the ZooKeeper Quorom instead of localhost.

cchahadoop · ‎10-19-2015

We have setup Oozie through CM in our Clustser. Lately, we noticed a workflow that runs longer than usual through Oozie. Ran the same Workflow through shell and it succeeded pretty quickly. Through the Oozie Web UI, I noticed that oozie.zookeeper.connection.string pointing to localhost:2181 instead of the ZooKeeper Quorom. Through CM, we are pointing it to use the ZooKeeper service so it should pick up the ZooKeeper hosts without having to over-ride in the oozie-site.xml file. Is this a bug?

cchahadoop · ‎10-10-2015

I'm running CDH5.4.3 using CM in my cluster. I have never used Search and I wanted to set it up using the latest SOLR package that includes Search. However, my parcels directory is pointing to an older version of SOLR. See this image here. In the parcels (using remote parcel repo), I'm pointing at http://archive.cloudera.com/search/parcels/latest/ . When going to this link, it doesn't seem to have the latest SOLR package for CDH5. I couldn't find the CDH5 related parcel for solr anywhere in archive.cloudera.

cchahadoop · ‎10-09-2015

I'm having the same issue. I want to store in the re-defined MySQL database as I don't want to use stats through Impala. The documentation at Cloudera isn't quite clear for the newer version. I followed for v5.2 but seems that Hive has changed its implementation of Stats since then. Can I use the older implementation still?

cchahadoop · ‎10-09-2015

I'm using CM 5.4.3 on my cluster which is also running Impala. I tried setting up Hive following the documentation here but it doesn't seem to be the recommended way anymore. Cloudera recommends to leverage Impala to enable stats. I want to use the additional functionalities that come with enabling Hive Stats (vectorization etc.). The other way is just to enable hive.stats.autogather in the Hive Safety Valve. Will only doing this suffice?

cchahadoop · ‎10-08-2015

I'm submitting an Oozie workflow through Hue but it returns a Server 500 error. I tried looking at the Oozie logs in (/var/log/oozie/) but they didn't show anything. The Hue logs aren't helpful either. Can anyone point me where I should be looking at? P.S. I use CM 5.4.x

cchahadoop · ‎10-08-2015

Through CM, You'd first have to enable the Capacity Scheduler by changing the yarn.resourcemanager.scheduler.class from Fair to Capacity in YARN Configuration. Then, add the required configuration in the Capacity Scheduler Configuration (xml) file through CM in YARN Configurations. See this for more info about it. After adding the required properties, you may have to restart YARN or the ResourceManager.

cchahadoop · ‎10-07-2015

Over the weekend, I noticed that the cluster was running really slow. All the jobs that otherwise ran fine took longer to complete. Even a simple HDFS lookup (hdfs dfs -ls /) took about 20+ seconds. When I ran the debug, I noticed that it took a lot of time to get the client out of cache [user@host ~]$ HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -ls / 15/10/07 14:37:05 DEBUG util.Shell: setsid exited with exit code 0 15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-common-2.6.0-cdh5.4.4.jar!/core-default.xml 15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7c594d5e 15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml 15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@22157ace 15/10/07 14:37:05 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops) . . . 15/10/07 14:37:06 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@7505062c 15/10/07 14:37:22 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... . . . ^ Please notice the time difference highlighted in bold above. It takes about 16s for it to get the client out of cache. Our CPU utilization is very low (96% idle) and every thing else seems fine. Also, note that we have made no configuration changes on this cluster whatsoever.

cchahadoop · ‎10-05-2015

I'm using CM v5.4.4 and I noticed this morning that the cluster had slowed down. The queries are taking longer to execute and even doing a simple list of the data (hdfs dfs -ls /) is taking more than 20seconds (it was quicker before). Doing a top on the DN and the NN's, I noticed that the cm-agent was taking about 150% of the CPU. Also, in the YARN logs, I noticed a lot of the following messages 2015-10-05 15:33:37,312 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...

cchahadoop · ‎09-30-2015

Do you have selinux enabled? What about the firewall (iptables)?

Online	Offline
Last Visited	‎02-28-2018 09:12 AM

Date Registered	‎09-03-2015 10:18 AM
Last Visited	‎02-28-2018 09:12 AM
Total Messages Posted	34
Total Kudos Received	3

Re: Some Services Showing Free Space Warning How t...

Re: HBase configuration via Cloudera Manager

Re: HDFS Configuration

Re: Moving CM Services to a new node

Re: Error Hive query - hive not allowed to do 'GET...

Re: How to resolve 'SASL negotiation failure' of ...

Re: How to resolve 'SASL negotiation failure' of ...

Re: How to resolve 'SASL negotiation failure' of ...

How does Impala handle heterogeneous hardware?

SHOW TABLE STATS not working for HBase tables afte...

Re: Hbase Soket TimeOut Exception

Re: Reports Manager - Aggregate Data has not been ...

Re: Reports Manager - Aggregate Data has not been ...

Re: Reports Manager - Aggregate Data has not been ...

Re: Impala, Llama, and YARNon CDH 5.5

Re: Reports Manager - Aggregate Data has not been ...

Re: Reports Manager - Aggregate Data has not been ...

Reports Manager - Aggregate Data has not been init...

Re: Cloudera 5.4.x cluster randomly reports "Clock...

Re: Some Services Showing Free Space Warning How t...

Re: HBase configuration via Cloudera Manager

Movie Oozier Server to a new Node

Re: HDFS Configuration

Tasks take long only on one node

Re: Oozie doesn't pickup the right ZooKeepers thro...

Oozie doesn't pickup the right ZooKeepers through ...

Can't find the latest SOLR package in CM

Re: Configuring Hive to Store Statistics in MySQL

Enable Hive Table Stats in CM 5.4

Oozie logs through Hue

Re: Fair Scheduler to Capacity Scheduler

Getting Client out of cache takes long time

CM Agent Using a lot of CPU

Re: Cloudmanager is not able to start