Member since
09-03-2015
34
Posts
3
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
982 | 10-30-2015 08:56 AM | |
2428 | 10-27-2015 09:47 AM | |
760 | 10-22-2015 01:42 PM | |
886 | 09-04-2015 08:28 AM |
03-08-2017
12:35 PM
I added hive user to the below two properties in the kms-acls.xml to make it work. hadoop.kms.acl.GET_METADATA whitelist.key.acl.READ Also, I noticed that this happens only when we are doing a JOIN on two tables. When we are querying a single table, it is fine. Are the GET_METADATA and READ operations happening on HDFS on the parquet metadata OR on the Hive Warehouse?
... View more
09-14-2016
08:24 AM
I looked in a little more depth on this issue. Here are the findings: 1. When I use 1 HS2 behind the LoadBalancer, it succeeds without any error 2. When I use 2 HS2 behind the LoadBalancer, I see one error but it retries and succeeds 3. When I use 3 or more than 3 HS2 behind the LoadBalancer, it errors out. I have tried turning the sticky connections for TCP inside the LoadBalancer but to no avail. Looks like there's some sort of Kerberos handshake for which it comes back to the LoadBalancer. Here, it'll then hit another server and then fail.
... View more
08-23-2016
08:21 AM
I was able to resolve this issue by using HS2 on DataNodes. I don't have HMS HA but I do have multiple Hive Servers behind a load balancer. The problem arose when I moved the HS2 roles to application nodes (non-Datanodes) but when I moved them back to the DataNodes, these errors swiftly disappeared.
... View more
08-19-2016
08:38 AM
I am facing the same issue as well. How are you passing the JDBC URL to Oozie?
... View more
07-13-2016
09:31 AM
Suppose we have 8 DataNodes with 128 GB RAM each of which 64 GB is allocated to Impala and we add 8 more nodes of 256 GB and we intend to allocated 128 GB to Impala. My concern is, will the coordinator be smart enough to know the mem_limit of each node it sends fragments to? Are there any other known issues that come up with such a configuration? NOTE: Current CDH version is CDH 5.4.4 (Impala v2.2)
... View more
04-05-2016
08:24 AM
We have recently enabled Sentry in our cluster. When I try to do a SHOW TABLE STATS on a HBase Table through the Impala shell, it runs into the following permission error: 0: jdbc:hive2://LB> show table stats database.table_summary; Error: RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=impala, access=EXECUTE, inode="/hbase":hbase:hbase:drwx------ I have tried adding the impala user to the hbase.superuser but still see the same issues. I have also granted impala and hive RWACX through the hbase shell. Is there something else that needs to be done? Thanks
... View more
12-15-2015
11:38 AM
Hi Dice, Have you had the chance to look at those logs?
... View more
12-04-2015
10:48 AM
You can find the Reports Manager logs here. Changed the hostnames for security reasons 🙂
... View more
12-03-2015
07:53 AM
This is the CM host. We do not have hadoop installed on this node. I ran the Host Inspector which alerts me that hdfs,hadoop and other such users are not present on this node and also this returns CDH version as "None". Could this be causing the Reports Manager to fail? We have a similar cluster (where CM host reports CDH version as NONE) but we do not see the same issue there. That cluster runs CM v5.4.7 and CDH 5.4.3.
... View more
12-02-2015
01:30 PM
Can you please confirm/deny the following conclusions I have made by reading the new 5.5 docs, your above post and testing "Static Service Pools" in a test environment: 1. Impala's integration with YARN through Llama is being discontinued from CDH 5.5/Impala 2.3 onwards? 2. Configuring "Static Service Pools" means that both Impala and YARN (and other services too) are managed by cgroups rather than Impala being managed by YARN? If point # 2 is true, it basically means that Impala can no longer be managed by YARN CDH 5.5 onwards. I tried enabling the "Static Service Pools" but I do not see the Impala queries I ran in YARN Applications.
... View more
12-02-2015
08:27 AM
Yes, this issue is still prevalent for me. [user@cmhost ~]$ rpm -qa | grep cloudera
cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server-db-2-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64
... View more
12-01-2015
09:49 AM
CM Version: Cloudera Enterprise 5.4.3 CDH Version: 5.4.4
... View more
11-05-2015
10:10 AM
I was trying to access some of the HDFS reports (through the Reports tab) and I ran into the following error
The Reports Manager (reportsmanager (cmhost)) could not complete the request: Aggregate data has not been initialized. Please retry later.
If problem persists for several hours, please check the Reports Manager log for errors.
When I check for the Report Manager's logs I see the following:
2015-11-05 18:01:27,865 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build
2015-11-05 18:01:27,877 ERROR com.cloudera.headlamp.AbstractIndexBuilder: Failed to build index from FSImage at /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage.tmp
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
... 10 more
2015-11-05 18:01:27,880 ERROR com.cloudera.headlamp.HeadlampIndex: Failed to build index
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
... 10 more
2015-11-05 18:01:27,882 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
... 10 more
Seems that it is failing to build index from FSImage? Can anyone highlight what this is?
... View more
Labels:
11-04-2015
01:25 PM
The asterisk (*) basically means that the NTP Daemon is in sync with the particular Remote Server (getting it's time from this server) Were you able to move past this issue? I'm facing a similar issue where the clocks randomly go out of sync on/off. I checked with my Network team for any firewall changes but there haven't been any.
... View more
10-30-2015
08:56 AM
You can either allocate more space to it or lower the monitoring thresholds through CM. Generally, you need 10+ GB free space for the logs. You can lower the monitoring thresholds by going to each service in the CM, and searching for "log space" in the Configuration To allocate more space, you can move the logs to another folder and create a symlink from the existing folder. This way, you don't need to change the log directory location in CM. Note that, you would need to stop each Service before moving the logs.
... View more
10-27-2015
09:47 AM
2 Kudos
Through CM, Go to HBase >> Configuration and search for hbase-site. You should see a HBase Advanced Configuration Snippet (Safety Valve). Enter the xml in these depending on wherever you want (RegionServer, Master, REST etc). Through CM, all the non-defaults that you want to add to the hbase-site.xml must be done through the "Safety Valve".
... View more
10-23-2015
02:11 PM
One of my nodes is over-loaded and I needed to move some of the services on it to a new node. I wanted to move the Oozier Server to another node but, since this is not a documented process and the cluster is in production right now, I wanted to know if there are some special cases to be taken care of? I understand that I need to do the following: 1. Stop Oozie Server 2. Add Oozie Server role on another host while deleting on the old one 3. Copy and paste the contents under /var/lib/oozie from the old host to the new host 4. Move the MySQL Connector Jar to under /usr/share/java 5. Start the new Oozie Server Am I missing something here?
... View more
10-22-2015
01:42 PM
Are you using Cloudera Manager? If so, these properties can be found in the "Configuration" page of the HDFS Service. Just search for these properties in the tab on the side. Let me know which folder are you looking at? The current configurations for each service are usually stored in /var/run/cloudera-scm-agent/process/XYZ-NameOfTheService. But, it can be only changed through the CM UI.
... View more
10-22-2015
01:33 PM
I have a cluster with 8 worker nodes (DN,NM and RS). The dev team are running a MapReduce program using an Oozie workflow. This step in the workflow is a MR job to enter the data into HBase tables. There are basically two things that happen 1. Heavy load on had01 causes the Region Server to shut down. The other Region Servers are working fine but the issue only seems to be on this one. I see a lot of JVM pauses in the log (Non GC) and it loses connection to the ZooKeepers before shutting down. 2. In the case the RS doesn't shut down, I still see heavy load on this node (121.2, 97.3, 87.3) and the map tasks that run on this node take much much more longer than on other nodes. Others nodes -> less than 2 mins Had01 -> 7 + mins Other Observations: 1. Heavy I/O (700 MB/s - 2 GB/s) 2. Number of blocks on this node is twice when compared to the other 7 nodes. 3. HBase Web UI shows the Write Request Count for this node as 0 Can someone point me where I can troubleshoot more? It only seems to happen when this step of the workflow is running. It comes back up anad is stable after this.
... View more
10-19-2015
06:20 PM
2015-10-19 00:15:11,477 INFO [main] org.apache.zookeeper.ZooKeeper: Client environment:host.name=host1
2015-10-19 00:15:11,478 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x57101ba40x0, quorum=localhost:2181, baseZNode=/hbase
2015-10-19 00:15:11,490 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-10-19 00:15:11,491 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused When I run through Oozie, I see a lot of above warnings of it using localhost. When I run it through the Shell, I see that it's using the correct hostnames from the ZooKeeper Quorom instead of localhost.
... View more
10-19-2015
02:22 PM
We have setup Oozie through CM in our Clustser. Lately, we noticed a workflow that runs longer than usual through Oozie. Ran the same Workflow through shell and it succeeded pretty quickly. Through the Oozie Web UI, I noticed that oozie.zookeeper.connection.string pointing to localhost:2181 instead of the ZooKeeper Quorom. Through CM, we are pointing it to use the ZooKeeper service so it should pick up the ZooKeeper hosts without having to over-ride in the oozie-site.xml file. Is this a bug?
... View more
10-10-2015
08:26 AM
I'm running CDH5.4.3 using CM in my cluster. I have never used Search and I wanted to set it up using the latest SOLR package that includes Search. However, my parcels directory is pointing to an older version of SOLR. See this image here. In the parcels (using remote parcel repo), I'm pointing at http://archive.cloudera.com/search/parcels/latest/ . When going to this link, it doesn't seem to have the latest SOLR package for CDH5. I couldn't find the CDH5 related parcel for solr anywhere in archive.cloudera.
... View more
10-09-2015
08:32 AM
I'm having the same issue. I want to store in the re-defined MySQL database as I don't want to use stats through Impala. The documentation at Cloudera isn't quite clear for the newer version. I followed for v5.2 but seems that Hive has changed its implementation of Stats since then. Can I use the older implementation still?
... View more
10-09-2015
07:56 AM
I'm using CM 5.4.3 on my cluster which is also running Impala. I tried setting up Hive following the documentation here but it doesn't seem to be the recommended way anymore. Cloudera recommends to leverage Impala to enable stats. I want to use the additional functionalities that come with enabling Hive Stats (vectorization etc.). The other way is just to enable hive.stats.autogather in the Hive Safety Valve. Will only doing this suffice?
... View more
10-08-2015
10:56 AM
I'm submitting an Oozie workflow through Hue but it returns a Server 500 error. I tried looking at the Oozie logs in (/var/log/oozie/) but they didn't show anything. The Hue logs aren't helpful either. Can anyone point me where I should be looking at? P.S. I use CM 5.4.x
... View more
10-08-2015
10:52 AM
Through CM, You'd first have to enable the Capacity Scheduler by changing the yarn.resourcemanager.scheduler.class from Fair to Capacity in YARN Configuration. Then, add the required configuration in the Capacity Scheduler Configuration (xml) file through CM in YARN Configurations. See this for more info about it. After adding the required properties, you may have to restart YARN or the ResourceManager.
... View more
10-07-2015
07:43 AM
Over the weekend, I noticed that the cluster was running really slow. All the jobs that otherwise ran fine took longer to complete. Even a simple HDFS lookup (hdfs dfs -ls /) took about 20+ seconds. When I ran the debug, I noticed that it took a lot of time to get the client out of cache [user@host ~]$ HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -ls /
15/10/07 14:37:05 DEBUG util.Shell: setsid exited with exit code 0
15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-5.4.4-1.cdh5.4.4.p0.4/jars/hadoop-common-2.6.0-cdh5.4.4.jar!/core-default.xml
15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7c594d5e
15/10/07 14:37:05 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
15/10/07 14:37:05 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@22157ace
15/10/07 14:37:05 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
.
.
.
15/10/07 14:37:06 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@7505062c
15/10/07 14:37:22 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... . . . ^ Please notice the time difference highlighted in bold above. It takes about 16s for it to get the client out of cache. Our CPU utilization is very low (96% idle) and every thing else seems fine. Also, note that we have made no configuration changes on this cluster whatsoever.
... View more
10-05-2015
08:45 AM
I'm using CM v5.4.4 and I noticed this morning that the cluster had slowed down. The queries are taking longer to execute and even doing a simple list of the data (hdfs dfs -ls /) is taking more than 20seconds (it was quicker before). Doing a top on the DN and the NN's, I noticed that the cm-agent was taking about 150% of the CPU. Also, in the YARN logs, I noticed a lot of the following messages 2015-10-05 15:33:37,312 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...
... View more