Member since
06-30-2018
12
Posts
0
Kudos Received
0
Solutions
08-14-2018
01:15 AM
Thanks. This makes sense given the behavior I am seeing. I haven't tried this from Beeline yet, but have from the Hive CLI. I'll take a look at that and post the results. Should Beeline, and the Hive CLI both produce the same results? Should they be allocated containers even though Hive view is not getting them?.. I've seen circumstances where the Hive CLI will also wait, and others, where it will be able to run, even when the Hive View is unable to run the query.
... View more
08-08-2018
02:39 AM
It is a small cluster. 4 nodes. Each 8v core, 16Gb. 3 nodes with master components. 1 data node, 4GB max container size. 12Gb total Yarn. Interestingly, Tez jobs from the Hive CLI will run and will be assigned containers.
... View more
08-06-2018
02:15 AM
Looking at Resource Manager UI shows that an AM has been assigned by yarn, however the AM is never assigned a container, since another Tez AM has never registered as completing it's job. Going into Tez view of the offending "running" application, reveals that all tasks of the application are marked as complete, but the job shows as running. Eventually Hive View 2 displays the following error: java.lang.Exception: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
java.lang.Exception: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.ambari.view.hive20.resources.jobs.JobService.getOne(JobService.java:147)
at sun.reflect.GeneratedMethodAccessor408.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
at org.apache.ambari.view.hive20.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49)
at org.apache.ambari.view.hive20.actor.StatementExecutor.runStatement(StatementExecutor.java:91)
at org.apache.ambari.view.hive20.actor.StatementExecutor.handleMessage(StatementExecutor.java:72)
at org.apache.ambari.view.hive20.actor.HiveActor.onReceive(HiveActor.java:38)
at akka.actor.UntypedActor$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
... View more
Labels:
08-06-2018
02:00 AM
This issue was likely caused by installing all master components (Named Node, RM, Hive Server 2, Druid, etc) on a single node. Redeploying a cluster with the master components distributed across 3 nodes resulted in a more stable cluster -- no more out of memory errors.
... View more
07-25-2018
03:09 AM
@Jay Kumar SenSharma....I just realized you had asked for the log file for this. Apologies for missing that. I have attached. hs-err-pid4314log.txt
... View more
07-12-2018
07:17 PM
Cache pool usage rights can be assigned to linux groups using -addPool - group. For Hive workloads where Ranger policies are in use, is there a way to constrain cache usage for different Hive users / groups via those Ranger policies, or is there a better way of managing cache resource usage in such a scenario.
... View more
- Tags:
- Hadoop Core
- HDFS
Labels:
07-12-2018
06:19 PM
In cache directives, the number of block replicas to cache are defined. What would be the benefit of caching replicas of a block? Why not cache a single copy of the block?
... View more
- Tags:
- Hadoop Core
- HDFS
Labels:
07-12-2018
04:58 PM
When assigning storage policies, and assigning storage to data nodes, dfs.datanode.data.dir is updated in hdfs-site.xml. Does this mean that all drives on the data node should be the same. I.e. best practice is to not mix drive types on a node, or is there additional granularity that can be applied to allow multiple storage types on the node.
... View more
Labels:
07-12-2018
02:26 AM
Hi @Jay Kumar SenSharma on the named node free-m gives total used free shared buff/cache available 14878 14088 250 198 540 325 Swap: 0 0 0 On a data node: total used free shared buff/cache available 14878 5085 872 24 8920 9434 Swap: 0 0 0 /proc/sys/vm/overcommit_memory was set to 0
# echo $JAVA_OPTIONS # echo $HADOOP_OPTS # echo $HADOOP_CLIENT_OPTS were all undefined the log shows: --------------- S Y S T E M ---------------
OS:CentOS Linux release 7.2.1511 (Core)
uname:Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 59369, NOFILE 4096, AS infinity
load average:2.56 1.96 2.06
/proc/meminfo:
MemTotal: 15235812 kB
MemFree: 157172 kB
MemAvailable: 109724 kB
Buffers: 0 kB
Cached: 219468 kB
SwapCached: 0 kB
Active: 14636256 kB
Inactive: 76244 kB
Active(anon): 14626136 kB
Inactive(anon): 67536 kB
Active(file): 10120 kB
Inactive(file): 8708 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 412 kB
Writeback: 0 kB
AnonPages: 14496064 kB
Mapped: 40920 kB
Shmem: 198124 kB
Slab: 130892 kB
SReclaimable: 55204 kB
SUnreclaim: 75688 kB
KernelStack: 55232 kB
PageTables: 57976 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 7617904 kB
Committed_AS: 27413956 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 167920 kB
VmallocChunk: 34359566840 kB
HardwareCorrupted: 0 kB
AnonHugePages: 8243200 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 94196 kB
DirectMap2M: 3051520 kB
DirectMap1G: 14680064 kB
... View more
07-12-2018
01:34 AM
E.g. hdfs dfs -put const.txt /qtests/file3 (file is <1MB) Error is: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f3400000, 123731968, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 123731968 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/centos/hs_err_pid4314.log ... Max Heap size on data nodes is 1GB.
... View more
Labels:
07-12-2018
01:08 AM
Hi @Vinicius Higa Murakami that gives the following output: sh-3.2# find / -name "hiveodbc" -type d -exec ls -ltrah {} \;
find: /private/var/db/ConfigurationProfiles/Store: Operation not permitted
find: /private/var/folders/6z/xqxn5gsd0h113x1hhj1csvhr0000gq/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/6z/xqxn5gsd0h113x1hhj1csvhr0000gq/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/6z/xqxn5gsd0h113x1hhj1csvhr0000gq/0/com.apple.routined: Operation not permitted
find: /private/var/folders/zb/mtpn5rwx3gb281dns8kxs1w00000gp/0/SafariFamily: Operation not permitted
find: /private/var/folders/zb/mtpn5rwx3gb281dns8kxs1w00000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/zb/mtpn5rwx3gb281dns8kxs1w00000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/zb/mtpn5rwx3gb281dns8kxs1w00000gp/0/com.apple.routined: Operation not permitted
find: /private/var/folders/mz/cl3l11ds58ggcs943fvj6cjh0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/mz/cl3l11ds58ggcs943fvj6cjh0000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/mz/cl3l11ds58ggcs943fvj6cjh0000gp/0/com.apple.routined: Operation not permitted
find: /private/var/folders/c_/qkgbxrpn3ps9glr5p9d1plwm0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/c_/qkgbxrpn3ps9glr5p9d1plwm0000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/c_/qkgbxrpn3ps9glr5p9d1plwm0000gp/0/com.apple.routined: Operation not permitted
find: /private/var/folders/18/9mbvnrk539x8gdykyhfl4k380000gp/0/SafariFamily: Operation not permitted
find: /private/var/folders/18/9mbvnrk539x8gdykyhfl4k380000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/18/9mbvnrk539x8gdykyhfl4k380000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/18/9mbvnrk539x8gdykyhfl4k380000gp/0/com.apple.routined: Operation not permitted
find: /private/var/folders/r9/w_42wmhn7dq4s6l7lrc69w6c0000gn/0/SafariFamily: Operation not permitted
find: /private/var/folders/r9/w_42wmhn7dq4s6l7lrc69w6c0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/r9/w_42wmhn7dq4s6l7lrc69w6c0000gn/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/r9/w_42wmhn7dq4s6l7lrc69w6c0000gn/0/com.apple.routined: Operation not permitted
find: /private/var/folders/5m/x60c1__j12373d31fd5bdftm0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/5m/x60c1__j12373d31fd5bdftm0000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/5m/x60c1__j12373d31fd5bdftm0000gp/0/com.apple.routined: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/SafariFamily: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.routined: Operation not permitted
find: /private/var/folders/r4/rkr68lhn2n33hhsbr8n6fpzh0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
find: /private/var/folders/r4/rkr68lhn2n33hhsbr8n6fpzh0000gp/0/com.apple.nsurlsessiond: Operation not permitted
find: /private/var/folders/r4/rkr68lhn2n33hhsbr8n6fpzh0000gp/0/com.apple.routined: Operation not permitted
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
... View more
07-06-2018
01:09 AM
When installing Hive ODBC Driver on OS X (10.13.5) - using instructions at : Hortonworks Hive ODBC Driver User Guide - July 12, 2016 pkgutil --info com.hortonworks.hiveodbc returns No receipt for 'com.hortonworks.hiveodbc' found at '/'. The driver files are described as being installed at /opt/hortonworks/hiveodbc, however, there is no such folder after install.
... View more
Labels:
