question Re: S020 Data storage error in Support Questions

S020 Data storage error

prakashpunj — Mon, 02 Oct 2017 09:27:54 GMT

How to fix it. Getting this in Hive View. Looked at the similar posting but not very sure what actually needs to be done. Recently upgraded from Ambari 2.2 to 2.5

Re: S020 Data storage error

jsensharma — Mon, 02 Oct 2017 11:19:23 GMT

@Prakash Punj

"S020 Data storage error" is a generic error, Hence in order to findout the actual cause of failure, we will need to look at the detailed stacktrace of this error.

So can you please check and share the "Hive View" logs and also the ambari-server.log.

Re: S020 Data storage error

kalaiselvan_moo — Mon, 02 Oct 2017 12:29:32 GMT

@Prakash Punj,

Can you check if any of the filesystem got full / any of your disks become bad.

Re: S020 Data storage error

prakashpunj — Mon, 02 Oct 2017 19:45:52 GMT

@kalai selvan.

I don't think space is a problem.

[centos@hdp-m:/usr/hdp/ssl ] $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       160G   73G   88G  46% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G   12K  7.8G   1% /dev/shm
tmpfs           7.8G  788M  7.0G  10% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/1000
tmpfs           1.6G     0  1.6G   0% /run/user/1011
tmpfs           1.6G     0  1.6G   0% /run/user/1003
tmpfs           1.6G     0  1.6G   0% /run/user/1008
tmpfs           1.6G     0  1.6G   0% /run/user/0

Re: S020 Data storage error

prakashpunj — Mon, 02 Oct 2017 19:57:57 GMT

Hive view log: This is the last 50 lines of text from the recent hiveserver2.log.2017-10-01

er-Pool: Thread-40
2017-10-01 23:54:23,382 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:54:23,401 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to be353dbf-a100-4c3b-85a9-f2e1f707e423 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:54:23,401 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:54:23,450 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to be353dbf-a100-4c3b-85a9-f2e1f707e423 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:54:23,450 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to be353dbf-a100-4c3b-85a9-f2e1f707e423 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:54:23,476 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:23,693 INFO  [HiveServer2-Handler-Pool: Thread-40]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
2017-10-01 23:57:23,696 INFO  [HiveServer2-Handler-Pool: Thread-40]: metastore.ObjectStore (ObjectStore.java:initialize(294)) - ObjectStore, initialize called
2017-10-01 23:57:23,726 INFO  [HiveServer2-Handler-Pool: Thread-40]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(140)) - Using direct SQL, underlying DB is MYSQL
2017-10-01 23:57:23,727 INFO  [HiveServer2-Handler-Pool: Thread-40]: metastore.ObjectStore (ObjectStore.java:setConf(277)) - Initialized ObjectStore
2017-10-01 23:57:23,858 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.SessionState (SessionState.java:createPath(613)) - Created local directory: /tmp/94104db2-4ff2-40b1-97eb-e6056fd7e376_resources
2017-10-01 23:57:23,876 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.SessionState (SessionState.java:createPath(613)) - Created HDFS directory: /tmp/hive/anonymous/94104db2-4ff2-40b1-97eb-e6056fd7e376
2017-10-01 23:57:23,883 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.SessionState (SessionState.java:createPath(613)) - Created local directory: /tmp/hive/94104db2-4ff2-40b1-97eb-e6056fd7e376
2017-10-01 23:57:23,905 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.SessionState (SessionState.java:createPath(613)) - Created HDFS directory: /tmp/hive/anonymous/94104db2-4ff2-40b1-97eb-e6056fd7e376/_tmp_space.db
2017-10-01 23:57:23,908 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(248)) - Operation log session directory is created: /tmp/hive/operation_logs/94104db2-4ff2-40b1-97eb-e6056fd7e376
2017-10-01 23:57:24,151 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to 94104db2-4ff2-40b1-97eb-e6056fd7e376 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,152 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,191 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to 94104db2-4ff2-40b1-97eb-e6056fd7e376 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,192 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,240 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to 94104db2-4ff2-40b1-97eb-e6056fd7e376 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,240 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:acquire(295)) - We are setting the hadoop caller context to 94104db2-4ff2-40b1-97eb-e6056fd7e376 for thread HiveServer2-Handler-Pool: Thread-40
2017-10-01 23:57:24,284 INFO  [HiveServer2-Handler-Pool: Thread-40]: session.HiveSessionImpl (HiveSessionImpl.java:release(309)) - We are resetting the hadoop caller context for thread HiveServer2-Handler-Pool: Thread-40

Re: S020 Data storage error

prakashpunj — Mon, 02 Oct 2017 20:00:14 GMT

@Jay SenSharma

This is the last 50 lines from ambari-server log

[centos@hdp-m:/var/log/ambari-server ] $ tail -50 ambari-server.log
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
        at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
        at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1045)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)
01 Oct 2017 23:20:07,110  INFO [pool-17-thread-1] MetricSinkWriteShardHostnameHashingStrategy:42 - Calculated collector shard hdp-m.asotc based on hostname: hdp-m.asotc
02 Oct 2017 00:30:10,487  INFO [pool-17-thread-1] MetricSinkWriteShardHostnameHashingStrategy:42 - Calculated collector shard hdp-m.asotc based on hostname: hdp-m.asotc
02 Oct 2017 01:40:14,577  INFO [pool-17-thread-1] MetricSinkWriteShardHostnameHashingStrategy:42 - Calculated collector shard hdp-m.asotc based on hostname: hdp-m.asotc
02 Oct 2017 02:50:18,344  INFO [pool-17-thread-1] MetricSinkWriteShardHostnameHashingStrategy:42 - Calculated

Re: S020 Data storage error

jsensharma — Mon, 02 Oct 2017 23:10:03 GMT

@Prakash Punj

The following error stack trace is not complete, We need to see from where this error begins. So can you please share the complete stackTrace of the following thread stack:

[centos@hdp-m:/var/log/ambari-server ] $ tail -50 ambari-server.log
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

Re: S020 Data storage error

prakashpunj — Tue, 03 Oct 2017 01:58:26 GMT

@Jay SenSharma

ambari-server log file is too big to be attached here. How you want me to deliver ?

Re: S020 Data storage error

prakashpunj — Tue, 03 Oct 2017 02:09:27 GMT

@Jay SenSharma

Sending you last 2 days of log trace of ambari-server.log file. It's a zip fileambari.zip

Re: S020 Data storage error

prakashpunj — Wed, 04 Oct 2017 22:49:23 GMT

@Jay SenSharma

Any update on this ?

Thanks

Re: S020 Data storage error

jsensharma — Wed, 04 Oct 2017 23:05:06 GMT

@Prakash Punj

Unfortunately in the "ambari.zip" i did not see any where the error that you mentioned "S020 Data storage error"

$ grep 'S020' ~/Downloads/ambari.txt

Re: S020 Data storage error

prakashpunj — Sun, 18 Aug 2019 04:57:53 GMT

@Jay SenSharma

Here is the situation, I upgraded from Ambari 2.2.0.0 to Ambari 2.5. After upgrade when I use Ambari Hive View and try to browse the data, I get this error S0202 data Storage error. If I do the same thing using Hive view 2.0, I don't see the error but there is no output for "Select * from table limit 10" command.

Re: S020 Data storage error

jsensharma — Wed, 04 Oct 2017 23:30:30 GMT

@Prakash Punj

In the above screenshot you will find a message box with 3 messages ... Can you share the complete output of the S020 error by clicking on the messages icon.

Re: S020 Data storage error

prakashpunj — Wed, 04 Oct 2017 23:33:52 GMT

Sure here you go @Jay SenSharma

org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin1, hiveQueryId='null, dagId='null, queryId='null}

org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin1, hiveQueryId='null, dagId='null, queryId='null}
	at org.apache.ambari.server.view.persistence.DataStoreImpl.throwPersistenceException(DataStoreImpl.java:657)
	at org.apache.ambari.server.view.persistence.DataStoreImpl.store(DataStoreImpl.java:157)
	at org.apache.ambari.view.hive2.persistence.DataStoreStorage.store(DataStoreStorage.java:61)
	at org.apache.ambari.view.hive2.resources.CRUDResourceManager.save(CRUDResourceManager.java:117)
	at org.apache.ambari.view.hive2.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:68)
	at org.apache.ambari.view.hive2.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:34)
	at org.apache.ambari.view.hive2.resources.CRUDResourceManager.create(CRUDResourceManager.java:59)
	at org.apache.ambari.view.hive2.resources.jobs.viewJobs.JobResourceManager.create(JobResourceManager.java:51)
	at org.apache.ambari.view.hive2.resources.jobs.JobService.create(JobService.java:519)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212)
	at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201)
	at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1045)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "ds_jobimpl_206_pkey"
  Detail: Key (ds_id)=(52) already exists.
Error Code: 0
Call: INSERT INTO DS_JOBIMPL_206 (DS_id, DS_applicationId, DS_confFile, DS_dagId, DS_dagName, DS_dataBase, DS_dateSubmitted, DS_duration, DS_forcedContent, DS_globalSettings, DS_guid, DS_hiveQueryId, DS_logFile, DS_owner, DS_queryFile, DS_queryId, DS_referrer, DS_sessionTag, DS_sqlState, DS_status, DS_statusDir, DS_statusMessage, DS_title) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
	bind => [23 parameters bound]
	at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
	at org.apache.ambari.server.view.persistence.DataStoreImpl.store(DataStoreImpl.java:142)
	... 101 more
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "ds_jobimpl_206_pkey"
  Detail: Key (ds_id)=(52) already exists.
Error Code: 0
Call: INSERT INTO DS_JOBIMPL_206 (DS_id, DS_applicationId, DS_confFile, DS_dagId, DS_dagName, DS_dataBase, DS_dateSubmitted, DS_duration, DS_forcedContent, DS_globalSettings, DS_guid, DS_hiveQueryId, DS_logFile, DS_owner, DS_queryFile, DS_queryId, DS_referrer, DS_sessionTag, DS_sqlState, DS_status, DS_statusDir, DS_statusMessage, DS_title) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
	bind => [23 parameters bound]
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
	at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
	at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
	at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
	at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
	at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
	... 102 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "ds_jobimpl_206_pkey"
  Detail: Key (ds_id)=(52) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
	... 114 more

Re: S020 Data storage error

jsensharma — Thu, 05 Oct 2017 00:00:00 GMT

@Prakash Punj

Looks like recently you have migrated some "Saved Queries" from Hive View 1.0 to Hive View 1.5

If we do the migration twice then we might see the same error when we next time try to run any query:

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "ds_jobimpl_206_pkey"  Detail: Key (ds_id)=(52) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)

As a workaround you can try the following:

1. Take a DB dump of your Ambari Database.

2. Find the Max Sequence ID in the "ambari_sequences" table of ambari Databse.

# psql -U ambari ambari
Password for user ambari: 
psql (9.2.18)
Type "help" for help.
ambari=> select * from ambari_sequences;

You will find a sequence ID as "ds_jobimpl_206_id_seq" you will need to increase it's value to a high value to avoid the "duplicate key value violates unique constraint" issue.

So update the value for this as following: (setting a very high value 12000 intentionally)

UPDATE ambari_sequences set sequence_value=12000 WHERE sequence_name='ds_jobimpl_206_id_seq';

Then restart ambari server

It looks related to the BUG: https://issues.apache.org/jira/browse/AMBARI-21977 (not resolved yet so we should try the workaround for now)

Re: S020 Data storage error

prakashpunj — Thu, 05 Oct 2017 01:03:33 GMT

Thanks a bunch @Jay SenSharma

That resolved the S020 Storage Error but I noticed two more error:

Cannot browse data stored on older tables
Created new tables from this tutorial (https://hortonworks.com/tutorial/how-to-process-data-with-apache-hive/#download-the-data) BUT not able to browse data after first "Insert command". I see error below when I give that insert command:

java.lang.Exception: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask


java.lang.Exception: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
	at org.apache.ambari.view.hive2.resources.jobs.JobService.getOne(JobService.java:143)
	at sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)

Re: S020 Data storage error

jsensharma — Thu, 05 Oct 2017 01:08:01 GMT

@Prakash Punj

The recent error is completely different from the previous one. Hence i will suggest you to open a separate thread that way we can make the HCC threads more useful for other HCC users by separating different issues/errors as part of different threads.

Please close this thread by accepting the answer that resolved the issue for other HCC users to quickly browse for a solution at a quick look.

Re: S020 Data storage error

jsensharma — Mon, 09 Oct 2017 23:12:27 GMT

@Prakash Punj

As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue.