Member since
08-08-2013
339
Posts
132
Kudos Received
27
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 14898 | 01-18-2018 08:38 AM | |
| 1591 | 05-11-2017 06:50 PM | |
| 9237 | 04-28-2017 11:00 AM | |
| 3457 | 04-12-2017 01:36 AM | |
| 2856 | 02-14-2017 05:11 AM |
05-16-2014
05:25 AM
Hi, I have a 'simple' usecase (I thought it is simple 😉 ). I want to call a hive script with a parameter in format YYYY-MM-DD. Inside the script I want to load data from HDFS path /a/b/c/d_${parameter} How to build the path dynamically to be able to place it into LOAD DATA INPATH '...' OVERWRITE INTO TABLE data_staging; Even setting the path to a variable (hardcoded) and place it as INPATH doesn't work ?!?! I tried: set p="/a/b/c/d_2013-07-08/data.tsv"; LOAD DATA INPATH "${hiveconf:p}" OVERWRITE INTO TABLE data_staging; => Error while compiling statement: FAILED: ParseException line 1:19 mismatched input '/' expecting INTO near '""' in load statement If I remove the quotes from the parameter placeholder I receive an error, too. File not found, even if it does exist in HDFS, for sure, I checked it many times... set p="/a/b/c/d_2013-07-08/data.tsv"; LOAD DATA INPATH ${hiveconf:p} OVERWRITE INTO TABLE data_staging; => Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path '"/a/b/c/d_2013-07-08/data.tsv"': No files matching path hdfs://nameservice1/a/b/c/d_2013-07-08/data.tsv How can I build the path variable inside the hive script to be able to load data files from that path ? I want to pass the day as parameter and read files from HDFS path /a/b/c/d_YYYY-MM-DD/ thanks in advance... PS: using latest CDH5.0.1
... View more
Labels:
- Labels:
-
Apache Hive
-
HDFS
05-07-2014
12:22 PM
Hi Darren, thank you so much, problem is solved by setting the hue_webhdfs parameter accordingly. Yes, I am using Namenode-HA in that cluster. Currently the service Hue doesn't show a concerning setting, but I'll try to reproduce the error (after we finished our current tasks 😉 ). Also big thanks to Harsh J for replying. br, Gerd
... View more
05-07-2014
03:49 AM
Things are getting worse.... I stopped the Hue service to try if I can configure it afterwards. Sadly not, and now Hue doesn't start up again. At starting service Hue I receive the following error: Command failed to run because service hue has invalid configuration. Review and correct its configuration. First error: Missing required value: HDFS Web Interface Role **bleep**, we need Hue urgently. Is it somehow possible to configure Hue via the REST API, or do I have to dive into the embedded PostgreSQL to fix the issue ?!?! Seems like I have to assign the availble HTTPFS service to Hue's "HDFS Web Interface Role" ...
... View more
05-06-2014
06:38 AM
and these are the entries from /var/log/cloudera-scm-server/cloudera-scm-server.log: ... 2014-05-06 15:34:35,618 INFO [WebServerImpl:mortbay.log@67] Started SelectChannelConnector@0.0.0.0:7180 2014-05-06 15:34:35,619 INFO [WebServerImpl:cmf.WebServerImpl@292] Started Jetty server. 2014-05-06 15:34:36,820 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@411] Num docs:3110 2014-05-06 15:34:36,826 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@352] Constructing repo:2014-05-06T13:34:36.826Z 2014-05-06 15:34:41,445 INFO [SearchRepositoryManager-0:components.SearchRepositoryManager@354] Finished constructing repo:2014-05-06T13:34:41.445Z 2014-05-06 15:34:42,055 INFO [454109838@scm-web-5:cmf.AuthenticationSuccessEventListener@32] Authentication success for user: admin 2014-05-06 15:34:44,776 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1913ms: GC pool 'PS MarkSweep' had collection(s): count=1 time=1851ms, GC pool 'PS Scavenge' had collection(s): count=1 time=183ms 2014-05-06 15:34:44,776 INFO [JvmPauseMonitor:debug.JvmPauseMonitor@236] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1700ms: GC pool 'PS MarkSweep' had collection(s): count=1 time=1851ms, GC pool 'PS Scavenge' had collection(s): count=1 time=183ms 2014-05-06 15:34:46,063 INFO [791948358@scm-web-8:common.JFrameException@167] Exception report generated accessing http://hadoop-pg-5:7180/cmf/services/landingPageStatusContent com.cloudera.enterprise.JsonUtil$JsonRuntimeException: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.google.common.collect.RegularImmutableList[6]->com.cloudera.server.web.cmf.actionables.Actionable["details"]) at com.cloudera.enterprise.JsonUtil.valueAsString(JsonUtil.java:168) at com.cloudera.enterprise.JsonUtil.valueAsString(JsonUtil.java:155) at com.cloudera.server.web.cmf.LandingPageController.servicesTable(LandingPageController.java:301) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.google.common.collect.RegularImmutableList[6]->com.cloudera.server.web.cmf.actionables.Actionable["details"]) at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) at org.codehaus.jackson.map.ser.SerializerBase.wrapAndThrow(SerializerBase.java:131) at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:183) at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:142) at org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer.serializeContents(ContainerSerializers.java:304) at org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer.serializeContents(ContainerSerializers.java:254) at org.codehaus.jackson.map.ser.ContainerSerializers$AsArraySerializer.serialize(ContainerSerializers.java:142) at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:600) at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:280) at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2267) at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:1836) at com.cloudera.enterprise.JsonUtil.valueAsString(JsonUtil.java:163) ... 72 more Caused by: java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) at com.cloudera.server.web.common.I18n.t(I18n.java:174) at com.cloudera.server.web.common.I18n.t(I18n.java:120) at com.cloudera.server.web.cmf.actionables.Actionable.getDetails(Actionable.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.jackson.map.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:467) at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:402) at org.codehaus.jackson.map.ser.BeanSerializer.serializeFields(BeanSerializer.java:175) ... 81 more
... View more
05-06-2014
06:31 AM
Hi, service Hue reported one concerning setting. It was the assignment of the WebHDFS role shall be HTTPFS instead of using a namenode (or similar to this...). In the related configuration entry I clicked "reset to default value" and after that I received a stack trace which I shall send to Cloudera 😉 Here it is: Error A server error has occurred. Send the following information to Cloudera. Path: http://hadoop-pg-5:7180/cmf/services/landingPageStatusContent Version: Cloudera Express 5.0.0 (#215 built by jenkins on 20140331-1424 git: 50c701f3e920b1fcf524bf5fa061d65902cde804) com.cloudera.enterprise.JsonUtil$JsonRuntimeException:org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: com.google.common.collect.RegularImmutableList[6]->com.cloudera.server.web.cmf.actionables.Actionable["details"]) at JsonUtil.java line 168 in com.cloudera.enterprise.JsonUtil valueAsString() Stack Trace: JsonUtil.java line 168 in com.cloudera.enterprise.JsonUtil valueAsString() JsonUtil.java line 155 in com.cloudera.enterprise.JsonUtil valueAsString() LandingPageController.java line 301 in com.cloudera.server.web.cmf.LandingPageController servicesTable() line -1 in sun.reflect.GeneratedMethodAccessor625 invoke() DelegatingMethodAccessorImpl.java line 43 in sun.reflect.DelegatingMethodAccessorImpl invoke() Method.java line 606 in java.lang.reflect.Method invoke() HandlerMethodInvoker.java line 176 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 436 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 424 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle() DispatcherServlet.java line 790 in org.springframework.web.servlet.DispatcherServlet doDispatch() DispatcherServlet.java line 719 in org.springframework.web.servlet.DispatcherServlet doService() FrameworkServlet.java line 669 in org.springframework.web.servlet.FrameworkServlet processRequest() FrameworkServlet.java line 585 in org.springframework.web.servlet.FrameworkServlet doPost() HttpServlet.java line 595 in javax.servlet.http.HttpServlet service() HttpServlet.java line 668 in javax.servlet.http.HttpServlet service() ServletHolder.java line 511 in org.mortbay.jetty.servlet.ServletHolder handle() ServletHandler.java line 1221 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() UserAgentFilter.java line 78 in org.mortbay.servlet.UserAgentFilter doFilter() GzipFilter.java line 131 in org.mortbay.servlet.GzipFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JAMonServletFilter.java line 48 in com.jamonapi.http.JAMonServletFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JavaMelodyFacade.java line 109 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() FilterChainProxy.java line 311 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterSecurityInterceptor.java line 116 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke() FilterSecurityInterceptor.java line 83 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ExceptionTranslationFilter.java line 113 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SessionManagementFilter.java line 101 in org.springframework.security.web.session.SessionManagementFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AnonymousAuthenticationFilter.java line 113 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RememberMeAuthenticationFilter.java line 146 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextHolderAwareRequestFilter.java line 54 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RequestCacheAwareFilter.java line 45 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AbstractAuthenticationProcessingFilter.java line 182 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() LogoutFilter.java line 105 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextPersistenceFilter.java line 87 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ConcurrentSessionFilter.java line 125 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterChainProxy.java line 173 in org.springframework.security.web.FilterChainProxy doFilter() DelegatingFilterProxy.java line 237 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate() DelegatingFilterProxy.java line 167 in org.springframework.web.filter.DelegatingFilterProxy doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() CharacterEncodingFilter.java line 88 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal() OncePerRequestFilter.java line 76 in org.springframework.web.filter.OncePerRequestFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() ServletHandler.java line 399 in org.mortbay.jetty.servlet.ServletHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() SessionHandler.java line 182 in org.mortbay.jetty.servlet.SessionHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() ContextHandler.java line 766 in org.mortbay.jetty.handler.ContextHandler handle() WebAppContext.java line 450 in org.mortbay.jetty.webapp.WebAppContext handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() StatisticsHandler.java line 53 in org.mortbay.jetty.handler.StatisticsHandler handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() Server.java line 326 in org.mortbay.jetty.Server handle() HttpConnection.java line 542 in org.mortbay.jetty.HttpConnection handleRequest() HttpConnection.java line 928 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete() HttpParser.java line 549 in org.mortbay.jetty.HttpParser parseNext() HttpParser.java line 212 in org.mortbay.jetty.HttpParser parseAvailable() HttpConnection.java line 404 in org.mortbay.jetty.HttpConnection handle() SelectChannelEndPoint.java line 410 in org.mortbay.io.nio.SelectChannelEndPoint run() QueuedThreadPool.java line 582 in org.mortbay.thread.QueuedThreadPool$PoolThread run() Caused by: org.codehaus.jackson.map.JsonMappingException:(was java.lang.NullPointerException) (through reference chain: com.google.common.collect.RegularImmutableList[6]->com.cloudera.server.web.cmf.actionables.Actionable["details"]) at JsonMappingException.java line 218 in org.codehaus.jackson.map.JsonMappingException wrapWithPath() Stack Trace: JsonMappingException.java line 218 in org.codehaus.jackson.map.JsonMappingException wrapWithPath() JsonMappingException.java line 183 in org.codehaus.jackson.map.JsonMappingException wrapWithPath() SerializerBase.java line 131 in org.codehaus.jackson.map.ser.SerializerBase wrapAndThrow() BeanSerializer.java line 183 in org.codehaus.jackson.map.ser.BeanSerializer serializeFields() BeanSerializer.java line 142 in org.codehaus.jackson.map.ser.BeanSerializer serialize() ContainerSerializers.java line 304 in org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer serializeContents() ContainerSerializers.java line 254 in org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer serializeContents() ContainerSerializers.java line 142 in org.codehaus.jackson.map.ser.ContainerSerializers$AsArraySerializer serialize() StdSerializerProvider.java line 600 in org.codehaus.jackson.map.ser.StdSerializerProvider _serializeValue() StdSerializerProvider.java line 280 in org.codehaus.jackson.map.ser.StdSerializerProvider serializeValue() ObjectMapper.java line 2267 in org.codehaus.jackson.map.ObjectMapper _configAndWriteValue() ObjectMapper.java line 1836 in org.codehaus.jackson.map.ObjectMapper writeValueAsString() JsonUtil.java line 163 in com.cloudera.enterprise.JsonUtil valueAsString() JsonUtil.java line 155 in com.cloudera.enterprise.JsonUtil valueAsString() LandingPageController.java line 301 in com.cloudera.server.web.cmf.LandingPageController servicesTable() line -1 in sun.reflect.GeneratedMethodAccessor625 invoke() DelegatingMethodAccessorImpl.java line 43 in sun.reflect.DelegatingMethodAccessorImpl invoke() Method.java line 606 in java.lang.reflect.Method invoke() HandlerMethodInvoker.java line 176 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 436 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 424 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle() DispatcherServlet.java line 790 in org.springframework.web.servlet.DispatcherServlet doDispatch() DispatcherServlet.java line 719 in org.springframework.web.servlet.DispatcherServlet doService() FrameworkServlet.java line 669 in org.springframework.web.servlet.FrameworkServlet processRequest() FrameworkServlet.java line 585 in org.springframework.web.servlet.FrameworkServlet doPost() HttpServlet.java line 595 in javax.servlet.http.HttpServlet service() HttpServlet.java line 668 in javax.servlet.http.HttpServlet service() ServletHolder.java line 511 in org.mortbay.jetty.servlet.ServletHolder handle() ServletHandler.java line 1221 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() UserAgentFilter.java line 78 in org.mortbay.servlet.UserAgentFilter doFilter() GzipFilter.java line 131 in org.mortbay.servlet.GzipFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JAMonServletFilter.java line 48 in com.jamonapi.http.JAMonServletFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JavaMelodyFacade.java line 109 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() FilterChainProxy.java line 311 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterSecurityInterceptor.java line 116 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke() FilterSecurityInterceptor.java line 83 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ExceptionTranslationFilter.java line 113 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SessionManagementFilter.java line 101 in org.springframework.security.web.session.SessionManagementFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AnonymousAuthenticationFilter.java line 113 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RememberMeAuthenticationFilter.java line 146 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextHolderAwareRequestFilter.java line 54 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RequestCacheAwareFilter.java line 45 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AbstractAuthenticationProcessingFilter.java line 182 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() LogoutFilter.java line 105 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextPersistenceFilter.java line 87 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ConcurrentSessionFilter.java line 125 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterChainProxy.java line 173 in org.springframework.security.web.FilterChainProxy doFilter() DelegatingFilterProxy.java line 237 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate() DelegatingFilterProxy.java line 167 in org.springframework.web.filter.DelegatingFilterProxy doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() CharacterEncodingFilter.java line 88 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal() OncePerRequestFilter.java line 76 in org.springframework.web.filter.OncePerRequestFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() ServletHandler.java line 399 in org.mortbay.jetty.servlet.ServletHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() SessionHandler.java line 182 in org.mortbay.jetty.servlet.SessionHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() ContextHandler.java line 766 in org.mortbay.jetty.handler.ContextHandler handle() WebAppContext.java line 450 in org.mortbay.jetty.webapp.WebAppContext handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() StatisticsHandler.java line 53 in org.mortbay.jetty.handler.StatisticsHandler handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() Server.java line 326 in org.mortbay.jetty.Server handle() HttpConnection.java line 542 in org.mortbay.jetty.HttpConnection handleRequest() HttpConnection.java line 928 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete() HttpParser.java line 549 in org.mortbay.jetty.HttpParser parseNext() HttpParser.java line 212 in org.mortbay.jetty.HttpParser parseAvailable() HttpConnection.java line 404 in org.mortbay.jetty.HttpConnection handle() SelectChannelEndPoint.java line 410 in org.mortbay.io.nio.SelectChannelEndPoint run() QueuedThreadPool.java line 582 in org.mortbay.thread.QueuedThreadPool$PoolThread run() Caused by: java.lang.NullPointerException: at Preconditions.java line 191 in com.google.common.base.Preconditions checkNotNull() Stack Trace: Preconditions.java line 191 in com.google.common.base.Preconditions checkNotNull() I18n.java line 174 in com.cloudera.server.web.common.I18n t() I18n.java line 120 in com.cloudera.server.web.common.I18n t() Actionable.java line 36 in com.cloudera.server.web.cmf.actionables.Actionable getDetails() line -1 in sun.reflect.GeneratedMethodAccessor497 invoke() DelegatingMethodAccessorImpl.java line 43 in sun.reflect.DelegatingMethodAccessorImpl invoke() Method.java line 606 in java.lang.reflect.Method invoke() BeanPropertyWriter.java line 467 in org.codehaus.jackson.map.ser.BeanPropertyWriter get() BeanPropertyWriter.java line 402 in org.codehaus.jackson.map.ser.BeanPropertyWriter serializeAsField() BeanSerializer.java line 175 in org.codehaus.jackson.map.ser.BeanSerializer serializeFields() BeanSerializer.java line 142 in org.codehaus.jackson.map.ser.BeanSerializer serialize() ContainerSerializers.java line 304 in org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer serializeContents() ContainerSerializers.java line 254 in org.codehaus.jackson.map.ser.ContainerSerializers$IndexedListSerializer serializeContents() ContainerSerializers.java line 142 in org.codehaus.jackson.map.ser.ContainerSerializers$AsArraySerializer serialize() StdSerializerProvider.java line 600 in org.codehaus.jackson.map.ser.StdSerializerProvider _serializeValue() StdSerializerProvider.java line 280 in org.codehaus.jackson.map.ser.StdSerializerProvider serializeValue() ObjectMapper.java line 2267 in org.codehaus.jackson.map.ObjectMapper _configAndWriteValue() ObjectMapper.java line 1836 in org.codehaus.jackson.map.ObjectMapper writeValueAsString() JsonUtil.java line 163 in com.cloudera.enterprise.JsonUtil valueAsString() JsonUtil.java line 155 in com.cloudera.enterprise.JsonUtil valueAsString() LandingPageController.java line 301 in com.cloudera.server.web.cmf.LandingPageController servicesTable() line -1 in sun.reflect.GeneratedMethodAccessor625 invoke() DelegatingMethodAccessorImpl.java line 43 in sun.reflect.DelegatingMethodAccessorImpl invoke() Method.java line 606 in java.lang.reflect.Method invoke() HandlerMethodInvoker.java line 176 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 436 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 424 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle() DispatcherServlet.java line 790 in org.springframework.web.servlet.DispatcherServlet doDispatch() DispatcherServlet.java line 719 in org.springframework.web.servlet.DispatcherServlet doService() FrameworkServlet.java line 669 in org.springframework.web.servlet.FrameworkServlet processRequest() FrameworkServlet.java line 585 in org.springframework.web.servlet.FrameworkServlet doPost() HttpServlet.java line 595 in javax.servlet.http.HttpServlet service() HttpServlet.java line 668 in javax.servlet.http.HttpServlet service() ServletHolder.java line 511 in org.mortbay.jetty.servlet.ServletHolder handle() ServletHandler.java line 1221 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() UserAgentFilter.java line 78 in org.mortbay.servlet.UserAgentFilter doFilter() GzipFilter.java line 131 in org.mortbay.servlet.GzipFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JAMonServletFilter.java line 48 in com.jamonapi.http.JAMonServletFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JavaMelodyFacade.java line 109 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() FilterChainProxy.java line 311 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterSecurityInterceptor.java line 116 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke() FilterSecurityInterceptor.java line 83 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ExceptionTranslationFilter.java line 113 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SessionManagementFilter.java line 101 in org.springframework.security.web.session.SessionManagementFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AnonymousAuthenticationFilter.java line 113 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RememberMeAuthenticationFilter.java line 146 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextHolderAwareRequestFilter.java line 54 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RequestCacheAwareFilter.java line 45 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AbstractAuthenticationProcessingFilter.java line 182 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() LogoutFilter.java line 105 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextPersistenceFilter.java line 87 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ConcurrentSessionFilter.java line 125 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterChainProxy.java line 173 in org.springframework.security.web.FilterChainProxy doFilter() DelegatingFilterProxy.java line 237 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate() DelegatingFilterProxy.java line 167 in org.springframework.web.filter.DelegatingFilterProxy doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() CharacterEncodingFilter.java line 88 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal() OncePerRequestFilter.java line 76 in org.springframework.web.filter.OncePerRequestFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() ServletHandler.java line 399 in org.mortbay.jetty.servlet.ServletHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() SessionHandler.java line 182 in org.mortbay.jetty.servlet.SessionHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() ContextHandler.java line 766 in org.mortbay.jetty.handler.ContextHandler handle() WebAppContext.java line 450 in org.mortbay.jetty.webapp.WebAppContext handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() StatisticsHandler.java line 53 in org.mortbay.jetty.handler.StatisticsHandler handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() Server.java line 326 in org.mortbay.jetty.Server handle() HttpConnection.java line 542 in org.mortbay.jetty.HttpConnection handleRequest() HttpConnection.java line 928 in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete() HttpParser.java line 549 in org.mortbay.jetty.HttpParser parseNext() HttpParser.java line 212 in org.mortbay.jetty.HttpParser parseAvailable() HttpConnection.java line 404 in org.mortbay.jetty.HttpConnection handle() SelectChannelEndPoint.java line 410 in org.mortbay.io.nio.SelectChannelEndPoint run() QueuedThreadPool.java line 582 in org.mortbay.thread.QueuedThreadPool$PoolThread run() The problem is, that this stack trace appears if I go to the dashboard view and also if I want to go to 'Service Hue' => 'Configuration'. Now I am unable to reconfigure Hue and the dashboard doesn't show the cluster overview 😞 What can I do to solve this issue? (restarting cloudera-scm-server didn't solve it) I am using latest CM+CDH5+NamenodeHA... thanks, Gerd
... View more
Labels:
05-01-2014
01:19 PM
Hi Wolfgang, finally I got it (almost) to work, just some schema.xml settings to adjust. Many thanks for your great support, let's celebrate that at BerlinBuzz 😉 br, Gerd
... View more
05-01-2014
09:05 AM
Hi Wolfgang, again me 😉 things are getting better. Seems like the increased memory setting saves my day. Now I am getting the expected DEBUG output from the morphline, but the document doesn't fit into Solr because the required field "id" is missing. Yes, of course Solr wouldn't accept such a document, but I configured flume to insert a header field called "id" filled by the UUIDInterceptor. The DEBUG output is: "" 2014-05-01 17:04:46,280 DEBUG org.kitesdk.morphline.stdlib.LogDebugBuilder$LogDebug: 1 : [{message=[<168>Feb 4 10:46:14 syslog sshd[607]: listening on 0.0.0.0 port 22.]}] "" This is the whole line of text read by ReadLine. Also the following Grok regex is parsing correctly but I have no idea how to access the header field "id" from the UUIDInterceptor. How do I have to handle that additional field to be put into the corresponding document-part being inserted into Solr ?!?! How can I check the header fields of the incoming event at all? best, Gerd
... View more
05-01-2014
06:09 AM
Hi Wolfgang, I tried a different approach, starting with a blank flume on a different cluster node. I configured the exact same settings for flume-/morphline-configuration. The behaviour is almost the same, data isn't being processed, but now I have an error message in the flume.log. Most probably you can interpret it much better 🙂 o.k., out of memory sounds obvious, but which setting does this affect ? 2014-05-01 15:05:11,724 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/1089-flume-AGENT/flume.conf for changes 2014-05-01 15:05:37,382 ERROR org.apache.flume.sink.hdfs.HDFSEventSink: process failed 2014-05-01 15:05:44,224 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/1089-flume-AGENT/flume.conf for changes 2014-05-01 15:05:44,977 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2ed7c530 counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.OutOfMemoryError: Java heap space at java.util.zip.ZipCoder.toString(ZipCoder.java:49) at java.util.zip.ZipFile.getZipEntry(ZipFile.java:531) at java.util.zip.ZipFile.access$900(ZipFile.java:56) at java.util.zip.ZipFile$1.nextElement(ZipFile.java:513) at java.util.zip.ZipFile$1.nextElement(ZipFile.java:483) at java.util.jar.JarFile$1.nextElement(JarFile.java:243) at java.util.jar.JarFile$1.nextElement(JarFile.java:238) at org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scanJar(ClassPath.java:343) at org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scanFrom(ClassPath.java:288) at org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$Scanner.scan(ClassPath.java:276) at org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath.from(ClassPath.java:84) at org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:134) at org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:77) at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43) at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) at org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-05-01 15:05:44,982 INFO org.apache.flume.sink.solr.morphline.MorphlineSink: Morphline Sink solrSink stopping... 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Component type: SINK, name: solrSink stopped 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.start.time == 1398949449905 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.stop.time == 1398949544982 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.batch.complete == 0 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.batch.empty == 0 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.batch.underflow == 0 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.connection.closed.count == 0 2014-05-01 15:05:44,982 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.connection.creation.count == 0 2014-05-01 15:05:44,983 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.connection.failed.count == 0 2014-05-01 15:05:44,983 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.event.drain.attempt == 0 2014-05-01 15:05:44,983 INFO org.apache.flume.instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: solrSink. sink.event.drain.sucess == 0 2014-05-01 15:05:44,983 INFO org.apache.flume.sink.solr.morphline.MorphlineSink: Morphline Sink solrSink stopped. Metrics: SINK:solrSink{sink.connection.closed.count=0, sink.event.drain.attempt=0, sink.batch.underflow=0, sink.connection.failed.count=0, sink.connection.creation.count=0, sink.event.drain.sucess=0, sink.batch.empty=0, sink.batch.complete=0}, {} 2014-05-01 15:05:44,983 WARN org.apache.flume.lifecycle.LifecycleSupervisor: Component SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2ed7c530 counterGroup:{ name:null counters:{} } } stopped, since it could not besuccessfully started due to missing dependencies 2014-05-01 15:06:14,979 DEBUG org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Checking file:/var/run/cloudera-scm-agent/process/1089-flume-AGENT/flume.conf for changes br, Gerd
... View more
05-01-2014
05:54 AM
I am using: ClouderaManager: Version: Cloudera Express 5.0.0 (#215 built by jenkins on 20140331-1424 git: 50c701f3e920b1fcf524bf5fa061d65902cde804) Hadoop stack: CDH5.0.0-1.cdh5.0.0.p0.47 (with the included Solr) I configured the TRACE in the corresponding "Advanced" section of the flume instance, but nothing has changed. After restarting the agent I cannot see any additional output in the /var/log/flume-ng/flume-cmf-flume-AGENT-hadoop-pg-7.cluster.log, and the data I insert into the file that is being watched (via tail -f ...) isn't being processed anyhow. The only thing what happens is that the the agent seems to reset/restart periodically and CM shows that agent in state 'BAD' due to a lot of unexpected exits. hmmm?!?!
... View more
05-01-2014
03:17 AM
Hi Wolfgang, thanks for answering. I modified that entry accordingly, but nothing has changed in the workflow (no data processing happens, no output in HDFS and solr). The flume.log remains at (same output as in the original post): ... 2014-05-01 12:10:37,893 INFO org.mortbay.log: jetty-6.1.26 2014-05-01 12:10:37,990 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:41414 2014-05-01 12:10:38,179 INFO org.kitesdk.morphline.api.MorphlineContext: Importing commands and restarts periodically, about every 90sec. Additionally the status of this agent in CM is "bad", because of " This role encountered 3 unexpected exit(s) in the previous 5 minute(s). Critical threshold: any." Where do I have some more log output of what is going on in the morphline pipeline? There is no log/debug output in the flume log, where does the morphline debug message go to?!?! getting confused 😉 , any help appreciated, Gerd PS: if I disable the solr-morphline-sink I receive data on HDFS, thereby the error must be somewhere in the morphline processing (assumingly)
... View more