@yshi: Hey, was hoping to hear back on the other issue I had created 🙂 . Is there a way I can bump that question on the forum to get some attention to it? ... View more

Thank you. I see that but I have always had this issue and is likely to happen again on increasing the number of clusters.Could it be some configurations or something that are causing this particular issue?   Also, look forward to a  response on the other question. Thank you 🙂 @yshi ... View more

Awesome. Using  -Dmapred.reduce.tasks=1  worked. I used it in my job with --meta-connect arguement as well and it still works 🙂 I would really like to understand what goes behind the scenes. Could you please explain why exactly I needed to use this argument?  @yshi    Also, I posted an issue here and it seems to not have gained any attention, Could you please have a look at it and let me know if this is supported at all? http://community.cloudera.com/t5/Data-Ingestion-Integration/change-sqoop-metastore-timezone-to-GMT/m-p/42596/highlight/true#M1701    Other than that, Thank you for keeping up all this time and helping out 🙂 . Happy holiday 🙂 🙂 ... View more

@yshi: I did sqoop job --show test_table    I get the following results:     verbose = false incremental.last.value = 2016-07-05 05:29:33.0 db.connect.string = jdbc:mysql://10.10.11.3/testdb?zeroDateTimeBehavior=convertToNull codegen.output.delimiters.escape = 0 codegen.output.delimiters.enclose.required = false codegen.input.delimiters.field = 0 split.limit = null hbase.create.table = false db.require.password = false hdfs.append.dir = false db.table = test_table codegen.input.delimiters.escape = 0 db.password = St@N!$$r3789L0vE accumulo.create.table = false import.fetch.size = null codegen.input.delimiters.enclose.required = false db.username = root reset.onemapper = false codegen.output.delimiters.record = 10 import.max.inline.lob.size = 16777216 hbase.bulk.load.enabled = false hcatalog.create.table = false db.clear.staging.table = false incremental.col = updated_at codegen.input.delimiters.record = 0 enable.compression = false hive.overwrite.table = false hive.import = false codegen.input.delimiters.enclose = 0 accumulo.batch.size = 10240000 hive.drop.delims = false customtool.options.jsonmap = {} codegen.output.delimiters.enclose = 0 hdfs.delete-target.dir = false codegen.output.dir = . codegen.auto.compile.dir = true relaxed.isolation = false mapreduce.num.mappers = 4 accumulo.max.latency = 5000 import.direct.split.size = 0 codegen.output.delimiters.field = 44 export.new.update = UpdateOnly incremental.mode = DateLastModified hdfs.file.format = TextFile codegen.compile.dir = /tmp/sqoop-root/compile/58af2027880638f91ea875a1c51b5de8 direct.import = false db.split.column = id hdfs.target.dir = /user/hue/ERP/testinc hive.fail.table.exists = false merge.key.col = id jdbc.driver.class = com.mysql.jdbc.Driver db.batch = false Value of merge.key.col as expected is id.    But I still do not get the value other attribute you asked for. @yshi   ... View more

Thank you.   I do get the logs now but none of the jobs have the two properties you asked for, Shall I share rest of the properties from the logs ? @yshi   Also, from the command line, I execute the job with both root user and hue user and the execution succedes in both the cases without any problem. My job is submitted to oozie using hue user only and that one fails.   ... View more

To enable the logs, I added it to my job :   sqoop job -Dmapreduce.map.log.level=DEUBG --meta-connect jdbc:hsqldb:hsql://FQDN:16000/sqoop --verbose --create test_table -- import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://ip3/testdb?zeroDateTimeBehavior=convertToNull --username root --password 'password' --table test_table --merge-key id --split-by id --target-dir location --incremental lastmodified --last-value 0 --check-column updated_at But trying to get the logs, I still get:   /tmp/logs/root/logs/application_1463739226103_5116does not exist. Log aggregation has not completed or is not enabled.   As a sidenote: I have also tried to change time zone of sqoop job in similar manner using -D option and that did not work either. Am I adding it in an incorrect way to the sqoop job? @yshi ... View more

@Markus Kemper: That's right. But then we already have the file in target dir when running the import for second time. It throws error that either import using incremental append or specify  merge key when target dir already exists and using incremental lastmodified.    Using sqoop merge command in a seprate job sounds okay but I would like to have --merge-key to work when it's available 😄 .   Also, the link https://community.hortonworks.com/questions/10710/sqoop-incremental-import-working-fine-now-i-want-k.html suggests to use --merge-key with import command only and has been confirmed that it works.   This link herehttp://stackoverflow.com/questions/34400973/apache-sqoop-incremental-import?rq=1 also confirms that it works   Also, it works when I run the sqoop job through CLI. It gives this problem only when running it through oozie ... View more

Hey, Tried collecting the diagnostics. I got an error: @yshi   A server error has occurred. Send the following information to Cloudera. Path: http://warehouse.swtched.com:7180/cmf/services/51/yarnDiagnosticsCollection Version: Cloudera Express 5.7.0 (#76 built by jenkins on 20160401-1334 git: ec0e7e69444280aa311511998bd83e8e6572f61c) java.lang.NullPointerException: at YarnController.java line 463 in com.cloudera.server.web.cmf.YarnController collectYarnApplicationDiagnostics() Stack Trace: YarnController.java line 463 in com.cloudera.server.web.cmf.YarnController collectYarnApplicationDiagnostics() <generated> line -1 in com.cloudera.server.web.cmf.YarnController$$FastClassByCGLIB$$ac91d355 invoke() MethodProxy.java line 191 in net.sf.cglib.proxy.MethodProxy invoke() Cglib2AopProxy.java line 688 in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint() ReflectiveMethodInvocation.java line 150 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed() MethodSecurityInterceptor.java line 61 in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke() ReflectiveMethodInvocation.java line 172 in org.springframework.aop.framework.ReflectiveMethodInvocation proceed() Cglib2AopProxy.java line 621 in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept() <generated> line -1 in com.cloudera.server.web.cmf.YarnController$$EnhancerByCGLIB$$f035552a collectYarnApplicationDiagnostics() NativeMethodAccessorImpl.java line -2 in sun.reflect.NativeMethodAccessorImpl invoke0() NativeMethodAccessorImpl.java line 57 in sun.reflect.NativeMethodAccessorImpl invoke() DelegatingMethodAccessorImpl.java line 43 in sun.reflect.DelegatingMethodAccessorImpl invoke() Method.java line 606 in java.lang.reflect.Method invoke() HandlerMethodInvoker.java line 176 in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 436 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod() AnnotationMethodHandlerAdapter.java line 424 in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle() DispatcherServlet.java line 790 in org.springframework.web.servlet.DispatcherServlet doDispatch() DispatcherServlet.java line 719 in org.springframework.web.servlet.DispatcherServlet doService() FrameworkServlet.java line 669 in org.springframework.web.servlet.FrameworkServlet processRequest() FrameworkServlet.java line 585 in org.springframework.web.servlet.FrameworkServlet doPost() HttpServlet.java line 727 in javax.servlet.http.HttpServlet service() HttpServlet.java line 820 in javax.servlet.http.HttpServlet service() ServletHolder.java line 511 in org.mortbay.jetty.servlet.ServletHolder handle() ServletHandler.java line 1221 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() UserAgentFilter.java line 78 in org.mortbay.servlet.UserAgentFilter doFilter() GzipFilter.java line 131 in org.mortbay.servlet.GzipFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JAMonServletFilter.java line 48 in com.jamonapi.http.JAMonServletFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() JavaMelodyFacade.java line 109 in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() FilterChainProxy.java line 311 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterSecurityInterceptor.java line 116 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke() FilterSecurityInterceptor.java line 83 in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ExceptionTranslationFilter.java line 113 in org.springframework.security.web.access.ExceptionTranslationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SessionManagementFilter.java line 101 in org.springframework.security.web.session.SessionManagementFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AnonymousAuthenticationFilter.java line 113 in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RememberMeAuthenticationFilter.java line 146 in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextHolderAwareRequestFilter.java line 54 in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() RequestCacheAwareFilter.java line 45 in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() AbstractAuthenticationProcessingFilter.java line 182 in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() LogoutFilter.java line 105 in org.springframework.security.web.authentication.logout.LogoutFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() SecurityContextPersistenceFilter.java line 87 in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() ConcurrentSessionFilter.java line 125 in org.springframework.security.web.session.ConcurrentSessionFilter doFilter() FilterChainProxy.java line 323 in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter() FilterChainProxy.java line 173 in org.springframework.security.web.FilterChainProxy doFilter() DelegatingFilterProxy.java line 237 in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate() DelegatingFilterProxy.java line 167 in org.springframework.web.filter.DelegatingFilterProxy doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() CharacterEncodingFilter.java line 88 in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal() OncePerRequestFilter.java line 76 in org.springframework.web.filter.OncePerRequestFilter doFilter() ServletHandler.java line 1212 in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter() ServletHandler.java line 399 in org.mortbay.jetty.servlet.ServletHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() SessionHandler.java line 182 in org.mortbay.jetty.servlet.SessionHandler handle() SecurityHandler.java line 216 in org.mortbay.jetty.security.SecurityHandler handle() ContextHandler.java line 767 in org.mortbay.jetty.handler.ContextHandler handle() WebAppContext.java line 450 in org.mortbay.jetty.webapp.WebAppContext handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() StatisticsHandler.java line 53 in org.mortbay.jetty.handler.StatisticsHandler handle() HandlerWrapper.java line 152 in org.mortbay.jetty.handler.HandlerWrapper handle() Server.java line 326 in org.mortbay.jetty.Server handle() HttpConnection.java line 542 in org.mortbay.jetty.HttpConnection handleRequest() HttpConnection.java line 945 in org.mortbay.jetty.HttpConnection$RequestHandler content() HttpParser.java line 756 in org.mortbay.jetty.HttpParser parseNext() HttpParser.java line 218 in org.mortbay.jetty.HttpParser parseAvailable() HttpConnection.java line 404 in org.mortbay.jetty.HttpConnection handle() SelectChannelEndPoint.java line 410 in org.mortbay.io.nio.SelectChannelEndPoint run() QueuedThreadPool.java line 582 in org.mortbay.thread.QueuedThreadPool$PoolThread run() ... View more