Spark Version: 2.4 Spark Streaming kafka: spark-streaming-kafka-0-10_2.11   Observe the  throw new Exception();  in while loop. Even if the batch fails because of the exception, i see the offset committed, I am expecting some lag here as the processing failed, what is wrong here ?   enable.auto.commit : false     JavaInputDStream<ConsumerRecord<String, String>> kafkaStream = KafkaUtils.createDirectStream(jssc, LocationStrategies.PreferConsistent(), ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)); kafkaStream.foreachRDD(kafkaStreamRDD -> { // fetch kafka offsets for manually committing it later OffsetRange[] offsetRanges = ((HasOffsetRanges) kafkaStreamRDD.rdd()).offsetRanges(); // filter unwanted data kafkaStreamRDD.filter(new Function<ConsumerRecord<String, String>, Boolean>() { //filter logic here }).foreachPartition(kafkaRecords -> { //Initializing DB connections while (kafkaRecords.hasNext()) { //doing some work here //-----> EXCEPTION throw new Exception(); } }); // commit offsets saveOffsets after processing ((CanCommitOffsets) kafkaStream.inputDStream()).commitAsync(offsetRanges, (offsets, exception) -> { if (exception != null) { System.out.println("-------------Unable to commit offsets, something went wrong, trace ------------"+ exception.getCause()); exception.printStackTrace(); // need this for driver } else { System.out.println("Successfully committed offsets"); // need this for driver for (OffsetRange offsetRange : offsetRanges) { System.out.println("Offset Info: paratition {}, fromOffset {} untilOffset {}: "+ offsetRange.partition() +":"+ offsetRange.fromOffset() +":"+ offsetRange.untilOffset()); } } });   ... View more

When I am trying to add cluster, I am gettring Bootstrap Fail issue Cloudera Director 2.2.0,  CDH 5.9 -> AWS AMI RHEL-7.3_HVM_GA-20161026-x86_64-1-Hourly2-GP2   Add Cluster - Instance Group Then Bootstrapping starts:     Then Bootstrapping Fails: Logs: 2017-08-11 05:46:55,011 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics. java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy106.writeMetrics(Unknown Source) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) ... 11 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 11 more 2017-08-11 05:47:11,367 INFO 2141295615@scm-web-93:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:47:33,815 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=39ms, min=19ms, max=1549ms. 2017-08-11 05:47:33,815 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=131ms. 2017-08-11 05:47:41,353 INFO 38788623@scm-web-96:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:47:50,082 WARN 120428605@scm-web-98:com.cloudera.server.web.cmf.StatusProvider: (325 skipped) Failed to get status from SERVICE_MONITORING java.util.concurrent.ExecutionException: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture.safeGetResponse(StatusProvider.java:645) at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:1080) at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getServicesByType(ServiceManagerDaoImpl.java:581) at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:608) at sun.reflect.GeneratedMethodAccessor1063.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:267) at com.cloudera.api.dao.impl.ManagerDaoBase.access$300(ManagerDaoBase.java:85) at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:308) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:30) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:219) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:343) at com.sun.proxy.$Proxy114.getMgmtService(Unknown Source) at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:42) at com.cloudera.api.v1.impl.MgmtServiceResourceImpl$RolesResourceWrapper.<init>(MgmtServiceResourceImpl.java:206) at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.getRolesResource(MgmtServiceResourceImpl.java:157) at sun.reflect.GeneratedMethodAccessor1062.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:44) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) at com.sun.proxy.$Proxy104.getSubjectStatus(Unknown Source) at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getSubjectStatus(TimeoutNozzleIPC.java:419) at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture$1.call(StatusProvider.java:734) at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture$1.call(StatusProvider.java:730) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 8 more 2017-08-11 05:48:11,414 INFO 120428605@scm-web-98:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:48:33,842 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=39ms, min=19ms, max=1549ms. 2017-08-11 05:48:33,842 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=131ms. 2017-08-11 05:48:41,360 INFO 38788623@scm-web-96:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:49:11,357 INFO 38788623@scm-web-96:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:49:33,893 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: execution stats: average=38ms, min=19ms, max=1549ms. 2017-08-11 05:49:33,894 INFO 1904007070@agentServer-0:com.cloudera.server.common.MonitoringThreadPool: agentServer: waiting in queue stats: average=0ms, min=0ms, max=131ms. 2017-08-11 05:49:41,359 INFO 1430665588@scm-web-95:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132 2017-08-11 05:49:55,937 INFO 1430665588@scm-web-95:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.31.20.132   ... View more

I am currently using CDH 5.8 in my env. I have to test the compatibility of my project jars before moving to CDH 5.9. To check the compatibility, I wanted to download quickstart VM 5.9 and get my projects working there.   It would be great if someone can share 5.9 quickstart vm archived link. ... View more

I created an external hive table pointing to directory   LOCATION '/user/hive/warehouse/logs' Inside logs directory I have multiple subfolders as follows:     /user/hive/warehouse/logs/test/day=20170606 /user/hive/warehouse/logs/test/day=20170607 /user/hive/warehouse/logs/test/day=20170608 I have also used the config:   set hive.mapred.supports.subdirectories=true; set mapred.input.dir.recursive=true; Using the 'hive' cli I am able to query the data and see the results   But when I perform query using Impala I see no results. Do i need to provide some configuration here   ... View more

I have flume sinks that access the database and filesystems. The connection parameters of these system change as I move from on env to other. Current Implementation: Provide configuration file at flume start up using the flume configuration. Java Configuration Options for Flume Agent -Dapp.config.file=hdfs://.../config.json But I don't think its a good HDFS application. Is there a way we can provide custom environment configuration at flume source channel sink configuration itself. I know we can provide above config at a sink level, but is there a way to provide it agent level and get the data in MycustomSink.java.   Please let me know if a better approach is available ... View more

Components Used for data ingestion: Kafka Channel -> Flume Custom Sink -> DB   Current implementation:  1. Before restart of flume ->  i need to export zookeeper offset for a kafka.source.groupId 2. Restart flume 3. import the offsets. -> reason: while restarting some messages fails at sink processing (custom sink) side, and I need to re-process them.   I am not using flume transactions rollback capability as for some errors(data issue) other that point 3, the message might be failed and rollbacked indefinately.    I wanted to know if there is a better approach available other the export/import offsets everytime I restart flume.       ... View more