Member since 
    
	
		
		
		10-28-2016
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                392
            
            
                Posts
            
        
                7
            
            
                Kudos Received
            
        
                20
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 3467 | 03-12-2018 02:28 AM | |
| 5186 | 12-18-2017 11:41 PM | |
| 3640 | 07-17-2017 07:01 PM | |
| 2567 | 07-13-2017 07:20 PM | |
| 8210 | 07-12-2017 08:31 PM | 
			
    
	
		
		
		07-12-2017
	
		
		08:12 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi All,  I've HDP 2.5.x, and i'm facing issue with the Node Managers going down.  Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again.  Now, i'm getting the following error in starting Yarn Node Managers  any ideas what needs to be done ?  ------------------------------------------------  java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect 
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135  ************************************************************/     
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		07-07-2017
	
		
		12:58 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @mqureshi, @liran nahum - loopig you in, any ideas on this ?  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		07-07-2017
	
		
		12:56 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi All,  I'm trying to integrate Prometheus to collect Kafka metrics.  I'm getting error in starting up Kafka, with the JMX exporter passed as java agent.
Error is as shown below, any ideas ?  ---------------------------------------------------  
[root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prometheus_javaagent-0.9.jar=7071:/usr/hdp/2.5.3.0-37/prometheus/jmx_exporter.yaml" ./bin/kafka-server-start.sh config/server.properties 
2017-07-07 00:44:31.781:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT
2017-07-07 00:44:31.866:INFO:ipjsoejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:7071
Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
  at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds  at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
  at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)
  at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:121)
  at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:38) 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		06-22-2017
	
		
		01:01 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @mqureshi, @Sriharsha Chintalapani , @Artem Ervits  -- any ideas on this ? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-22-2017
	
		
		12:59 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi All,  
I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality.   Here is what i did :   1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2   2) On Cluster1, I publish 100 messages on Topic1   3) I've 2 consumers reading messages from the 2 topics on Cluster1 & Cluster2   
4) I start MirrorMaker AFTER THE MESSAGES HAVE BEEN PUBLISHED ON Cluster1   $KAFKA10_HOME1/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config $KAFKA10_HOME1/config/mmConsumer.config --num.streams 3 --producer.config $KAFKA10_HOME1/config/mmProducer.config --whitelist="mmtopic1" --abort.on.send.failure true   I expected that Once the MirrorMaker was started, the 100 messages published on Cluster1,
would be published on Clueter2   However that is not happening...   What needs to be done to enable this ?  Pls see my consumer & producer configs.  ---------------------------------------------------------------  mmConsumer.config :  bootstrap.servers=localhost:6092,localhost:6093,localhost:6094,localhost:6095
  group.id=mmgroup  exclude.internal.topics=true  #mirror.topics.whitelist=mmtopic1  client.id=mmconsumer  auto.commit.enabled=false  mmProducer.config :  bootstrap.servers=localhost:5092,localhost:5093,localhost:5094,localhost:5095
  acks=1  batch.size=100  client.id=mmproducer  max.in.flight.requests.per.connection=1  #retries=Int.MaxValue  retries=1000000  block.on.buffer.full=true 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		06-15-2017
	
		
		07:07 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @mqureshi, @amankumbare - looping you in, any ideas ? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-15-2017
	
		
		07:01 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi All - I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to use the ConsumerOffsetChecker &  bin/kafka-consumer-groups.sh to check for offsets.   I'm seeing different behavior in using the above 2 scrripts.   Here is what i did ->
a) When i use ConsumerOffsetChecker  $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group myGroup --topic newBroker1  [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)  Group           Topic                          Pid Offset          logSize         Lag             Owner  myGroup         newBroker1                     0   224216          507732          283516          none  myGroup         newBroker1                     1   224888          508978          284090          none  myGroup         newBroker1                     2   141104          424038          282934          none  myGroup         newBroker1                     3   11829           295110          283281          none  myGroup         newBroker1                     4   11580           294510          282930          none  b) When i run the kafka-consumer-groups.sh (since the WARNING message above says - ConsumerOffsetChecker is deprecated & we need to use ConsumerGroupCommand instead)   $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe  --bootstrap-server localhost:9092,localhost:9093,localhost:9094,localhost:9095 --new-consumer --group myGroup   Consumer group `myGroup` does not exist or is rebalancing.  OR  $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe  --zookeeper localhost:2181  --group myGroup
  No topic available for consumer group provided  GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER  The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why is the script not giving me Offset details of the group - myGroup & Topic - newBroker1  What needs to be done for this (using script $KAFKA_HOME/bin/kafka-consumer-groups.sh) ? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		06-15-2017
	
		
		06:57 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @mqureshi , thanks ! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-14-2017
	
		
		05:24 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 @mqureshi, @aromero,   @Neeraj Sabharwal, @Kuldeep Kulkarni, @Josh Elser - any ideas on this ? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-13-2017
	
		
		10:00 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 
			Hello All,
			
			I've streaming data flowing into Kafka topics.   Can I add Broker & re-balance the topics while the data flows into Kafka topic ?
   What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?
  
				
Pls. advise.    Thanks!
			
  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
 
        


