Member since
10-28-2016
392
Posts
7
Kudos Received
20
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3485 | 03-12-2018 02:28 AM | |
| 5209 | 12-18-2017 11:41 PM | |
| 3649 | 07-17-2017 07:01 PM | |
| 2579 | 07-13-2017 07:20 PM | |
| 8246 | 07-12-2017 08:31 PM |
07-12-2017
08:12 PM
Hi All, I've HDP 2.5.x, and i'm facing issue with the Node Managers going down. Initially, the error was - /tmp/keystore.jks was not available.. so i generated the keystore.jks file again. Now, i'm getting the following error in starting Yarn Node Managers any ideas what needs to be done ? ------------------------------------------------ java.lang.RuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:504)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:178)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:457)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:547)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594)Caused by: java.io.IOException: Keystore was tampered with, or password was incorrectat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)at java.security.KeyStore.load(KeyStore.java:1445)at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:173)at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)at org.apache.hadoop.mapred.ShuffleHandler$HttpPipelineFactory.<init>(ShuffleHandler.java:757)at org.apache.hadoop.mapred.ShuffleHandler.serviceStart(ShuffleHandler.java:502)... 11 moreCaused by: java.security.UnrecoverableKeyException: Password verification failedat sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)... 19 more2017-07-12 20:05:18,245 INFOtimeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:run(416)) - Closing HadoopTimelineMetricSink. Flushing metrics to collector...2017-07-12 20:05:18,249 INFOnodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NodeManager at nwk2-bdp-hadoop-08.gdcs-qa.apple.com/10.78.130.135 ************************************************************/
... View more
Labels:
07-07-2017
12:58 AM
@mqureshi, @liran nahum - loopig you in, any ideas on this ?
... View more
07-07-2017
12:56 AM
Hi All, I'm trying to integrate Prometheus to collect Kafka metrics. I'm getting error in starting up Kafka, with the JMX exporter passed as java agent.
Error is as shown below, any ideas ? ---------------------------------------------------
[root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prometheus_javaagent-0.9.jar=7071:/usr/hdp/2.5.3.0-37/prometheus/jmx_exporter.yaml" ./bin/kafka-server-start.sh config/server.properties
2017-07-07 00:44:31.781:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT
2017-07-07 00:44:31.866:INFO:ipjsoejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:7071
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:54)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:128)
at io.prometheus.jmx.shaded.io.prometheus.client.Collector.register(Collector.java:121)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:38)
... View more
Labels:
06-22-2017
01:01 AM
@mqureshi, @Sriharsha Chintalapani , @Artem Ervits -- any ideas on this ?
... View more
06-22-2017
12:59 AM
Hi All,
I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality. Here is what i did : 1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2 2) On Cluster1, I publish 100 messages on Topic1 3) I've 2 consumers reading messages from the 2 topics on Cluster1 & Cluster2
4) I start MirrorMaker AFTER THE MESSAGES HAVE BEEN PUBLISHED ON Cluster1 $KAFKA10_HOME1/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config $KAFKA10_HOME1/config/mmConsumer.config --num.streams 3 --producer.config $KAFKA10_HOME1/config/mmProducer.config --whitelist="mmtopic1" --abort.on.send.failure true I expected that Once the MirrorMaker was started, the 100 messages published on Cluster1,
would be published on Clueter2 However that is not happening... What needs to be done to enable this ? Pls see my consumer & producer configs. --------------------------------------------------------------- mmConsumer.config : bootstrap.servers=localhost:6092,localhost:6093,localhost:6094,localhost:6095
group.id=mmgroup exclude.internal.topics=true #mirror.topics.whitelist=mmtopic1 client.id=mmconsumer auto.commit.enabled=false mmProducer.config : bootstrap.servers=localhost:5092,localhost:5093,localhost:5094,localhost:5095
acks=1 batch.size=100 client.id=mmproducer max.in.flight.requests.per.connection=1 #retries=Int.MaxValue retries=1000000 block.on.buffer.full=true
... View more
Labels:
06-15-2017
07:07 PM
@mqureshi, @amankumbare - looping you in, any ideas ?
... View more
06-15-2017
07:01 PM
Hi All - I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for offsets. I'm seeing different behavior in using the above 2 scrripts. Here is what i did ->
a) When i use ConsumerOffsetChecker $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group myGroup --topic newBroker1 [2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$) Group Topic Pid Offset logSize Lag Owner myGroup newBroker1 0 224216 507732 283516 none myGroup newBroker1 1 224888 508978 284090 none myGroup newBroker1 2 141104 424038 282934 none myGroup newBroker1 3 11829 295110 283281 none myGroup newBroker1 4 11580 294510 282930 none b) When i run the kafka-consumer-groups.sh (since the WARNING message above says - ConsumerOffsetChecker is deprecated & we need to use ConsumerGroupCommand instead) $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092,localhost:9093,localhost:9094,localhost:9095 --new-consumer --group myGroup Consumer group `myGroup` does not exist or is rebalancing. OR $KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper localhost:2181 --group myGroup
No topic available for consumer group provided GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why is the script not giving me Offset details of the group - myGroup & Topic - newBroker1 What needs to be done for this (using script $KAFKA_HOME/bin/kafka-consumer-groups.sh) ?
... View more
Labels:
06-15-2017
06:57 PM
@mqureshi , thanks !
... View more
06-14-2017
05:24 PM
@mqureshi, @aromero, @Neeraj Sabharwal, @Kuldeep Kulkarni, @Josh Elser - any ideas on this ?
... View more
06-13-2017
10:00 PM
Hello All,
I've streaming data flowing into Kafka topics. Can I add Broker & re-balance the topics while the data flows into Kafka topic ?
What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?
Pls. advise. Thanks!
... View more
Labels: