Member since
06-14-2016
69
Posts
28
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7048 | 07-30-2018 06:45 PM | |
5582 | 06-22-2018 10:28 PM | |
1436 | 06-20-2018 04:29 AM | |
1423 | 06-20-2018 04:24 AM | |
2916 | 06-15-2018 08:24 PM |
07-20-2017
07:52 AM
@Anil Reddy It looks like large processing time between poll calls ( when processor processes large volume of data) can exceed session.timeout.ms and cause group rebalancing. One way you can try is increasing the group.max.session.timeout.ms on the brokers side and add increased value of request.timeout.ms and session.timeout.ms in the ConsumeKafka configs. Please let me know if it helps. Thank you!
... View more
07-12-2017
07:24 PM
@Zhao Chaofeng So the deletion might not go through in the following scenarios: 1. broker hosting one of the replicas for that topic goes down 2. partition reassignment for partitions of that topic is in progress 3. preferred replica election for partitions of that topic is in progress Kindly check if your brokers are up and running fine. Also you can check state-change logs to monitor the state of the topic after you ran delete command. If you can share the state change logs here as well that would be helpful. Also the last option would be to delete it manually but I would suggest to check the cause first. Thank you!
... View more
07-10-2017
10:57 AM
@Zhao Chaofeng When you delete a topic, it is marked for deletion and it eventually gets deleted. In the screenshot that you provided, it is working as design. The 'Note' that you see in the delete command output is a general reminder to set delete.enable.topic=true. If it is already set to true you can ignore that 'Note' From the screenshot it looks like deletion is working fine because after deleting, I do not see that topic when you list topics, only when you are creating the topic again shows it in the list. May I know how are you verifying that topic is not getting deleted?
... View more
06-29-2017
07:35 AM
4 Kudos
In a non-Kerberized environment, for Kafka console consumer command displays all messages, perform the following: Note: Here, <broker_host> is Kafka broker hostname, <zk_host> is zookeeper hostname and Kafka port will be 6667:
Using Old consumer API (prior to HDP 2.5.x), run the following: bin/kafka-console-consumer.sh --zookeeper <zk_host>:2181 --topic test --from-beginning
Using New consumer API ( From HDP 2.5.x onwards), run the following: bin/kafka-console-consumer.sh --bootstrap-server <broker_host>:6667 --topic test --from-beginnin
To view a specific number of message in a Kafka topic, use the --max-messages option. To view the oldest message, run the console consumer with --from-beginning and --max-messages 1:
Using Old consumer API ( Prior to HDP 2.5.x): bin/kafka-console-consumer.sh --zookeeper <zk_host>:2181 --topic test --from-beginning --max-messages 1
Using New consumer API ( From HDP 2.5.x onwards): bin/kafka-console-consumer.sh --bootstrap-server <broker_host>:6667 --topic test --from-beginning --max-messages
... View more
Labels:
06-29-2017
07:31 AM
3 Kudos
Problem: While implementing Auto-Hdfs, following errors were thrown in Nimbus log: Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: https://da0gdal202.match.corp:9393/kms/v1/?op=GETDELEGATIONTOKEN&doAs=gdsreader&renewer=hdfs-hdpprod%40MATCH.CORP&user.name=hdfs, status: 403, message: Forbidden
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1024)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1019)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
Cause: 1. No symlink of Ranger KMS conf to core-site and hdfs-site 2. Missing 'kms.proxyuser.hdfs.groups' and 'hadoop.kms.proxyuser.hdfs.hosts' in Kms-site.xml Solution: 1. Created symlink of ranger kms conf to core site and hdfs site
2. Added following properties in Kms-site.xml: <property>
<name>hadoop.kms.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.kms.proxyuser.hdfs.hosts</name>
<value>*</value>
</property>
... View more
Labels:
06-29-2017
07:25 AM
3 Kudos
Problem: In HDP 2.3.4 and Ambari 2.4.2, Storm nimbus was not coming up: 2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message.
java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?? Following is the error from Nimbus log: 2017-05-30 14:35:18.750 b.s.d.nimbus [WARN] principal: storm@EXAMPLE.COM is trying to impersonate principal: ambari-server@EXAMPLE.COM
2017-05-30 14:35:18.751 b.s.d.nimbus [WARN] impersonation attempt but nimbus.impersonation.authorizer has no authorizer configured. potential
security risk, please see SECURITY.MD to learn how to configure impersonation authorizer.
2017-05-30 14:35:18.756 o.a.t.s.TThreadPoolServer [ERROR] Error occurred during processing of message.
java.lang.RuntimeException: No nimbus leader participant host found, have you started your nimbus hosts?
Cause: Property ''nimbus.impersonation.authorizer" was set to 'org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer'. Prior to HDP 2.5, the storm package was backtype instead of org.apache. Solution: Modify the property as follows: nimbus.impersonation.authorizer=backtype.storm.security.auth.authorizer.ImpersonationAuthorizer
... View more
Labels:
06-29-2017
07:06 AM
3 Kudos
Problem Description: Unable to start storm nimbus from Ambari, throwing following error: raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh ln -s /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-*.jar /usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' returned 1. ln: target `/usr/hdp/current/storm-nimbus/lib/ambari-metrics-storm-sink.jar' is not a directory Cause: Two legacy jars under storm lib: # ls -ld /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common*
-rw-r--r-- 1 root root 1937430 May 11 15:57 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar
-rw-r--r-- 1 root root 1933392 Nov 23 2016 /usr/lib/storm/lib/ambari-metrics-storm-sink-legacy-with-common-2.4.2.0.136.jar Solution: As the Ambari version was 2.4.2 so, moved ambari-metrics-storm-sink-legacy-with-common-2.4.1.0.22.jar to a different location and started storm nimbus.
... View more
Labels:
06-29-2017
06:51 AM
1 Kudo
Following was the error in supervisor log: 2017-06-09 14:40:18.348 o.a.s.d.supervisor [ERROR] Error on initialization of server mk-supervisor
java.lang.RuntimeException: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83)
at org.apache.storm.utils.LocalState.get(LocalState.java:130)
at org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61)
at org.apache.storm.daemon.supervisor$standalone_supervisor$reify__7977.prepare(supervisor.clj:1216)
at org.apache.storm.daemon.supervisor$fn__7833$exec_fn__3537__auto____7834.invoke(supervisor.clj:766)
at clojure.lang.AFn.applyToHelper(AFn.java:160)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invoke(core.clj:630)
at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204)
at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at org.apache.storm.daemon.supervisor.main(Unknown Source)
Caused by: java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78)
... 14 more
2017-06-09 14:40:18.351 o.a.s.util [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at org.apache.storm.daemon.supervisor$fn__7833$mk_supervisor__7878.doInvoke(supervisor.clj:764)
at clojure.lang.RestFn.invoke(RestFn.java:436)
at org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1204)
at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1237)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at org.apache.storm.daemon.supervisor.main(Unknown Source)
Cause: 1. Few storm configs were still configured with backtype packages 2. Stale states in zookeeper and local data Solution: 1. Filtered in the configs and changed all the backtype packages to org.apache as Storm from HDP 2.5 onwards uses org.apache packages.
2. Follow below steps to clear stale states and local data:
-> Deactivate all running topologies.
-> Stop Storm service
-> Delete all states under zookeeper: $/usr/hdp/current/zookeeper-client/bin/zkCli.sh (optionally in secure environment specify -server zk.server:port)
> rmr /storm -> Delete all states under the storm-local directory. Please make sure to run this on all storm hosts: $ rm -rf <value of stormlocal.dir> -> Start storm service.
... View more
Labels:
06-28-2017
10:27 PM
2 Kudos
Problem Description: On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data. Getting Below error: 17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181] Cause: The zookeeper ACL for state znode was not correct : getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state
‘auth,’
: cdrwa,
‘world,’anyone
:r The following property was set in zookeeper env : -Dzookeeper.skipACL=yes Solution:
Deleted the topic manually as deletion was stuck Removed the property -Dzookeeper.skipACL=yes Restart zookeeper and Kafka services NOTE- It is dangerous to use -Dzookeeper.skipACL=yes property and instead it is recommended to use kafka service principal if in case you need to delete the znode for kafka topics.
... View more
Labels:
06-27-2017
06:44 AM
2 Kudos
PROBLEM: While implementing AutoHDFS for storm-hdfs integration, observed following error: 2017-05-19 11:21:44.865 o.a.s.h.c.s.AutoHDFS [ERROR] Could not populate HDFS credentials.
java.lang.RuntimeException: Failed to get delegation tokens.
at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:242)
at org.apache.storm.hdfs.common.security.AutoHDFS.populateCredentials(AutoHDFS.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__11226.submitTopologyWithOpts(nimbus.clj:1544)
at org.apache.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2940)
at org.apache.storm.generated.Nimbus$Processor$submitTopologyWithOpts.getResult(Nimbus.java:2924)
at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:138)
at org.apache.storm.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2746)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2759)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386)
at org.apache.storm.hdfs.common.security.AutoHDFS$1.run(AutoHDFS.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1704)
at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:213)
... 17 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120)
CAUSE: Missing Hadoop dependencies in pom.xml caused the following exception: Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hdfs.DistributedFileSystem not found
SOLUTION: To resolve this issue, added the following dependency in pom.xml: <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.3.2.5.3.0-37</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.3.2.5.3.0-37</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
... View more
Labels: