Member since
08-14-2015
6
Posts
6
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 8614 | 11-18-2016 07:50 PM | |
| 51922 | 08-19-2015 03:47 PM |
11-18-2016
07:50 PM
Finally I am able to resolve this issue - it is more related to improper usage of kafka dependencies in my pom.xml This is what I did : 1. Created a new Cloudera Quickstart VM 5.8 2. Installed Kafka by following the commands mentioned in https://www.cloudera.com/documentation/kafka/latest/topics/kafka_packaging.html In the above HTML link read carefully on maven dependency we have to use. Also check below pom.xml where kafka_clients is excluded and added from Cloudera Repository 3. In my maven pom.xml I have below configuration Please note below kafka_clients exclusion while defining spark-streaming-kafka <repositories> <repository> <id>cloudera</id> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> </repository> </repositories> <properties> <java.version>1.8</java.version> <scala.minor.version>2.10</scala.minor.version> <scala.complete.version>${scala.minor.version}.6</scala.complete.version> <spark.version>1.6.1</spark.version> </properties> ..... <dependencies> ............... <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_${scala.minor.version}</artifactId> <version>${spark.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.9.0-kafka-2.0.1</version> </dependency> ............... <dependencies> Hope this will help others Thanks, TrainingMyHobby
... View more
11-17-2016
10:38 PM
Dear All,
On Mac -> Virtual Box -> Clouderq Quickstart VM 5.8
Installed Kafka using packages from command line (followed the Cloudera Kafka installation document i.e. yum install kafka)
Changed the /etc/yum.repos.d/cloudera-kafka.repo to use "baseurl=http://archive.cloudera.com/kafka/redhat/6/x86_64/kafka/1/"
Developed a Spark Kafka Streaming Consumer. My maven dependencies are
<spark.version>1.6.1</spark.version>
<scala.minor.version>2.10</scala.minor.version>
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_${scala.minor.version}</artifactId> <version>${spark.version}</version> </dependency>
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka_${scala.minor.version}</artifactId> <version>${spark.version}</version> </dependency>
Issue:
I keep getting this below error in my Spark-Kafka code ....
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder]( streamingCtx, config.asScala.toMap, kafkaTopics.toSet)
logger.info("Count in stream is " + stream.count())
stream.foreachRDD { rdd => rdd.collect.foreach(x => logger.info("KAFKA_VALUE_IS " + x)) }
I have only 1 partition and 1 broker in my Cloudera Quickstart 5.8 VM
Please help me in resolving below issue.
Thanks,
Training My Hobby
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, quickstart.cloudera): java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:267) at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.TopicData$.readFrom(FetchResponse.scala:98) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator$class.foreach(Iterator.scala:727)
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
-
Quickstart VM
08-19-2015
03:47 PM
5 Kudos
I am able to resolve this issue and below Oracle link helped me to resolve it http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Ticket) Solution is we need to specify -Djavax.security.auth.useSubjectCredsOnly=false while executing my Java program from command line That means java -Djavax.security.auth.useSubjectCredsOnly=false ........... My Java program internally uses Hive JDBC API. This is what I did: 1. kinit from command line 2. Run the Java program with above -D property and also in the JDBC URL specify the appropriate Hive JDBC URL with principal name etc
... View more
08-14-2015
04:01 PM
Here is the additionl stack trace 2015-08-14 18:44:40 INFO Utils:285 - Supplied authorities: hiveserver-ip-address:10000 2015-08-14 18:44:40 WARN Utils:401 - ***** JDBC param deprecation ***** 2015-08-14 18:44:40 WARN Utils:402 - The use of sasl.qop is deprecated. 2015-08-14 18:44:40 WARN Utils:403 - Please use saslQop like so: jdbc:hive2://<host>:<port>/dbName;saslQop=<qop_value> 2015-08-14 18:44:40 INFO Utils:372 - Resolved authority: hiveserver-ip-address:10000 2015-08-14 18:44:40 DEBUG MutableMetricsFactory:42 - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)]) 2015-08-14 18:44:40 DEBUG MutableMetricsFactory:42 - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)]) 2015-08-14 18:44:40 DEBUG MutableMetricsFactory:42 - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups]) 2015-08-14 18:44:40 DEBUG MetricsSystemImpl:231 - UgiMetrics, User and group related metrics 2015-08-14 18:44:40 DEBUG Groups:301 - Creating new Groups object 2015-08-14 18:44:40 DEBUG NativeCodeLoader:46 - Trying to load the custom-built native-hadoop library... 2015-08-14 18:44:40 DEBUG NativeCodeLoader:55 - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path 2015-08-14 18:44:40 DEBUG NativeCodeLoader:56 - java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2015-08-14 18:44:40 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-08-14 18:44:40 DEBUG PerformanceAdvisory:41 - Falling back to shell based 2015-08-14 18:44:40 DEBUG JniBasedUnixGroupsMappingWithFallback:45 - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping 2015-08-14 18:44:40 DEBUG Groups:112 - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 2015-08-14 18:44:40 DEBUG UserGroupInformation:221 - hadoop login 2015-08-14 18:44:40 DEBUG UserGroupInformation:156 - hadoop login commit 2015-08-14 18:44:40 DEBUG UserGroupInformation:186 - using local user:UnixPrincipal: <my login name> 2015-08-14 18:44:40 DEBUG UserGroupInformation:192 - Using user: "UnixPrincipal: <my login name>" with name <my login name> 2015-08-14 18:44:40 DEBUG UserGroupInformation:202 - User entry: "<my login name>" 2015-08-14 18:44:40 DEBUG UserGroupInformation:840 - UGI loginUser:<my login name> (auth:SIMPLE) 2015-08-14 18:44:40 DEBUG HadoopThriftAuthBridge:155 - Current authMethod = SIMPLE 2015-08-14 18:44:40 DEBUG HadoopThriftAuthBridge:93 - Setting UGI conf as passed-in authMethod of kerberos != current. 2015-08-14 18:44:40 INFO HiveConnection:189 - Will try to open client transport with JDBC Uri: jdbc:hive2://hiveserver-ip-address:10000/default;principal=hive/_HOST@A.B.COM;sasl.qop=auth-conf 2015-08-14 18:44:40 DEBUG UserGroupInformation:1693 - PrivilegedAction as:<my login name> (auth:SIMPLE) from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) 2015-08-14 18:44:40 DEBUG TSaslTransport:261 - opening transport org.apache.thrift.transport.TSaslClientTransport@41c2284a 2015-08-14 18:44:40 ERROR TSaslTransport:315 - SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:208)
... View more
08-14-2015
03:37 PM
1 Kudo
Hi All, In my project I have a need to connect to Hive through JDBC. Developed a JDBC client program and connecting to it I am getting below error. What I need to do? Cloudera CDH version (Hadoop 2.6.0-cdh5.4.3) TUGIAssumingTransport.java:49) 2015-08-14 18:16:55 DEBUG TSaslTransport:261 - opening transport org.apache.thrift.transport.TSaslClientTransport@41c2284a 2015-08-14 18:16:55 ERROR TSaslTransport:315 - SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] Below is a snippet of HiveClient program and below explained the steps I am executing MyHiveClient { public static Connection createConnection() { Class.forName("org.apache.hive.jdbc.HiveDriver") String hive2JDBCConnectionURL = "jdbc:hive2://hiveserver-ip-address:10000/default;principal=hive/_HOST@A.B.COM;sasl.qop=auth-conf"" return DriverManager.getConnection(hive2JDBCConnectionURL, new Properties()) } My java class is invoked through a shell script (say run_metrics.sh ). This Java class internally creates a Hive JDBC connection by invoking above MyHiveClient.createConnection This is what I do : kinit <enter password here> ./run_metrics.sh <Now I get above error>
... View more
Labels: