@lweichberger, I suspect the same about the version mismatch. https://spark.apache.org/docs/2.3.0/streaming-kafka-0-8-integration.html As per the example given in above link, when I create the streaming context ssc = StreamingContext(sc, 5) I get the JavaAbstractMethodError from py4j. Using spark-sql-kafka, I can get the structured stream with new API. But I like to fix the existing version issue. ... View more

@lweichberger Thanks for your article. It really helps a lot. I am facing similar kind of issue with pyspark. I am using the spark version 2.3.0.2.6.5-292.I want to read the kakfa stream. My code was working fine in spark version 2.2.0 with this package p.p1 org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 Support for spark-streaming-kafka-0-8.2.11 is deprecated in spark 2.3.0 per documentation. I am able to import the StreamingContext from pyspark but I am not able to create the StreamingContext. I am getting the same error of Java AbstractMethod. It seems something has changed in the underneath. I am following the official guide from here: https://spark.apache.org/docs/2.3.0/structured-streaming-kafka-integration.html # pyspark --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.3.0.2.6.5.0-292 /_/ Using Python version 2.7.5 (default, Sep 15 2016 22:37:39) SparkSession available as 'spark'. >>> from pyspark.streaming import StreamingContext >>> ssc = StreamingContext(sc, 5) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/hdp/current/spark2-client/python/pyspark/streaming/context.py", line 61, in __init__ self._jssc = jssc or self._initialize_context(self._sc, batchDuration) File "/usr/hdp/current/spark2-client/python/pyspark/streaming/context.py", line 65, in _initialize_context return self._jvm.JavaStreamingContext(sc._jsc, self._jduration(duration)) File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1428, in __call__ File "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.streaming.api.java.JavaStreamingContext. : java.lang.AbstractMethodError at org.apache.spark.util.ListenerBus$class.$init$(ListenerBus.scala:35) at org.apache.spark.streaming.scheduler.StreamingListenerBus.<init>(StreamingListenerBus.scala:30) at org.apache.spark.streaming.scheduler.JobScheduler.<init>(JobScheduler.scala:57) at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:184) at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:76) at org.apache.spark.streaming.api.java.JavaStreamingContext.<init>(JavaStreamingContext.scala:130) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:238) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:745) >>> ... View more