Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

pyspark2 kinesis streaming, AttributeError: 'SparkSession' object has no attribute '_getJavaStorageLevel'

Highlighted

pyspark2 kinesis streaming, AttributeError: 'SparkSession' object has no attribute '_getJavaStorageLevel'

New Contributor

Am using spark 2.3.0 and python 2.7.13. Am trying to build a streaming pipeline but am getting error while trying to createStream. below is the code snippet am using . Am trying it pyspark2 --master="local[*]". I have the jars in my local which am setting using config option (read online that kinesis asl & spark should have compatible jar versions:

 

from __future__ import print_function

import sys

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.storagelevel import StorageLevel
from pyspark.streaming import DStream
from pyspark.streaming import StreamingContext
from pyspark.streaming.kinesis import KinesisUtils, InitialPositionInStream


spark = SparkSession.builder.appName("TestStreaming").config("spark.jars","spark-streaming-kinesis-asl_2.11-2.3.0.jar")\
.config("spark.jars","spark-streaming-kinesis-asl-assembly_2.11-2.0.0.jar")\
.config("spark.jars","aws-java-sdk-1.11.310.jar")\
.config("spark.jars","amazon-kinesis-client-1.9.0.jar")\
.getOrCreate()
ssc = StreamingContext(spark, 2)


lines = KinesisUtils.createStream(ssc, "TestStreaming", "cwlog-kinesis-stream-SDLC-catalina", "https://kinesis.us-east-1.amazonaws.com", "us-east-1", InitialPositionInStream.LATEST, 2, StorageLevel.MEMORY_AND_DISK_2)

 

 

Am getting below on executing "lines":

 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/python/pyspark/streaming/kinesis.py", line 79, in createStream
    jlevel = ssc._sc._getJavaStorageLevel(storageLevel)
AttributeError: 'SparkSession' object has no attribute '_getJavaStorageLevel'