Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

pyspark exception error handling ?

I am newbie for pyspark , i could not able to get pyspark exception handling in transformations .

for example i am calling a function on each line of map transformation , i would like to handle few exceptions in that functions and log them .

exampleof my code : .map(lambda eachone : ultility.function1(eachone , somesrgs)) , here in function1 i would like to handle exceptions .

please provide an example for better understanding . thank you .


Expert Contributor


I couldn't find any special exception handling behavior implemented for pyspark.

But you can do a native python try/except in utility.function1, returning None if there is an exception and then filter the resulting rdd on that. (or handle as you want to)

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.