Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

pyspark exception error handling ?

Highlighted

pyspark exception error handling ?

I am newbie for pyspark , i could not able to get pyspark exception handling in transformations .

for example i am calling a function on each line of map transformation , i would like to handle few exceptions in that functions and log them .

exampleof my code : .map(lambda eachone : ultility.function1(eachone , somesrgs)) , here in function1 i would like to handle exceptions .

please provide an example for better understanding . thank you .

1 REPLY 1
Highlighted

Re: pyspark exception error handling ?

Expert Contributor

@srini

I couldn't find any special exception handling behavior implemented for pyspark.

But you can do a native python try/except in utility.function1, returning None if there is an exception and then filter the resulting rdd on that. (or handle as you want to)

Don't have an account?
Coming from Hortonworks? Activate your account here