Member since
03-17-2020
1
Post
0
Kudos Received
0
Solutions
03-17-2020
11:53 PM
Hello,
We are getting below exception while reading table from netezza using spark,
py4j.protocol.Py4JJavaError: An error occurred while calling o287.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 10.0 failed 4 times, most recent failure: Lost task 0.3 in stage 10.0 (TID 13, executor 10): org.netezza.error.NzSQLException: netezza.bad.value
at org.netezza.sql.NzResultSet.getDbosTimestamp(NzResultSet.java:4053)
at org.netezza.sql.NzResultSet.getTimestamp(NzResultSet.java:1578)
at org.netezza.sql.NzResultSet.getTimestamp(NzResultSet.java:1528).
Attaching full stackstarce.
we are using nzjdbc3.jar for netezza and spark connection and below are the connection string,
input_df = spark.read.format('jdbc').options(url='jdbc:netezza://server_name:port/dbname', user='', password='’, driver='org.netezza.Driver',dbtable="(select * from schema_name.table_name limit 100) as t").load()
I am able to print schema of dataframe but when i performed some action like show(),count() It is failing for timestamp column for selected tables, for other tables it is working fine. Also i am able to select other columns other than timestamp columns.
The below workaround we tried,
1) convert timestamp to stringType() still failing.
What will be the fix for this issue?
Thanks
... View more
Labels:
- Labels:
-
Apache Spark