Created on 10-22-2018 08:58 PM - edited 09-16-2022 06:49 AM
I want to get subquery from impala table as one dataset.
Code like this:
String subQuery = "(select to_timestamp(unix_timestamp(now())) as ts from my_table) t" Dataset<Row> ds = spark.read().jdbc(myImpalaUrl, subQuery, prop);
But result is error:
Caused by: java.sql.SQLDataException: [Cloudera][JDBC](10140) Error converting value to Timestamp.
to_timestmap() function failed ,but unix_timestmap() , now() work fine.
ps. I found another problem, when I use hive udf in the "jdbc" api still failed.
Can anyone help me?
Created 06-24-2020 09:13 AM
Getting same conversion issues for below schema.
root
|-- instance: string (nullable = true)
|-- count(*): long (nullable = true)
Error Message:
java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to long.
at com.cloudera.exceptions.ExceptionConverter.toSQLException(Unknown Source)
at com.cloudera.utilities.conversion.TypeConverter.toLong(Unknown Source)
at com.cloudera.jdbc.common.SForwardResultSet.getLong(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$8.apply(JdbcUtils.scala:409)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$8.apply(JdbcUtils.scala:408)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:330)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:312)
Can anyone please help on resolving this issue in spark java.
Created on 10-02-2021 12:58 AM - edited 10-02-2021 01:00 AM
Hey cloudera, @Nathan @Shushruth @is this issue addressed getting the same issue with python3 and pyspark2.3 while fetching data from Impala using spark jdbc.
Created 10-04-2021 01:58 AM
Hello,
Are you trying to connect impala from spark via JDBC?
if yes, we don't support this feature yet. please refer to the below document.
Created 10-04-2021 02:02 AM
@ShankerSharma is cloudera planning to include this functionality anytime soon. If not, what other ways do we have to read tables from Impala using pyspark 2.3.