question Re: DataFrame.insertInto() an existing hive table encounter "Table Not Found" error in Archives of Support Questions (Read Only)

DataFrame.insertInto() an existing hive table encounter "Table Not Found" error

rxu — Wed, 09 Dec 2015 03:44:10 GMT

Java code:

DataFrame peopleDataFrame = sqlContext.createDataFrame(rowRDD, schema);

HiveContext hiveContext = new org.apache.spark.sql.hive.HiveContext( jsc.sc() );

hiveContext.sql("CREATE TABLE IF NOT EXISTS people_t1 (emp_id string, first_name string, last_name string, job_title string, mgr_emp_id string)");

// Register the DataFrame as a table.

peopleDataFrame.registerTempTable("people");

....

peopleDataFrame.insertInto("default.people_t1", true);

Got:

java.lang.RuntimeException: Table Not Found: default.people_t1

While table does exist in Hive:

hive> describe people_t1;

emp_id string

first_name string

last_name string

job_title string

mgr_emp_id string

Time taken: 0.284 seconds, Fetched: 5 row(s)

Re: DataFrame.insertInto() an existing hive table encounter "Table Not Found" error

phargis — Wed, 09 Dec 2015 04:12:54 GMT

What version of Spark? I assume you are using Spark 1.5.1 from Tech Preview. What if you try using the DataFrameWriter class, saveAsTable() method. That may not work for you if you really are INSERTing records to an existing table, but could work if you mean to OVERWRITE the table.

Documentation here:

http://spark.apache.org/docs/latest/api/scala/#org.apache.spark.sql.DataFrameWriter

Re: DataFrame.insertInto() an existing hive table encounter "Table Not Found" error

rxu — Wed, 09 Dec 2015 04:45:21 GMT

I am using Spark 1.3.1. Seems that saveAsTable() creates internal Spark table source.

http://stackoverflow.com/questions/31482798/save-spark-dataframe-to-hive-table-not-readable-because-parquet-not-a-sequence

Re: DataFrame.insertInto() an existing hive table encounter "Table Not Found" error

rxu — Wed, 09 Dec 2015 05:50:25 GMT

Figured it out, it has to be HiveContext, not SQLContext, after making below change, it works:

HiveContext hiveContext = new org.apache.spark.sql.hive.HiveContext(sc.sc());

//SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);