Table schema is missing on Sqooping data from Teradata to Hive

[#1]

Hi,

Agend:

When I Sqoop data from Teradata to Hive, it should import both data and create adhoc schema same as Teradata schema in my Hive.

Issue:

When I'm trying to Sqoop the data from the Teradata to Hive table, that data is getting successfully sqooped data from Teradata to database directory on HDFS but the is unable able to create the table schema in Hive.

Below is my sqoop command:

sqoop import -libjars  /usr/hdp/current/sqoop-server/lib/ --connect jdbc:teradata://example/Database=ABCDE --create-hive-table --connection-manager org.apache.sqoop.teradata.TeradataConnManager --create-hive-table --username user --password xxxxx --table cricket_location_store --split-by storeid --target-dir /apps/hive/warehouse/user.db/cricket_location_store

Below is the error message I'm getting:

Processor is: 0s16/08/25 15:58:40 ERROR teradata.TeradataSqoopImportHelper:
Exception running Teradata import jobcom.teradata.connector.common.exception.ConnectorException:
Import Hive table's column schema is missing  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)  at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)  at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)  at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)  at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)  at org.apache.sqoop.Sqoop.run(Sqoop.java:148)  at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)  at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)  at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)16/08/25 15:58:40 INFO teradata.TeradataSqoopImportHelper:
Teradata import job completed with exit code 116/08/25 15:58:40 ERROR tool.ImportTool: Encountered
IOException running import job: java.io.IOException: Exception running Teradata
import job   at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:373)  at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)  at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)  at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)  at org.apache.sqoop.Sqoop.run(Sqoop.java:148)  at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)  at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)  at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)Caused by:
com.teradata.connector.common.exception.ConnectorException: Import Hive table's
column schema is missing  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)  at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)  ... 9 more

Any help is highly appreciated and thanks in advance.

[#2]

As I last used in TDCH 1.4.2, create-hive-table as parameter in "sqoop import" doesn't work. Though you can use "sqoop create-hive-table" to create hive based on the schema from Teradata. Keep in mind, the sqoop for Teradata use TDCH as internal driver, so some of parameters are supported in sqoop, are not supported in TDCH.

[#3]

@Frank Lu

Sorry I didn't get you. Are you saying it's working or not working. If it's working can you please rewrite above sqoop command let me know how to pass create-hive-table parameter?

[#4]

It didn't work if you use "sqoop import --create-hive-table ......"

It worked if you use "sqoop create-hive-table .......". In this way, you only create the table based on Teradata schema, but not ingest the data at same time.

At the end of last time I use TDCH, I had to create table first in hive, and then TDCH the data over.

[#5]

@Frank Lu

This doesn't my problem. Even unable to execute the sqoop command. I'm using 1.4.1 teradata connector

[#6]

@Frank Lu @Kit Menke

Below is the error message I'm getting:

Processor is: 0s
16/08/25 15:58:40 ERROR teradata.TeradataSqoopImportHelper:
Exception running Teradata import job
com.teradata.connector.common.exception.ConnectorException:
Import Hive table's column schema is missing
  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)
  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)
  at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)
  at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)
  at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
  at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
  at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
  at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
  at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)
16/08/25 15:58:40 INFO teradata.TeradataSqoopImportHelper:
Teradata import job completed with exit code 1
16/08/25 15:58:40 ERROR tool.ImportTool: Encountered
IOException running import job: java.io.IOException: Exception running Teradata
import job
   at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:373)
  at
org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:504)
  at
org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
  at
org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
  at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
  at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at
org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
  at
org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
  at
org.apache.sqoop.Sqoop.main(Sqoop.java:244)
Caused by:
com.teradata.connector.common.exception.ConnectorException: Import Hive table's
column schema is missing
  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:140)
  at
com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:56)
  at
org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:370)
  ... 9 more	

Hope this helps you.

[#7]

@SBandaru

Are you using default connector that comes with Sqoop? If yes, then you need to use Hortonworks connector for Teradata. Please see the instructions including download link here.

[#8]

@mqureshi

I have seen this link, it doesn't provide much information to problem.

[#9]

Hi @SBandaru

What I meant was this link has a link to download Hortonworks Connector for Teradata. So what I was wondering is if you are using the default sqoop connector for Teradata or Hortonworks Connector for Teradata.

[#10]

@SBandaru We have been struggling with this as well. I documented some of what we've found so far here: Importing data from Teradata into Hive. Even if you get the schema created automatically with Sqoop the column types are pretty bad (all dates/timestamps get converted to string as an example).