Support Questions

Find answers, ask questions, and share your expertise

Soop failing due to column name having hyphen (note-id)

New Contributor

I am trying to ingest data from progress datasource and I have few fields with names like Sales-uk due to hyphen sqoop is throwing me an error It seems like you are looking up a column that does not

[2018-05-10 15:05:09,149] {base_task_runner.py:95} INFO - Subtask: [2018-05-10 15:05:09,148] {sqoop_hook.py:101} INFO - 18/05/10 15:05:09 ERROR util.SqlTypeMap: exist in the table. Please ensure that you've specified
[2018-05-10 15:05:09,149] {base_task_runner.py:95} INFO - Subtask: [2018-05-10 15:05:09,148] {sqoop_hook.py:101} INFO - 18/05/10 15:05:09 ERROR util.SqlTypeMap: correct column names in Sqoop options.
[2018-05-10 15:05:09,149] {base_task_runner.py:95} INFO - Subtask: [2018-05-10 15:05:09,148] {sqoop_hook.py:101} INFO - 18/05/10 15:05:09 ERROR tool.ImportTool: Import failed: column not found: `notes-id`

I can do this through query option but my framework doesn't allow me to use query function in sqoop, can you please suggest is there any other way around this please?
2 REPLIES 2

Expert Contributor

If you are using a SQL statement in sqoop, try bring the columns as new alias like select notes-id notesid, ...

New Contributor

I am creating a DAG in Json format and ingesting data to hive and then to impala.

I can't use SQL statement or --query option as my framework doesn't support it.I am using Airflow to run these jobs.

PS:I am able to run this using Select statement in sqooo command when running manually not in airflow.As said my framework for Airflow doesnt support query so looking for any other alternatives like using '' or ``