Support Questions

Find answers, ask questions, and share your expertise

Sqoop import is failing to import more than 536 columns

Explorer

I'm trying to import a table (Oracle DB) which has 706 columns. Sqoop is failing to import if the number of columns is more than 536, getting NPE exception. I'm using Apache Sqoop 1.4.6 (HDP-2.5.3.0-37). Is this a known issue ??

Below is the sqoop command.

sqoop import --connect <connection string > --username XXXXXXXXXX --password XXXXXXXXX --target-dir <hive_table_path> --fields-terminated-by '~' --null-string '\\N' --null-non-string '\\N' -m 1

Error message:

17/04/07 16:46:32 INFO mapreduce.Job: Task Id : attempt_1491519071779_1849_m_000000_1, Status : FAILED

Error: java.lang.NullPointerException

at oracle.jdbc.driver.T4CVarcharAccessor.unmarshalOneRow(T4CVarcharAccessor.java:312)

at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:945)

at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:865)

at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:790)

at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:403)

at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:227)

at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)

3 REPLIES 3

Super Guru
@Amit Kumar Agarwal

Are you able to import any small table using this? Want to make sure your driver is in the classpath.

Explorer

Yes, able to ingest any table less than 536 columns using the sqoop command.

Super Guru
@Amit Kumar Agarwal

Can you increase the number of mappers. Why do you have only one mapper. Can you try with ten mappers? How much total data are you trying to import?