java.lang.NullPointerException at org.apache.hive.hcatalog.data.schema.HCatSchema.get

[#1]

HDP version: 2.3.0 upgraded from 2.2.4.2

A user reported his sqoop command fails if database column name and HCatalog column name are different. If he uses same column names for both side, it works. But he thinks it used to work as long as the number of columns and column types match.

Is this true?

The part of the error is:

java.lang.NullPointerException at org.apache.hive.hcatalog.data.schema.HCatSchema.get(HCatSchema.java:105) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureHCat(SqoopHCatUtilities.java:390) at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.configureImportOutputFormat(SqoopHCatUtilities.java:783) at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:98) ...

[#2]

what's the schema on the database side?

what's the schema on the hcatalog side?

what's the sqoop query?

[#3]

ORACLE:

CREATE TABLE HAJIME.SQ_TEST
(
  COLUMN_NUMBER                  NUMBER(5,0),
  P_SC                           VARCHAR2(16 CHAR),
  P_YMDH                         NUMBER(14,0)
);
INSERT INTO HAJIME.SQ_TEST (P_YMDH, P_SC, COLUMN_NUMBER) VALUES (5, 't', 4);
INSERT INTO HAJIME.SQ_TEST (P_YMDH, P_SC, COLUMN_NUMBER) VALUES (2, 'jlfkooakywhsc', 1);
INSERT INTO HAJIME.SQ_TEST (P_YMDH, P_SC, COLUMN_NUMBER) VALUES (3, 'vp', 3);

HIVE:

create table sq_test (
 column_number       bigint,
 logacc_no           string,
 ymdh                bigint
);

SQOOP COMMAND:

sqoop import --verbose --username hajime --password ***** --connect jdbc:oracle:thin:@ORACLE_SERVER:SID --query "select * from hajime.SQ_TEST WHERE \$CONDITIONS" --split-by COLUMN_NUMBER --hcatalog-table sq_test

Then I get NullPointerException with the following error:

15/10/28 00:47:40 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Caught Exception checking database column p_sc in hcatalog table.

[#4]

@Hajime Can you post the solution you used?

[#5 · Accepted solution]

Looks like if I wanted to use HCat, the column names need to match...