Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Single record spitted as 2 lines after imported using sqoop from oracle-- count mismatch in hive external tables,hdfs textfile showing duplicate data inserting data as nextline

Single record spitted as 2 lines after imported using sqoop from oracle-- count mismatch in hive external tables,hdfs textfile showing duplicate data inserting data as nextline

New Contributor

Hi ,

I have imported a table using Sqoop from oracle data base which contains the 80000 records. After import done I shocked total around 85000 records are available in the outputdir -m-part000 .

I used the textfile as format with delimited "," and newline "\n" , I look into the output file ..I found that a single record data splits and comes as 2nd row. but in the source table first and second rows data is a single record.

I thought some newline or delimited issues occurred using sqoop..please advice how to handle..

,

3 REPLIES 3

Re: Single record spitted as 2 lines after imported using sqoop from oracle-- count mismatch in hive external tables,hdfs textfile showing duplicate data inserting data as nextline

Guru

@Mahesh K get the output of DBMS_METADATA.GET_DDL('TABLE','EMP','SCOTT') for the source table in Oracle, you'd have to compare both the layouts first, for delimiter, nulls, quotes if any. Text is certainly a good choice in hive, however, the issue looks like field delimiter or linefeed.

Can you share one or two rows of the records you believe were split as two records ?

Re: Single record spitted as 2 lines after imported using sqoop from oracle-- count mismatch in hive external tables,hdfs textfile showing duplicate data inserting data as nextline

New Contributor

Re: Single record spitted as 2 lines after imported using sqoop from oracle-- count mismatch in hive external tables,hdfs textfile showing duplicate data inserting data as nextline

New Contributor

Hi Shyam ,

Thanks for the repsonse...

I have looked into the DB table one of the single column having text as below ..

"Not directly related to this loss event, but controls have been in place as below

- Have set up hard limit to prevent fat finger order error

- Have double checked the message poped up or alarmed from the system"

Hence , The same data in part-m-00000 coming as below 3 records..

1- Not directly related to this loss event, but controls have been in place as below

2 - Have set up hard limit to prevent fat finger order error

3- Have double checked the message poped up or alarmed from the system

at runtime how we can avoid..since i am using - --import-all-tables using sqoop.

Don't have an account?
Coming from Hortonworks? Activate your account here