Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

incremental load into ORC table

Highlighted

incremental load into ORC table

Master Collaborator

I asked this question 3 times and got no valid respone , for some reason no hortonworks employee is addressing it but I am hoping someone must have done similar thing and will answer it .

I am unable to do incremental loads in ORC table.

sqoop job --create incjob  -- import --connect "jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=patronQA)(port=1526))(connect_data=(service_name=patron)))" --username RON --password xxxx --incremental append --check-column txn_process_date --table PATRON.TAB1 --split-by TAB1.TXN_ID --hcatalog-database default --hcatalog-table PA_LANE_TXN_orc --create-hcatalog-table --hcatalog-storage-stanza 'stored as orc tblproperties ("orc.compress"="SNAPPY")' --map-column-hive 'txn_process_date=date ,ext_date_time=date ,ent_date_time=date' --columns 'TXN_ID,TXN_PROCESS_DATE,TRANSP_ID,TRANSP_CLASS,EXT_PLAZA_ID,EXT_LANE_ID,EXT_LANE_TYPE_CODE,EXT_DATE_TIME,TRANSP_CUR_BAL,AVC_CLASS,TOLL_AMT_CHARGED,TOLL_AMT_COLLECTED,TOLL_AMT_FULL,CREDIT_LIST_UPDATE_FLAG,TRANS_SOURCE,REVCLASS_REV_CLASS_CODE,PAYMENT_METHOD_CODE,ENT_PLAZA_ID,ENT_DATE_TIME,ENT_LANE_ID,ENT_LANE_TYPE_CODE,AGENCY_REJECT_CODE,MSG_ID,TRANSP_INTERNAL_NUM,UFM_PAYMENT_CODE,VEH_LIC_NUM,STATE_ID_CODE,ORIG_TXN_ID'

Append mode for imports is not  compatible with HCatalog. Please remove the parameter--append-mode
[root@hadoop1 ~]#

3 REPLIES 3
Highlighted

Re: incremental load into ORC table

Highlighted

Re: incremental load into ORC table

Super Guru

@Sami Ahmad

As the error message says, you cannot do incremental updates using Hcatalog. See the example on this page:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_dataintegration/content/incrementally-up...

Use jdbc driver to connect. Also, on the following page, read the best answer to remove any confusion you may have (there is too much redundancy on this page. Simply focus on best answer and comments on best answer).

https://community.hortonworks.com/questions/10710/sqoop-incremental-import-working-fine-now-i-want-k...

Highlighted

Re: incremental load into ORC table

Explorer

As I know, sqoop import "--append" options is not compatible with "--hcatalog-*". Just remove "--append".

Don't have an account?
Coming from Hortonworks? Activate your account here