Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

sqoop import

Solved Go to solution
Highlighted

sqoop import

Super Collaborator

Hi:

after run this script, the map reduce doesnt finish, here the script and the log:

sqoop import -D oraoop.disabled=true  --connect jdbc:oracle:thin:@HOSTNAME:2521/CIP_BATCH --username=XXXXXX --password=XXXXXX --table RDWC01.MI_CLTE_ECO_GEN \
--columns "MI_FECHA_FIN_MES,COD_NRBE_EN,COD_LINEA,ID_GRP_PD,MI_NUM_TOT_AC_ACT,MI_NUM_AC_SUS,MI_SDO_AC_P,MI_NUM_AC_P,MI_DIA_AC_P,MI_INT_DEV_ACR_D,MI_INT_DEV_DEU_D,MI_COMIS_APL_D,MI_TOT_MOV_D,MI_TOT_MOV_H,MI_TOT_IMP_MOV_D,MI_TOT_IMP_MOV_H" \
--where "COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD')" \
--target-dir=/RSI/datalake/desercion/2016/1 --verbose --split-by COD_NRBE_EN
16/06/07 13:11:02 WARN db.TextSplitter: Generating splits for a textual index column.
16/06/07 13:11:02 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
16/06/07 13:11:02 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
16/06/07 13:11:02 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to MIN_INCREMENT
16/06/07 13:11:02 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound 'COD_NRBE_EN >= '3159'' and upper bound 'COD_NRBE_EN <= '3159''
16/06/07 13:11:02 INFO mapreduce.JobSubmitter: number of splits:1
16/06/07 13:11:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464163049638_0629
16/06/07 13:11:03 INFO impl.YarnClientImpl: Submitted application application_1464163049638_0629
16/06/07 13:11:03 INFO mapreduce.Job: The url to track the job: http://lnxbig05.cajarural.gcr:8088/proxy/application_1464163049638_0629/
16/06/07 13:11:03 INFO mapreduce.Job: Running job: job_1464163049638_0629
16/06/07 13:11:12 INFO mapreduce.Job: Job job_1464163049638_0629 running in uber mode : false
16/06/07 13:11:12 INFO mapreduce.Job:  map 0% reduce 0%

any suggestions???

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: sqoop import

Super Collaborator

Hi:

its ok now, its work with this;

--query "select ID_INTERNO_PE,MI_FECHA_FIN_MES,COD_NRBE_EN,COD_LINEA,ID_GRP_PD,MI_NUM_TOT_AC_ACT,MI_NUM_AC_SUS,MI_SDO_AC_P,MI_NUM_AC_P,MI_DIA_AC_P,MI_INT_DEV_ACR_D,MI_INT_DEV_DEU_D,MI_COMIS_APL_D,MI_TOT_MOV_D,MI_TOT_MOV_H,MI_TOT_IMP_MOV_D,MI_TOT_IMP_MOV_H from RDWC01.MI_CLTE_ECO_GEN where \$CONDITIONS AND COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD')" \


NoteIf you are issuing the query wrapped with double quotes ("), you will have to use \$CONDITIONS instead of just $CONDITIONS to disallow your shell from treating it as a shell variable. For example, a double quoted query may look like: "SELECT * FROM x WHERE a='foo' AND \$CONDITIONS"

Many thanks all of you.

View solution in original post

18 REPLIES 18
Highlighted

Re: sqoop import

@Roberto Sancho

Can you please check the application log of "job_1464163049638_0629" if there are any errors?

Also, resource manager log to understand if the containers are allocated or not.

Thanks and Regards,

Sindhu

Highlighted

Re: sqoop import

Hi @Roberto Sancho. Any relevant information in ressources manager logs?

Highlighted

Re: sqoop import

Super Collaborator

its posible i need user and pass proxy inside my company???

2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: Retrying request
Highlighted

Re: sqoop import

Super Collaborator

4810-captura.png

hi:

its important:

2016-06-07 13:11:17,313 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: SELECT MI_FECHA_FIN_MES, COD_NRBE_EN, COD_LINEA, ID_GRP_PD, MI_NUM_TOT_AC_ACT, MI_NUM_AC_SUS, MI_SDO_AC_P, MI_NUM_AC_P, MI_DIA_AC_P, MI_INT_DEV_ACR_D, MI_INT_DEV_DEU_D, MI_COMIS_APL_D, MI_TOT_MOV_D, MI_TOT_MOV_H, MI_TOT_IMP_MOV_D, MI_TOT_IMP_MOV_H FROM RDWC01.MI_CLTE_ECO_GEN WHERE ( COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD') ) AND ( COD_NRBE_EN >= '3159' ) AND ( COD_NRBE_EN <= '3159' )
2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: Retrying request

Highlighted

Re: sqoop import

@Roberto Sancho

Seems like connectivity issue for Application timeline server, please check if timeline server is up and running.

Highlighted

Re: sqoop import

Super Collaborator
Hi: app time is up and here the log
2016-06-07 13:05:47,152 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619147147 and earlier in 0.005 seconds
2016-06-07 13:10:47,157 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619447152 and earlier in 0.005 seconds
2016-06-07 13:15:47,163 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619747158 and earlier in 0.005 seconds
2016-06-07 13:20:47,168 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620047163 and earlier in 0.005 seconds
2016-06-07 13:25:47,173 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620347168 and earlier in 0.005 seconds
2016-06-07 13:30:47,178 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620647173 and earlier in 0.005 seconds
2016-06-07 13:35:47,183 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620947178 and earlier in 0.005 seconds
2016-06-07 13:40:47,189 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462621247184 and earlier in 0.005 seconds
2016-06-07 13:45:47,194 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462621547189 and earlier in 0.005 seconds

4821-captura.png

Highlighted

Re: sqoop import

Please share the complete error log of application, the above message seems to be misleading.

Highlighted

Re: sqoop import

Super Collaborator

hi:

now is working, but just 1 split and very very slow, a need help please

sqoop import -D oraoop.disabled=true \
--connect jdbc:oracle:thin:@hostname:2521/CIP_BATCH  \
--username=U029550 \
--password=Mayo2016 \
--table RDWC01.MI_CLTE_ECO_GEN \
--columns "ID_INTERNO_PE,MI_FECHA_FIN_MES,COD_NRBE_EN,COD_LINEA,ID_GRP_PD,MI_NUM_TOT_AC_ACT,MI_NUM_AC_SUS,MI_SDO_AC_P,MI_NUM_AC_P,MI_DIA_AC_P,MI_INT_DEV_ACR_D,MI_INT_DEV_DEU_D,MI_COMIS_APL_D,MI_TOT_MOV_D,MI_TOT_MOV_H,MI_TOT_IMP_MOV_D,MI_TOT_IMP_MOV_H" \
--where "COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD')" \
--target-dir=/RSI/datalake/desercion/2016/1
--verbose
--split-by MI_FECHA_FIN_MES \
--fetch-size 10000 \
--num-mappers 5 \
--direct \
--direct-split-size 1000000000 \



16/06/07 19:17:54 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/07 19:17:54 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(COD_NRBE_EN), MAX(COD_NRBE_EN) FROM RDWC01.MI_CLTE_ECO_GEN WHERE ( COD_RL_                                    PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD') )


16/06/07 19:22:52 WARN db.TextSplitter: Generating splits for a textual index column.
16/06/07 19:22:52 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
16/06/07 19:22:52 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
16/06/07 19:22:52 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to MIN_INCREMENT
16/06/07 19:22:52 INFO mapreduce.JobSubmitter: number of splits:1
16/06/07 19:22:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464163049638_0638
16/06/07 19:22:53 INFO impl.YarnClientImpl: Submitted application application_1464163049638_0638
16/06/07 19:22:53 INFO mapreduce.Job: The url to track the job: http://lnxbig05.cajarural.gcr:8088/proxy/application_1464163049638_0638/
16/06/07 19:22:53 INFO mapreduce.Job: Running job: job_1464163049638_0638
16/06/07 19:23:04 INFO mapreduce.Job: Job job_1464163049638_0638 running in uber mode : false
16/06/07 19:23:04 INFO mapreduce.Job:  map 0% reduce 0%



Highlighted

Re: sqoop import

Super Collaborator

Hi:

can i split by a integer colum????

thanks

Don't have an account?
Coming from Hortonworks? Activate your account here