sqoop import

[#1]

Hi:

after run this script, the map reduce doesnt finish, here the script and the log:

sqoop import -D oraoop.disabled=true  --connect jdbc:oracle:thin:@HOSTNAME:2521/CIP_BATCH --username=XXXXXX --password=XXXXXX --table RDWC01.MI_CLTE_ECO_GEN \
--columns "MI_FECHA_FIN_MES,COD_NRBE_EN,COD_LINEA,ID_GRP_PD,MI_NUM_TOT_AC_ACT,MI_NUM_AC_SUS,MI_SDO_AC_P,MI_NUM_AC_P,MI_DIA_AC_P,MI_INT_DEV_ACR_D,MI_INT_DEV_DEU_D,MI_COMIS_APL_D,MI_TOT_MOV_D,MI_TOT_MOV_H,MI_TOT_IMP_MOV_D,MI_TOT_IMP_MOV_H" \
--where "COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD')" \
--target-dir=/RSI/datalake/desercion/2016/1 --verbose --split-by COD_NRBE_EN

16/06/07 13:11:02 WARN db.TextSplitter: Generating splits for a textual index column.
16/06/07 13:11:02 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
16/06/07 13:11:02 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
16/06/07 13:11:02 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to MIN_INCREMENT
16/06/07 13:11:02 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound 'COD_NRBE_EN >= '3159'' and upper bound 'COD_NRBE_EN <= '3159''
16/06/07 13:11:02 INFO mapreduce.JobSubmitter: number of splits:1
16/06/07 13:11:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464163049638_0629
16/06/07 13:11:03 INFO impl.YarnClientImpl: Submitted application application_1464163049638_0629
16/06/07 13:11:03 INFO mapreduce.Job: The url to track the job: http://lnxbig05.cajarural.gcr:8088/proxy/application_1464163049638_0629/
16/06/07 13:11:03 INFO mapreduce.Job: Running job: job_1464163049638_0629
16/06/07 13:11:12 INFO mapreduce.Job: Job job_1464163049638_0629 running in uber mode : false
16/06/07 13:11:12 INFO mapreduce.Job:  map 0% reduce 0%

any suggestions???

Thanks

[#2]

@Roberto Sancho

Can you please check the application log of "job_1464163049638_0629" if there are any errors?

Also, resource manager log to understand if the containers are allocated or not.

Thanks and Regards,

Sindhu

[#3]

Hi @Roberto Sancho. Any relevant information in ressources manager logs?

[#4]

its posible i need user and pass proxy inside my company???

2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: Retrying request

[#5]

hi:

its important:

2016-06-07 13:11:17,313 INFO [main] org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: SELECT MI_FECHA_FIN_MES, COD_NRBE_EN, COD_LINEA, ID_GRP_PD, MI_NUM_TOT_AC_ACT, MI_NUM_AC_SUS, MI_SDO_AC_P, MI_NUM_AC_P, MI_DIA_AC_P, MI_INT_DEV_ACR_D, MI_INT_DEV_DEU_D, MI_COMIS_APL_D, MI_TOT_MOV_D, MI_TOT_MOV_H, MI_TOT_IMP_MOV_D, MI_TOT_IMP_MOV_H FROM RDWC01.MI_CLTE_ECO_GEN WHERE ( COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD') ) AND ( COD_NRBE_EN >= '3159' ) AND ( COD_NRBE_EN <= '3159' )
2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2016-06-07 13:13:15,466 INFO [timeline] org.apache.commons.httpclient.HttpMethodDirector: Retrying request

[#6]

@Roberto Sancho

Seems like connectivity issue for Application timeline server, please check if timeline server is up and running.

[#7]

Hi: app time is up and here the log

2016-06-07 13:05:47,152 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619147147 and earlier in 0.005 seconds
2016-06-07 13:10:47,157 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619447152 and earlier in 0.005 seconds
2016-06-07 13:15:47,163 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462619747158 and earlier in 0.005 seconds
2016-06-07 13:20:47,168 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620047163 and earlier in 0.005 seconds
2016-06-07 13:25:47,173 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620347168 and earlier in 0.005 seconds
2016-06-07 13:30:47,178 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620647173 and earlier in 0.005 seconds
2016-06-07 13:35:47,183 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462620947178 and earlier in 0.005 seconds
2016-06-07 13:40:47,189 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462621247184 and earlier in 0.005 seconds
2016-06-07 13:45:47,194 INFO  timeline.LeveldbTimelineStore (LeveldbTimelineStore.java:discardOldEntities(1527)) - Discarded 0 entities for timestamp 1462621547189 and earlier in 0.005 seconds

[#8]

Please share the complete error log of application, the above message seems to be misleading.

[#9]

hi:

now is working, but just 1 split and very very slow, a need help please

sqoop import -D oraoop.disabled=true \
--connect jdbc:oracle:thin:@hostname:2521/CIP_BATCH  \
--username=U029550 \
--password=Mayo2016 \
--table RDWC01.MI_CLTE_ECO_GEN \
--columns "ID_INTERNO_PE,MI_FECHA_FIN_MES,COD_NRBE_EN,COD_LINEA,ID_GRP_PD,MI_NUM_TOT_AC_ACT,MI_NUM_AC_SUS,MI_SDO_AC_P,MI_NUM_AC_P,MI_DIA_AC_P,MI_INT_DEV_ACR_D,MI_INT_DEV_DEU_D,MI_COMIS_APL_D,MI_TOT_MOV_D,MI_TOT_MOV_H,MI_TOT_IMP_MOV_D,MI_TOT_IMP_MOV_H" \
--where "COD_RL_PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD')" \
--target-dir=/RSI/datalake/desercion/2016/1
--verbose
--split-by MI_FECHA_FIN_MES \
--fetch-size 10000 \
--num-mappers 5 \
--direct \
--direct-split-size 1000000000 \

16/06/07 19:17:54 INFO db.DBInputFormat: Using read commited transaction isolation
16/06/07 19:17:54 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(COD_NRBE_EN), MAX(COD_NRBE_EN) FROM RDWC01.MI_CLTE_ECO_GEN WHERE ( COD_RL_                                    PERS_AC = 01 AND COD_LINEA in ('01','03','04','05') AND COD_NRBE_EN = '3159' AND TRUNC(MI_FECHA_FIN_MES) >=TO_DATE('2010-01-01', 'YYYY-MM-DD') )


16/06/07 19:22:52 WARN db.TextSplitter: Generating splits for a textual index column.
16/06/07 19:22:52 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
16/06/07 19:22:52 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
16/06/07 19:22:52 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to MIN_INCREMENT
16/06/07 19:22:52 INFO mapreduce.JobSubmitter: number of splits:1
16/06/07 19:22:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1464163049638_0638
16/06/07 19:22:53 INFO impl.YarnClientImpl: Submitted application application_1464163049638_0638
16/06/07 19:22:53 INFO mapreduce.Job: The url to track the job: http://lnxbig05.cajarural.gcr:8088/proxy/application_1464163049638_0638/
16/06/07 19:22:53 INFO mapreduce.Job: Running job: job_1464163049638_0638
16/06/07 19:23:04 INFO mapreduce.Job: Job job_1464163049638_0638 running in uber mode : false
16/06/07 19:23:04 INFO mapreduce.Job:  map 0% reduce 0%

[#10]

Hi:

can i split by a integer colum????

thanks