This is the script. sqoop_sql.sh <code>query=$(cat ${SQL_SCRIPT}) where_clause=" where dateadded >= '2016-05-01' and dateadded < '2016-06-01' and \$CONDITIONS" sqoop import -D mapreduce.job.queuename=s_sourcedata \--connect 'jdbc:sqlserver://connection' \--compression-codec org.apache.hadoop.io.compress.SnappyCodec \--username name \--password pas \--query "${query}${where_clause}" \--as-parquetfile \--split-by dateadded \--delete-target-dir \--target-dir prioritypass_history \-m 1 It doesn't work this way, but if I change first string to <code>query="select * FROM smth.[dbo].[tablename]" it works. My action looks like this <code><action name="history" cred="hv_cred"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${JOB_TRACKER}</job-tracker> <name-node>${NAME_NODE}</name-node> <exec>sqoop_sql.sh</exec> <env-var>SQL_SCRIPT=${SQL_SCRIPT_HISTORY}</env-var> ... <file>${WORKFLOW_APPLICATION_PATH}/bash/sqoop_sql.sh#sqoop_sql.sh</file> <file>${WORKFLOW_APPLICATION_PATH}/oracle/${SQL_SCRIPT_HISTORY}#${SQL_SCRIPT_HISTORY}</file> </shell><ok to="end"/> <error to="kill"/></action> The thing is I used this same code to import data from oracle, changing only connection details. My only guess is that oozie doesn't like the fact the script is in folder oracle , but I'm not sure and don't know what to change it to if that's the case. PS I don't use sqoop action because there are some libs missing on the claster and it doesn't work. ... View more

I have a distcp action which copies files of certain type from one directory to another. <action name="copy_mta"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <prepare> <delete path="${NAME_NODE}{PATH_M}/*"/> </prepare> <arg>${NAME_NODE}${PATH_COMVERSE}${DATES}_*/*mta.gz</arg> <arg>${NAME_NODE}{PATH_M}</arg> </distcp> <ok to="end"/> <error to="kill"/> </action> In this script ${DATES} is 20170901 . This works. Now I want to copy data for a couple of days. I tried 20170901|20170902 and 20170901||20170902 and 20170901 or 20170902 . It doesn't work. The error is Returned value from distcp is non-zero (-1) And in the logs I only see 2017-10-06 12:17:19,887 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1503284561269_666022_m_000000_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 The thing is there are folders for other days 20170903 and so on, but I only need those 2, so * wouldn't do. How do I do it? ... View more

I have a sqoop script that works sqoop export -Dmapreduce.job.queuename=name --password pwd --username login --connect jdbc:oracle:thin:A/B@//host:port/B --hcatalog-database db --hcatalog-table t --hive-partition-key time_key --hive-partition-value 2017-01-01 --table A.HIVE_TO_ORACLE -m 1 when I try to run it through oozie I get the error from above <action name="export_table" cred="hv_cred"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${JOB_TRACKER}</job-tracker> <name-node>${NAME_NODE}</name-node> <configuration> <property> <name>mapred.task.timeout</name> <value>1200000</value> </property> <property> <name>oozie.sqoop.log.level</name> <value>DEBUG</value> </property> </configuration> <command> export -Dmapreduce.job.queuename=name --password pwd --username login --connect jdbc:oracle:thin:A/B@//host:port/B --hcatalog-database db --hcatalog-table t --hive-partition-key time_key --hive-partition-value 2017-01-01 --table A.HIVE_TO_ORACLE -m 1 </command> </sqoop> <ok to="end"/> <error to="kill"/> </action> I check the logs from job tracker. STDERR is almost empty and has no actual errors. STDOUT has some info Log Type: syslog Log Upload Time: Mon Sep 18 17:40:58 +0300 2017 Log Length: 39181 2017-09-18 17:40:06,309 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1503284561269_416308_000001 2017-09-18 17:40:06,993 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-09-18 17:40:07,085 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2017-09-18 17:40:07,085 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 416308 cluster_timestamp: 1503284561269 } attemptId: 1 } keyId: -165026979) 2017-09-18 17:40:07,440 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.31.160.247:8188, Ident: (owner=user, renewer=yarn, realUser=oozie, issueDate=1505745475695, maxDate=1506350275695, sequenceNumber=2038136, masterKeyId=343) 2017-09-18 17:40:07,442 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: MR_DELEGATION_TOKEN, Service: 10.31.160.247:10020, Ident: (owner=user, renewer=yarn, realUser=oozie/address@address, issueDate=1505745475492, maxDate=1506350275492, sequenceNumber=1371685, masterKeyId=336) 2017-09-18 17:40:07,442 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nn, Ident: (HDFS_DELEGATION_TOKEN token 4628530 for user) 2017-09-18 17:40:07,443 WARN [main] org.apache.hadoop.security.token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN 2017-09-18 17:40:07,443 WARN [main] org.apache.hadoop.security.token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 08 65 76 6b 75 7a 6d 69 6e 05 6f 6f 7a 69 65 2d 6f 6f 7a 69 65 2f 68 64 2d 68 61 73 30 31 35 2e 76 69 6d 70 65 6c 63 6f 6d 2e 72 75 40 42 45 45 2e 56 49 4d 50 45 4c 43 4f 4d 2e 52 55 8a 01 5e 95 6c 94 22 8a 01 5e b9 79 18 22 8d 01 2e 24 8e 03 5b 2017-09-18 17:40:07,443 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.31.160.247:8032,10.31.160.246:8032, Ident: (owner=user, renewer=rm, realUser=oozie/address@address, issueDate=1505745474806, maxDate=1506350274806, sequenceNumber=11544145, masterKeyId=773) 2017-09-18 17:40:07,466 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2017-09-18 17:40:07,468 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter 2017-09-18 17:40:08,246 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2017-09-18 17:40:08,331 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1443) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:291) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486) 2017-09-18 17:40:08,509 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2017-09-18 17:40:08,510 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2017-09-18 17:40:08,511 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2017-09-18 17:40:08,512 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2017-09-18 17:40:08,512 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2017-09-18 17:40:08,513 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2017-09-18 17:40:08,514 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2017-09-18 17:40:08,514 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2017-09-18 17:40:08,570 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nn:port] 2017-09-18 17:40:08,605 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nn:port] 2017-09-18 17:40:08,641 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nn:port] 2017-09-18 17:40:08,656 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757) at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618) at org.apache.hadoop.ipc.Client.call(Client.java:1449) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423) at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1443) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.mkdir(JobHistoryEventHandler.java:267) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:166) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:459) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486) 2017-09-18 17:40:08,673 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2017-09-18 17:40:08,711 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2017-09-18 17:40:08,801 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2017-09-18 17:40:08,863 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-09-18 17:40:08,863 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2017-09-18 17:40:08,873 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1503284561269_416308 to jobTokenSecretManager 2017-09-18 17:40:09,021 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1503284561269_416308 because: not enabled; 2017-09-18 17:40:09,043 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1503284561269_416308 = 0. Number of splits = 1 2017-09-18 17:40:09,044 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1503284561269_416308 = 0 2017-09-18 17:40:09,044 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1503284561269_416308Job Transitioned from NEW to INITED 2017-09-18 17:40:09,045 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1503284561269_416308. 2017-09-18 17:40:09,077 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-09-18 17:40:09,086 INFO [Socket Reader #1 for port 42172] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 42172 2017-09-18 17:40:09,109 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2017-09-18 17:40:09,163 INFO [IPC Server listener on 42172] org.apache.hadoop.ipc.Server: IPC Server listener on 42172: starting 2017-09-18 17:40:09,163 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-09-18 17:40:09,164 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at hd199.address/10.31.160.110:42172 2017-09-18 17:40:09,274 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-09-18 17:40:09,287 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-09-18 17:40:09,296 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2017-09-18 17:40:09,306 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-09-18 17:40:09,378 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2017-09-18 17:40:09,378 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2017-09-18 17:40:09,383 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2017-09-18 17:40:09,383 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2017-09-18 17:40:09,834 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-09-18 17:40:09,835 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 36767 2017-09-18 17:40:09,836 INFO [main] org.mortbay.log: jetty-6.1.26.hwx 2017-09-18 17:40:09,878 INFO [main] org.mortbay.log: Extract jar:file:/dfs/3/hadoop/yarn/local/filecache/19228/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.3.2.5.0.0-1245.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_36767_mapreduce____h3o2ck/webapp 2017-09-18 17:40:11,231 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:36767 2017-09-18 17:40:11,232 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 36767 2017-09-18 17:40:11,236 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-09-18 17:40:11,237 INFO [Socket Reader #1 for port 33433] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33433 2017-09-18 17:40:11,296 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2017-09-18 17:40:11,296 INFO [IPC Server listener on 33433] org.apache.hadoop.ipc.Server: IPC Server listener on 33433: starting 2017-09-18 17:40:11,318 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2017-09-18 17:40:11,318 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2017-09-18 17:40:11,318 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2017-09-18 17:40:11,438 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:76800, vCores:24> 2017-09-18 17:40:11,438 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: adhoc 2017-09-18 17:40:11,443 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2017-09-18 17:40:11,443 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2017-09-18 17:40:11,450 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 2017-09-18 17:40:11,458 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1503284561269_416308Job Transitioned from INITED to SETUP 2017-09-18 17:40:11,460 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2017-09-18 17:40:11,465 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2017-09-18 17:40:11,465 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2017-09-18 17:40:32,662 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1503284561269_416308Job Transitioned from SETUP to RUNNING 2017-09-18 17:40:32,685 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1503284561269_416308_m_000000 Task Transitioned from NEW to SCHEDULED 2017-09-18 17:40:32,687 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2017-09-18 17:40:32,688 INFO [Thread-55] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1> 2017-09-18 17:40:32,745 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1503284561269_416308, File: hdfs://nn:port/user/user/.staging/job_1503284561269_416308/job_1503284561269_416308_1.jhist 2017-09-18 17:40:33,520 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2017-09-18 17:40:33,524 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1503284561269_416308: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:73728, vCores:2225> knownNMs=254 2017-09-18 17:40:34,535 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2017-09-18 17:40:34,695 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved hd225.address to /yr02 2017-09-18 17:40:34,697 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e80_1503284561269_416308_01_000002 to attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:34,698 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2017-09-18 17:40:34,742 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved hd225.address to /yr02 2017-09-18 17:40:34,742 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Job jar is not present. Not adding any jar to the list of resources. 2017-09-18 17:40:34,759 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/user/.staging/job_1503284561269_416308/job.xml 2017-09-18 17:40:34,859 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/aws-java-sdk-core-1.10.6.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/aws-java-sdk-core-1.10.6.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,860 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/aws-java-sdk-kms-1.10.6.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/aws-java-sdk-kms-1.10.6.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,861 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/aws-java-sdk-s3-1.10.6.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/aws-java-sdk-s3-1.10.6.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,862 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/azure-storage-2.2.0.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/azure-storage-2.2.0.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,864 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/commons-lang3-3.3.2.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/commons-lang3-3.3.2.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,865 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/guava-11.0.2.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/guava-11.0.2.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,866 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/hadoop-aws-2.7.3.2.5.0.0-1245.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/hadoop-aws-2.7.3.2.5.0.0-1245.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,867 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/hadoop-azure-2.7.3.2.5.0.0-1245.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/hadoop-azure-2.7.3.2.5.0.0-1245.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,868 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/jackson-annotations-2.4.0.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/jackson-annotations-2.4.0.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,869 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/jackson-core-2.4.4.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/jackson-core-2.4.4.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,870 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/jackson-databind-2.4.4.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/jackson-databind-2.4.4.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,872 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/sqoop/joda-time-2.1.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://nn/user/oozie/share/lib/lib_20161025162659/oozie/joda-time-2.1.jar This will be an error in Hadoop 2.0 2017-09-18 17:40:34,876 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #5 tokens and #1 secret keys for NM use for launching container 2017-09-18 17:40:34,876 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 6 2017-09-18 17:40:34,876 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2017-09-18 17:40:35,149 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2017-09-18 17:40:35,153 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e80_1503284561269_416308_01_000002 taskAttempt attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:35,155 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:35,156 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : hd225.address:45454 2017-09-18 17:40:35,213 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1503284561269_416308_m_000000_0 : 13562 2017-09-18 17:40:35,214 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1503284561269_416308_m_000000_0] using containerId: [container_e80_1503284561269_416308_01_000002 on NM: [hd225.address:45454] 2017-09-18 17:40:35,217 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2017-09-18 17:40:35,217 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1503284561269_416308_m_000000 Task Transitioned from SCHEDULED to RUNNING 2017-09-18 17:40:35,714 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1503284561269_416308: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:104448, vCores:2232> knownNMs=254 2017-09-18 17:40:46,141 INFO [Socket Reader #1 for port 33433] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1503284561269_416308 (auth:SIMPLE) 2017-09-18 17:40:46,160 INFO [Socket Reader #1 for port 33433] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1503284561269_416308 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol 2017-09-18 17:40:46,170 INFO [IPC Server handler 3 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1503284561269_416308_m_87960930222082 asked for a task 2017-09-18 17:40:46,170 INFO [IPC Server handler 3 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1503284561269_416308_m_87960930222082 given task: attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,435 INFO [IPC Server handler 3 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1503284561269_416308_m_000000_0 is : 0.0 2017-09-18 17:40:49,587 INFO [IPC Server handler 9 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,588 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2017-09-18 17:40:49,588 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1503284561269_416308_m_000000_0 given a go for committing the task output. 2017-09-18 17:40:49,590 INFO [IPC Server handler 5 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,590 INFO [IPC Server handler 5 on 33433] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1503284561269_416308_m_000000_0:true 2017-09-18 17:40:49,721 INFO [IPC Server handler 8 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1503284561269_416308_m_000000_0 is : 1.0 2017-09-18 17:40:49,729 INFO [IPC Server handler 7 on 33433] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,730 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2017-09-18 17:40:49,730 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e80_1503284561269_416308_01_000002 taskAttempt attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,730 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,731 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : hd225.address:45454 2017-09-18 17:40:49,747 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1503284561269_416308_m_000000_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2017-09-18 17:40:49,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1503284561269_416308_m_000000_0 2017-09-18 17:40:49,754 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1503284561269_416308_m_000000 Task Transitioned from RUNNING to SUCCEEDED 2017-09-18 17:40:49,755 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2017-09-18 17:40:49,756 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1503284561269_416308Job Transitioned from RUNNING to COMMITTING 2017-09-18 17:40:49,756 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT 2017-09-18 17:40:50,012 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 2017-09-18 17:40:50,013 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1503284561269_416308Job Transitioned from COMMITTING to SUCCEEDED 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2017-09-18 17:40:50,013 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2017-09-18 17:40:50,014 INFO [Thread-72] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0 2017-09-18 17:40:50,180 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nn:port/user/user/.staging/job_1503284561269_416308/job_1503284561269_416308_1.jhist to hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308-1505745475697-user-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DEXPORT_HIVE%3AA%3Dexport_tabl-1505745650010-1-0-SUCCEEDED-adhoc-1505745611452.jhist_tmp 2017-09-18 17:40:50,312 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308-1505745475697-user-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DEXPORT_HIVE%3AA%3Dexport_tabl-1505745650010-1-0-SUCCEEDED-adhoc-1505745611452.jhist_tmp 2017-09-18 17:40:50,344 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nn:port/user/user/.staging/job_1503284561269_416308/job_1503284561269_416308_1_conf.xml to hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308_conf.xml_tmp 2017-09-18 17:40:50,465 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308_conf.xml_tmp 2017-09-18 17:40:50,525 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308.summary_tmp to hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308.summary 2017-09-18 17:40:50,555 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308_conf.xml_tmp to hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308_conf.xml 2017-09-18 17:40:50,585 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308-1505745475697-user-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DEXPORT_HIVE%3AA%3Dexport_tabl-1505745650010-1-0-SUCCEEDED-adhoc-1505745611452.jhist_tmp to hdfs://nn:port/mr-history/tmp/user/job_1503284561269_416308-1505745475697-user-oozie%3Alauncher%3AT%3Dsqoop%3AW%3DEXPORT_HIVE%3AA%3Dexport_tabl-1505745650010-1-0-SUCCEEDED-adhoc-1505745611452.jhist 2017-09-18 17:40:50,586 INFO [Thread-72] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2017-09-18 17:40:50,587 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to 2017-09-18 17:40:50,593 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is a/jobhistory/job/job_1503284561269_416308 2017-09-18 17:40:50,598 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered. 2017-09-18 17:40:51,599 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2017-09-18 17:40:51,600 INFO [Thread-72] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nn /user/user/.staging/job_1503284561269_416308 2017-09-18 17:40:51,813 INFO [Thread-72] org.apache.hadoop.ipc.Server: Stopping server on 33433 2017-09-18 17:40:51,814 INFO [IPC Server listener on 33433] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 33433 2017-09-18 17:40:51,814 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2017-09-18 17:40:51,814 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted there is the READ is not supported in state standby which I was told can be ignored because everything is redirected to an active node and nothing else. I'm not even sure if sqoop fails at creation or execution. Has someone encountered this before? If not maybe you know how I can get the log for sqoop(like the one I get in CLI), so that can at least find the error? EDIT The problem had nothing to do with the workflow or the sqoop script itself, but rather some inner mechanics I'm not aware of. When I run the script in terminal, a java class is generated. This class seems to describe how the hive table is parsed to oracle table or smth. When I start oozie I need to add this autogenerated file to the root directory(the one with coordinator and workflow). If I don't oozie fails. The damn thing won't even give any errors( Anyway, after adding the java class to root, I can run the sqoop from a shell action. Sqoop action still doesn't work. Another thing I noticed, the shell action will execute if in sqoop I use --export-dir /apps/hive/warehouse/db.db/table/ But if I try --hcatalog-database db \ --hcatalog-table table \ in shell, it will fail anyway. It probably tries to add some libraries to class path and can't or smth. ... View more

I have some hive scripts running on the cluster. They are started by oozie. I want to know if they can stop working if the cluster is updated. Is there a place where I can see what words became reserved words(like date) or what settings we have to write differently(for example SET hive.execution.engine=mr changing to hive.exec.engine) and other such things? I tried looking at the official site and there are some descriptions, but they are not what I need. ... View more

My actions start_fair_usage ends with status okey, but test_copy returns Main class [org.apache.oozie.action.hadoop.DistcpMain], main() threw exception, null In /user/comverse/data/${1}_B I have a lot of different files, some of which I want to copy to ${NAME_NODE}/user/evkuzmin/output . For that I try to pass paths from copy_files.sh which holds an array of paths to the files I need. <action name="start_fair_usage"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${JOB_TRACKER}</job-tracker> <name-node>${NAME_NODE}</name-node> <exec>${copy_file}</exec> <argument>${today_without_dash}</argument> <argument>${mta}</argument> <!-- <file>${path}#${start_fair_usage}</file> --> <file>${path}${copy_file}#${copy_file}</file> <capture-output/> </shell> <ok to="test_copy"/> <error to="KILL"/> </action> <action name="test_copy"> <distcp xmlns="uri:oozie:distcp-action:0.2"> <job-tracker>${JOB_TRACKER}</job-tracker> <name-node>${NAME_NODE}</name-node> <arg>${wf:actionData('start_fair_usage')['paths']}</arg> <!-- <arg>${NAME_NODE}/user/evkuzmin/input/*</arg> --> <arg>${NAME_NODE}/user/evkuzmin/output</arg> </distcp> <ok to="END"/> <error to="KILL"/> </action> start_fair_usage starts copy_file.sh echo ${1} echo ${2} dirs=(/user/comverse/data/${1}_B ) args=()for i in $(hadoop fs -ls "${dirs[@]}"| egrep ${2}.gz | awk -F " "'{print $8}')do args+=("$i") echo "copy file - "${i}done paths=${args} echo ${paths} ... View more

Sometimes I get this. And sometimes I get this. There is also a third case where I get a mix of both. What is happening? My settings. set hive.execution.engine=tez; set tez.queue.name=adhoc; set hive.tez.container.size=4096; set hive.auto.convert.join=true; set hive.exec.parallel=true; set hive.tez.auto.reducer.parallelism=true; ... View more

I run a script with tez and it worked, but then I tried to run it without tez and there was this line in the logs tez session was closed reopenning the settings I had with tez set hive.execution.engine=tez; set tez.queue.name=adhoc; set hive.tez.container.size=4096; set hive.auto.convert.join=true; set hive.exec.parallel=true; set hive.tez.auto.reducer.parallelism=true; without tez SET mapreduce.job.queuename = adhoc; SET mapreduce.job.reduces = 100; SET hive.exec.parallel.thread.number = 8; SET hive.cli.print.header=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.dynamic.partition=true; SET hive.exec.parallel=true; SET hive.cli.print.current.db=true; set hive.auto.convert.join=false; set hive.resultset.use.unique.column.names=false; The code was identical in both cases. I run scripts from HUE. Why did it happen? ... View more