Hello bgooley, I will going to office on Tuesday now, will do the troubleshooting on that day and if still the same issue can provide you full haproxy config file. If u can provide ur full haproxy config file then I will compare and will correct the things. - Vijay M ... View more

@bgooley,   Kindly find below updated hzproxy configuration.   frontend hiveserver2_front bind *:10001 option tcplog mode tcp default_backend hiveserver2 # This is the setup for HS2. beeline client connect to load_balancer_host:10001. # HAProxy will balance connections among the list of servers listed below. backend hiveserver2 mode tcp balance source #option ssl-hello-chk server hiveserver2_1 abc:10000 server hiveserver2_2 xyz:10000   Updated hiveserver2 configuration with hive load balancer.   HiveServer2 Load Balancer  -  abc:10001     Tried to connect through beeline but it still gives the same TLS error.   beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM scan complete in 2ms Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)     Tried to test TLS connectivity with openssl command which you suggested. below snap  for the same. Kindly check and suggest.   [root@abc ~]# openssl s_client -connect abc:10001 -msg -debug CONNECTED(00000003) write to 0x14e0e00 [0x14f73b0] (289 bytes => 289 (0x121)) 0000 - 16 03 01 01 1c 01 00 01-18 03 03 f2 83 c8 e8 a0 ................ 0010 - 24 eb da e4 31 75 05 76-62 91 34 b1 4b cc ce 21 $...1u.vb.4.K..! 0020 - 15 99 d7 cc 76 93 78 7d-5e bb 8f 00 00 ac c0 30 ....v.x}^......0 0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1 .,.(.$.......... 0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37 ...k.j.i.h.9.8.7 0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a .6.........2...* 0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../ 0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0 .+.'.#.......... 0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31 ...g.@.?.>.3.2.1 0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43 .0.........E.D.C 00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c .B.1.-.).%...... 00b0 - 00 3c 00 2f 00 96 00 41-c0 12 c0 08 00 16 00 13 .<./...A........ 00c0 - 00 10 00 0d c0 0d c0 03-00 0a 00 07 c0 11 c0 07 ................ 00d0 - c0 0c c0 02 00 05 00 04-00 ff 01 00 00 43 00 0b .............C.. 00e0 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 17 00 19 ................ 00f0 - 00 18 00 16 00 23 00 00-00 0d 00 20 00 1e 06 01 .....#..... .... 0100 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03 ................ 0110 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01 ................ 0120 - 01 . >>> TLS 1.2 [length 0005] 16 03 01 01 1c >>> TLS 1.2 Handshake [length 011c], ClientHello 01 00 01 18 03 03 f2 83 c8 e8 a0 24 eb da e4 31 75 05 76 62 91 34 b1 4b cc ce 21 15 99 d7 cc 76 93 78 7d 5e bb 8f 00 00 ac c0 30 c0 2c c0 28 c0 24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00 6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00 87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0 05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0 23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00 40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00 99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0 2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00 96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0 0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01 02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06 03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03 03 02 01 02 02 02 03 00 0f 00 01 01 read from 0x14e0e00 [0x14fc910] (7 bytes => 0 (0x0)) 140683941742480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1542369743 Timeout : 300 (sec) Verify return code: 0 (ok)     Tried to shutdown 1 of hiveserver2 instance and tested with beeline but still same issue.   Hiveserver2 TLS certificates are in JKS format.   Kindly suggest.   - Vijay M   ... View more

Hello bgooley, Sure, let me try what u suggested. I would like to use Ssl pass through in haproxy config. Kindly provide exact configuration needs to define in haproxy config. - Vijay M ... View more

@bgooley   Also if i configure haproxy using below configuration its does not starts.   frontend hiveserver2_front bind *:10001 ssl crt /app/bds/security/x509/cmserver.pem option tcplog mode tcp default_backend hiveserver2 # This is the setup for HS2. beeline client connect to load_balancer_host:10001. # HAProxy will balance connections among the list of servers listed below. backend hiveserver2 mode tcp balance source option ssl-hello-chk server hiveserver2_1 abc:10000 server hiveserver2_2 xyz:10000   below ask for passhphrase, not sure why   [root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg Enter PEM pass phrase:   When i entered the password/passphrase it gives invalid configuration file   [root@abc conf]# /usr/sbin/haproxy -c -f /etc/haproxy/haproxy.cfg Enter PEM pass phrase: [WARNING] 318/081440 (2708462) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear. Configuration file is valid     Kindly suggest what wrong i m doing ?   - Vijay M ... View more

@bgooley   Same issue with port 10001 or port 443.   Below snap confirms haproxy started and running on port 10001.   [root@abc ~]# ps -ef | grep -i haproxy root 2620129 1 0 06:19 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid haproxy 2620130 2620129 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds haproxy 2620131 2620130 0 06:19 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds [root@abc ~]# netstat -tunlp | grep 10001 tcp 0 0 0.0.0.0:10001 0.0.0.0:* LISTEN 2620131/haproxy [root@abc ~]#     Below are 2 scenarios which i tried and explained. Kindly check and suggest to fix.     Case1:     When i removed haproxy load balancer porperty from hive configuration and trying to connect individual haproxy services through beeline. i am able to connect. Below snap for the same.   beeline> !connect jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:1>       beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM scan complete in 2ms Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://a301-8883-2675.gdzd.ubs.net:1>         Case2:   With haproxy load balancer property in hive configuration with port 10001 configured in haproxy configuration its not working and throws error.   beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0) beeline>       No logs of both hiveserver2 instance and haproxy gets updated for above error.     with above scnario when i am trying to connect individual hiveserver2 instances i am able to connect to hiveserver2 instance on haproxy running but unable to connect to other hiveserver2 instance and gets TLS error.    Below snap for both.   --  Successfully able to connect to hiveserrver2 where haproxy also running.   beeline> !connect jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:1>     --- Unable to connect to other hiveserver2 instance   beeline> !connect jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM scan complete in 2ms Connecting to jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://xyz:10000/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM: Peer indicated failure: GSS initiate failed (state=08S01,code=0) beeline>     hiveserver2 log shows below error.   2018-11-15 06:46:08,217 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-40]: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)   Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ... 14 more Caused by: KrbException: Checksum failed   2018-11-15 06:46:08,220 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-40]: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: GSS initiate failed   Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed     - Vijay M ... View more

@bgooley   Below find details of certificates which i have on cluster.   below certificate is from root CA -rwxr-xr-x. 1 cloudera-scm cloudera-scm 8152 Oct  5 10:36 cacerts.pem   Below certificate are keystore and trustore used by Hive service TLS enabled.   -rwxr-xr-x. 1 cloudera-scm cloudera-scm 9624 Oct 5 10:38 cloudera_keystore.jks -rwxr-xr-x. 1 cloudera-scm cloudera-scm 4048 Oct 5 10:39 cloudera_truststore.jks     Below find configuration of haproxy   #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend hiveserver2_front bind *:443 option tcplog mode tcp default_backend hiveserver2 # This is the setup for HS2. beeline client connect to load_balancer_host:10001. # HAProxy will balance connections among the list of servers listed below. backend hiveserver2 mode tcp balance source option ssl-hello-chk server hiveserver2_1 abc:10000 check server hiveserver2_2 xyz:10000 check     --- hive server2 configuration from cloudera manager configured with below property   HiveServer2 Load Balancer      abc:443     1. Kindly confirm does in above property do i have to add https  or http?  Is it require? 2. Kindly review the configuration and let me know if anything more details require? 3. one of hiveserver2 instance and haproxy services configured on same server i.e. abc, Is it an issue?     Kindly suggest?   - Vijay M   ... View more

@bgooley, Haproxy log doesn't shows anything and even Hiveserver2 logs. Will send you configuration post Monday as I am on leave. - Vijay M ... View more

@bgooley,   I was engaged in some other projects so unable to reply on it. Started working on it today.   When iam connecting both hiveserver2 instance without haproxy, removed the load balance entry from hive confiugration.  I am able to connect to both hiveserver2 instance from beeline.   I have hiveserver2 TLS enable using CA signed certificates. and Hiveserver2 certificates are in Java format i.e. .jks(keystore.jks and truststore.jks).   In my haproxy configuration at bind line i am giving keystore.jks entry and for backend entry i am giving truststore.jks entry for both server.   Kindly confirm is it correct or suggest?   - VIjay M ... View more

I able to fix the authentication error, It was because user was not present on all nodes of cluster.   Ran the job again and its got failed. Below provided job application log.  I able to figure out few warnings and GC allocation errors.    Can someone also review and confirm what the problem is ?   $ yarn logs -applicationId application_1540368383755_0019 Container: container_e50_1540368383755_0019_01_000002 on psbx05xv00app.she.pwj.com_8041 ========================================================================================= LogType:container-localizer-syslog Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:0 Log Contents: LogType:stderr Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:886 Log Contents: Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.YarnChild). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/sandbox/bds/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/sandbox/bds/data/yarn/nm/01/usercache/p_diku/appcache/application_1540368383755_0019/filecache/10/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] LogType:stdout Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:1493 Log Contents: 1.294: [GC (Allocation Failure) [PSYoungGen: 131584K->13538K(153088K)] 131584K->13626K(502784K), 0.0173503 secs] [Times: user=0.15 sys=0.04, real=0.01 secs] 1.555: [GC (Metadata GC Threshold) [PSYoungGen: 63824K->7859K(153088K)] 63912K->7955K(502784K), 0.0127835 secs] [Times: user=0.20 sys=0.03, real=0.01 secs] 1.568: [Full GC (Metadata GC Threshold) [PSYoungGen: 7859K->0K(153088K)] [ParOldGen: 96K->7129K(349696K)] 7955K->7129K(502784K), [Metaspace: 20971K->20971K(1069056K)], 0.0369262 secs] [Times: user=0.44 sys=0.01, real=0.04 secs] POST 0: Entered run POST 11: After setEmbeddedH2OConfig 32.093: [GC (Allocation Failure) [PSYoungGen: 131584K->8382K(153088K)] 138713K->15519K(502784K), 0.0095201 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] Heap PSYoungGen total 153088K, used 30980K [0x00000000f5580000, 0x0000000100000000, 0x0000000100000000) eden space 131584K, 17% used [0x00000000f5580000,0x00000000f6b91878,0x00000000fd600000) from space 21504K, 38% used [0x00000000fd600000,0x00000000fde2f9a0,0x00000000feb00000) to space 21504K, 0% used [0x00000000feb00000,0x00000000feb00000,0x0000000100000000) ParOldGen total 349696K, used 7137K [0x00000000e0000000, 0x00000000f5580000, 0x00000000f5580000) object space 349696K, 2% used [0x00000000e0000000,0x00000000e06f85a8,0x00000000f5580000) Metaspace used 28606K, capacity 28822K, committed 29104K, reserved 1075200K class space used 3348K, capacity 3437K, committed 3504K, reserved 1048576K   Container: container_e50_1540368383755_0019_01_000001 on psbx13xv00app.she.pwj.com_8041 ========================================================================================= LogType:container-localizer-syslog Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:0 Log Contents: LogType:stderr Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:2933 Log Contents: Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/sandbox/bds/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/sandbox/bds/data/yarn/nm/01/usercache/p_diku/appcache/application_1540368383755_0019/filecache/11/job.jar/job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Oct 24, 2018 8:09:29 AM com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider get WARNING: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice documentation at http://code.google.com/p/google-guice/wiki/Servlets for more information. Oct 24, 2018 8:09:29 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class Oct 24, 2018 8:09:29 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 24, 2018 8:09:29 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class Oct 24, 2018 8:09:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 24, 2018 8:09:29 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 24, 2018 8:09:30 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 24, 2018 8:09:30 AM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. LogType:stdout Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:0 Log Contents: LogType:syslog Log Upload Time:Wed Oct 24 08:11:19 -0400 2018 LogLength:25758 Log Contents: 2018-10-24 08:09:16,307 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1540368383755_0019_000001 2018-10-24 08:09:16,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2018-10-24 08:09:16,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@6caf0677) 2018-10-24 08:09:16,690 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:hdfsha, Ident: (token for p_diku: HDFS_DELEGATION_TOKEN owner=p_diku@BDS.UBS.COM, renewer=yarn, realUser=, issueDate=1540382949785, maxDate=1540987749785, sequenceNumber=104, masterKeyId=137) 2018-10-24 08:09:16,704 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2018-10-24 08:09:16,705 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2018-10-24 08:09:16,736 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2018-10-24 08:09:16,736 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2018-10-24 08:09:17,156 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-10-24 08:09:17,174 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2018-10-24 08:09:17,312 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2018-10-24 08:09:17,313 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2018-10-24 08:09:17,314 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2018-10-24 08:09:17,315 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2018-10-24 08:09:17,315 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2018-10-24 08:09:17,316 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2018-10-24 08:09:17,316 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2018-10-24 08:09:17,317 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2018-10-24 08:09:17,349 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hdfsha:8020] 2018-10-24 08:09:17,367 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hdfsha:8020] 2018-10-24 08:09:17,383 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hdfsha:8020] 2018-10-24 08:09:17,396 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2018-10-24 08:09:17,423 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2018-10-24 08:09:17,558 WARN [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: org.apache.commons.configuration.ConfigurationException: Unable to load the configuration from the URL file:/run/cloudera-scm-agent/process/1785-yarn-NODEMANAGER/hadoop-metrics2.properties 2018-10-24 08:09:17,598 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1540368383755_0019 to jobTokenSecretManager 2018-10-24 08:09:17,700 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1540368383755_0019 because: not enabled; 2018-10-24 08:09:17,712 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1540368383755_0019 = 0. Number of splits = 1 2018-10-24 08:09:17,712 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1540368383755_0019 = 0 2018-10-24 08:09:17,712 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from NEW to INITED 2018-10-24 08:09:17,713 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1540368383755_0019. 2018-10-24 08:09:17,732 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 2018-10-24 08:09:17,737 INFO [Socket Reader #1 for port 38385] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 38385 2018-10-24 08:09:17,750 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2018-10-24 08:09:17,774 INFO [IPC Server listener on 38385] org.apache.hadoop.ipc.Server: IPC Server listener on 38385: starting 2018-10-24 08:09:17,774 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-10-24 08:09:17,775 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at psbx13xv00app.she.pwj.com/10.79.0.71:38385 2018-10-24 08:09:17,820 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-10-24 08:09:17,825 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2018-10-24 08:09:17,833 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2018-10-24 08:09:17,851 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2018-10-24 08:09:17,887 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2018-10-24 08:09:17,887 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2018-10-24 08:09:17,891 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2018-10-24 08:09:17,891 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2018-10-24 08:09:17,900 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 36102 2018-10-24 08:09:17,900 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 2018-10-24 08:09:17,930 INFO [main] org.mortbay.log: Extract jar:file:/sandbox/bds/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/hadoop-yarn-common-2.6.0-cdh5.13.3.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_36102_mapreduce____v067m1/webapp 2018-10-24 08:09:18,168 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:36102 2018-10-24 08:09:18,169 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 36102 2018-10-24 08:09:18,378 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2018-10-24 08:09:18,383 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 2018-10-24 08:09:18,384 INFO [Socket Reader #1 for port 34682] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34682 2018-10-24 08:09:18,411 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-10-24 08:09:18,411 INFO [IPC Server listener on 34682] org.apache.hadoop.ipc.Server: IPC Server listener on 34682: starting 2018-10-24 08:09:18,428 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2018-10-24 08:09:18,428 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2018-10-24 08:09:18,428 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2018-10-24 08:09:18,508 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:248832, vCores:76> 2018-10-24 08:09:18,509 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.wma_default_grp 2018-10-24 08:09:18,514 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2018-10-24 08:09:18,514 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2018-10-24 08:09:18,525 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from INITED to SETUP 2018-10-24 08:09:18,527 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2018-10-24 08:09:18,539 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from SETUP to RUNNING 2018-10-24 08:09:18,564 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1540368383755_0019_m_000000 Task Transitioned from NEW to SCHEDULED 2018-10-24 08:09:18,565 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-10-24 08:09:18,566 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:563, vCores:1> 2018-10-24 08:09:18,675 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1540368383755_0019, File: hdfs://hdfsha:8020/user/p_diku/.staging/job_1540368383755_0019/job_1540368383755_0019_1.jhist 2018-10-24 08:09:18,752 INFO [Socket Reader #1 for port 38385] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for p_diku@BDS.UBS.COM (auth:SIMPLE) 2018-10-24 08:09:18,766 INFO [Socket Reader #1 for port 38385] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for p_diku@BDS.UBS.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB 2018-10-24 08:09:18,884 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://hdfsha:8020] 2018-10-24 08:09:19,512 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-10-24 08:09:19,554 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1540368383755_0019: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:1479680, vCores:453> knownNMs=6 2018-10-24 08:09:20,566 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2018-10-24 08:09:20,605 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1540368383755_0019_01_000002 to attempt_1540368383755_0019_m_000000_0 2018-10-24 08:09:20,607 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2018-10-24 08:09:20,674 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://hdfsha/user/p_diku/.staging/job_1540368383755_0019/job.jar 2018-10-24 08:09:20,676 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/p_diku/.staging/job_1540368383755_0019/job.xml 2018-10-24 08:09:20,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #1 tokens and #1 secret keys for NM use for launching container 2018-10-24 08:09:20,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2 2018-10-24 08:09:20,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2018-10-24 08:09:20,703 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-10-24 08:09:20,706 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1540368383755_0019_01_000002 taskAttempt attempt_1540368383755_0019_m_000000_0 2018-10-24 08:09:20,709 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1540368383755_0019_m_000000_0 2018-10-24 08:09:20,779 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1540368383755_0019_m_000000_0 : 13562 2018-10-24 08:09:20,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1540368383755_0019_m_000000_0] using containerId: [container_e50_1540368383755_0019_01_000002 on NM: [psbx05xv00app.she.pwj.com:8041] 2018-10-24 08:09:20,785 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-10-24 08:09:20,785 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1540368383755_0019_m_000000 Task Transitioned from SCHEDULED to RUNNING 2018-10-24 08:09:21,609 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1540368383755_0019: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:1476608, vCores:452> knownNMs=6 2018-10-24 08:09:25,403 INFO [Socket Reader #1 for port 34682] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1540368383755_0019 (auth:SIMPLE) 2018-10-24 08:09:25,418 INFO [Socket Reader #1 for port 34682] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1540368383755_0019 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol 2018-10-24 08:09:25,427 INFO [IPC Server handler 0 on 34682] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1540368383755_0019_m_54975581388802 asked for a task 2018-10-24 08:09:25,428 INFO [IPC Server handler 0 on 34682] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1540368383755_0019_m_54975581388802 given task: attempt_1540368383755_0019_m_000000_0 2018-10-24 08:09:38,161 INFO [IPC Server handler 7 on 34682] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1540368383755_0019_m_000000_0 is : 0.0 2018-10-24 08:10:38,261 INFO [IPC Server handler 11 on 34682] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1540368383755_0019_m_000000_0 is : 0.0 2018-10-24 08:11:12,164 INFO [IPC Server handler 0 on 38385] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Kill job job_1540368383755_0019 received from p_diku@BDS.UBS.COM (auth:TOKEN) at 10.32.44.96 2018-10-24 08:11:12,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from RUNNING to KILL_WAIT 2018-10-24 08:11:12,166 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1540368383755_0019_m_000000 Task Transitioned from RUNNING to KILL_WAIT 2018-10-24 08:11:12,167 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP 2018-10-24 08:11:12,168 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1540368383755_0019_01_000002 taskAttempt attempt_1540368383755_0019_m_000000_0 2018-10-24 08:11:12,168 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1540368383755_0019_m_000000_0 2018-10-24 08:11:12,189 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 2018-10-24 08:11:12,190 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2018-10-24 08:11:12,206 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1540368383755_0019_m_000000_0 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 2018-10-24 08:11:12,222 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1540368383755_0019_m_000000 Task Transitioned from KILL_WAIT to KILLED 2018-10-24 08:11:12,224 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 2018-10-24 08:11:12,225 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from KILL_WAIT to KILL_ABORT 2018-10-24 08:11:12,226 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 2018-10-24 08:11:12,233 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1540368383755_0019Job Transitioned from KILL_ABORT to KILLED 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2018-10-24 08:11:12,234 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2018-10-24 08:11:12,235 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 2 2018-10-24 08:11:12,242 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED 2018-10-24 08:11:12,246 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_KILLED 2018-10-24 08:11:12,295 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://hdfsha:8020/user/p_diku/.staging/job_1540368383755_0019/job_1540368383755_0019_1.jhist to hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019-1540382951606-p_diku-H2O_31727-1540383072224-0-0-KILLED-root.wma_default_grp-1540382958519.jhist_tmp 2018-10-24 08:11:12,333 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019-1540382951606-p_diku-H2O_31727-1540383072224-0-0-KILLED-root.wma_default_grp-1540382958519.jhist_tmp 2018-10-24 08:11:12,336 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://hdfsha:8020/user/p_diku/.staging/job_1540368383755_0019/job_1540368383755_0019_1_conf.xml to hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019_conf.xml_tmp 2018-10-24 08:11:12,371 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019_conf.xml_tmp 2018-10-24 08:11:12,395 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019.summary_tmp to hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019.summary 2018-10-24 08:11:12,397 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019_conf.xml_tmp to hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019_conf.xml 2018-10-24 08:11:12,399 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019-1540382951606-p_diku-H2O_31727-1540383072224-0-0-KILLED-root.wma_default_grp-1540382958519.jhist_tmp to hdfs://hdfsha:8020/user/history/done_intermediate/p_diku/job_1540368383755_0019-1540382951606-p_diku-H2O_31727-1540383072224-0-0-KILLED-root.wma_default_grp-1540382958519.jhist 2018-10-24 08:11:12,399 INFO [Thread-71] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2018-10-24 08:11:12,400 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to Kill job job_1540368383755_0019 received from p_diku@BDS.UBS.COM (auth:TOKEN) at 10.32.44.96 Job received Kill while in RUNNING state. 2018-10-24 08:11:12,401 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is https://a302-0044-9735.she.pwj.com:19890/jobhistory/job/job_1540368383755_0019 2018-10-24 08:11:12,406 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2018-10-24 08:11:13,407 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2018-10-24 08:11:13,409 INFO [Thread-71] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://hdfsha /user/p_diku/.staging/job_1540368383755_0019 2018-10-24 08:11:13,412 INFO [Thread-71] org.apache.hadoop.ipc.Server: Stopping server on 34682 2018-10-24 08:11:13,413 INFO [IPC Server listener on 34682] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34682 2018-10-24 08:11:13,413 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2018-10-24 08:11:13,413 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2018-10-24 08:11:13,414 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted 2018-10-24 08:11:18,414 INFO [Thread-71] org.apache.hadoop.ipc.Server: Stopping server on 38385 2018-10-24 08:11:18,415 INFO [IPC Server listener on 38385] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 38385 2018-10-24 08:11:18,415 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2018-10-24 08:11:18,418 INFO [Thread-71] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0 ... View more