Container: container_1539782982241_0020_01_000001 on testnode-2.bdlocal_45454_1539843785407 LogAggregationType: AGGREGATED =========================================================================================== LogType:container-localizer-syslog LogLastModifiedTime:Thu Oct 18 06:23:05 +0000 2018 LogLength:506 LogContents: 2018-10-18 06:22:54,809 INFO [main] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer: Disk Validator: yarn.nodemanager.disk-validator is loaded. 2018-10-18 06:22:56,282 WARN [ContainerLocalizer Downloader] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error End of LogType:container-localizer-syslog ******************************************************************************************* Container: container_1539782982241_0020_01_000001 on testnode-2.bdlocal_45454_1539843785407 LogAggregationType: AGGREGATED =========================================================================================== LogType:directory.info LogLastModifiedTime:Thu Oct 18 06:23:05 +0000 2018 LogLength:36902 LogContents: ls -l: total 12 -rw-------. 1 spark hadoop 367 Oct 17 23:22 container_tokens lrwxrwxrwx. 1 spark hadoop 62 Oct 17 23:22 __hive_libs__ -> /hadoop/yarn/local/filecache/11/spark2-hdp-hive-archive.tar.gz -rwx------. 1 spark hadoop 5218 Oct 17 23:22 launch_container.sh lrwxrwxrwx. 1 spark hadoop 66 Oct 17 23:22 __spark_conf__ -> /hadoop/yarn/local/usercache/spark/filecache/10/__spark_conf__.zip lrwxrwxrwx. 1 spark hadoop 62 Oct 17 23:22 __spark_libs__ -> /hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz drwxr-s---. 2 spark hadoop 6 Oct 17 23:22 tmp find -L . -maxdepth 5 -ls: 134274332 0 drwxr-s--- 3 spark hadoop 133 Oct 17 23:22 . 138412577 0 drwxr-s--- 2 spark hadoop 6 Oct 17 23:22 ./tmp 134274333 8 -rwx------ 1 spark hadoop 5218 Oct 17 23:22 ./launch_container.sh 134274334 4 -rw------- 1 spark hadoop 367 Oct 17 23:22 ./container_tokens 1002443541 0 drwxr-xr-x 2 yarn hadoop 64 Oct 17 06:39 ./__hive_libs__ 1002443542 46828 -r-xr-xr-x 1 yarn hadoop 47950336 Jul 12 13:47 ./__hive_libs__/standalone-metastore-1.21.2.3.0.0.0-1634-hive3.jar 117458993 0 drwx------ 3 spark hadoop 145 Oct 17 23:22 ./__spark_conf__ 117458994 4 -r-x------ 1 spark hadoop 620 Oct 17 23:22 ./__spark_conf__/log4j.properties 117458995 8 -r-x------ 1 spark hadoop 4956 Oct 17 23:22 ./__spark_conf__/metrics.properties 121673216 4 drwx------ 2 spark hadoop 4096 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__ 121673217 4 -r-x------ 1 spark hadoop 2591 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarnservice-log4j.properties 121673218 12 -r-x------ 1 spark hadoop 10630 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/log4j.properties 121673219 8 -r-x------ 1 spark hadoop 7332 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hadoop-env.sh 121673228 8 -r-x------ 1 spark hadoop 7934 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/mapred-site.xml 121681422 4 -r-x------ 1 spark hadoop 1493 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hadoop-metrics2.properties 121681430 32 -r-x------ 1 spark hadoop 29150 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn-site.xml 121681441 4 -r-x------ 1 spark hadoop 3999 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hadoop-env.cmd 121681451 12 -r-x------ 1 spark hadoop 8742 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/core-site.xml 121681452 4 -r-x------ 1 spark hadoop 1 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn.exclude 121681453 4 -r-x------ 1 spark hadoop 10 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/workers 121681454 4 -r-x------ 1 spark hadoop 244 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/spark-thrift-fairscheduler.xml 121681455 4 -r-x------ 1 spark hadoop 1 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/dfs.exclude 121681456 4 -r-x------ 1 spark hadoop 1747 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/kms-log4j.properties 121681457 4 -r-x------ 1 spark hadoop 2250 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn-env.cmd 121681458 4 -r-x------ 1 spark hadoop 845 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ssl-client.xml 121681459 4 -r-x------ 1 spark hadoop 3943 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/capacity-scheduler.xml 121681460 4 -r-x------ 1 spark hadoop 2591 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-hdfs-audit.xml 121681461 4 -r-x------ 1 spark hadoop 3518 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/kms-acls.xml 121681462 4 -r-x------ 1 spark hadoop 2358 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/topology_script.py 121681463 4 -r-x------ 1 spark hadoop 543 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn_nm_jaas.conf 121681464 4 -r-x------ 1 spark hadoop 1335 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/configuration.xsl 121681465 4 -r-x------ 1 spark hadoop 64 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-security.xml 121681466 8 -r-x------ 1 spark hadoop 5489 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn-env.sh 121681467 12 -r-x------ 1 spark hadoop 11026 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hdfs-site.xml 121681468 4 -r-x------ 1 spark hadoop 551 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn_ats_jaas.conf 121681469 4 -r-x------ 1 spark hadoop 1020 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/commons-logging.properties 121681470 4 -r-x------ 1 spark hadoop 1591 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/container-executor.cfg 121681471 4 -r-x------ 1 spark hadoop 305 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hdfs_dn_jaas.conf 122087962 4 -r-x------ 1 spark hadoop 305 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hdfs_nn_jaas.conf 122087963 4 -r-x------ 1 spark hadoop 515 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn_jaas.conf 122087964 4 -r-x------ 1 spark hadoop 311 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/resource-types.xml 122087965 8 -r-x------ 1 spark hadoop 4221 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/task-log4j.properties 122087966 4 -r-x------ 1 spark hadoop 1418 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/mapred-env.sh 122087967 4 -r-x------ 1 spark hadoop 1602 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/health_check 122087968 4 -r-x------ 1 spark hadoop 1067 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-policymgr-ssl-yarn.xml 122087969 4 -r-x------ 1 spark hadoop 2125 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hive-site.xml 122087970 4 -r-x------ 1 spark hadoop 292 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/mapred_jaas.conf 122087971 4 -r-x------ 1 spark hadoop 2316 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ssl-client.xml.example 122087972 4 -r-x------ 1 spark hadoop 1351 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/kms-env.sh 122087973 4 -r-x------ 1 spark hadoop 309 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/yarn_registry_dns_jaas.conf 122087974 4 -r-x------ 1 spark hadoop 1352 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hadoop-policy.xml 122087975 4 -r-x------ 1 spark hadoop 3507 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-yarn-audit.xml 122087976 4 -r-x------ 1 spark hadoop 77 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/slaves 122087977 4 -r-x------ 1 spark hadoop 255 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/topology_mappings.data 122087978 4 -r-x------ 1 spark hadoop 1063 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-hdfs-security.xml 122087979 4 -r-x------ 1 spark hadoop 1044 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ssl-server.xml 122087980 4 -r-x------ 1 spark hadoop 951 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/mapred-env.cmd 122087981 4 -r-x------ 1 spark hadoop 305 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/hdfs_jn_jaas.conf 122087982 4 -r-x------ 1 spark hadoop 1061 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-policymgr-ssl.xml 122087983 4 -r-x------ 1 spark hadoop 2697 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ssl-server.xml.example 122087984 4 -r-x------ 1 spark hadoop 1060 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/ranger-yarn-security.xml 122087985 4 -r-x------ 1 spark hadoop 945 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/taskcontroller.cfg 122087986 8 -r-x------ 1 spark hadoop 4113 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/mapred-queues.xml.template 122087987 4 -r-x------ 1 spark hadoop 682 Oct 17 23:22 ./__spark_conf__/__hadoop_conf__/kms-site.xml 117461609 152 -r-x------ 1 spark hadoop 153167 Oct 17 23:22 ./__spark_conf__/__spark_hadoop_conf__.xml 117461610 4 -r-x------ 1 spark hadoop 2549 Oct 17 23:22 ./__spark_conf__/__spark_conf__.properties 998246408 16 drwxr-xr-x 2 yarn hadoop 12288 Oct 17 06:39 ./__spark_libs__ 998246410 1848 -r-xr-xr-x 1 yarn hadoop 1890075 Jul 12 13:47 ./__spark_libs__/datanucleus-core-3.2.10.jar 998246414 132 -r-xr-xr-x 1 yarn hadoop 134308 Jul 12 13:47 ./__spark_libs__/HikariCP-java7-2.4.12.jar 998246424 176 -r-xr-xr-x 1 yarn hadoop 178947 Jul 12 13:47 ./__spark_libs__/hk2-api-2.4.0-b34.jar 998246425 20 -r-xr-xr-x 1 yarn hadoop 16993 Jul 12 13:47 ./__spark_libs__/JavaEWAH-0.3.2.jar 998246426 1768 -r-xr-xr-x 1 yarn hadoop 1809447 Jul 12 13:47 ./__spark_libs__/datanucleus-rdbms-3.2.9.jar 998246427 200 -r-xr-xr-x 1 yarn hadoop 201928 Jul 12 13:47 ./__spark_libs__/RoaringBitmap-0.5.11.jar 998246428 180 -r-xr-xr-x 1 yarn hadoop 181271 Jul 12 13:47 ./__spark_libs__/hk2-locator-2.4.0-b34.jar 998246429 232 -r-xr-xr-x 1 yarn hadoop 236660 Jul 12 13:47 ./__spark_libs__/ST4-4.0.4.jar 998246430 3152 -r-xr-xr-x 1 yarn hadoop 3224708 Jul 12 13:47 ./__spark_libs__/derby-10.12.1.1.jar 998246431 32 -r-xr-xr-x 1 yarn hadoop 30035 Jul 12 13:47 ./__spark_libs__/accessors-smart-1.2.jar 998246432 304 -r-xr-xr-x 1 yarn hadoop 307637 Jul 12 13:47 ./__spark_libs__/dnsjava-2.1.7.jar 998246433 68 -r-xr-xr-x 1 yarn hadoop 69409 Jul 12 13:47 ./__spark_libs__/activation-1.1.1.jar 998246434 1688 -r-xr-xr-x 1 yarn hadoop 1726527 Jul 12 13:47 ./__spark_libs__/ehcache-3.3.1.jar 998246435 128 -r-xr-xr-x 1 yarn hadoop 130802 Jul 12 13:47 ./__spark_libs__/aircompressor-0.8.jar 998246436 120 -r-xr-xr-x 1 yarn hadoop 118973 Jul 12 13:47 ./__spark_libs__/hk2-utils-2.4.0-b34.jar 998246437 436 -r-xr-xr-x 1 yarn hadoop 445288 Jul 12 13:47 ./__spark_libs__/antlr-2.7.7.jar 998246438 20 -r-xr-xr-x 1 yarn hadoop 18482 Jul 12 13:47 ./__spark_libs__/eigenbase-properties-1.1.5.jar 998246439 164 -r-xr-xr-x 1 yarn hadoop 164368 Jul 12 13:47 ./__spark_libs__/antlr-runtime-3.4.jar 998246440 12 -r-xr-xr-x 1 yarn hadoop 10166 Jul 12 13:47 ./__spark_libs__/flatbuffers-1.2.0-3f79e055.jar 998246441 328 -r-xr-xr-x 1 yarn hadoop 334662 Jul 12 13:47 ./__spark_libs__/antlr4-runtime-4.7.jar 998246442 1632 -r-xr-xr-x 1 yarn hadoop 1671083 Jul 12 13:47 ./__spark_libs__/hppc-0.7.2.jar 998246443 8 -r-xr-xr-x 1 yarn hadoop 4467 Jul 12 13:47 ./__spark_libs__/aopalliance-1.0.jar 998246444 1988 -r-xr-xr-x 1 yarn hadoop 2035066 Jul 12 13:47 ./__spark_libs__/commons-math3-3.4.1.jar 998246445 12 -r-xr-xr-x 1 yarn hadoop 12131 Jul 12 13:47 ./__spark_libs__/jpam-1.1.jar 998246446 16 -r-xr-xr-x 1 yarn hadoop 14766 Jul 12 13:47 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar 998246447 132 -r-xr-xr-x 1 yarn hadoop 134595 Jul 12 13:47 ./__spark_libs__/commons-crypto-1.0.0.jar 998246448 440 -r-xr-xr-x 1 yarn hadoop 448794 Jul 12 13:47 ./__spark_libs__/apache-log4j-extras-1.2.17.jar 998246449 2140 -r-xr-xr-x 1 yarn hadoop 2189117 Jul 12 13:47 ./__spark_libs__/guava-14.0.1.jar 998246450 1168 -r-xr-xr-x 1 yarn hadoop 1194003 Jul 12 13:47 ./__spark_libs__/arpack_combined_all-0.1.jar 998246451 24 -r-xr-xr-x 1 yarn hadoop 24239 Jul 12 13:47 ./__spark_libs__/commons-daemon-1.0.13.jar 998246452 52 -r-xr-xr-x 1 yarn hadoop 52178 Jul 12 13:47 ./__spark_libs__/arrow-format-0.8.0.3.0.0.0-1634.jar 998246453 160 -r-xr-xr-x 1 yarn hadoop 160519 Jul 12 13:47 ./__spark_libs__/commons-dbcp-1.4.jar 998246454 16 -r-xr-xr-x 1 yarn hadoop 15071 Jul 12 13:47 ./__spark_libs__/jta-1.1.jar 998246455 80 -r-xr-xr-x 1 yarn hadoop 79365 Jul 12 13:47 ./__spark_libs__/arrow-memory-0.8.0.3.0.0.0-1634.jar 998246456 300 -r-xr-xr-x 1 yarn hadoop 305001 Jul 12 13:47 ./__spark_libs__/commons-httpclient-3.1.jar 998246457 1240 -r-xr-xr-x 1 yarn hadoop 1267435 Jul 12 13:47 ./__spark_libs__/arrow-vector-0.8.0.3.0.0.0-1634.jar 998246458 764 -r-xr-xr-x 1 yarn hadoop 781831 Jul 12 13:47 ./__spark_libs__/httpclient-4.5.4.jar 998246459 428 -r-xr-xr-x 1 yarn hadoop 436303 Jul 12 13:47 ./__spark_libs__/avro-1.7.7.jar 998246460 1256 -r-xr-xr-x 1 yarn hadoop 1282424 Jul 12 13:47 ./__spark_libs__/ivy-2.4.0.jar 998246461 192 -r-xr-xr-x 1 yarn hadoop 192993 Jul 12 13:47 ./__spark_libs__/avro-ipc-1.7.7.jar 998246462 80 -r-xr-xr-x 1 yarn hadoop 79845 Jul 12 13:47 ./__spark_libs__/compress-lzf-1.0.3.jar 998246463 180 -r-xr-xr-x 1 yarn hadoop 180736 Jul 12 13:47 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar 998244878 96 -r-xr-xr-x 1 yarn hadoop 96221 Jul 12 13:47 ./__spark_libs__/commons-pool-1.5.4.jar 998246912 84200 -r-xr-xr-x 1 yarn hadoop 86220098 Jul 12 13:47 ./__spark_libs__/aws-java-sdk-bundle-1.11.271.jar 998246913 164 -r-xr-xr-x 1 yarn hadoop 164422 Jul 12 13:47 ./__spark_libs__/core-1.1.2.jar 998246914 128 -r-xr-xr-x 1 yarn hadoop 128414 Jul 12 13:47 ./__spark_libs__/re2j-1.1.jar 998246915 92 -r-xr-xr-x 1 yarn hadoop 92683 Jul 12 13:47 ./__spark_libs__/azure-data-lake-store-sdk-2.2.7.jar 998246916 2368 -r-xr-xr-x 1 yarn hadoop 2423138 Jul 12 13:47 ./__spark_libs__/curator-client-2.12.0.jar 998246917 12 -r-xr-xr-x 1 yarn hadoop 10288 Jul 12 13:47 ./__spark_libs__/azure-keyvault-core-1.0.0.jar 998246918 188 -r-xr-xr-x 1 yarn hadoop 190432 Jul 12 13:47 ./__spark_libs__/gson-2.2.4.jar 998246919 796 -r-xr-xr-x 1 yarn hadoop 812977 Jul 12 13:47 ./__spark_libs__/azure-storage-7.0.0.jar 998246920 320 -r-xr-xr-x 1 yarn hadoop 324565 Jul 12 13:47 ./__spark_libs__/httpcore-4.4.8.jar 998246921 20 -r-xr-xr-x 1 yarn hadoop 17008 Jul 12 13:47 ./__spark_libs__/base64-2.3.8.jar 998246922 656 -r-xr-xr-x 1 yarn hadoop 668235 Jul 12 13:47 ./__spark_libs__/guice-4.0.jar 998246923 3864 -r-xr-xr-x 1 yarn hadoop 3955990 Jul 12 13:47 ./__spark_libs__/bcprov-jdk15on-1.58.jar 998246924 76 -r-xr-xr-x 1 yarn hadoop 76983 Jul 12 13:47 ./__spark_libs__/guice-servlet-4.0.jar 998246925 112 -r-xr-xr-x 1 yarn hadoop 110600 Jul 12 13:47 ./__spark_libs__/bonecp-0.8.0.RELEASE.jar 998246926 200 -r-xr-xr-x 1 yarn hadoop 201953 Jul 12 13:47 ./__spark_libs__/curator-framework-2.12.0.jar 998246927 184 -r-xr-xr-x 1 yarn hadoop 187882 Jul 12 13:47 ./__spark_libs__/breeze-macros_2.11-0.13.2.jar 998246928 448 -r-xr-xr-x 1 yarn hadoop 456674 Jul 12 13:47 ./__spark_libs__/hadoop-aws-3.1.0.3.0.0.0-1634.jar 998246929 14760 -r-xr-xr-x 1 yarn hadoop 15113382 Jul 12 13:47 ./__spark_libs__/breeze_2.11-0.13.2.jar 998246930 604 -r-xr-xr-x 1 yarn hadoop 616888 Jul 12 13:47 ./__spark_libs__/commons-configuration2-2.1.1.jar 998246931 256 -r-xr-xr-x 1 yarn hadoop 258370 Jul 12 13:47 ./__spark_libs__/calcite-avatica-1.2.0-incubating.jar 998246932 280 -r-xr-xr-x 1 yarn hadoop 283598 Jul 12 13:47 ./__spark_libs__/curator-recipes-2.12.0.jar 998246933 3440 -r-xr-xr-x 1 yarn hadoop 3519262 Jul 12 13:47 ./__spark_libs__/calcite-core-1.2.0-incubating.jar 998246934 3420 -r-xr-xr-x 1 yarn hadoop 3500456 Jul 12 13:47 ./__spark_libs__/gcs-connector-1.9.0.3.0.0.0-1634-shaded.jar 998246935 436 -r-xr-xr-x 1 yarn hadoop 442406 Jul 12 13:47 ./__spark_libs__/calcite-linq4j-1.2.0-incubating.jar 998246936 136 -r-xr-xr-x 1 yarn hadoop 138300 Jul 12 13:47 ./__spark_libs__/hadoop-auth-3.1.0.3.0.0.0-1634.jar 998246937 60 -r-xr-xr-x 1 yarn hadoop 60483 Jul 12 13:47 ./__spark_libs__/chill-java-0.8.4.jar 998246938 300 -r-xr-xr-x 1 yarn hadoop 304172 Jul 12 13:47 ./__spark_libs__/hadoop-azure-3.1.0.3.0.0.0-1634.jar 998246939 220 -r-xr-xr-x 1 yarn hadoop 224167 Jul 12 13:47 ./__spark_libs__/chill_2.11-0.8.4.jar 998246940 44 -r-xr-xr-x 1 yarn hadoop 43891 Jul 12 13:47 ./__spark_libs__/hadoop-client-3.1.0.3.0.0.0-1634.jar 998246941 244 -r-xr-xr-x 1 yarn hadoop 246174 Jul 12 13:47 ./__spark_libs__/commons-beanutils-1.9.3.jar 998246942 228 -r-xr-xr-x 1 yarn hadoop 232248 Jul 12 13:47 ./__spark_libs__/jackson-core-asl-1.9.13.jar 998246943 44 -r-xr-xr-x 1 yarn hadoop 41123 Jul 12 13:47 ./__spark_libs__/commons-cli-1.2.jar 998246944 3940 -r-xr-xr-x 1 yarn hadoop 4031772 Jul 12 13:47 ./__spark_libs__/hadoop-common-3.1.0.3.0.0.0-1634.jar 998246945 280 -r-xr-xr-x 1 yarn hadoop 284184 Jul 12 13:47 ./__spark_libs__/commons-codec-1.10.jar 998246946 332 -r-xr-xr-x 1 yarn hadoop 339666 Jul 12 13:47 ./__spark_libs__/datanucleus-api-jdo-3.2.6.jar 998246947 576 -r-xr-xr-x 1 yarn hadoop 588337 Jul 12 13:47 ./__spark_libs__/commons-collections-3.2.2.jar 998246948 136 -r-xr-xr-x 1 yarn hadoop 138978 Jul 12 13:47 ./__spark_libs__/hive-beeline-1.21.2.3.0.0.0-1634.jar 998246949 40 -r-xr-xr-x 1 yarn hadoop 37887 Jul 12 13:47 ./__spark_libs__/commons-compiler-3.0.8.jar 998246950 40 -r-xr-xr-x 1 yarn hadoop 40878 Jul 12 13:47 ./__spark_libs__/hive-cli-1.21.2.3.0.0.0-1634.jar 998246951 236 -r-xr-xr-x 1 yarn hadoop 241367 Jul 12 13:47 ./__spark_libs__/commons-compress-1.4.1.jar 998246952 184 -r-xr-xr-x 1 yarn hadoop 185140 Jul 12 13:47 ./__spark_libs__/commons-io-2.4.jar 998246953 11232 -r-xr-xr-x 1 yarn hadoop 11498146 Jul 12 13:47 ./__spark_libs__/hive-exec-1.21.2.3.0.0.0-1634.jar 998246954 280 -r-xr-xr-x 1 yarn hadoop 284220 Jul 12 13:47 ./__spark_libs__/commons-lang-2.6.jar 998246955 104 -r-xr-xr-x 1 yarn hadoop 102713 Jul 12 13:47 ./__spark_libs__/hive-jdbc-1.21.2.3.0.0.0-1634.jar 998246956 472 -r-xr-xr-x 1 yarn hadoop 479881 Jul 12 13:47 ./__spark_libs__/commons-lang3-3.5.jar 998246957 1468 -r-xr-xr-x 1 yarn hadoop 1502280 Jul 12 13:47 ./__spark_libs__/htrace-core4-4.1.0-incubating.jar 998246958 64 -r-xr-xr-x 1 yarn hadoop 62050 Jul 12 13:47 ./__spark_libs__/commons-logging-1.1.3.jar 998246959 256 -r-xr-xr-x 1 yarn hadoop 258919 Jul 12 13:47 ./__spark_libs__/jackson-core-2.6.7.jar 998246960 208 -r-xr-xr-x 1 yarn hadoop 212453 Jul 12 13:47 ./__spark_libs__/commons-net-2.2.jar 998246961 780 -r-xr-xr-x 1 yarn hadoop 796326 Jul 12 13:47 ./__spark_libs__/janino-3.0.8.jar 998246962 72 -r-xr-xr-x 1 yarn hadoop 72733 Jul 12 13:47 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar 998246963 96 -r-xr-xr-x 1 yarn hadoop 94672 Jul 12 13:47 ./__spark_libs__/xz-1.0.jar 998246964 56 -r-xr-xr-x 1 yarn hadoop 55236 Jul 12 13:47 ./__spark_libs__/geronimo-jcache_1.0_spec-1.0-alpha-1.jar 998246965 932 -r-xr-xr-x 1 yarn hadoop 951701 Jul 12 13:47 ./__spark_libs__/jersey-server-2.22.2.jar 998246966 60 -r-xr-xr-x 1 yarn hadoop 59840 Jul 12 13:47 ./__spark_libs__/hadoop-annotations-3.1.0.3.0.0.0-1634.jar 998246967 684 -r-xr-xr-x 1 yarn hadoop 698375 Jul 12 13:47 ./__spark_libs__/jersey-common-2.22.2.jar 998246968 308 -r-xr-xr-x 1 yarn hadoop 313702 Jul 12 13:47 ./__spark_libs__/libfb303-0.9.3.jar 998246969 68 -r-xr-xr-x 1 yarn hadoop 66835 Jul 12 13:47 ./__spark_libs__/hadoop-azure-datalake-3.1.0.3.0.0.0-1634.jar 998246970 2000 -r-xr-xr-x 1 yarn hadoop 2046361 Jul 12 13:47 ./__spark_libs__/jets3t-0.9.4.jar 998246971 232 -r-xr-xr-x 1 yarn hadoop 234201 Jul 12 13:47 ./__spark_libs__/libthrift-0.9.3.jar 998246972 44 -r-xr-xr-x 1 yarn hadoop 43578 Jul 12 13:47 ./__spark_libs__/hadoop-cloud-storage-3.1.0.3.0.0.0-1634.jar 998246973 444 -r-xr-xr-x 1 yarn hadoop 452019 Jul 12 13:47 ./__spark_libs__/jetty-util-9.3.20.v20170531.jar 998246974 4872 -r-xr-xr-x 1 yarn hadoop 4985225 Jul 12 13:47 ./__spark_libs__/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar 998246975 200 -r-xr-xr-x 1 yarn hadoop 201124 Jul 12 13:47 ./__spark_libs__/jdo-api-3.0.1.jar 998267904 64 -r-xr-xr-x 1 yarn hadoop 65464 Jul 12 13:47 ./__spark_libs__/kerb-common-1.0.1.jar 998267905 788 -r-xr-xr-x 1 yarn hadoop 803498 Jul 12 13:47 ./__spark_libs__/hadoop-mapreduce-client-common-3.1.0.3.0.0.0-1634.jar 998267906 20 -r-xr-xr-x 1 yarn hadoop 18098 Jul 12 13:47 ./__spark_libs__/jersey-container-servlet-2.22.2.jar 998267907 1616 -r-xr-xr-x 1 yarn hadoop 1654648 Jul 12 13:47 ./__spark_libs__/hadoop-mapreduce-client-core-3.1.0.3.0.0.0-1634.jar 998267908 164 -r-xr-xr-x 1 yarn hadoop 167421 Jul 12 13:47 ./__spark_libs__/jersey-client-2.22.2.jar 998267909 200 -r-xr-xr-x 1 yarn hadoop 204650 Jul 12 13:47 ./__spark_libs__/kerby-pkix-1.0.1.jar 998267910 84 -r-xr-xr-x 1 yarn hadoop 85410 Jul 12 13:47 ./__spark_libs__/hadoop-mapreduce-client-jobclient-3.1.0.3.0.0.0-1634.jar 998267911 40 -r-xr-xr-x 1 yarn hadoop 38705 Jul 12 13:47 ./__spark_libs__/jetty-util-ajax-9.3.20.v20170531.jar 998267912 156 -r-xr-xr-x 1 yarn hadoop 159700 Jul 12 13:47 ./__spark_libs__/hadoop-openstack-3.1.0.3.0.0.0-1634.jar 998267913 112 -r-xr-xr-x 1 yarn hadoop 113390 Jul 12 13:47 ./__spark_libs__/jetty-webapp-9.3.20.v20170531.jar 998267914 3028 -r-xr-xr-x 1 yarn hadoop 3097487 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-api-3.1.0.3.0.0.0-1634.jar 998267915 52 -r-xr-xr-x 1 yarn hadoop 50771 Jul 12 13:47 ./__spark_libs__/jetty-xml-9.3.20.v20170531.jar 998267916 296 -r-xr-xr-x 1 yarn hadoop 300538 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-client-3.1.0.3.0.0.0-1634.jar 998267917 212 -r-xr-xr-x 1 yarn hadoop 213911 Jul 12 13:47 ./__spark_libs__/jline-2.12.1.jar 998267918 480 -r-xr-xr-x 1 yarn hadoop 489884 Jul 12 13:47 ./__spark_libs__/log4j-1.2.17.jar 998267919 2752 -r-xr-xr-x 1 yarn hadoop 2814261 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-common-3.1.0.3.0.0.0-1634.jar 998267920 616 -r-xr-xr-x 1 yarn hadoop 627814 Jul 12 13:47 ./__spark_libs__/joda-time-2.9.3.jar 998267921 364 -r-xr-xr-x 1 yarn hadoop 370119 Jul 12 13:47 ./__spark_libs__/lz4-java-1.4.0.jar 998267922 220 -r-xr-xr-x 1 yarn hadoop 224598 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-registry-3.1.0.3.0.0.0-1634.jar 998267923 68 -r-xr-xr-x 1 yarn hadoop 66270 Jul 12 13:47 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar 998267924 1308 -r-xr-xr-x 1 yarn hadoop 1336802 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-server-common-3.1.0.3.0.0.0-1634.jar 998267925 952 -r-xr-xr-x 1 yarn hadoop 971310 Jul 12 13:47 ./__spark_libs__/jersey-guava-2.22.2.jar 998267926 8 -r-xr-xr-x 1 yarn hadoop 5711 Jul 12 13:47 ./__spark_libs__/minlog-1.3.0.jar 998267927 80 -r-xr-xr-x 1 yarn hadoop 80404 Jul 12 13:47 ./__spark_libs__/hadoop-yarn-server-web-proxy-3.1.0.3.0.0.0-1634.jar 998267928 420 -r-xr-xr-x 1 yarn hadoop 427780 Jul 12 13:47 ./__spark_libs__/jodd-core-3.5.2.jar 998267929 44 -r-xr-xr-x 1 yarn hadoop 41755 Jul 12 13:47 ./__spark_libs__/objenesis-2.1.jar 998267930 5344 -r-xr-xr-x 1 yarn hadoop 5471277 Jul 12 13:47 ./__spark_libs__/hive-metastore-1.21.2.3.0.0.0-1634.jar 998267931 40 -r-xr-xr-x 1 yarn hadoop 40341 Jul 12 13:47 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar 998267932 48 -r-xr-xr-x 1 yarn hadoop 46986 Jul 12 13:47 ./__spark_libs__/jackson-annotations-2.6.7.jar 998267933 100 -r-xr-xr-x 1 yarn hadoop 100636 Jul 12 13:47 ./__spark_libs__/jsp-api-2.1.jar 998267934 1140 -r-xr-xr-x 1 yarn hadoop 1165323 Jul 12 13:47 ./__spark_libs__/jackson-databind-2.6.7.1.jar 998267935 36 -r-xr-xr-x 1 yarn hadoop 33015 Jul 12 13:47 ./__spark_libs__/jsr305-1.3.9.jar 998267936 48 -r-xr-xr-x 1 yarn hadoop 48468 Jul 12 13:47 ./__spark_libs__/jackson-dataformat-cbor-2.6.7.jar 998267937 748 -r-xr-xr-x 1 yarn hadoop 764569 Jul 12 13:47 ./__spark_libs__/jtransforms-2.4.0.jar 998267938 32 -r-xr-xr-x 1 yarn hadoop 32373 Jul 12 13:47 ./__spark_libs__/jackson-jaxrs-base-2.9.5.jar 998267939 120 -r-xr-xr-x 1 yarn hadoop 120316 Jul 12 13:47 ./__spark_libs__/json-smart-2.3.jar 998267940 324 -r-xr-xr-x 1 yarn hadoop 331034 Jul 12 13:47 ./__spark_libs__/okhttp-2.7.5.jar 998267941 16 -r-xr-xr-x 1 yarn hadoop 15864 Jul 12 13:47 ./__spark_libs__/jackson-jaxrs-json-provider-2.9.5.jar 998267942 8 -r-xr-xr-x 1 yarn hadoop 4596 Jul 12 13:47 ./__spark_libs__/jul-to-slf4j-1.7.16.jar 998267943 764 -r-xr-xr-x 1 yarn hadoop 780664 Jul 12 13:47 ./__spark_libs__/jackson-mapper-asl-1.9.13.jar 998267944 84 -r-xr-xr-x 1 yarn hadoop 82421 Jul 12 13:47 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar 998267945 32 -r-xr-xr-x 1 yarn hadoop 32612 Jul 12 13:47 ./__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar 998267946 80 -r-xr-xr-x 1 yarn hadoop 80980 Jul 12 13:47 ./__spark_libs__/kerb-admin-1.0.1.jar 998267947 44 -r-xr-xr-x 1 yarn hadoop 42858 Jul 12 13:47 ./__spark_libs__/jackson-module-paranamer-2.7.9.jar 998267948 576 -r-xr-xr-x 1 yarn hadoop 589462 Jul 12 13:47 ./__spark_libs__/json4s-core_2.11-3.2.11.jar 998267949 504 -r-xr-xr-x 1 yarn hadoop 515645 Jul 12 13:47 ./__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jar 998267950 224 -r-xr-xr-x 1 yarn hadoop 226672 Jul 12 13:47 ./__spark_libs__/kerb-core-1.0.1.jar 998267951 20 -r-xr-xr-x 1 yarn hadoop 17316 Jul 12 13:47 ./__spark_libs__/java-xmlbuilder-1.1.jar 998267952 116 -r-xr-xr-x 1 yarn hadoop 116120 Jul 12 13:47 ./__spark_libs__/kerb-crypto-1.0.1.jar 998267953 700 -r-xr-xr-x 1 yarn hadoop 714194 Jul 12 13:47 ./__spark_libs__/javassist-3.18.1-GA.jar 998267954 112 -r-xr-xr-x 1 yarn hadoop 113017 Jul 12 13:47 ./__spark_libs__/kerb-client-1.0.1.jar 998267955 28 -r-xr-xr-x 1 yarn hadoop 26366 Jul 12 13:47 ./__spark_libs__/javax.annotation-api-1.2.jar 998267956 40 -r-xr-xr-x 1 yarn hadoop 40554 Jul 12 13:47 ./__spark_libs__/kerby-util-1.0.1.jar 998267957 4 -r-xr-xr-x 1 yarn hadoop 2497 Jul 12 13:47 ./__spark_libs__/javax.inject-1.jar 998267958 20 -r-xr-xr-x 1 yarn hadoop 20046 Jul 12 13:47 ./__spark_libs__/kerb-identity-1.0.1.jar 998267959 8 -r-xr-xr-x 1 yarn hadoop 5950 Jul 12 13:47 ./__spark_libs__/javax.inject-2.4.0-b34.jar 998267960 84 -r-xr-xr-x 1 yarn hadoop 82756 Jul 12 13:47 ./__spark_libs__/kerb-server-1.0.1.jar 998267961 96 -r-xr-xr-x 1 yarn hadoop 95806 Jul 12 13:47 ./__spark_libs__/javax.servlet-api-3.1.0.jar 998267962 20 -r-xr-xr-x 1 yarn hadoop 20409 Jul 12 13:47 ./__spark_libs__/kerb-simplekdc-1.0.1.jar 998267963 116 -r-xr-xr-x 1 yarn hadoop 115534 Jul 12 13:47 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar 998267964 36 -r-xr-xr-x 1 yarn hadoop 36708 Jul 12 13:47 ./__spark_libs__/kerb-util-1.0.1.jar 998267965 388 -r-xr-xr-x 1 yarn hadoop 395195 Jul 12 13:47 ./__spark_libs__/javolution-5.5.1.jar 998267966 352 -r-xr-xr-x 1 yarn hadoop 358390 Jul 12 13:47 ./__spark_libs__/kryo-shaded-3.0.3.jar 998267967 100 -r-xr-xr-x 1 yarn hadoop 102244 Jul 12 13:47 ./__spark_libs__/jaxb-api-2.2.11.jar 998267968 100 -r-xr-xr-x 1 yarn hadoop 102174 Jul 12 13:47 ./__spark_libs__/kerby-asn1-1.0.1.jar 998267969 8 -r-xr-xr-x 1 yarn hadoop 4722 Jul 12 13:47 ./__spark_libs__/jcip-annotations-1.0-1.jar 998267970 32 -r-xr-xr-x 1 yarn hadoop 30674 Jul 12 13:47 ./__spark_libs__/kerby-config-1.0.1.jar 998267971 20 -r-xr-xr-x 1 yarn hadoop 16430 Jul 12 13:47 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar 998267972 1024 -r-xr-xr-x 1 yarn hadoop 1045744 Jul 12 13:47 ./__spark_libs__/leveldbjni-all-1.8.jar 998267973 32 -r-xr-xr-x 1 yarn hadoop 29134 Jul 12 13:47 ./__spark_libs__/kerby-xdr-1.0.1.jar 998267974 9888 -r-xr-xr-x 1 yarn hadoop 10121868 Jul 12 13:47 ./__spark_libs__/spire_2.11-0.13.0.jar 998267975 36 -r-xr-xr-x 1 yarn hadoop 34603 Jul 12 13:47 ./__spark_libs__/machinist_2.11-0.6.1.jar 998267976 28 -r-xr-xr-x 1 yarn hadoop 26514 Jul 12 13:47 ./__spark_libs__/stax-api-1.0.1.jar 998267977 4 -r-xr-xr-x 1 yarn hadoop 3142 Jul 12 13:47 ./__spark_libs__/macro-compat_2.11-1.1.1.jar 998267978 160 -r-xr-xr-x 1 yarn hadoop 161867 Jul 12 13:47 ./__spark_libs__/stax2-api-3.1.4.jar 998267979 120 -r-xr-xr-x 1 yarn hadoop 120465 Jul 12 13:47 ./__spark_libs__/metrics-core-3.1.5.jar 998267980 172 -r-xr-xr-x 1 yarn hadoop 174351 Jul 12 13:47 ./__spark_libs__/stream-2.7.0.jar 998267981 24 -r-xr-xr-x 1 yarn hadoop 21247 Jul 12 13:47 ./__spark_libs__/metrics-graphite-3.1.5.jar 998267982 148 -r-xr-xr-x 1 yarn hadoop 148627 Jul 12 13:47 ./__spark_libs__/stringtemplate-3.2.1.jar 998267983 16 -r-xr-xr-x 1 yarn hadoop 15824 Jul 12 13:47 ./__spark_libs__/metrics-json-3.1.5.jar 998267984 92 -r-xr-xr-x 1 yarn hadoop 93210 Jul 12 13:47 ./__spark_libs__/super-csv-2.2.0.jar 998267985 40 -r-xr-xr-x 1 yarn hadoop 39283 Jul 12 13:47 ./__spark_libs__/metrics-jvm-3.1.5.jar 998267986 20 -r-xr-xr-x 1 yarn hadoop 18763 Jul 12 13:47 ./__spark_libs__/token-provider-1.0.1.jar 998267987 776 -r-xr-xr-x 1 yarn hadoop 792442 Jul 12 13:47 ./__spark_libs__/mssql-jdbc-6.2.1.jre7.jar 998267988 376 -r-xr-xr-x 1 yarn hadoop 384150 Jul 12 13:47 ./__spark_libs__/univocity-parsers-2.5.9.jar 998267989 1300 -r-xr-xr-x 1 yarn hadoop 1330219 Jul 12 13:47 ./__spark_libs__/netty-3.9.9.Final.jar 998267990 64 -r-xr-xr-x 1 yarn hadoop 63777 Jul 12 13:47 ./__spark_libs__/validation-api-1.1.0.Final.jar 998267991 3692 -r-xr-xr-x 1 yarn hadoop 3780056 Jul 12 13:47 ./__spark_libs__/netty-all-4.1.17.Final.jar 998267992 504 -r-xr-xr-x 1 yarn hadoop 512742 Jul 12 13:47 ./__spark_libs__/woodstox-core-5.0.3.jar 998267993 296 -r-xr-xr-x 1 yarn hadoop 299508 Jul 12 13:47 ./__spark_libs__/nimbus-jose-jwt-4.41.1.jar 998267994 68 -r-xr-xr-x 1 yarn hadoop 65928 Jul 12 13:47 ./__spark_libs__/okio-1.6.0.jar 998267995 20 -r-xr-xr-x 1 yarn hadoop 19827 Jul 12 13:47 ./__spark_libs__/opencsv-2.3.jar 998267996 144 -r-xr-xr-x 1 yarn hadoop 144660 Jul 12 13:47 ./__spark_libs__/xbean-asm5-shaded-4.4.jar 998267997 1412 -r-xr-xr-x 1 yarn hadoop 1442702 Jul 12 13:47 ./__spark_libs__/orc-core-1.4.4-nohive.jar 998267998 120 -r-xr-xr-x 1 yarn hadoop 120715 Jul 12 13:47 ./__spark_libs__/spark-repl_2.11-2.3.1.3.0.0.0-1634.jar 998267999 740 -r-xr-xr-x 1 yarn hadoop 757253 Jul 12 13:47 ./__spark_libs__/orc-mapreduce-1.4.4-nohive.jar 998268000 64 -r-xr-xr-x 1 yarn hadoop 65261 Jul 12 13:47 ./__spark_libs__/oro-2.0.8.jar 998268001 32 -r-xr-xr-x 1 yarn hadoop 30101 Jul 12 13:47 ./__spark_libs__/spark-sketch_2.11-2.3.1.3.0.0.0-1634.jar 998268002 20 -r-xr-xr-x 1 yarn hadoop 20235 Jul 12 13:47 ./__spark_libs__/osgi-resource-locator-1.0.1.jar 998268003 36 -r-xr-xr-x 1 yarn hadoop 34654 Jul 12 13:47 ./__spark_libs__/paranamer-2.8.jar 998268004 788 -r-xr-xr-x 1 yarn hadoop 803549 Jul 12 13:47 ./__spark_libs__/zookeeper-3.4.6.3.0.0.0-1634.jar 998268005 940 -r-xr-xr-x 1 yarn hadoop 961731 Jul 12 13:47 ./__spark_libs__/parquet-column-1.8.3.jar 998268006 2280 -r-xr-xr-x 1 yarn hadoop 2333186 Jul 12 13:47 ./__spark_libs__/zstd-jni-1.3.2-2.jar 998268007 44 -r-xr-xr-x 1 yarn hadoop 42064 Jul 12 13:47 ./__spark_libs__/parquet-common-1.8.3.jar 998268008 280 -r-xr-xr-x 1 yarn hadoop 286071 Jul 12 13:47 ./__spark_libs__/parquet-encoding-1.8.3.jar 998268009 384 -r-xr-xr-x 1 yarn hadoop 390733 Jul 12 13:47 ./__spark_libs__/parquet-format-2.3.1.jar 998268010 248 -r-xr-xr-x 1 yarn hadoop 250989 Jul 12 13:47 ./__spark_libs__/parquet-hadoop-1.8.3.jar 998268011 8496 -r-xr-xr-x 1 yarn hadoop 8698795 Jul 12 13:47 ./__spark_libs__/spark-sql_2.11-2.3.1.3.0.0.0-1634.jar 998268012 2732 -r-xr-xr-x 1 yarn hadoop 2796935 Jul 12 13:47 ./__spark_libs__/parquet-hadoop-bundle-1.6.0.jar 998268013 1024 -r-xr-xr-x 1 yarn hadoop 1048115 Jul 12 13:47 ./__spark_libs__/parquet-jackson-1.8.3.jar 998268014 524 -r-xr-xr-x 1 yarn hadoop 533455 Jul 12 13:47 ./__spark_libs__/protobuf-java-2.5.0.jar 998268015 120 -r-xr-xr-x 1 yarn hadoop 122774 Jul 12 13:47 ./__spark_libs__/py4j-0.10.7.jar 998268016 96 -r-xr-xr-x 1 yarn hadoop 94796 Jul 12 13:47 ./__spark_libs__/pyrolite-4.13.jar 998268017 15128 -r-xr-xr-x 1 yarn hadoop 15487351 Jul 12 13:47 ./__spark_libs__/scala-compiler-2.11.8.jar 998268018 5612 -r-xr-xr-x 1 yarn hadoop 5744974 Jul 12 13:47 ./__spark_libs__/scala-library-2.11.8.jar 998268019 80 -r-xr-xr-x 1 yarn hadoop 80180 Jul 12 13:47 ./__spark_libs__/spark-launcher_2.11-2.3.1.3.0.0.0-1634.jar 998268020 416 -r-xr-xr-x 1 yarn hadoop 423753 Jul 12 13:47 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar 998268021 4468 -r-xr-xr-x 1 yarn hadoop 4573750 Jul 12 13:47 ./__spark_libs__/scala-reflect-2.11.8.jar 998268022 656 -r-xr-xr-x 1 yarn hadoop 671138 Jul 12 13:47 ./__spark_libs__/scala-xml_2.11-1.0.5.jar 998268023 788 -r-xr-xr-x 1 yarn hadoop 802818 Jul 12 13:47 ./__spark_libs__/scalap-2.11.8.jar 998268024 3444 -r-xr-xr-x 1 yarn hadoop 3522616 Jul 12 13:47 ./__spark_libs__/shapeless_2.11-2.3.2.jar 998268025 40 -r-xr-xr-x 1 yarn hadoop 40509 Jul 12 13:47 ./__spark_libs__/slf4j-api-1.7.16.jar 998268026 12 -r-xr-xr-x 1 yarn hadoop 9939 Jul 12 13:47 ./__spark_libs__/slf4j-log4j12-1.7.16.jar 998268027 48 -r-xr-xr-x 1 yarn hadoop 48720 Jul 12 13:47 ./__spark_libs__/snappy-0.2.jar 998268028 1032 -r-xr-xr-x 1 yarn hadoop 1056168 Jul 12 13:47 ./__spark_libs__/snappy-java-1.1.2.6.jar 998268029 180 -r-xr-xr-x 1 yarn hadoop 183877 Jul 12 13:47 ./__spark_libs__/spark-mllib-local_2.11-2.3.1.3.0.0.0-1634.jar 998268030 8800 -r-xr-xr-x 1 yarn hadoop 9008594 Jul 12 13:47 ./__spark_libs__/spark-catalyst_2.11-2.3.1.3.0.0.0-1634.jar 998268031 7592 -r-xr-xr-x 1 yarn hadoop 7773635 Jul 12 13:47 ./__spark_libs__/spark-mllib_2.11-2.3.1.3.0.0.0-1634.jar 998268032 12748 -r-xr-xr-x 1 yarn hadoop 13049902 Jul 12 13:47 ./__spark_libs__/spark-core_2.11-2.3.1.3.0.0.0-1634.jar 998268033 2328 -r-xr-xr-x 1 yarn hadoop 2382241 Jul 12 13:47 ./__spark_libs__/spark-network-common_2.11-2.3.1.3.0.0.0-1634.jar 998268034 696 -r-xr-xr-x 1 yarn hadoop 709061 Jul 12 13:47 ./__spark_libs__/spark-graphx_2.11-2.3.1.3.0.0.0-1634.jar 998268035 1300 -r-xr-xr-x 1 yarn hadoop 1330567 Jul 12 13:47 ./__spark_libs__/spark-hive_2.11-2.3.1.3.0.0.0-1634.jar 998268036 496 -r-xr-xr-x 1 yarn hadoop 506840 Jul 12 13:47 ./__spark_libs__/spark-hadoop-cloud_2.11-2.3.1.3.0.0.0-1634.jar 998268037 52 -r-xr-xr-x 1 yarn hadoop 53052 Jul 12 13:47 ./__spark_libs__/spark-kvstore_2.11-2.3.1.3.0.0.0-1634.jar 998268038 1772 -r-xr-xr-x 1 yarn hadoop 1814228 Jul 12 13:47 ./__spark_libs__/spark-hive-thriftserver_2.11-2.3.1.3.0.0.0-1634.jar 998268039 88 -r-xr-xr-x 1 yarn hadoop 87192 Jul 12 13:47 ./__spark_libs__/spire-macros_2.11-0.13.0.jar 998268040 64 -r-xr-xr-x 1 yarn hadoop 64901 Jul 12 13:47 ./__spark_libs__/spark-network-shuffle_2.11-2.3.1.3.0.0.0-1634.jar 998268041 2124 -r-xr-xr-x 1 yarn hadoop 2171390 Jul 12 13:47 ./__spark_libs__/spark-streaming_2.11-2.3.1.3.0.0.0-1634.jar 998268042 16 -r-xr-xr-x 1 yarn hadoop 15488 Jul 12 13:47 ./__spark_libs__/spark-tags_2.11-2.3.1.3.0.0.0-1634.jar 998268043 48 -r-xr-xr-x 1 yarn hadoop 48976 Jul 12 13:47 ./__spark_libs__/spark-unsafe_2.11-2.3.1.3.0.0.0-1634.jar 998268044 656 -r-xr-xr-x 1 yarn hadoop 670108 Jul 12 13:47 ./__spark_libs__/spark-yarn_2.11-2.3.1.3.0.0.0-1634.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info ******************************************************************************* Container: container_1539782982241_0020_01_000001 on testnode-2.bdlocal_45454_1539843785407 LogAggregationType: AGGREGATED =========================================================================================== LogType:launch_container.sh LogLastModifiedTime:Thu Oct 18 06:23:05 +0000 2018 LogLength:5218 LogContents: #!/bin/bash set -o pipefail -e export PRELAUNCH_OUT="/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/prelaunch.out" exec >"${PRELAUNCH_OUT}" export PRELAUNCH_ERR="/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/prelaunch.err" exec 2>"${PRELAUNCH_ERR}" echo "Setting up env variables" export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/3.0.0.0-1634/hadoop/conf"} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/3.0.0.0-1634/hadoop-yarn"} export HADOOP_HOME=${HADOOP_HOME:-"/usr/hdp/3.0.0.0-1634/hadoop"} export PATH=${PATH:-"/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent"} export LANG=${LANG:-"en_US.UTF-8"} export HADOOP_TOKEN_FILE_LOCATION="/hadoop/yarn/local/usercache/spark/appcache/application_1539782982241_0020/container_1539782982241_0020_01_000001/container_tokens" export CONTAINER_ID="container_1539782982241_0020_01_000001" export NM_PORT="45454" export NM_HOST="testnode-2.bdlocal" export NM_HTTP_PORT="8042" export LOCAL_DIRS="/hadoop/yarn/local/usercache/spark/appcache/application_1539782982241_0020" export LOCAL_USER_DIRS="/hadoop/yarn/local/usercache/spark/" export LOG_DIRS="/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001" export USER="spark" export LOGNAME="spark" export HOME="/home/" export PWD="/hadoop/yarn/local/usercache/spark/appcache/application_1539782982241_0020/container_1539782982241_0020_01_000001" export JVM_PID="$$" export MALLOC_ARENA_MAX="4" export NM_AUX_SERVICE_timeline_collector="" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" export NM_AUX_SERVICE_spark2_shuffle="" export SPARK_YARN_STAGING_DIR="hdfs://hortonha/user/spark/.sparkStaging/application_1539782982241_0020" export APP_SUBMIT_TIME_ENV="1539843773415" export TIMELINE_FLOW_NAME_TAG="Thrift JDBC/ODBC Server" export TIMELINE_FLOW_VERSION_TAG="1" export APPLICATION_WEB_PROXY_BASE="/proxy/application_1539782982241_0020" export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/3.0.0.0-1634/hadoop/*:/usr/hdp/3.0.0.0-1634/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/3.0.0.0-1634/hadoop/lib/hadoop-lzo-0.6.0.3.0.0.0-1634.jar:/etc/hadoop/conf/secure:$PWD/__spark_conf__/__hadoop_conf__" export SPARK_USER="spark" export TIMELINE_FLOW_RUN_ID_TAG="1539843773416" echo "Setting up job resources" ln -sf "/hadoop/yarn/local/filecache/11/spark2-hdp-hive-archive.tar.gz" "__hive_libs__" ln -sf "/hadoop/yarn/local/usercache/spark/filecache/10/__spark_conf__.zip" "__spark_conf__" ln -sf "/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz" "__spark_libs__" echo "Copying debugging information" # Creating copy of launch script cp "launch_container.sh" "/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/launch_container.sh" chmod 640 "/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" ls -l 1>>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/directory.info" echo "Launching container" exec /bin/bash -c "$JAVA_HOME/bin/java -server -Xmx512m -Djava.io.tmpdir=$PWD/tmp -Dhdp.version=3.0.0.0-1634 -Dspark.yarn.app.container.log.dir=/hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 'testnode-1.bdlocal:33094' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/stdout 2> /hadoop/yarn/log/application_1539782982241_0020/container_1539782982241_0020_01_000001/stderr" End of LogType:launch_container.sh ************************************************************************************ End of LogType:prelaunch.err ****************************************************************************** Container: container_1539782982241_0020_01_000001 on testnode-2.bdlocal_45454_1539843785407 LogAggregationType: AGGREGATED =========================================================================================== LogType:prelaunch.out LogLastModifiedTime:Thu Oct 18 06:23:05 +0000 2018 LogLength:100 LogContents: Setting up env variables Setting up job resources Copying debugging information Launching container End of LogType:prelaunch.out ****************************************************************************** Container: container_1539782982241_0020_01_000001 on testnode-2.bdlocal_45454_1539843785407 LogAggregationType: AGGREGATED =========================================================================================== LogType:stderr LogLastModifiedTime:Thu Oct 18 06:23:05 +0000 2018 LogLength:6437 LogContents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.0.0-1634/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/10/18 06:22:57 INFO SignalUtils: Registered signal handler for TERM 18/10/18 06:22:57 INFO SignalUtils: Registered signal handler for HUP 18/10/18 06:22:57 INFO SignalUtils: Registered signal handler for INT 18/10/18 06:22:58 INFO SecurityManager: Changing view acls to: spark 18/10/18 06:22:58 INFO SecurityManager: Changing modify acls to: spark 18/10/18 06:22:58 INFO SecurityManager: Changing view acls groups to: 18/10/18 06:22:58 INFO SecurityManager: Changing modify acls groups to: 18/10/18 06:22:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set() 18/10/18 06:22:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 18/10/18 06:22:58 INFO ApplicationMaster: Preparing Local resources 18/10/18 06:22:59 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 18/10/18 06:23:00 WARN Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error 18/10/18 06:23:00 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1539782982241_0020_000001 18/10/18 06:23:00 INFO ApplicationMaster: Waiting for Spark driver to be reachable. 18/10/18 06:23:00 INFO ApplicationMaster: Driver now available: testnode-1.bdlocal:33094 18/10/18 06:23:00 INFO TransportClientFactory: Successfully created connection to testnode-1.bdlocal/10.39.246.3:33094 after 101 ms (0 ms spent in bootstraps) 18/10/18 06:23:00 INFO ApplicationMaster: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark_libs__/*$HADOOP_CONF_DIR/usr/hdp/3.0.0.0-1634/hadoop/*/usr/hdp/3.0.0.0-1634/hadoop/lib/*/usr/hdp/current/hadoop-hdfs-client/*/usr/hdp/current/hadoop-hdfs-client/lib/*/usr/hdp/current/hadoop-yarn-client/*/usr/hdp/current/hadoop-yarn-client/lib/*$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/3.0.0.0-1634/hadoop/lib/hadoop-lzo-0.6.0.3.0.0.0-1634.jar:/etc/hadoop/conf/secure{{PWD}}/__spark_conf__/__hadoop_conf__ SPARK_YARN_STAGING_DIR -> hdfs://hortonha/user/spark/.sparkStaging/application_1539782982241_0020 SPARK_USER -> spark command: LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" \ {{JAVA_HOME}}/bin/java \ -server \ -Xmx1024m \ '-XX:+UseNUMA' \ -Djava.io.tmpdir={{PWD}}/tmp \ '-Dspark.driver.port=33094' \ -Dspark.yarn.app.container.log.dir= \ -XX:OnOutOfMemoryError='kill %p' \ org.apache.spark.executor.CoarseGrainedExecutorBackend \ --driver-url \ spark://CoarseGrainedScheduler@testnode-1.bdlocal:33094 \ --executor-id \ \ --hostname \ \ --cores \ 1 \ --app-id \ application_1539782982241_0020 \ --user-class-path \ file:$PWD/__app__.jar \ 1>/stdout \ 2>/stderr resources: __spark_libs__ -> resource { scheme: "hdfs" host: "hortonha" port: -1 file: "/hdp/apps/3.0.0.0-1634/spark2/spark2-hdp-yarn-archive.tar.gz" } size: 283900156 timestamp: 1537552371339 type: ARCHIVE visibility: PUBLIC __spark_conf__ -> resource { scheme: "hdfs" host: "hortonha" port: -1 file: "/user/spark/.sparkStaging/application_1539782982241_0020/__spark_conf__.zip" } size: 317132 timestamp: 1539843773252 type: ARCHIVE visibility: PRIVATE __hive_libs__ -> resource { scheme: "hdfs" host: "hortonha" port: -1 file: "/hdp/apps/3.0.0.0-1634/spark2/spark2-hdp-hive-archive.tar.gz" } size: 43028314 timestamp: 1537552375468 type: ARCHIVE visibility: PUBLIC =============================================================================== 18/10/18 06:23:01 INFO YarnRMClient: Registering the ApplicationMaster 18/10/18 06:23:01 INFO ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/10/18 06:23:01 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 18/10/18 06:23:01 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.0.0.0-1634/0/resource-types.xml 18/10/18 06:23:01 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 18/10/18 06:23:04 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. testnode-1.bdlocal:33094 18/10/18 06:23:04 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. testnode-1.bdlocal:33094 18/10/18 06:23:04 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0 18/10/18 06:23:04 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED 18/10/18 06:23:04 INFO AMRMClientImpl: Waiting for application to be successfully unregistered. 18/10/18 06:23:04 INFO ApplicationMaster: Deleting staging directory hdfs://hortonha/user/spark/.sparkStaging/application_1539782982241_0020 18/10/18 06:23:04 INFO ShutdownHookManager: Shutdown hook called End of LogType:stderr *********************************************************************** End of LogType:stdout ***********************************************************************