Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark on yarn in Cluster mdoe -> java.io.IOException: Filesystem closed

Highlighted

Spark on yarn in Cluster mdoe -> java.io.IOException: Filesystem closed

Contributor

Dear All,

 

Spark on yarn running in cluster mode is unable to perform wider transformations. are there any guidelines on memory optimization and tuning

 

Attaching the logs as well. I am facing below errors at different times of execution

 

  • SparkContext did not initialize after waiting for 720000 ms.
  • org.apache.spark.SparkException: Job 3 cancelled because SparkContext was shut down
  • java.io.IOException: Filesystem closed
  • Caused by: java.lang.InterruptedException

 

2017-10-02 18:23:35,624 INFO  [main] yarn.ApplicationMaster (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM, HUP, INT]
2017-10-02 18:23:36,661 INFO  [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - ApplicationAttemptId: appattempt_1506703761089_3300_000001
2017-10-02 18:23:37,349 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing view acls to: xxxx
2017-10-02 18:23:37,350 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing modify acls to: xxxx
2017-10-02 18:23:37,350 INFO  [main] spark.SecurityManager (Logging.scala:logInfo(58)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xxxx); users with modify permissions: Set(xxxx)
2017-10-02 18:23:37,372 INFO  [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Starting the user application in a separate Thread
2017-10-02 18:23:37,420 INFO  [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization
2017-10-02 18:23:37,420 INFO  [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ... 
2017-10-02 18:23:40,030 INFO  [Driver] commons.Environment$ (Environment.scala:parseEnv(21)) - yarnHS is: 
2017-10-02 18:23:40,037 INFO  [Driver] commons.Environment$ (Environment.scala:master(42)) - Master is: xx
2017-10-02 18:23:40,048 INFO  [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Running Spark version 1.6.0
2017-10-02 18:23:40,078 INFO  [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing view acls to: xxxx
2017-10-02 18:23:40,078 INFO  [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing modify acls to: xxxx
2017-10-02 18:23:40,078 INFO  [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xxxx); users with modify permissions: Set(xxxx)
2017-10-02 18:23:40,248 INFO  [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'sparkDriver' on port 54663.
2017-10-02 18:23:40,472 INFO  [sparkDriverActorSystem-akka.actor.default-dispatcher-3] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
2017-10-02 18:23:40,515 INFO  [sparkDriverActorSystem-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
2017-10-02 18:23:40,652 INFO  [sparkDriverActorSystem-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@namenode:55316]
2017-10-02 18:23:40,654 INFO  [sparkDriverActorSystem-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@3namenode55316]
2017-10-02 18:23:40,661 INFO  [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'sparkDriverActorSystem' on port 55316.
2017-10-02 18:23:40,680 INFO  [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering MapOutputTracker
2017-10-02 18:23:40,698 INFO  [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering BlockManagerMaster
...
2017-10-02 18:23:40,715 INFO [Driver] storage.DiskBlockManager (Logging.scala:logInfo(58)) - Created local directory at /data9/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/blockmgr-7df05782-ba29-42cd-8825-0e9c2640e045 2017-10-02 18:23:40,721 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - MemoryStore started with capacity 983.1 MB 2017-10-02 18:23:40,885 INFO [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering OutputCommitCoordinator 2017-10-02 18:23:41,045 INFO [Driver] ui.JettyUtils (Logging.scala:logInfo(58)) - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 2017-10-02 18:23:41,071 INFO [Driver] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT 2017-10-02 18:23:41,115 INFO [Driver] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:45099 2017-10-02 18:23:41,116 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'SparkUI' on port 45099. 2017-10-02 18:23:41,118 INFO [Driver] ui.SparkUI (Logging.scala:logInfo(58)) - Started SparkUI at http://namenode:45099 2017-10-02 18:23:41,238 INFO [Driver] executor.Executor (Logging.scala:logInfo(58)) - Starting executor ID driver on host localhost 2017-10-02 18:23:41,277 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55535. 2017-10-02 18:23:41,278 INFO [Driver] netty.NettyBlockTransferService (Logging.scala:logInfo(58)) - Server created on 55535 2017-10-02 18:23:41,279 INFO [Driver] storage.BlockManager (Logging.scala:logInfo(58)) - external shuffle service port = 7337 2017-10-02 18:23:41,280 INFO [Driver] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - Trying to register BlockManager 2017-10-02 18:23:41,282 INFO [dispatcher-event-loop-10] storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(58)) - Registering block manager localhost:55535 with 983.1 MB RAM, BlockManagerId(driver, localhost, 55535) 2017-10-02 18:23:41,284 INFO [Driver] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - Registered BlockManager 2017-10-02 18:23:41,496 WARN [Driver] security.UserGroupInformation (UserGroupInformation.java:doAs(1701)) - PriviledgedActionException as:xxxx (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error 2017-10-02 18:23:41,497 WARN [Driver] ipc.Client (Client.java:run(682)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error 2017-10-02 18:23:41,497 WARN [Driver] security.UserGroupInformation (UserGroupInformation.java:doAs(1701)) - PriviledgedActionException as:xxxx (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error 2017-10-02 18:23:41,700 INFO [Driver] scheduler.EventLoggingListener (Logging.scala:logInfo(58)) - Logging events to hdfs://namenode/user/spark/applicationHistory/local-1506961421183 2017-10-02 18:23:41,741 INFO [Driver] properties (LoadedProperties.scala:fromFile(53)) - LOOKING FOR test.conf FOR ENVIRONMENT: xxx 2017-10-02 18:23:41,851 INFO [Driver] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1174)) - mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 2017-10-02 18:23:41,964 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x7c1651ef connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 ...
2017-10-02 18:23:41,976 INFO [Driver] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/xxxx 2017-10-02 18:23:41,976 INFO [Driver] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001 2017-10-02 18:23:41,977 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x7c1651ef0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:41,992 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-2/39.7.48.2:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:41,993 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:38451, server: namenode-230-2/39.7.48.2:2181 2017-10-02 18:23:42,005 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-2/39.7.48.2:2181, sessionid = 0x15ece8a28d39372, negotiated timeout = 60000 2017-10-02 18:23:42,077 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 2017-10-02 18:23:42,077 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:42,079 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-3/39.7.48.3:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:42,079 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:57307, server: namenode-230-3/39.7.48.3:2181 2017-10-02 18:23:42,093 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-3/39.7.48.3:2181, sessionid = 0x35ece8a34bb9324, negotiated timeout = 60000 2017-10-02 18:23:42,100 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x35ece8a34bb9324 closed 2017-10-02 18:23:42,100 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:42,100 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x15ece8a28d39372 2017-10-02 18:23:42,105 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:42,105 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x15ece8a28d39372 closed 2017-10-02 18:23:42,247 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_0 stored as values in memory (estimated size 358.7 KB, free 358.7 KB) 2017-10-02 18:23:42,441 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 390.7 KB) 2017-10-02 18:23:42,444 INFO [dispatcher-event-loop-12] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_0_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB) 2017-10-02 18:23:42,447 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 0 from broadcast at HBaseContext.scala:71 2017-10-02 18:23:42,449 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 393.6 KB) 2017-10-02 18:23:42,458 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_1_piece0 stored as bytes in memory (estimated size 789.0 B, free 394.4 KB) 2017-10-02 18:23:42,459 INFO [dispatcher-event-loop-13] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_1_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB) 2017-10-02 18:23:42,460 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 1 from broadcast at HBaseContext.scala:72 2017-10-02 18:23:43,216 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - Initializing execution hive, version 1.1.0 2017-10-02 18:23:43,272 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Inspected Hadoop version: 2.6.0-cdh5.9.0 2017-10-02 18:23:43,273 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.9.0 2017-10-02 18:23:44,027 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(386)) - Trying to connect to metastore with URI thrift://namenode-230-2:9083 2017-10-02 18:23:44,057 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(431)) - Opened a connection to metastore, current connections: 1 2017-10-02 18:23:44,057 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(483)) - Connected to metastore. 2017-10-02 18:23:44,140 INFO [Driver] metadata.Hive (Hive.java:reloadFunctions(220)) - Registering function indicfmt com.teradata.dynaload.hcatalog.hiveudf.TDIndicRowTbl 2017-10-02 18:23:44,188 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: file:/data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/spark-d24114da-2c76-463d-a55d-08017491fd15/scratch/xxxx ...
2017-10-02 18:23:44,194 INFO [Driver] session.SessionState (SessionState.java:start(557)) - No Tez session required at this point. hive.execution.engine=mr. 2017-10-02 18:23:44,289 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - default warehouse location is /user/hive/warehouse 2017-10-02 18:23:44,298 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - Initializing metastore client version 1.1.0 using Spark classes. 2017-10-02 18:23:44,316 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Inspected Hadoop version: 2.6.0-cdh5.9.0 2017-10-02 18:23:44,327 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.9.0 2017-10-02 18:23:45,061 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(386)) - Trying to connect to metastore with URI thrift://namenode-230-2:9083 2017-10-02 18:23:45,089 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(431)) - Opened a connection to metastore, current connections: 1 2017-10-02 18:23:45,089 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(483)) - Connected to metastore. 2017-10-02 18:23:45,163 INFO [Driver] metadata.Hive (Hive.java:reloadFunctions(220)) - Registering function indicfmt com.teradata.dynaload.hcatalog.hiveudf.TDIndicRowTbl 2017-10-02 18:23:45,170 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created local directory: /data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/54c056c8-6309-438d-ab2a-936b1bbfd978_resources 2017-10-02 18:23:45,185 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: /tmp/hive/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978 2017-10-02 18:23:45,186 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created local directory: /data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978 2017-10-02 18:23:45,194 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: /tmp/hive/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978/_tmp_space.db 2017-10-02 18:23:45,198 INFO [Driver] session.SessionState (SessionState.java:start(557)) - No Tez session required at this point. hive.execution.engine=mr. 2017-10-02 18:23:45,411 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x13d283f7 connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 2017-10-02 18:23:45,412 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x13d283f70x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:45,414 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-3/39.7.48.3:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:45,414 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:57350, server: namenode-230-3/39.7.48.3:2181 2017-10-02 18:23:45,420 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-3/39.7.48.3:2181, sessionid = 0x35ece8a34bb9325, negotiated timeout = 60000 2017-10-02 18:23:45,421 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 2017-10-02 18:23:45,421 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:45,423 INFO [Driver-SendThread(namenode-230-8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-8/39.7.48.8:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:45,436 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x25ece8a288c91e5 closed 2017-10-02 18:23:45,437 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x35ece8a34bb9325 2017-10-02 18:23:45,437 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:45,442 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x35ece8a34bb9325 closed 2017-10-02 18:23:45,442 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:45,449 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_2 stored as values in memory (estimated size 358.7 KB, free 753.1 KB) 2017-10-02 18:23:45,480 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_2_piece0 stored as bytes in memory (estimated size 32.0 KB, free 785.1 KB) 2017-10-02 18:23:45,481 INFO [dispatcher-event-loop-14] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_2_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB) 2017-10-02 18:23:45,482 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 2 from broadcast at HBaseContext.scala:71 2017-10-02 18:23:45,483 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_3 stored as values in memory (estimated size 2.9 KB, free 788.0 KB) 2017-10-02 18:23:45,489 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_3_piece0 stored as bytes in memory (estimated size 789.0 B, free 788.8 KB) 2017-10-02 18:23:45,489 INFO [dispatcher-event-loop-15] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_3_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB) 2017-10-02 18:23:45,490 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 3 from broadcast at HBaseContext.scala:72 2017-10-02 18:23:45,556 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x12ec67c connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 2017-10-02 18:23:45,556 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x12ec67c0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:45,558 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-2/39.7.48.2:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:45,564 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 2017-10-02 18:23:45,564 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase 2017-10-02 18:23:45,565 INFO [Driver-SendThread(namenode-230-8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-8/39.7.48.8:2181. Will not attempt to authenticate using SASL (unknown error) 2017-10-02 18:23:45,716 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x25ece8a288c91e6 closed 2017-10-02 18:23:45,717 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:45,717 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x15ece8a28d39374 2017-10-02 18:23:45,722 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x15ece8a28d39374 closed 2017-10-02 18:23:45,722 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down 2017-10-02 18:23:45,730 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_4 stored as values in memory (estimated size 358.7 KB, free 1147.4 KB) 2017-10-02 18:23:45,758 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_4_piece0 stored as bytes in memory (estimated size 32.0 KB, free 1179.4 KB) 2017-10-02 18:23:45,759 INFO [dispatcher-event-loop-16] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_4_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB) 2017-10-02 18:23:45,760 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 4 from broadcast at HBaseContext.scala:71 2017-10-02 18:23:45,761 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_5 stored as values in memory (estimated size 2.9 KB, free 1182.3 KB) 2017-10-02 18:23:45,768 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_5_piece0 stored as bytes in memory (estimated size 789.0 B, free 1183.1 KB) 2017-10-02 18:23:45,768 INFO [dispatcher-event-loop-17] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_5_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB) 2017-10-02 18:23:45,769 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 5 from broadcast at HBaseContext.scala:72 2017-10-02 18:23:45,776 INFO [Driver] properties (LoadedProperties.scala:fromFile(53)) - LOOKING FOR ingestion.conf FOR ENVIRONMENT: not set 2017-10-02 18:23:45,830 INFO [Driver] ingestion (ExampleUtils.scala:apply(204)) - File pattern matched for test_20170927_201709282324.csv.gz 2017-10-02 18:23:45,830 INFO [Driver] ingestion (ExampleUtils.scala:apply(204)) - File pattern matched for test_20170928_201709291238.csv.gz 2017-10-02 18:23:46,228 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_6 stored as values in memory (estimated size 277.8 KB, free 1460.9 KB) 2017-10-02 18:23:46,247 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_6_piece0 stored as bytes in memory (estimated size 25.9 KB, free 1486.8 KB) 2017-10-02 18:23:46,247 INFO [dispatcher-event-loop-18] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_6_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB) 2017-10-02 18:23:46,248 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 6 from textFile at TextFile.scala:30 2017-10-02 18:23:46,450 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1 2017-10-02 18:23:46,463 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004 2017-10-02 18:23:46,500 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: isEmpty at ExampleUtils.scala:117 2017-10-02 18:23:46,516 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 0 (isEmpty at ExampleUtils.scala:117) with 1 output partitions ...
2017-10-02 18:23:46,552 INFO [dispatcher-event-loop-19] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_7_piece0 in memory on localhost:55535 (size: 4.4 KB, free: 982.9 MB) 2017-10-02 18:23:46,553 INFO [dag-scheduler-event-loop] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 7 from broadcast at DAGScheduler.scala:1006 ...
2017-10-02 18:23:46,594 INFO [dispatcher-event-loop-20] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0,ANY, 2206 bytes) 2017-10-02 18:23:46,600 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 0.0 (TID 0) 2017-10-02 18:23:46,609 INFO [SparkListenerBus] spark.ExecutorAllocationManager (Logging.scala:logInfo(58)) - New executor driver has registered (new total is 1) 2017-10-02 18:23:46,628 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652 2017-10-02 18:23:46,637 INFO [Executor task launch worker-0] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1174)) - mapred.tip.id is deprecated. Instead, use mapreduce.task.id ...
2017-10-02 18:23:46,770 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 0.0 (TID 0). 3075 bytes result sent to driver 2017-10-02 18:23:46,791 INFO [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 0.0 (TID 0) in 207 ms on localhost (executor driver) (1/1) 2017-10-02 18:23:46,793 INFO [task-result-getter-0] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 0.0, whose tasks have all completed, from pool 2017-10-02 18:23:46,795 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 0 (isEmpty at ExampleUtils.scala:117) finished in 0.227 s 2017-10-02 18:23:46,802 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: isEmpty at ExampleUtils.scala:117, took 0.301981 s 2017-10-02 18:23:46,841 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_8 stored as values in memory (estimated size 277.9 KB, free 1777.3 KB) 2017-10-02 18:23:46,857 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_8_piece0 stored as bytes in memory (estimated size 25.9 KB, free 1803.2 KB) 2017-10-02 18:23:46,858 INFO [dispatcher-event-loop-1] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_8_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB) 2017-10-02 18:23:46,858 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 8 from textFile at TextFile.scala:30 2017-10-02 18:23:46,930 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1 2017-10-02 18:23:46,936 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.15:1004 2017-10-02 18:23:46,940 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: head at ExampleUtils.scala:118 2017-10-02 18:23:46,942 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 1 (head at ExampleUtils.scala:118) with 1 output partitions ...
2017-10-02 18:23:46,954 INFO [dispatcher-event-loop-2] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_9_piece0 in memory on localhost:55535 (size: 4.2 KB, free: 982.9 MB) 2017-10-02 18:23:46,955 INFO [dag-scheduler-event-loop] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Adding task set 1.0 with 1 tasks 2017-10-02 18:23:46,956 INFO [dispatcher-event-loop-3] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0,ANY, 2206 bytes) 2017-10-02 18:23:46,956 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 1.0 (TID 1) 2017-10-02 18:23:46,965 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652 2017-10-02 18:23:46,996 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 1.0 (TID 1). 2571 bytes result sent to driver 2017-10-02 18:23:47,003 INFO [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 1.0 (TID 1) in 47 ms on localhost (executor driver) (1/1) 2017-10-02 18:23:47,003 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 1 (head at ExampleUtils.scala:118) finished in 0.048 s 2017-10-02 18:23:47,003 INFO [task-result-getter-1] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 1.0, whose tasks have all completed, from pool 2017-10-02 18:23:47,004 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 1 finished: head at ExampleUtils.scala:118, took 0.063267 s ..... 2017-10-02 18:23:47,079 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_10 stored as values in memory (estimated size 277.9 KB, free 2.0 MB) 2017-10-02 18:23:47,095 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_10_piece0 stored as bytes in memory (estimated size 25.9 KB, free 2.1 MB) 2017-10-02 18:23:47,095 INFO [dispatcher-event-loop-7] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_10_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB) 2017-10-02 18:23:47,096 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 10 from textFile at TextFile.scala:30 2017-10-02 18:23:47,157 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1 2017-10-02 18:23:47,160 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004 2017-10-02 18:23:47,160 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.17:1004 2017-10-02 18:23:47,183 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: collect at ExampleUtils.scala:320 2017-10-02 18:23:47,185 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Missing parents: List() 2017-10-02 18:23:47,200 INFO [dispatcher-event-loop-9] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0,ANY, 2206 bytes) 2017-10-02 18:23:47,200 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 2.0 (TID 2) 2017-10-02 18:23:47,211 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652 2017-10-02 18:23:47,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ... 2017-10-02 18:23:47,442 INFO [Executor task launch worker-0] codegen.GeneratePredicate (Logging.scala:logInfo(58)) - Code generated in 197.295072 ms 2017-10-02 18:23:47,774 INFO [dispatcher-event-loop-13] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_9_piece0 on localhost:55535 in memory (size: 4.2 KB, free: 982.9 MB) 2017-10-02 18:23:47,778 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(58)) - Cleaned accumulator 2 2017-10-02 18:23:47,779 INFO [dispatcher-event-loop-16] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_8_piece0 on localhost:55535 in memory (size: 25.9 KB, free: 982.9 MB) 2017-10-02 18:23:47,780 INFO [dispatcher-event-loop-19] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_7_piece0 on localhost:55535 in memory (size: 4.4 KB, free: 982.9 MB) 2017-10-02 18:23:47,781 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(58)) - Cleaned accumulator 1 2017-10-02 18:23:57,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ... 2017-10-02 18:24:07,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ... ...
2017-10-02 18:32:35,305 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 2.0 (TID 2). 2694 bytes result sent to driver 2017-10-02 18:32:35,315 INFO [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 2.0 (TID 2) in 528116 ms on localhost (executor driver) (1/1) 2017-10-02 18:32:35,315 INFO [task-result-getter-2] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 2.0, whose tasks have all completed, from pool 2017-10-02 18:32:35,316 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 2 (collect at ExampleUtils.scala:320) finished in 528.117 s 2017-10-02 18:32:35,317 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 2 finished: collect at ExampleUtils.scala:320, took 528.133409 s 2017-10-02 18:32:35,319 INFO [Driver] ingestion (ExampleUtils.scala:rowCountValidation(322)) - Trailer record value: 72304806 2017-10-02 18:32:35,350 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_12 stored as values in memory (estimated size 277.9 KB, free 2.0 MB) 2017-10-02 18:32:35,364 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_12_piece0 stored as bytes in memory (estimated size 25.9 KB, free 2.1 MB) 2017-10-02 18:32:35,365 INFO [dispatcher-event-loop-15] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_12_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB) 2017-10-02 18:32:35,365 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 12 from textFile at TextFile.scala:30 2017-10-02 18:32:35,468 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1 2017-10-02 18:32:35,471 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004 ...
2017-10-02 18:32:35,505 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: count at ExampleUtils.scala:324 2017-10-02 18:32:35,509 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Registering RDD 25 (count at ExampleUtils.scala:324) ...
2017-10-02 18:32:35,519 INFO [dag-scheduler-event-loop] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_13_piece0 stored as bytes in memory (estimated size 7.4 KB, free 2.1 MB) 2017-10-02 18:32:35,520 INFO [dispatcher-event-loop-14] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_13_piece0 in memory on localhost:55535 (size: 7.4 KB, free: 982.9 MB) 2017-10-02 18:32:35,520 INFO [dag-scheduler-event-loop] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 13 from broadcast at DAGScheduler.scala:1006 2017-10-02 18:32:35,524 INFO [dispatcher-event-loop-16] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0,ANY, 2195 bytes) 2017-10-02 18:32:35,525 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 3.0 (TID 3) 2017-10-02 18:32:35,651 INFO [Executor task launch worker-0] codegen.GenerateUnsafeProjection (Logging.scala:logInfo(58)) - Code generated in 9.558917 ms 2017-10-02 18:32:37,442 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ... 2017-10-02 18:35:37,458 INFO [Thread-5] spark.SparkContext (Logging.scala:logInfo(58)) - Invoking stop() from shutdown hook 2017-10-02 18:35:37,526 INFO [Thread-5] handler.ContextHandler (ContextHandler.java:doStop(843)) - stopped o.s.j.s.ServletContextHandler{/static/sql,null} 2017-10-02 18:35:37,587 INFO [Thread-5] ui.SparkUI (Logging.scala:logInfo(58)) - Stopped Spark web UI at http://xxx.xxx.xxx.xx:45099 2017-10-02 18:35:37,630 INFO [Thread-5] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ShuffleMapStage 3 (count at ExampleUtils.scala:324) failed in 182.106 s due to Stage cancelled because SparkContext was shut down 2017-10-02 18:35:37,631 ERROR [Thread-5] scheduler.LiveListenerBus (Logging.scala:logError(74)) - SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@5e8b246) 2017-10-02 18:35:37,631 ERROR [Thread-5] scheduler.LiveListenerBus (Logging.scala:logError(74)) - SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(3,1506962137631,JobFailed(org.apache.spark.SparkException: Job 3 cancelled because SparkContext was shut down)) 2017-10-02 18:35:37,635 INFO [dispatcher-event-loop-13] spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(58)) - MapOutputTrackerMasterEndpoint stopped! 2017-10-02 18:35:37,643 INFO [Thread-5] storage.MemoryStore (Logging.scala:logInfo(58)) - MemoryStore cleared 2017-10-02 18:35:37,644 INFO [Thread-5] storage.BlockManager (Logging.scala:logInfo(58)) - BlockManager stopped 2017-10-02 18:35:37,645 INFO [Thread-5] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - BlockManagerMaster stopped 2017-10-02 18:35:37,648 INFO [dispatcher-event-loop-17] scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint (Logging.scala:logInfo(58)) - OutputCommitCoordinator stopped! 2017-10-02 18:35:37,651 INFO [Thread-5] spark.SparkContext (Logging.scala:logInfo(58)) - Successfully stopped SparkContext 2017-10-02 18:35:37,654 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data9/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-79c559be-e744-4a2f-85fe-f224c10969c0 2017-10-02 18:35:37,655 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-4] remote.RemoteActorRefProvider$RemotingTerminator (Slf4jLogger.scala:apply$mcV$sp(74)) - Shutting down remote daemon. 2017-10-02 18:35:37,655 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data3/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-0cdba60a-e381-4149-ade1-567b98502fe5 2017-10-02 18:35:37,656 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data2/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-5a06a64d-3db7-4b56-a075-12657a0aa432 2017-10-02 18:35:37,658 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-4] remote.RemoteActorRefProvider$RemotingTerminator (Slf4jLogger.scala:apply$mcV$sp(74)) - Remote daemon shut down; proceeding with flushing remote transports. 2017-10-02 18:35:37,716 ERROR [Executor task launch worker-0] util.Utils (Logging.scala:logError(95)) - Uncaught exception in thread Executor task launch worker-0 java.lang.NullPointerException at org.apache.spark.scheduler.Task$$anonfun$run$1.apply$mcV$sp(Task.scala:95) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1230) at org.apache.spark.scheduler.Task.run(Task.scala:93) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-10-02 18:35:37,718 WARN [Executor task launch worker-0] memory.TaskMemoryManager (TaskMemoryManager.java:cleanUpAllAllocatedMemory(386)) - leak 32.3 MB memory from org.apache.spark.unsafe.map.BytesToBytesMap@5b7017a5 2017-10-02 18:35:37,719 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(74)) - Managed memory leak detected; size = 33816576 bytes, TID = 3 2017-10-02 18:35:37,719 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(95)) - Exception in task 0.0 in stage 3.0 (TID 3) java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:857) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:875) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179) at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163) at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105) at java.io.InputStream.read(InputStream.java:101) at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180) at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174) at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:248) at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:504) at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686) at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95) at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-10-02 18:35:37,739 WARN [Executor task launch worker-0] netty.NettyRpcEnv (Logging.scala:logWarning(70)) - RpcEnv already stopped.

 

Don't have an account?
Coming from Hortonworks? Activate your account here