Member since
03-02-2017
31
Posts
1
Kudos Received
6
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3058 | 09-08-2017 05:49 AM | |
| 1616 | 06-05-2017 04:29 AM | |
| 2013 | 06-05-2017 04:24 AM | |
| 2207 | 03-21-2017 02:23 AM | |
| 1601 | 03-11-2017 06:18 PM |
04-10-2018
02:10 AM
Hi The issue was related to customized environment and it is fixed now. regards Shafi
... View more
10-05-2017
01:49 AM
Dear All, Spark on yarn running in cluster mode is unable to perform wider transformations. are there any guidelines on memory optimization and tuning Attaching the logs as well. I am facing below errors at different times of execution SparkContext did not initialize after waiting for 720000 ms. org.apache.spark.SparkException: Job 3 cancelled because SparkContext was shut down java.io.IOException: Filesystem closed Caused by: java.lang.InterruptedException 2017-10-02 18:23:35,624 INFO [main] yarn.ApplicationMaster (SignalLogger.scala:register(47)) - Registered signal handlers for [TERM, HUP, INT]
2017-10-02 18:23:36,661 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - ApplicationAttemptId: appattempt_1506703761089_3300_000001
2017-10-02 18:23:37,349 INFO [main] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing view acls to: xxxx
2017-10-02 18:23:37,350 INFO [main] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing modify acls to: xxxx
2017-10-02 18:23:37,350 INFO [main] spark.SecurityManager (Logging.scala:logInfo(58)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xxxx); users with modify permissions: Set(xxxx)
2017-10-02 18:23:37,372 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Starting the user application in a separate Thread
2017-10-02 18:23:37,420 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization
2017-10-02 18:23:37,420 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ...
2017-10-02 18:23:40,030 INFO [Driver] commons.Environment$ (Environment.scala:parseEnv(21)) - yarnHS is:
2017-10-02 18:23:40,037 INFO [Driver] commons.Environment$ (Environment.scala:master(42)) - Master is: xx
2017-10-02 18:23:40,048 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Running Spark version 1.6.0
2017-10-02 18:23:40,078 INFO [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing view acls to: xxxx
2017-10-02 18:23:40,078 INFO [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - Changing modify acls to: xxxx
2017-10-02 18:23:40,078 INFO [Driver] spark.SecurityManager (Logging.scala:logInfo(58)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(xxxx); users with modify permissions: Set(xxxx)
2017-10-02 18:23:40,248 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'sparkDriver' on port 54663.
2017-10-02 18:23:40,472 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-3] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started
2017-10-02 18:23:40,515 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting
2017-10-02 18:23:40,652 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@namenode:55316]
2017-10-02 18:23:40,654 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-3] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@3namenode55316]
2017-10-02 18:23:40,661 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'sparkDriverActorSystem' on port 55316.
2017-10-02 18:23:40,680 INFO [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering MapOutputTracker
2017-10-02 18:23:40,698 INFO [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering BlockManagerMaster
... 2017-10-02 18:23:40,715 INFO [Driver] storage.DiskBlockManager (Logging.scala:logInfo(58)) - Created local directory at /data9/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/blockmgr-7df05782-ba29-42cd-8825-0e9c2640e045
2017-10-02 18:23:40,721 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - MemoryStore started with capacity 983.1 MB
2017-10-02 18:23:40,885 INFO [Driver] spark.SparkEnv (Logging.scala:logInfo(58)) - Registering OutputCommitCoordinator
2017-10-02 18:23:41,045 INFO [Driver] ui.JettyUtils (Logging.scala:logInfo(58)) - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2017-10-02 18:23:41,071 INFO [Driver] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT
2017-10-02 18:23:41,115 INFO [Driver] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:45099
2017-10-02 18:23:41,116 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'SparkUI' on port 45099.
2017-10-02 18:23:41,118 INFO [Driver] ui.SparkUI (Logging.scala:logInfo(58)) - Started SparkUI at http://namenode:45099
2017-10-02 18:23:41,238 INFO [Driver] executor.Executor (Logging.scala:logInfo(58)) - Starting executor ID driver on host localhost
2017-10-02 18:23:41,277 INFO [Driver] util.Utils (Logging.scala:logInfo(58)) - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55535.
2017-10-02 18:23:41,278 INFO [Driver] netty.NettyBlockTransferService (Logging.scala:logInfo(58)) - Server created on 55535
2017-10-02 18:23:41,279 INFO [Driver] storage.BlockManager (Logging.scala:logInfo(58)) - external shuffle service port = 7337
2017-10-02 18:23:41,280 INFO [Driver] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - Trying to register BlockManager
2017-10-02 18:23:41,282 INFO [dispatcher-event-loop-10] storage.BlockManagerMasterEndpoint (Logging.scala:logInfo(58)) - Registering block manager localhost:55535 with 983.1 MB RAM, BlockManagerId(driver, localhost, 55535)
2017-10-02 18:23:41,284 INFO [Driver] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - Registered BlockManager
2017-10-02 18:23:41,496 WARN [Driver] security.UserGroupInformation (UserGroupInformation.java:doAs(1701)) - PriviledgedActionException as:xxxx (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
2017-10-02 18:23:41,497 WARN [Driver] ipc.Client (Client.java:run(682)) - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
2017-10-02 18:23:41,497 WARN [Driver] security.UserGroupInformation (UserGroupInformation.java:doAs(1701)) - PriviledgedActionException as:xxxx (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
2017-10-02 18:23:41,700 INFO [Driver] scheduler.EventLoggingListener (Logging.scala:logInfo(58)) - Logging events to hdfs://namenode/user/spark/applicationHistory/local-1506961421183
2017-10-02 18:23:41,741 INFO [Driver] properties (LoadedProperties.scala:fromFile(53)) - LOOKING FOR test.conf FOR ENVIRONMENT: xxx
2017-10-02 18:23:41,851 INFO [Driver] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1174)) - mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
2017-10-02 18:23:41,964 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x7c1651ef connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
... 2017-10-02 18:23:41,976 INFO [Driver] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/xxxx
2017-10-02 18:23:41,976 INFO [Driver] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001
2017-10-02 18:23:41,977 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x7c1651ef0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:41,992 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-2/39.7.48.2:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:41,993 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:38451, server: namenode-230-2/39.7.48.2:2181
2017-10-02 18:23:42,005 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-2/39.7.48.2:2181, sessionid = 0x15ece8a28d39372, negotiated timeout = 60000
2017-10-02 18:23:42,077 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
2017-10-02 18:23:42,077 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:42,079 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-3/39.7.48.3:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:42,079 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:57307, server: namenode-230-3/39.7.48.3:2181
2017-10-02 18:23:42,093 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-3/39.7.48.3:2181, sessionid = 0x35ece8a34bb9324, negotiated timeout = 60000
2017-10-02 18:23:42,100 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x35ece8a34bb9324 closed
2017-10-02 18:23:42,100 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:42,100 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x15ece8a28d39372
2017-10-02 18:23:42,105 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:42,105 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x15ece8a28d39372 closed
2017-10-02 18:23:42,247 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_0 stored as values in memory (estimated size 358.7 KB, free 358.7 KB)
2017-10-02 18:23:42,441 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_0_piece0 stored as bytes in memory (estimated size 32.0 KB, free 390.7 KB)
2017-10-02 18:23:42,444 INFO [dispatcher-event-loop-12] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_0_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB)
2017-10-02 18:23:42,447 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 0 from broadcast at HBaseContext.scala:71
2017-10-02 18:23:42,449 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 393.6 KB)
2017-10-02 18:23:42,458 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_1_piece0 stored as bytes in memory (estimated size 789.0 B, free 394.4 KB)
2017-10-02 18:23:42,459 INFO [dispatcher-event-loop-13] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_1_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB)
2017-10-02 18:23:42,460 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 1 from broadcast at HBaseContext.scala:72
2017-10-02 18:23:43,216 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - Initializing execution hive, version 1.1.0
2017-10-02 18:23:43,272 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Inspected Hadoop version: 2.6.0-cdh5.9.0
2017-10-02 18:23:43,273 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.9.0
2017-10-02 18:23:44,027 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(386)) - Trying to connect to metastore with URI thrift://namenode-230-2:9083
2017-10-02 18:23:44,057 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(431)) - Opened a connection to metastore, current connections: 1
2017-10-02 18:23:44,057 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(483)) - Connected to metastore.
2017-10-02 18:23:44,140 INFO [Driver] metadata.Hive (Hive.java:reloadFunctions(220)) - Registering function indicfmt com.teradata.dynaload.hcatalog.hiveudf.TDIndicRowTbl
2017-10-02 18:23:44,188 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: file:/data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/spark-d24114da-2c76-463d-a55d-08017491fd15/scratch/xxxx
... 2017-10-02 18:23:44,194 INFO [Driver] session.SessionState (SessionState.java:start(557)) - No Tez session required at this point. hive.execution.engine=mr.
2017-10-02 18:23:44,289 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - default warehouse location is /user/hive/warehouse
2017-10-02 18:23:44,298 INFO [Driver] hive.HiveContext (Logging.scala:logInfo(58)) - Initializing metastore client version 1.1.0 using Spark classes.
2017-10-02 18:23:44,316 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Inspected Hadoop version: 2.6.0-cdh5.9.0
2017-10-02 18:23:44,327 INFO [Driver] client.ClientWrapper (Logging.scala:logInfo(58)) - Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.9.0
2017-10-02 18:23:45,061 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(386)) - Trying to connect to metastore with URI thrift://namenode-230-2:9083
2017-10-02 18:23:45,089 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(431)) - Opened a connection to metastore, current connections: 1
2017-10-02 18:23:45,089 INFO [Driver] hive.metastore (HiveMetaStoreClient.java:open(483)) - Connected to metastore.
2017-10-02 18:23:45,163 INFO [Driver] metadata.Hive (Hive.java:reloadFunctions(220)) - Registering function indicfmt com.teradata.dynaload.hcatalog.hiveudf.TDIndicRowTbl
2017-10-02 18:23:45,170 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created local directory: /data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/54c056c8-6309-438d-ab2a-936b1bbfd978_resources
2017-10-02 18:23:45,185 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: /tmp/hive/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978
2017-10-02 18:23:45,186 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created local directory: /data10/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/container_e152_1506703761089_3300_01_000001/tmp/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978
2017-10-02 18:23:45,194 INFO [Driver] session.SessionState (SessionState.java:createPath(655)) - Created HDFS directory: /tmp/hive/xxxx/54c056c8-6309-438d-ab2a-936b1bbfd978/_tmp_space.db
2017-10-02 18:23:45,198 INFO [Driver] session.SessionState (SessionState.java:start(557)) - No Tez session required at this point. hive.execution.engine=mr.
2017-10-02 18:23:45,411 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x13d283f7 connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
2017-10-02 18:23:45,412 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x13d283f70x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:45,414 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-3/39.7.48.3:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:45,414 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established, initiating session, client: /namenode:57350, server: namenode-230-3/39.7.48.3:2181
2017-10-02 18:23:45,420 INFO [Driver-SendThread(namenode-230-3:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server namenode-230-3/39.7.48.3:2181, sessionid = 0x35ece8a34bb9325, negotiated timeout = 60000
2017-10-02 18:23:45,421 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
2017-10-02 18:23:45,421 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:45,423 INFO [Driver-SendThread(namenode-230-8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-8/39.7.48.8:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:45,436 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x25ece8a288c91e5 closed
2017-10-02 18:23:45,437 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x35ece8a34bb9325
2017-10-02 18:23:45,437 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:45,442 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x35ece8a34bb9325 closed
2017-10-02 18:23:45,442 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:45,449 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_2 stored as values in memory (estimated size 358.7 KB, free 753.1 KB)
2017-10-02 18:23:45,480 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_2_piece0 stored as bytes in memory (estimated size 32.0 KB, free 785.1 KB)
2017-10-02 18:23:45,481 INFO [dispatcher-event-loop-14] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_2_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB)
2017-10-02 18:23:45,482 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 2 from broadcast at HBaseContext.scala:71
2017-10-02 18:23:45,483 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_3 stored as values in memory (estimated size 2.9 KB, free 788.0 KB)
2017-10-02 18:23:45,489 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_3_piece0 stored as bytes in memory (estimated size 789.0 B, free 788.8 KB)
2017-10-02 18:23:45,489 INFO [dispatcher-event-loop-15] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_3_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB)
2017-10-02 18:23:45,490 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 3 from broadcast at HBaseContext.scala:72
2017-10-02 18:23:45,556 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=hconnection-0x12ec67c connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
2017-10-02 18:23:45,556 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=hconnection-0x12ec67c0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:45,558 INFO [Driver-SendThread(namenode-230-2:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-2/39.7.48.2:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:45,564 INFO [Driver] zookeeper.RecoverableZooKeeper (RecoverableZooKeeper.java:<init>(120)) - Process identifier=TokenUtil-getAuthToken connecting to ZooKeeper ensemble=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181
2017-10-02 18:23:45,564 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181 sessionTimeout=60000 watcher=TokenUtil-getAuthToken0x0, quorum=namenode-230-2:2181,namenode-230-8:2181,namenode-230-3:2181, baseZNode=/hbase
2017-10-02 18:23:45,565 INFO [Driver-SendThread(namenode-230-8:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server namenode-230-8/39.7.48.8:2181. Will not attempt to authenticate using SASL (unknown error)
2017-10-02 18:23:45,716 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x25ece8a288c91e6 closed
2017-10-02 18:23:45,717 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:45,717 INFO [Driver] client.ConnectionManager$HConnectionImplementation (ConnectionManager.java:closeZooKeeperWatcher(1830)) - Closing zookeeper sessionid=0x15ece8a28d39374
2017-10-02 18:23:45,722 INFO [Driver] zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x15ece8a28d39374 closed
2017-10-02 18:23:45,722 INFO [Driver-EventThread] zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2017-10-02 18:23:45,730 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_4 stored as values in memory (estimated size 358.7 KB, free 1147.4 KB)
2017-10-02 18:23:45,758 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_4_piece0 stored as bytes in memory (estimated size 32.0 KB, free 1179.4 KB)
2017-10-02 18:23:45,759 INFO [dispatcher-event-loop-16] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_4_piece0 in memory on localhost:55535 (size: 32.0 KB, free: 983.0 MB)
2017-10-02 18:23:45,760 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 4 from broadcast at HBaseContext.scala:71
2017-10-02 18:23:45,761 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_5 stored as values in memory (estimated size 2.9 KB, free 1182.3 KB)
2017-10-02 18:23:45,768 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_5_piece0 stored as bytes in memory (estimated size 789.0 B, free 1183.1 KB)
2017-10-02 18:23:45,768 INFO [dispatcher-event-loop-17] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_5_piece0 in memory on localhost:55535 (size: 789.0 B, free: 983.0 MB)
2017-10-02 18:23:45,769 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 5 from broadcast at HBaseContext.scala:72
2017-10-02 18:23:45,776 INFO [Driver] properties (LoadedProperties.scala:fromFile(53)) - LOOKING FOR ingestion.conf FOR ENVIRONMENT: not set
2017-10-02 18:23:45,830 INFO [Driver] ingestion (ExampleUtils.scala:apply(204)) - File pattern matched for test_20170927_201709282324.csv.gz
2017-10-02 18:23:45,830 INFO [Driver] ingestion (ExampleUtils.scala:apply(204)) - File pattern matched for test_20170928_201709291238.csv.gz
2017-10-02 18:23:46,228 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_6 stored as values in memory (estimated size 277.8 KB, free 1460.9 KB)
2017-10-02 18:23:46,247 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_6_piece0 stored as bytes in memory (estimated size 25.9 KB, free 1486.8 KB)
2017-10-02 18:23:46,247 INFO [dispatcher-event-loop-18] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_6_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB)
2017-10-02 18:23:46,248 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 6 from textFile at TextFile.scala:30
2017-10-02 18:23:46,450 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1
2017-10-02 18:23:46,463 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004
2017-10-02 18:23:46,500 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: isEmpty at ExampleUtils.scala:117
2017-10-02 18:23:46,516 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 0 (isEmpty at ExampleUtils.scala:117) with 1 output partitions
... 2017-10-02 18:23:46,552 INFO [dispatcher-event-loop-19] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_7_piece0 in memory on localhost:55535 (size: 4.4 KB, free: 982.9 MB)
2017-10-02 18:23:46,553 INFO [dag-scheduler-event-loop] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 7 from broadcast at DAGScheduler.scala:1006
... 2017-10-02 18:23:46,594 INFO [dispatcher-event-loop-20] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0,ANY, 2206 bytes)
2017-10-02 18:23:46,600 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 0.0 (TID 0)
2017-10-02 18:23:46,609 INFO [SparkListenerBus] spark.ExecutorAllocationManager (Logging.scala:logInfo(58)) - New executor driver has registered (new total is 1)
2017-10-02 18:23:46,628 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652
2017-10-02 18:23:46,637 INFO [Executor task launch worker-0] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1174)) - mapred.tip.id is deprecated. Instead, use mapreduce.task.id
... 2017-10-02 18:23:46,770 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 0.0 (TID 0). 3075 bytes result sent to driver
2017-10-02 18:23:46,791 INFO [task-result-getter-0] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 0.0 (TID 0) in 207 ms on localhost (executor driver) (1/1)
2017-10-02 18:23:46,793 INFO [task-result-getter-0] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 0.0, whose tasks have all completed, from pool
2017-10-02 18:23:46,795 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 0 (isEmpty at ExampleUtils.scala:117) finished in 0.227 s
2017-10-02 18:23:46,802 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: isEmpty at ExampleUtils.scala:117, took 0.301981 s
2017-10-02 18:23:46,841 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_8 stored as values in memory (estimated size 277.9 KB, free 1777.3 KB)
2017-10-02 18:23:46,857 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_8_piece0 stored as bytes in memory (estimated size 25.9 KB, free 1803.2 KB)
2017-10-02 18:23:46,858 INFO [dispatcher-event-loop-1] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_8_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB)
2017-10-02 18:23:46,858 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 8 from textFile at TextFile.scala:30
2017-10-02 18:23:46,930 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1
2017-10-02 18:23:46,936 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.15:1004
2017-10-02 18:23:46,940 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: head at ExampleUtils.scala:118
2017-10-02 18:23:46,942 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 1 (head at ExampleUtils.scala:118) with 1 output partitions
... 2017-10-02 18:23:46,954 INFO [dispatcher-event-loop-2] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_9_piece0 in memory on localhost:55535 (size: 4.2 KB, free: 982.9 MB)
2017-10-02 18:23:46,955 INFO [dag-scheduler-event-loop] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Adding task set 1.0 with 1 tasks
2017-10-02 18:23:46,956 INFO [dispatcher-event-loop-3] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0,ANY, 2206 bytes)
2017-10-02 18:23:46,956 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 1.0 (TID 1)
2017-10-02 18:23:46,965 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652
2017-10-02 18:23:46,996 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 1.0 (TID 1). 2571 bytes result sent to driver
2017-10-02 18:23:47,003 INFO [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 1.0 (TID 1) in 47 ms on localhost (executor driver) (1/1)
2017-10-02 18:23:47,003 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 1 (head at ExampleUtils.scala:118) finished in 0.048 s
2017-10-02 18:23:47,003 INFO [task-result-getter-1] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 1.0, whose tasks have all completed, from pool
2017-10-02 18:23:47,004 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 1 finished: head at ExampleUtils.scala:118, took 0.063267 s
.....
2017-10-02 18:23:47,079 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_10 stored as values in memory (estimated size 277.9 KB, free 2.0 MB)
2017-10-02 18:23:47,095 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_10_piece0 stored as bytes in memory (estimated size 25.9 KB, free 2.1 MB)
2017-10-02 18:23:47,095 INFO [dispatcher-event-loop-7] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_10_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB)
2017-10-02 18:23:47,096 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 10 from textFile at TextFile.scala:30
2017-10-02 18:23:47,157 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1
2017-10-02 18:23:47,160 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004
2017-10-02 18:23:47,160 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.17:1004
2017-10-02 18:23:47,183 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: collect at ExampleUtils.scala:320
2017-10-02 18:23:47,185 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Missing parents: List()
2017-10-02 18:23:47,200 INFO [dispatcher-event-loop-9] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0,ANY, 2206 bytes)
2017-10-02 18:23:47,200 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 2.0 (TID 2)
2017-10-02 18:23:47,211 INFO [Executor task launch worker-0] rdd.HadoopRDD (Logging.scala:logInfo(58)) - Input split: hdfs://namenode/test_20170927_201709282324.csv.gz:0+333448652
2017-10-02 18:23:47,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ...
2017-10-02 18:23:47,442 INFO [Executor task launch worker-0] codegen.GeneratePredicate (Logging.scala:logInfo(58)) - Code generated in 197.295072 ms
2017-10-02 18:23:47,774 INFO [dispatcher-event-loop-13] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_9_piece0 on localhost:55535 in memory (size: 4.2 KB, free: 982.9 MB)
2017-10-02 18:23:47,778 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(58)) - Cleaned accumulator 2
2017-10-02 18:23:47,779 INFO [dispatcher-event-loop-16] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_8_piece0 on localhost:55535 in memory (size: 25.9 KB, free: 982.9 MB)
2017-10-02 18:23:47,780 INFO [dispatcher-event-loop-19] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Removed broadcast_7_piece0 on localhost:55535 in memory (size: 4.4 KB, free: 982.9 MB)
2017-10-02 18:23:47,781 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(58)) - Cleaned accumulator 1
2017-10-02 18:23:57,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ...
2017-10-02 18:24:07,421 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ...
... 2017-10-02 18:32:35,305 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 2.0 (TID 2). 2694 bytes result sent to driver
2017-10-02 18:32:35,315 INFO [task-result-getter-2] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished task 0.0 in stage 2.0 (TID 2) in 528116 ms on localhost (executor driver) (1/1)
2017-10-02 18:32:35,315 INFO [task-result-getter-2] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 2.0, whose tasks have all completed, from pool
2017-10-02 18:32:35,316 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ResultStage 2 (collect at ExampleUtils.scala:320) finished in 528.117 s
2017-10-02 18:32:35,317 INFO [Driver] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 2 finished: collect at ExampleUtils.scala:320, took 528.133409 s
2017-10-02 18:32:35,319 INFO [Driver] ingestion (ExampleUtils.scala:rowCountValidation(322)) - Trailer record value: 72304806
2017-10-02 18:32:35,350 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_12 stored as values in memory (estimated size 277.9 KB, free 2.0 MB)
2017-10-02 18:32:35,364 INFO [Driver] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_12_piece0 stored as bytes in memory (estimated size 25.9 KB, free 2.1 MB)
2017-10-02 18:32:35,365 INFO [dispatcher-event-loop-15] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_12_piece0 in memory on localhost:55535 (size: 25.9 KB, free: 982.9 MB)
2017-10-02 18:32:35,365 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 12 from textFile at TextFile.scala:30
2017-10-02 18:32:35,468 INFO [Driver] mapred.FileInputFormat (FileInputFormat.java:listStatus(249)) - Total input paths to process : 1
2017-10-02 18:32:35,471 INFO [Driver] net.NetworkTopology (NetworkTopology.java:add(426)) - Adding a new node: /default/39.7.48.18:1004
... 2017-10-02 18:32:35,505 INFO [Driver] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: count at ExampleUtils.scala:324
2017-10-02 18:32:35,509 INFO [dag-scheduler-event-loop] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Registering RDD 25 (count at ExampleUtils.scala:324)
... 2017-10-02 18:32:35,519 INFO [dag-scheduler-event-loop] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_13_piece0 stored as bytes in memory (estimated size 7.4 KB, free 2.1 MB)
2017-10-02 18:32:35,520 INFO [dispatcher-event-loop-14] storage.BlockManagerInfo (Logging.scala:logInfo(58)) - Added broadcast_13_piece0 in memory on localhost:55535 (size: 7.4 KB, free: 982.9 MB)
2017-10-02 18:32:35,520 INFO [dag-scheduler-event-loop] spark.SparkContext (Logging.scala:logInfo(58)) - Created broadcast 13 from broadcast at DAGScheduler.scala:1006
2017-10-02 18:32:35,524 INFO [dispatcher-event-loop-16] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0,ANY, 2195 bytes)
2017-10-02 18:32:35,525 INFO [Executor task launch worker-0] executor.Executor (Logging.scala:logInfo(58)) - Running task 0.0 in stage 3.0 (TID 3)
2017-10-02 18:32:35,651 INFO [Executor task launch worker-0] codegen.GenerateUnsafeProjection (Logging.scala:logInfo(58)) - Code generated in 9.558917 ms
2017-10-02 18:32:37,442 INFO [main] yarn.ApplicationMaster (Logging.scala:logInfo(58)) - Waiting for spark context initialization ...
2017-10-02 18:35:37,458 INFO [Thread-5] spark.SparkContext (Logging.scala:logInfo(58)) - Invoking stop() from shutdown hook
2017-10-02 18:35:37,526 INFO [Thread-5] handler.ContextHandler (ContextHandler.java:doStop(843)) - stopped o.s.j.s.ServletContextHandler{/static/sql,null}
2017-10-02 18:35:37,587 INFO [Thread-5] ui.SparkUI (Logging.scala:logInfo(58)) - Stopped Spark web UI at http://xxx.xxx.xxx.xx:45099
2017-10-02 18:35:37,630 INFO [Thread-5] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - ShuffleMapStage 3 (count at ExampleUtils.scala:324) failed in 182.106 s due to Stage cancelled because SparkContext was shut down
2017-10-02 18:35:37,631 ERROR [Thread-5] scheduler.LiveListenerBus (Logging.scala:logError(74)) - SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@5e8b246)
2017-10-02 18:35:37,631 ERROR [Thread-5] scheduler.LiveListenerBus (Logging.scala:logError(74)) - SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(3,1506962137631,JobFailed(org.apache.spark.SparkException: Job 3 cancelled because SparkContext was shut down))
2017-10-02 18:35:37,635 INFO [dispatcher-event-loop-13] spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(58)) - MapOutputTrackerMasterEndpoint stopped!
2017-10-02 18:35:37,643 INFO [Thread-5] storage.MemoryStore (Logging.scala:logInfo(58)) - MemoryStore cleared
2017-10-02 18:35:37,644 INFO [Thread-5] storage.BlockManager (Logging.scala:logInfo(58)) - BlockManager stopped
2017-10-02 18:35:37,645 INFO [Thread-5] storage.BlockManagerMaster (Logging.scala:logInfo(58)) - BlockManagerMaster stopped
2017-10-02 18:35:37,648 INFO [dispatcher-event-loop-17] scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint (Logging.scala:logInfo(58)) - OutputCommitCoordinator stopped!
2017-10-02 18:35:37,651 INFO [Thread-5] spark.SparkContext (Logging.scala:logInfo(58)) - Successfully stopped SparkContext
2017-10-02 18:35:37,654 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data9/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-79c559be-e744-4a2f-85fe-f224c10969c0
2017-10-02 18:35:37,655 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-4] remote.RemoteActorRefProvider$RemotingTerminator (Slf4jLogger.scala:apply$mcV$sp(74)) - Shutting down remote daemon.
2017-10-02 18:35:37,655 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data3/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-0cdba60a-e381-4149-ade1-567b98502fe5
2017-10-02 18:35:37,656 INFO [Thread-5] util.ShutdownHookManager (Logging.scala:logInfo(58)) - Deleting directory /data2/yarn/nm/usercache/xxxx/appcache/application_1506703761089_3300/spark-5a06a64d-3db7-4b56-a075-12657a0aa432
2017-10-02 18:35:37,658 INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-4] remote.RemoteActorRefProvider$RemotingTerminator (Slf4jLogger.scala:apply$mcV$sp(74)) - Remote daemon shut down; proceeding with flushing remote transports.
2017-10-02 18:35:37,716 ERROR [Executor task launch worker-0] util.Utils (Logging.scala:logError(95)) - Uncaught exception in thread Executor task launch worker-0
java.lang.NullPointerException
at org.apache.spark.scheduler.Task$$anonfun$run$1.apply$mcV$sp(Task.scala:95)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1230)
at org.apache.spark.scheduler.Task.run(Task.scala:93)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-02 18:35:37,718 WARN [Executor task launch worker-0] memory.TaskMemoryManager (TaskMemoryManager.java:cleanUpAllAllocatedMemory(386)) - leak 32.3 MB memory from org.apache.spark.unsafe.map.BytesToBytesMap@5b7017a5
2017-10-02 18:35:37,719 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(74)) - Managed memory leak detected; size = 33816576 bytes, TID = 3
2017-10-02 18:35:37,719 ERROR [Executor task launch worker-0] executor.Executor (Logging.scala:logError(95)) - Exception in task 0.0 in stage 3.0 (TID 3)
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:857)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:875)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:179)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:163)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:248)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:246)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:208)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:504)
at org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.<init>(TungstenAggregationIterator.scala:686)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
at org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-02 18:35:37,739 WARN [Executor task launch worker-0] netty.NettyRpcEnv (Logging.scala:logWarning(70)) - RpcEnv already stopped.
... View more
Labels:
09-20-2017
03:33 AM
Hi I was using 5.12 express edition. It was not easy to startup with services after installation. The configurations needs to be changed and High Availability configurations, Journal nodes. I have updated Java heap size for HDFS instances. Namenode, Secondary Name node and Journal nodes only retained regards Shafi
... View more
09-13-2017
06:50 AM
Hi Yes through oozie in kerberised env, it shows Spark job as FAILED. Application Manager tries two attempts and it fails but actually in Yarn logs i see the spark dataframe output and message as SUCCEEDED regards Shafi
... View more
09-08-2017
05:49 AM
Hi Added it as dependent project to a spark project.
... View more
09-08-2017
05:48 AM
Hi getting this issue every time on spark submit. in 5.9, 5.11 and 5.12 is it kerberos problem ??? SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:ERROR Could not read configuration file from URL [file:/var/run/cloudera-scm-agent/process/18060-yarn-NODEMANAGER/log4j.properties].
java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/18060-yarn-NODEMANAGER/log4j.properties (Permission denied)
... View more
Labels:
09-06-2017
12:56 PM
When i submit a spark job i am the spark job is shown as failed in Yarn logs but in detailed logs it actually succeeds. Why does the application master is failing after making attempts. I have reduced the number of attempts to 1. User: dev Name: com.example Application Type: SPARK Application Tags: State: FAILED FinalStatus: FAILED Started: Wed Sep 06 09:12:15 -0500 2017 Elapsed: 21sec Tracking URL: History Diagnostics: Application application_1504705933896_0004 failed 1 times due to AM Container for appattempt_1504705933896_0004_000001 exited with exitCode: 0 Log Type: stderr Log Upload Time: Wed Sep 06 09:12:38 -0500 2017 Log Length: 119115 Showing 4096 bytes of 119115 total. Click here for the full log. heduler: ResultStage 3 (foreachPartition at HBaseContext.scala:216) finished in 0.142 s 17/09/06 09:12:36 INFO scheduler.DAGScheduler: Job 3 finished: foreachPartition at HBaseContext.scala:216, took 0.157439 s 17/09/06 09:12:36 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15e029e15d2ff0e 17/09/06 09:12:36 INFO zookeeper.ZooKeeper: Session: 0x15e029e15d2ff0e closed 17/09/06 09:12:36 INFO zookeeper.ClientCnxn: EventThread shut down |COL1|COL2|COL3| +----------+-------------+-------------+--------------+--------------------------+-------------------------+------------+ | Data is printed here 17/09/06 09:12:18 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT] 17/09/06 09:12:18 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1504705933896_0004_000001 17/09/06 09:12:19 INFO spark.SecurityManager: Changing view acls to: yarn,dev 17/09/06 09:12:19 INFO spark.SecurityManager: Changing modify acls to: yarn,dev 17/09/06 09:12:19 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, dev); users with modify permissions: Set(yarn, dev) 17/09/06 09:12:19 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread 17/09/06 09:12:19 INFO yarn.ApplicationMaster: Waiting for spark context initialization... 17/09/06 09:12:36 INFO ingestion: mysamplespark job executed successfully 17/09/06 09:12:36 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0 17/09/06 09:12:36 INFO spark.SparkContext: Invoking stop() from shutdown hook 17/09/06 09:12:36 INFO ui.SparkUI: Stopped Spark web UI at http://10.6.0.10:43467 17/09/06 09:12:37 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/09/06 09:12:37 INFO storage.MemoryStore: MemoryStore cleared 17/09/06 09:12:37 INFO storage.BlockManager: BlockManager stopped 17/09/06 09:12:37 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 17/09/06 09:12:37 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/09/06 09:12:37 INFO spark.SparkContext: Successfully stopped SparkContext 17/09/06 09:12:37 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED 17/09/06 09:12:37 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/09/06 09:12:37 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1504705933896_0004 17/09/06 09:12:37 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 17/09/06 09:12:37 INFO util.ShutdownHookManager: Shutdown hook called 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data1/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-de7ee7f3-5e8f-49a2-b99f-37e1c0a0122c 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data0/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-b0ebe098-f97c-49c8-bc0e-317af738619c 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data8/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/container_1504705933896_0004_01_000001/tmp/spark-afaac501-7ff7-42e9-a175-6f3ab2da8465 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data6/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-0508bde0-58ff-42ce-8cca-b15e07551e05 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data4/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-7fd2da4a-1c79-426b-bbbe-16c9c7b1aeaf 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data9/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-c1b3d37f-e394-4049-9874-62a2504c4d6b 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data5/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-ce45d5fd-970b-41cb-9678-855b86254285 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data2/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-53786e4f-f263-4876-926a-ed7b98822930 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data3/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-c7ab8793-95d6-4c21-9eac-a7407375005f 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data7/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-4669faae-b4d1-4ae8-8751-194b476eb6dd 17/09/06 09:12:37 INFO util.ShutdownHookManager: Deleting directory /data8/yarn/nm/usercache/dev/appcache/application_1504705933896_0004/spark-c267166f-0c4b-483d-a7e1-6621afb3eb90 17/09/06 09:12:37 INFO Remoting: Remoting shut down 17/09/06 09:12:37 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
... View more
Labels:
08-25-2017
03:57 AM
Hi I have a scala program with a main class and main function. It is not a spark project. it is a utility project how can i schedule the scala program as a oozie action regards Shafi
... View more
Labels:
08-25-2017
03:55 AM
Dear All, I am facing this issue in Cloudera 5.9. 5.11 and 5.12 when i am submitting a spark job through oozie this exception comes /var/run/cloudera-scm-agent/process/578-yarn-NODEMANAGER/log4j.properties (permission denied) The spark project is built with scala 2.10.6 and java 1.8. regards Shafi
... View more
08-07-2017
10:04 PM
Spark2 is not yet supported. But we can install in separately. you will have two submit commands after installation. one for 1.6.0 (default) and spark2 version 2.0
... View more
08-07-2017
03:21 AM
Hi I resolved this issue by configuring proper java heap memory for name node & Configuring HA on namenode regards Shafi
... View more
08-07-2017
03:17 AM
Hi Check the High Availability is configured properly or not. regards Shafi
... View more
06-05-2017
04:29 AM
It was due to the rogue node. i had to delete that node from cluster. This is fine now.
... View more
06-05-2017
04:24 AM
Hi, I have to delete that node from azure. then Cloudera was not able to find it and i was able to stop and delete the service. Earlier i only stopped that node in Azure. Somehow it still was able to attached to it. Anyhow, that node was of no use so i deleted it. I was able to move the Activity monitor to different host by creating the database for it. Also, the de-commissioning and deletion of host is also done from cluster. The parcel distribution is also fine now.
... View more
06-01-2017
04:00 AM
The screenshot is given below: The highlighted service 'host' is not available in the cluster. The service is is still started condition and it is not getting stopped. I am not able to move Activity Monitor to another role instance either, there is no option. Can you assist me in resolving this. Do i need to create the am and nav databases on different host. At present i want to stop service and delete this host. add this service to new role instance
... View more
06-01-2017
03:56 AM
Hi Recently i have resized the disks on the node and restarted the cluster. My parcels screen looks like below. I have to install spark 2 and kafka, can you please suggest how to resolve this issue The parcels are in activating state for 2 days.
... View more
Labels:
05-30-2017
12:44 AM
Hi I have tried that. The service is not getting stopped. Even the inspect hosts on that node results into error. So when i want to decommission node, it says Cloudera Manager Activity Monitor is running stop it first. When i try to stop activity monitor on that node it times out after 90 seconds.
... View more
05-28-2017
12:48 PM
Hi I have a 4 node cluster and the node on which Activity monitor service of Cloudera manager service is running is gone down. In cloudera manager service, i am not getting option to change this role to another host. It is frustrating. In the configuration page i have changed the property also to different host. still it is not getting out. I need to decomission this host and add another node. How to resolve this issue ? i want to change the role instance of Actvity monitor to another role. The wizard is also not showing any option
... View more
Labels:
05-03-2017
08:15 AM
Hi I have only Masternode and Data nodes on my cloudera Cluster. I want to add a new host and assign it Gateway node which the team will connect and work Please help on documentation links and optimal hardware configuration for gateway node.
... View more
04-13-2017
05:22 AM
1 Kudo
I have installed Clouder Enterprise data hub in azure. The CDH is version 5.10.0 The spark version i have is 1.6.0 there Can i upgrade my spark to stable verion of spark 2.0 or spark 2.1 It is a parcel based installation. Can i upgrade the parcels. Please share any related documentation or links
... View more
Labels:
04-12-2017
09:50 AM
Hi I want to organize my hbase tables under namespaces. Also, while creating a table i want to select a namespace and create it. I cannot see these options in Hue. Has anyone faced this issue. Please help how to achieve this regards Shafi
... View more
03-21-2017
11:41 PM
Hi I am removing hive service from Cloudera manager. i am getting following notification from hue. is it tightly integrated with hue "Missing Required Value: HiveService" How to resolve this. I am not seeing "none" option
... View more
03-21-2017
02:23 AM
MRS uses a directory called /var/RevoShare for its execution. Once it executes, it creates folders by the user name who executes along with its data inside this folder. Somehow it is not deleting older files. I have manually deleted the files & then it started working.
... View more
03-19-2017
01:25 AM
Hi I am trying to execute the below code with "Revo64-9.0 -f testrxspark.R" command list.files(system.file("SampleData", package = "RevoScaleR")) myHadoopCluster <- RxSpark(namenode="zzz.westeurope.cloudapp.azure.com", port=8020,consoleOutput=TRUE) rxSetComputeContext(myHadoopCluster) file.exists(system.file("SampleData/AirlineDemoSmall.csv", package="RevoScaleR")) bigDataDirRoot <- "/user/RevoShare" # HDFS location of the example data rxHadoopListFiles(bigDataDirRoot) # There will be no files at this point. source <-system.file("SampleData/AirlineDemoSmall.csv", package="RevoScaleR") inputDir <- file.path(bigDataDirRoot,"AirlineDemoSmall") rxHadoopMakeDir(inputDir) rxHadoopListFiles(bigDataDirRoot) rxHadoopCopyFromLocal(source, inputDir) rxHadoopListFiles(inputDir) hdfsFS <- RxHdfsFileSystem(hostName="zzz.westeurope.cloudapp.azure.com", port=8020) colInfo <- list(DayOfWeek = list(type = "factor", levels = c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"))) airDS <- RxTextData(file = inputDir, missingValueString = "M", colInfo = colInfo, fileSystem = hdfsFS) rxSummary(~ArrDelay:DayOfWeek, data = airDS) I got following error: 17/03/19 00:00:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where appli$ Warning: libjvm.so not found in /log/cloudera/parcels/MRS-9.0.1/hadoop, searching system-wide Internal Error: Cannot reset hdfs internal params while connected to an hdfs file system. Error in try({ : Internal Error: Cannot reset hdfs internal params while connected to an hdfs file system. Error: Error in try({ : Internal Error: Cannot reset hdfs internal params while connected to an hdfs file system. The ScaleR library scaleR-hadoop-0.1-SNAPSHOT.jar is copied to Cloudera Parcel hadoop lib folders on all nodes. Please help me fix this at the earliest
... View more
03-11-2017
06:18 PM
Hi I have opted for Cloudera Enterprise Data Hub from Azure Market place. Once installed i have customized it. Azure installed the Cluster in VNet with Subnets. I am able to get it working. However, the above one is not working. I am using this option now.
... View more
03-04-2017
09:29 AM
Thank you so much. I have tried the installation on Google Cloud it works like a charm and installs in an hour. No issues at all. There is something with Azure platform. The installation is not straight forward. I tried Path B as well as manual installation. Getting lot of problems. Is there any reference link to install cloudera latest one on Azure Platform. Please share the link. Otherwise no issues at all on Google Cloud
... View more
03-03-2017
05:57 PM
Hi I have tried all possible combinations to install cloudera express edition. everytime i get stuck at parcel distribution. I am installing on azure using ubuntu 14.04 LTS. The port 7191 and other parcel configuration are in place, no firewall, all mentioned ports are working. The heart beating of thread is creating frustration from last two days. The logs of agents are fine only the server log has one exception. please resolve at the earliest as i need to make choice between opting for enterprise edition vs other distribution
... View more
Labels:
03-03-2017
01:54 AM
hi I have resolved this by keeping /etc/hosts with only localhost entries a) updating listening host and Ip in config.ini of all nodes it worked
... View more
03-02-2017
10:35 PM
i have same problem. can u help what u did to resolve this
... View more
03-02-2017
05:47 PM
Hi I am installing latest Cloudera Express Edition on Azure using Ubuntu trusty 14.04 LTS. i am getting this error. /etc/hosts is having public IPs defined in it. The dig command is able to forward lookup, the revese lookup is not working properly. All the ports are open in Azure portal. I kills the 9001 everytime i re-try the installation. The SCM agent is waiting for heartbeat. This is frustrating. I have followed and crosschecked various links in this forum. Nothing is working. Kindly suggest how to resolve this.
... View more
Labels:
