Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/08/25 15:00:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/25 15:00:34 INFO CheckpointReader: Checkpoint files found: hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000.bk,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694680000,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694680000.bk,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503693120000,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503693120000.bk,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503693000000,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503693000000.bk,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503692880000,hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503692880000.bk
17/08/25 15:00:34 INFO CheckpointReader: Attempting to load checkpoint from file hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000
17/08/25 15:00:34 INFO Checkpoint: Checkpoint for time 1503694800000 ms validated
17/08/25 15:00:34 INFO CheckpointReader: Checkpoint successfully loaded from file hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000
17/08/25 15:00:34 INFO CheckpointReader: Checkpoint was generated at time 1503694800000 ms
17/08/25 15:00:34 INFO SparkContext: Running Spark version 2.1.0
17/08/25 15:00:34 INFO SecurityManager: Changing view acls to: cmuthukumaran
17/08/25 15:00:34 INFO SecurityManager: Changing modify acls to: cmuthukumaran
17/08/25 15:00:34 INFO SecurityManager: Changing view acls groups to: 
17/08/25 15:00:34 INFO SecurityManager: Changing modify acls groups to: 
17/08/25 15:00:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(cmuthukumaran); groups with view permissions: Set(); users  with modify permissions: Set(cmuthukumaran); groups with modify permissions: Set()
17/08/25 15:00:34 INFO Utils: Successfully started service 'sparkDriver' on port 49494.
17/08/25 15:00:35 INFO SparkEnv: Registering MapOutputTracker
17/08/25 15:00:35 INFO SparkEnv: Registering BlockManagerMaster
17/08/25 15:00:35 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/08/25 15:00:35 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/08/25 15:00:35 INFO DiskBlockManager: Created local directory at C:\Users\cmuthukumaran\AppData\Local\Temp\blockmgr-11b13289-1b53-42f8-bb3f-6b6d8923214a
17/08/25 15:00:35 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
17/08/25 15:00:35 INFO SparkEnv: Registering OutputCommitCoordinator
17/08/25 15:00:35 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/08/25 15:00:35 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.71.52.119:4040
17/08/25 15:00:35 INFO Executor: Starting executor ID driver on host localhost
17/08/25 15:00:35 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49503.
17/08/25 15:00:35 INFO NettyBlockTransferService: Server created on 10.71.52.119:49503
17/08/25 15:00:35 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/08/25 15:00:35 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.71.52.119, 49503, None)
17/08/25 15:00:35 INFO BlockManagerMasterEndpoint: Registering block manager 10.71.52.119:49503 with 912.3 MB RAM, BlockManagerId(driver, 10.71.52.119, 49503, None)
17/08/25 15:00:35 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.71.52.119, 49503, None)
17/08/25 15:00:35 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.71.52.119, 49503, None)
17/08/25 15:00:35 WARN StreamingContext: spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.
17/08/25 15:00:35 INFO ForEachDStream: Set context for org.apache.spark.streaming.dstream.ForEachDStream@4c4f4365
17/08/25 15:00:35 INFO MappedDStream: Set context for org.apache.spark.streaming.dstream.MappedDStream@7ce7e83c
17/08/25 15:00:35 INFO MappedDStream: Set context for org.apache.spark.streaming.dstream.MappedDStream@4a05d8ae
17/08/25 15:00:35 INFO WindowedDStream: Set context for org.apache.spark.streaming.dstream.WindowedDStream@3c904f1e
17/08/25 15:00:35 INFO TransformedDStream: Set context for org.apache.spark.streaming.dstream.TransformedDStream@4eb30d44
17/08/25 15:00:35 INFO MappedDStream: Set context for org.apache.spark.streaming.dstream.MappedDStream@d56aaa6
17/08/25 15:00:35 INFO DirectKafkaInputDStream: Set context for org.apache.spark.streaming.kafka010.DirectKafkaInputDStream@6972c30a
17/08/25 15:00:35 INFO DStreamGraph: Restoring checkpoint data
17/08/25 15:00:35 INFO ForEachDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO WindowedDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO TransformedDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO DirectKafkaInputDStream: Restoring checkpoint data
17/08/25 15:00:35 INFO DirectKafkaInputDStream$DirectKafkaInputDStreamCheckpointData: Restoring KafkaRDD for time 1503694680000 ms [(REST,0,2283640,2283640)]
17/08/25 15:00:35 INFO DirectKafkaInputDStream$DirectKafkaInputDStreamCheckpointData: Restoring KafkaRDD for time 1503694800000 ms [(REST,0,2283640,2283646)]
17/08/25 15:00:35 INFO DirectKafkaInputDStream: Restored checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restored checkpoint data
17/08/25 15:00:35 INFO TransformedDStream: Restored checkpoint data
17/08/25 15:00:35 INFO WindowedDStream: Restored checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restored checkpoint data
17/08/25 15:00:35 INFO MappedDStream: Restored checkpoint data
17/08/25 15:00:35 INFO ForEachDStream: Restored checkpoint data
17/08/25 15:00:35 INFO DStreamGraph: Restored checkpoint data
17/08/25 15:00:35 INFO FileBasedWriteAheadLog_ReceivedBlockTracker: Recovered 21 write ahead log files from hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering from write ahead logs in hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint
17/08/25 15:00:35 INFO FileBasedWriteAheadLog_ReceivedBlockTracker: Reading from the logs:
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503683160957-1503683220957
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503683285175-1503683345175
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503688201884-1503688261884
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503688320555-1503688380555
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503690600919-1503690660919
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503690725415-1503690785415
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503690841000-1503690901000
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503690965221-1503691025221
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503691925849-1503691985849
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692041287-1503692101287
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692165274-1503692225274
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692281719-1503692341719
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692401676-1503692461676
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692521629-1503692581629
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692641535-1503692701535
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692761021-1503692821021
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503692885389-1503692945389
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503693006037-1503693066037
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503693120979-1503693180979
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503694680957-1503694740957
hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/receivedBlockMetadata/log-1503694805097-1503694865097
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:35 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO ReceivedBlockTracker: Recovering record java.nio.HeapByteBuffer[pos=0 lim=262 cap=262]
17/08/25 15:00:36 INFO JobGenerator: Batches during down time (1 batches): 1503694800000 ms
17/08/25 15:00:36 INFO JobGenerator: Batches pending processing (0 batches): 
17/08/25 15:00:36 INFO JobGenerator: Batches to reschedule (1 batches): 1503694800000 ms
17/08/25 15:00:36 INFO TransformedDStream: Slicing from 1503651720000 ms to 1503694800000 ms (aligned to 1503651720000 ms and 1503694800000 ms)
17/08/25 15:00:36 INFO TransformedDStream: Time 1503694560000 ms is invalid as zeroTime is 1503694560000 ms , slideDuration is 120000 ms and difference is 0 ms
17/08/25 15:00:36 INFO JobScheduler: Added jobs for time 1503694800000 ms
17/08/25 15:00:36 INFO RecurringTimer: Started timer for JobGenerator at time 1503694920000
17/08/25 15:00:36 INFO JobGenerator: Restarted JobGenerator at 1503694920000 ms
17/08/25 15:00:36 INFO JobScheduler: Started JobScheduler
17/08/25 15:00:36 INFO JobScheduler: Starting job streaming job 1503694800000 ms.0 from job set of time 1503694800000 ms
17/08/25 15:00:36 INFO StreamingContext: StreamingContext started
17/08/25 15:00:36 INFO SparkContext: Starting job: isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56
17/08/25 15:00:36 INFO DAGScheduler: Registering RDD 8 (repartition at DStream.scala:576)
17/08/25 15:00:36 INFO DAGScheduler: Registering RDD 3 (repartition at DStream.scala:576)
17/08/25 15:00:36 INFO DAGScheduler: Got job 0 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56) with 1 output partitions
17/08/25 15:00:36 INFO DAGScheduler: Final stage: ResultStage 2 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56)
17/08/25 15:00:36 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0, ShuffleMapStage 1)
17/08/25 15:00:36 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0, ShuffleMapStage 1)
17/08/25 15:00:36 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[8] at repartition at DStream.scala:576), which has no missing parents
17/08/25 15:00:36 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.3 KB, free 912.3 MB)
17/08/25 15:00:36 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.7 KB, free 912.3 MB)
17/08/25 15:00:36 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.71.52.119:49503 (size: 2.7 KB, free: 912.3 MB)
17/08/25 15:00:36 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:996
17/08/25 15:00:36 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[8] at repartition at DStream.scala:576)
17/08/25 15:00:36 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/08/25 15:00:36 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[3] at repartition at DStream.scala:576), which has no missing parents
17/08/25 15:00:36 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.3 KB, free 912.3 MB)
17/08/25 15:00:36 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.7 KB, free 912.3 MB)
17/08/25 15:00:36 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.71.52.119:49503 (size: 2.7 KB, free: 912.3 MB)
17/08/25 15:00:36 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:996
17/08/25 15:00:36 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[3] at repartition at DStream.scala:576)
17/08/25 15:00:36 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
17/08/25 15:00:36 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 6207 bytes)
17/08/25 15:00:36 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
17/08/25 15:00:36 INFO KafkaRDD: Computing topic REST, partition 0 offsets 2283640 -> 2283646
17/08/25 15:00:36 INFO ConsumerConfig: ConsumerConfig values: 
	metric.reporters = []
	metadata.max.age.ms = 300000
	partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
	reconnect.backoff.ms = 50
	sasl.kerberos.ticket.renew.window.factor = 0.8
	max.partition.fetch.bytes = 1048576
	bootstrap.servers = [10.24.18.36:6667]
	ssl.keystore.type = JKS
	enable.auto.commit = false
	sasl.mechanism = GSSAPI
	interceptor.classes = null
	exclude.internal.topics = true
	ssl.truststore.password = null
	client.id = 
	ssl.endpoint.identification.algorithm = null
	max.poll.records = 2147483647
	check.crcs = true
	request.timeout.ms = 40000
	heartbeat.interval.ms = 3000
	auto.commit.interval.ms = 5000
	receive.buffer.bytes = 65536
	ssl.truststore.type = JKS
	ssl.truststore.location = null
	ssl.keystore.password = null
	fetch.min.bytes = 1
	send.buffer.bytes = 131072
	value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	group.id = spark-executor-2
	retry.backoff.ms = 100
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	ssl.trustmanager.algorithm = PKIX
	ssl.key.password = null
	fetch.max.wait.ms = 500
	sasl.kerberos.min.time.before.relogin = 60000
	connections.max.idle.ms = 540000
	session.timeout.ms = 30000
	metrics.num.samples = 2
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	ssl.protocol = TLS
	ssl.provider = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.keystore.location = null
	ssl.cipher.suites = null
	security.protocol = PLAINTEXT
	ssl.keymanager.algorithm = SunX509
	metrics.sample.window.ms = 30000
	auto.offset.reset = none

17/08/25 15:00:36 INFO ConsumerConfig: ConsumerConfig values: 
	metric.reporters = []
	metadata.max.age.ms = 300000
	partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
	reconnect.backoff.ms = 50
	sasl.kerberos.ticket.renew.window.factor = 0.8
	max.partition.fetch.bytes = 1048576
	bootstrap.servers = [10.24.18.36:6667]
	ssl.keystore.type = JKS
	enable.auto.commit = false
	sasl.mechanism = GSSAPI
	interceptor.classes = null
	exclude.internal.topics = true
	ssl.truststore.password = null
	client.id = consumer-1
	ssl.endpoint.identification.algorithm = null
	max.poll.records = 2147483647
	check.crcs = true
	request.timeout.ms = 40000
	heartbeat.interval.ms = 3000
	auto.commit.interval.ms = 5000
	receive.buffer.bytes = 65536
	ssl.truststore.type = JKS
	ssl.truststore.location = null
	ssl.keystore.password = null
	fetch.min.bytes = 1
	send.buffer.bytes = 131072
	value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	group.id = spark-executor-2
	retry.backoff.ms = 100
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	ssl.trustmanager.algorithm = PKIX
	ssl.key.password = null
	fetch.max.wait.ms = 500
	sasl.kerberos.min.time.before.relogin = 60000
	connections.max.idle.ms = 540000
	session.timeout.ms = 30000
	metrics.num.samples = 2
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	ssl.protocol = TLS
	ssl.provider = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.keystore.location = null
	ssl.cipher.suites = null
	security.protocol = PLAINTEXT
	ssl.keymanager.algorithm = SunX509
	metrics.sample.window.ms = 30000
	auto.offset.reset = none

17/08/25 15:00:36 INFO AppInfoParser: Kafka version : 0.10.0.1
17/08/25 15:00:36 INFO AppInfoParser: Kafka commitId : a7a17cdec9eaa6c5
17/08/25 15:00:37 INFO CachedKafkaConsumer: Initial fetch for spark-executor-2 REST 0 2283640
17/08/25 15:00:37 INFO AbstractCoordinator: Discovered coordinator siaihdpmaster1a.coc.ca:6667 (id: 2147482645 rack: null) for group spark-executor-2.
17/08/25 15:00:37 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1415 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 6207 bytes)
17/08/25 15:00:37 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
17/08/25 15:00:37 INFO KafkaRDD: Beginning offset 2283640 is the same as ending offset skipping REST 0
17/08/25 15:00:37 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 817 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 539 ms on localhost (executor driver) (1/1)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
17/08/25 15:00:37 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 12 ms on localhost (executor driver) (1/1)
17/08/25 15:00:37 INFO DAGScheduler: ShuffleMapStage 0 (repartition at DStream.scala:576) finished in 0.567 s
17/08/25 15:00:37 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
17/08/25 15:00:37 INFO DAGScheduler: looking for newly runnable stages
17/08/25 15:00:37 INFO DAGScheduler: running: Set(ShuffleMapStage 1)
17/08/25 15:00:37 INFO DAGScheduler: waiting: Set(ResultStage 2)
17/08/25 15:00:37 INFO DAGScheduler: failed: Set()
17/08/25 15:00:37 INFO DAGScheduler: ShuffleMapStage 1 (repartition at DStream.scala:576) finished in 0.527 s
17/08/25 15:00:37 INFO DAGScheduler: looking for newly runnable stages
17/08/25 15:00:37 INFO DAGScheduler: running: Set()
17/08/25 15:00:37 INFO DAGScheduler: waiting: Set(ResultStage 2)
17/08/25 15:00:37 INFO DAGScheduler: failed: Set()
17/08/25 15:00:37 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54), which has no missing parents
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.3 KB, free 912.3 MB)
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.1 KB, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.71.52.119:49503 (size: 3.1 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:996
17/08/25 15:00:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/08/25 15:00:37 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 6522 bytes)
17/08/25 15:00:37 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_6_0 stored as bytes in memory (estimated size 0.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_6_0 in memory on 10.71.52.119:49503 (size: 0.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2159 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 56 ms on localhost (executor driver) (1/1)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
17/08/25 15:00:37 INFO DAGScheduler: ResultStage 2 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56) finished in 0.057 s
17/08/25 15:00:37 INFO DAGScheduler: Job 0 finished: isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56, took 0.840038 s
17/08/25 15:00:37 INFO SparkContext: Starting job: isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56
17/08/25 15:00:37 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 152 bytes
17/08/25 15:00:37 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 149 bytes
17/08/25 15:00:37 INFO DAGScheduler: Got job 1 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56) with 4 output partitions
17/08/25 15:00:37 INFO DAGScheduler: Final stage: ResultStage 5 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56)
17/08/25 15:00:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4)
17/08/25 15:00:37 INFO DAGScheduler: Missing parents: List()
17/08/25 15:00:37 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54), which has no missing parents
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 5.3 KB, free 912.3 MB)
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.1 KB, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.71.52.119:49503 (size: 3.1 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:996
17/08/25 15:00:37 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Adding task set 5.0 with 4 tasks
17/08/25 15:00:37 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 3, localhost, executor driver, partition 1, PROCESS_LOCAL, 6522 bytes)
17/08/25 15:00:37 INFO Executor: Running task 0.0 in stage 5.0 (TID 3)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_6_1 stored as bytes in memory (estimated size 0.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_6_1 in memory on 10.71.52.119:49503 (size: 0.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 0.0 in stage 5.0 (TID 3). 2080 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 4, localhost, executor driver, partition 2, PROCESS_LOCAL, 6522 bytes)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 3) in 15 ms on localhost (executor driver) (1/4)
17/08/25 15:00:37 INFO Executor: Running task 1.0 in stage 5.0 (TID 4)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_6_2 stored as bytes in memory (estimated size 0.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_6_2 in memory on 10.71.52.119:49503 (size: 0.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 1.0 in stage 5.0 (TID 4). 2080 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 5, localhost, executor driver, partition 3, PROCESS_LOCAL, 6522 bytes)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 4) in 14 ms on localhost (executor driver) (2/4)
17/08/25 15:00:37 INFO Executor: Running task 2.0 in stage 5.0 (TID 5)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_6_3 stored as bytes in memory (estimated size 0.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_6_3 in memory on 10.71.52.119:49503 (size: 0.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 2.0 in stage 5.0 (TID 5). 2170 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 6, localhost, executor driver, partition 4, ANY, 6522 bytes)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 5) in 13 ms on localhost (executor driver) (3/4)
17/08/25 15:00:37 INFO Executor: Running task 3.0 in stage 5.0 (TID 6)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_11_0 stored as bytes in memory (estimated size 498.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_11_0 in memory on 10.71.52.119:49503 (size: 498.0 B, free: 912.3 MB)
17/08/25 15:00:37 WARN Executor: 1 block locks were not released by TID = 6:
[rdd_11_0]
17/08/25 15:00:37 INFO Executor: Finished task 3.0 in stage 5.0 (TID 6). 2629 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Finished task 3.0 in stage 5.0 (TID 6) in 33 ms on localhost (executor driver) (4/4)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
17/08/25 15:00:37 INFO DAGScheduler: ResultStage 5 (isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56) finished in 0.073 s
17/08/25 15:00:37 INFO DAGScheduler: Job 1 finished: isEmpty at NGINXLogProcessingWindowedwithcheckpointappl.scala:56, took 0.092085 s
17/08/25 15:00:37 INFO SparkContext: Starting job: count at NGINXLogProcessingWindowedwithcheckpointappl.scala:57
17/08/25 15:00:37 INFO DAGScheduler: Got job 2 (count at NGINXLogProcessingWindowedwithcheckpointappl.scala:57) with 8 output partitions
17/08/25 15:00:37 INFO DAGScheduler: Final stage: ResultStage 8 (count at NGINXLogProcessingWindowedwithcheckpointappl.scala:57)
17/08/25 15:00:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6, ShuffleMapStage 7)
17/08/25 15:00:37 INFO DAGScheduler: Missing parents: List()
17/08/25 15:00:37 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54), which has no missing parents
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 5.1 KB, free 912.3 MB)
17/08/25 15:00:37 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.0 KB, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.71.52.119:49503 (size: 3.0 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:996
17/08/25 15:00:37 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 8 (MapPartitionsRDD[14] at map at NGINXLogProcessingWindowedwithcheckpointappl.scala:54)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Adding task set 8.0 with 8 tasks
17/08/25 15:00:37 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 7, localhost, executor driver, partition 0, PROCESS_LOCAL, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 0.0 in stage 8.0 (TID 7)
17/08/25 15:00:37 INFO BlockManager: Found block rdd_6_0 locally
17/08/25 15:00:37 INFO Executor: Finished task 0.0 in stage 8.0 (TID 7). 952 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 8, localhost, executor driver, partition 1, PROCESS_LOCAL, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 1.0 in stage 8.0 (TID 8)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 7) in 9 ms on localhost (executor driver) (1/8)
17/08/25 15:00:37 INFO BlockManager: Found block rdd_6_1 locally
17/08/25 15:00:37 INFO Executor: Finished task 1.0 in stage 8.0 (TID 8). 873 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 9, localhost, executor driver, partition 2, PROCESS_LOCAL, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 2.0 in stage 8.0 (TID 9)
17/08/25 15:00:37 INFO BlockManager: Found block rdd_6_2 locally
17/08/25 15:00:37 INFO Executor: Finished task 2.0 in stage 8.0 (TID 9). 873 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 10, localhost, executor driver, partition 3, PROCESS_LOCAL, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 3.0 in stage 8.0 (TID 10)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 9) in 7 ms on localhost (executor driver) (2/8)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 8) in 14 ms on localhost (executor driver) (3/8)
17/08/25 15:00:37 INFO BlockManager: Found block rdd_6_3 locally
17/08/25 15:00:37 INFO Executor: Finished task 3.0 in stage 8.0 (TID 10). 873 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 4.0 in stage 8.0 (TID 11, localhost, executor driver, partition 4, PROCESS_LOCAL, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 4.0 in stage 8.0 (TID 11)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 10) in 7 ms on localhost (executor driver) (4/8)
17/08/25 15:00:37 INFO BlockManager: Found block rdd_11_0 locally
17/08/25 15:00:37 INFO Executor: Finished task 4.0 in stage 8.0 (TID 11). 954 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 5.0 in stage 8.0 (TID 12, localhost, executor driver, partition 5, ANY, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 5.0 in stage 8.0 (TID 12)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 4.0 in stage 8.0 (TID 11) in 6 ms on localhost (executor driver) (5/8)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_11_1 stored as bytes in memory (estimated size 246.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_11_1 in memory on 10.71.52.119:49503 (size: 246.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 5.0 in stage 8.0 (TID 12). 2147 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 6.0 in stage 8.0 (TID 13, localhost, executor driver, partition 6, ANY, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 6.0 in stage 8.0 (TID 13)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 5.0 in stage 8.0 (TID 12) in 12 ms on localhost (executor driver) (6/8)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_11_2 stored as bytes in memory (estimated size 231.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_11_2 in memory on 10.71.52.119:49503 (size: 231.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 6.0 in stage 8.0 (TID 13). 2147 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Starting task 7.0 in stage 8.0 (TID 14, localhost, executor driver, partition 7, ANY, 6451 bytes)
17/08/25 15:00:37 INFO Executor: Running task 7.0 in stage 8.0 (TID 14)
17/08/25 15:00:37 INFO TaskSetManager: Finished task 6.0 in stage 8.0 (TID 13) in 13 ms on localhost (executor driver) (7/8)
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
17/08/25 15:00:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
17/08/25 15:00:37 INFO MemoryStore: Block rdd_11_3 stored as bytes in memory (estimated size 516.0 B, free 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Added rdd_11_3 in memory on 10.71.52.119:49503 (size: 516.0 B, free: 912.3 MB)
17/08/25 15:00:37 INFO Executor: Finished task 7.0 in stage 8.0 (TID 14). 2068 bytes result sent to driver
17/08/25 15:00:37 INFO TaskSetManager: Finished task 7.0 in stage 8.0 (TID 14) in 13 ms on localhost (executor driver) (8/8)
17/08/25 15:00:37 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
17/08/25 15:00:37 INFO DAGScheduler: ResultStage 8 (count at NGINXLogProcessingWindowedwithcheckpointappl.scala:57) finished in 0.067 s
17/08/25 15:00:37 INFO DAGScheduler: Job 2 finished: count at NGINXLogProcessingWindowedwithcheckpointappl.scala:57, took 0.076602 s
count received 6
17/08/25 15:00:37 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.71.52.119:49503 in memory (size: 2.7 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.71.52.119:49503 in memory (size: 2.7 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 10.71.52.119:49503 in memory (size: 3.1 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 10.71.52.119:49503 in memory (size: 3.1 KB, free: 912.3 MB)
17/08/25 15:00:37 INFO BlockManagerInfo: Removed broadcast_4_piece0 on 10.71.52.119:49503 in memory (size: 3.0 KB, free: 912.3 MB)
17/08/25 15:00:38 INFO JobScheduler: Finished job streaming job 1503694800000 ms.0 from job set of time 1503694800000 ms
17/08/25 15:00:38 INFO JobScheduler: Total delay: 38.591 s for time 1503694800000 ms (execution: 2.137 s)
17/08/25 15:00:38 ERROR JobScheduler: Error running job streaming job 1503694800000 ms.0
java.lang.NullPointerException
	at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:165)
	at ca.twitter2.NGINXLogProcessingWindowedwithcheckpointappl$$anonfun$creatingFunc$1.apply(NGINXLogProcessingWindowedwithcheckpointappl.scala:61)
	at ca.twitter2.NGINXLogProcessingWindowedwithcheckpointappl$$anonfun$creatingFunc$1.apply(NGINXLogProcessingWindowedwithcheckpointappl.scala:55)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:254)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
	at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:165)
	at ca.twitter2.NGINXLogProcessingWindowedwithcheckpointappl$$anonfun$creatingFunc$1.apply(NGINXLogProcessingWindowedwithcheckpointappl.scala:61)
	at ca.twitter2.NGINXLogProcessingWindowedwithcheckpointappl$$anonfun$creatingFunc$1.apply(NGINXLogProcessingWindowedwithcheckpointappl.scala:55)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:627)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:254)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
17/08/25 15:00:38 INFO JobGenerator: Checkpointing graph for time 1503694800000 ms
17/08/25 15:00:38 INFO DStreamGraph: Updating checkpoint data for time 1503694800000 ms
17/08/25 15:00:38 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook
17/08/25 15:00:38 INFO BatchedWriteAheadLog: BatchedWriteAheadLog shutting down at time: 1503694838606.
17/08/25 15:00:38 WARN BatchedWriteAheadLog: BatchedWriteAheadLog Writer queue interrupted.
17/08/25 15:00:38 INFO BatchedWriteAheadLog: BatchedWriteAheadLog Writer thread exiting.
17/08/25 15:00:38 INFO FileBasedWriteAheadLog_ReceivedBlockTracker: Stopped write ahead log manager
17/08/25 15:00:38 INFO DStreamGraph: Updated checkpoint data for time 1503694800000 ms
17/08/25 15:00:38 INFO ReceiverTracker: ReceiverTracker stopped
17/08/25 15:00:38 INFO JobGenerator: Stopping JobGenerator immediately
17/08/25 15:00:38 INFO RecurringTimer: Stopped timer for JobGenerator after time -1
17/08/25 15:00:38 INFO CheckpointWriter: Submitted checkpoint of time 1503694800000 ms to writer queue
17/08/25 15:00:38 INFO CheckpointWriter: Saving checkpoint for time 1503694800000 ms to file 'hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000'
17/08/25 15:00:38 INFO CheckpointWriter: Checkpoint for time 1503694800000 ms saved to file 'hdfs://ln461.alt.calgary.ca:8020/user/NGINXAccesslogcheckpoint/checkpoint-1503694800000', took 5075 bytes and 208 ms
17/08/25 15:00:38 INFO CheckpointWriter: CheckpointWriter executor terminated? true, waited for 197 ms.
17/08/25 15:00:38 INFO JobGenerator: Stopped JobGenerator
17/08/25 15:00:38 INFO JobScheduler: Stopped JobScheduler
17/08/25 15:00:38 INFO StreamingContext: StreamingContext stopped successfully
17/08/25 15:00:38 INFO SparkContext: Invoking stop() from shutdown hook
17/08/25 15:00:38 INFO SparkUI: Stopped Spark web UI at http://10.71.52.119:4040
17/08/25 15:00:38 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/08/25 15:00:38 INFO MemoryStore: MemoryStore cleared
17/08/25 15:00:38 INFO BlockManager: BlockManager stopped
17/08/25 15:00:38 INFO BlockManagerMaster: BlockManagerMaster stopped
17/08/25 15:00:38 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/08/25 15:00:38 INFO SparkContext: Successfully stopped SparkContext
17/08/25 15:00:38 INFO ShutdownHookManager: Shutdown hook called
17/08/25 15:00:38 INFO ShutdownHookManager: Deleting directory C:\Users\cmuthukumaran\AppData\Local\Temp\spark-50bb323a-5adf-4b73-804c-3203526a4a93