question Re: Debug Spark program in Eclipse Data in AWS in Archives of Support Questions (Read Only) https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48482#M48706 <P>It's also possible to establish an ssl tunnel in order to connect to a remote debug session. &nbsp;Take a look at the -L option for ssh, you will be able to open a local port and setup the remote port within the ssh command. &nbsp;This will work for private IPs as long as you can connect to a server from a public IP that has access to the private network. &nbsp;Note though that there can be extreme latency and still be difficult to debug in setups like this.</P> Tue, 13 Dec 2016 19:11:25 GMT hubbarja 2016-12-13T19:11:25Z Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48472#M48704 <P>hi All</P><P>&nbsp; &nbsp; &nbsp; &nbsp;i am trying to run debug spark program in eclipse on Cloudera cluster on AWS EC2. i tried&nbsp;</P><P>val conf = new SparkConf().setAppName("WordCount").setMaster("yarn-client")</P><P><SPAN>val conf = new SparkConf()</SPAN><SPAN>.setAppName("WordCount")</SPAN><SPAN>.setMaster("local[3]")</SPAN></P><P><SPAN>&nbsp; &nbsp; &nbsp; &nbsp;i find out i am facing an issue . the namenode in the AWS EC2 cluster return me the private IP in AWS.like&nbsp;</SPAN></P><P><SPAN>172.31.26.79,172.31.26.80 etc.. which my local windows mechine not able to resolve .&nbsp; </SPAN></P><P><SPAN>&nbsp; &nbsp; &nbsp; Any idea how to handle all this ?</SPAN></P><P>&nbsp;</P> Fri, 16 Sep 2022 10:51:05 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48472#M48704 ranan 2022-09-16T10:51:05Z Re: Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48477#M48705 <P>I know move the program inside the cluster network will help ~ but sometime you can't move the program into the cluster&nbsp;</P> Tue, 13 Dec 2016 18:34:15 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48477#M48705 ranan 2016-12-13T18:34:15Z Re: Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48482#M48706 <P>It's also possible to establish an ssl tunnel in order to connect to a remote debug session. &nbsp;Take a look at the -L option for ssh, you will be able to open a local port and setup the remote port within the ssh command. &nbsp;This will work for private IPs as long as you can connect to a server from a public IP that has access to the private network. &nbsp;Note though that there can be extreme latency and still be difficult to debug in setups like this.</P> Tue, 13 Dec 2016 19:11:25 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48482#M48706 hubbarja 2016-12-13T19:11:25Z Re: Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48496#M48707 <P>Thank you for your reply .&nbsp;</P><P>I solve the issue by create another node in the AWS EC2 as workspace through that node to connect to other AWS EC2 cluster .&nbsp;</P> Tue, 13 Dec 2016 22:56:54 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/48496#M48707 ranan 2016-12-13T22:56:54Z Re: Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/49574#M48708 <P><A href="http://aws.amazon.com/elasticmapreduce/" target="_blank">Amazon Elastic MapReduce</A><SPAN> (EMR) builds proprietary versions of Apache Hadoop, Hive, and Pig optimized for running on Amazon Web Services. Amazon EMR provides a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (EC2) or Simple Storage Service (S3)</SPAN></P> Wed, 18 Jan 2017 01:39:50 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/49574#M48708 ZachRoes 2017-01-18T01:39:50Z Re: Debug Spark program in Eclipse Data in AWS https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/292132#M48709 <P>can you help me out this error....</P><P>[Stage 0:&gt; (0 + 1) / 1]20/03/19 21:40:24 WARN BlockReaderFactory: I/O error constructing remote block reader.<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:40:24 WARN DFSClient: Failed to connect to /172.31.34.22:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:40:45 WARN BlockReaderFactory: I/O error constructing remote block reader.<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:40:45 WARN DFSClient: Failed to connect to /172.31.37.160:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />[Stage 0:&gt; (0 + 1) / 1]20/03/19 21:41:06 WARN BlockReaderFactory: I/O error constructing remote block reader.<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:41:06 WARN DFSClient: Failed to connect to /172.31.37.41:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:41:06 WARN DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 2719.6350079908057 msec.<BR />20/03/19 21:41:30 WARN BlockReaderFactory: I/O error constructing remote block reader.<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)<BR />20/03/19 21:41:30 WARN DFSClient: Failed to connect to /172.31.37.160:50010 for block, add to deadNodes and continue. java.net.ConnectException: Connection timed out: no further information<BR />java.net.ConnectException: Connection timed out: no further information<BR />at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)<BR />at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)<BR />at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)<BR />at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)<BR />at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3436)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:777)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:694)<BR />at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)<BR />at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)<BR />at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)<BR />at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)<BR />at java.io.DataInputStream.read(Unknown Source)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)<BR />at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)<BR />at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)<BR />at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)<BR />at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)<BR />at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)<BR />at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.hasNext(HadoopFileLinesReader.scala:69)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:181)<BR />at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)<BR />at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)<BR />at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)<BR />at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)<BR />at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)<BR />at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)<BR />at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:123)<BR />at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)<BR />at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<BR />at java.lang.Thread.run(Unknown Source)</P> Thu, 19 Mar 2020 16:38:55 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Debug-Spark-program-in-Eclipse-Data-in-AWS/m-p/292132#M48709 ParthiCyberPunk 2020-03-19T16:38:55Z