Since the "list" commands gets the apps from the ResourceManager and doesn't set any explicit filters and limits (except those provided with it) on the request, technically it returns all the applications which are present with RM at the moment. That number is controlled by "yarn.resourcemanager.max-completed-applications" config. Hope that clarifies. ... View more

As your intent seems to capture the driver logs in a separate file while executing the app in the cluster mode, make sure that '/some/path/to/edgeNode/' dir is present on all of the NodeManager essentially as in cluster mode the driver will be running in the Yarn app's application master. If you can't make sure that follow a general practice to provide log file path to some pre-existing paths e.g. "/var/log/SparkDriver.log".     ... View more

It is the below line which is setting the data types for both the fields as StringType:    val schema = StructType( schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true))) You can define your custom schema as follows :    val customSchema = StructType(Array( StructField("name", StringType, true), StructField("age", IntegerType, true))) You can add additional fields as well in the above schema definition.   And then you can use this customSchema while creating the dataframe as follows:    val peopleDataFrame = sqlContext.createDataFrame(rowRDD, customSchema)   Also for details, please see this page.  ... View more

You are getting this exception because "sc.testFile" reads a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.    You said that you want to get the data from URL and want to save it to HDFS, then you should do:   val data = scala.io.Source.fromURL("http://10.3.9.34:9900/messages").mkString val list = data.split("\n").filter(_ != "") val rdds = sc.parallelize(list) rdds.saveAsTextFile(outputDirectory) ... View more

The attached log indicates that application is accepted by cluster manager (YARN) but unable to execute due to resource crunch. Please make sure there are enough resources available in your cluster while submitting the job.   Do check following and configure based on hosts: yarn.nodemanager.resource.memory-mb yarn.nodemanager.resource.cpu-vcores yarn.scheduler.maximum-allocation-mb yarn.scheduler.minimum-allocation-mb    yarn-scheduler.maximum-allocation-vcores yarn.scheduler.minimum-allocation-vcores     ... View more