Member since
04-05-2016
37
Posts
8
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2847 | 07-30-2019 11:52 PM | |
5270 | 06-07-2019 01:01 AM | |
9163 | 04-14-2017 08:31 PM | |
5604 | 08-03-2016 12:52 AM | |
2892 | 06-22-2016 02:10 AM |
02-29-2024
10:24 PM
How about submitting a CDE job from CML to a private cloud base cluster?
... View more
07-30-2019
11:52 PM
1 Kudo
Since the "list" commands gets the apps from the ResourceManager and doesn't set any explicit filters and limits (except those provided with it) on the request, technically it returns all the applications which are present with RM at the moment. That number is controlled by "yarn.resourcemanager.max-completed-applications" config. Hope that clarifies.
... View more
06-07-2019
01:01 AM
1 Kudo
As your intent seems to capture the driver logs in a separate file while executing the app in the cluster mode, make sure that '/some/path/to/edgeNode/' dir is present on all of the NodeManager essentially as in cluster mode the driver will be running in the Yarn app's application master. If you can't make sure that follow a general practice to provide log file path to some pre-existing paths e.g. "/var/log/SparkDriver.log".
... View more
05-14-2019
02:42 AM
Please check if numpy is actually installed on all of the nodemanagers, if not, install it using below command (for python2.x) : pip install numpy If already installed, let us know the following: 1) Can you execute the same command outside of hue i.e. using Spark2-submit ? Mention the full command here. 2) What spark command you use in Hue?
... View more
04-14-2017
08:31 PM
It is the below line which is setting the data types for both the fields as StringType: val schema =
StructType(
schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true))) You can define your custom schema as follows : val customSchema = StructType(Array(
StructField("name", StringType, true),
StructField("age", IntegerType, true))) You can add additional fields as well in the above schema definition. And then you can use this customSchema while creating the dataframe as follows: val peopleDataFrame = sqlContext.createDataFrame(rowRDD, customSchema) Also for details, please see this page.
... View more
03-06-2017
10:30 PM
By Spark 2.1 do you mean Cloudera Spark 2.0 Release 1 or Apache Spark 2.1 ? Regarding Cloudera Spark 2.0 Release 1 or Release 2, I would like to tell that minimum required CDH version is CDH 5.7.x but you are on CDH 5.5.4.
... View more
08-30-2016
10:04 PM
The valuable information is at very bottom: NameError: name 'master' is not defined Please make sure you have defined variable "master" in your code. Or if you are specifying master via spark-submit, you should not set it in code.
... View more
08-03-2016
08:08 AM
You don't need to export as a JAR for unit testing. You can do : SparkConf().setMaster(local[2]) and run the program as usual Java application in IDE. Also make sure that you have all the dependent libraries in the classpath.
... View more
08-03-2016
12:52 AM
1 Kudo
You are getting this exception because "sc.testFile" reads a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. You said that you want to get the data from URL and want to save it to HDFS, then you should do: val data = scala.io.Source.fromURL("http://10.3.9.34:9900/messages").mkString
val list = data.split("\n").filter(_ != "")
val rdds = sc.parallelize(list)
rdds.saveAsTextFile(outputDirectory)
... View more
06-22-2016
02:10 AM
1 Kudo
The attached log indicates that application is accepted by cluster manager (YARN) but unable to execute due to resource crunch. Please make sure there are enough resources available in your cluster while submitting the job. Do check following and configure based on hosts: yarn.nodemanager.resource.memory-mb yarn.nodemanager.resource.cpu-vcores yarn.scheduler.maximum-allocation-mb yarn.scheduler.minimum-allocation-mb yarn-scheduler.maximum-allocation-vcores yarn.scheduler.minimum-allocation-vcores
... View more