Member since
03-05-2015
2
Posts
0
Kudos Received
0
Solutions
03-05-2015
04:54 AM
I am using the broadcast variable within the class Test, which implements the VoidFunction (this is a closure rght ?) , there I am trying to print its value . And the Broadcast variable is coming as null there. Please suggest. Please note, when I am running the program locally , it worked fine.
... View more
03-05-2015
03:24 AM
Hi All I have a simple spark application, where I am trying to broadcast a String type variable on YARN Cluster. But every time I am trying to access the broadcast-ed variable value , I am getting null within the Task. It will be really helpful, if you guys can suggest, what I am doing wrong here. My code is like follows:- public class TestApp implements Serializable{ static Broadcast<String[]> mongoConnectionString; public static void main( String[] args ) { String mongoBaseURL = args[0]; SparkConf sparkConf = new SparkConf().setAppName(Constants.appName); JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf); mongoConnectionString = javaSparkContext.broadcast(args); JavaSQLContext javaSQLContext = new JavaSQLContext(javaSparkContext); JavaSchemaRDD javaSchemaRDD = javaSQLContext.jsonFile(hdfsBaseURL+Constants.hdfsInputDirectoryPath); if(javaSchemaRDD!=null){ javaSchemaRDD.registerTempTable("LogAction"); javaSchemaRDD.cache(); pageSchemaRDD = javaSQLContext.sql(SqlConstants.getLogActionPage); pageSchemaRDD.foreach(new Test()); } } private static class Test implements VoidFunction<Row>{ /** * */ private static final long serialVersionUID = 1L; public void call(Row t) throws Exception { // TODO Auto-generated method stub logger.info("mongoConnectionString "+mongoConnectionString.value()); } } Thanks and Regards Samriddha
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN