Member since
10-14-2016
4
Posts
4
Kudos Received
0
Solutions
11-23-2016
03:04 AM
1 Kudo
One more helpfull link http://stackoverflow.com/questions/28712420/how-to-run-concurrent-jobsactions-in-apache-spark-using-single-spark-context
... View more
11-23-2016
03:01 AM
1 Kudo
Please use Future interface in your code. Request to explorebelow mentioned object or interfaces. java.util.concurrent.ExecutionException; java.util.concurrent.ExecutorService; java.util.concurrent.Executors; java.util.concurrent.Future; You can get more information from below mentioned URLs. https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html
... View more
10-24-2016
03:10 AM
1 Kudo
Thanks Hubbarja, Will check and respond very shortly.
... View more
10-14-2016
07:25 AM
1 Kudo
We are doing spark programming in java language. The ‘DataFrame’ has been stored in temporary table and we are running multiple queries from this temporary table inside loop. The quires are running in sequential order. We need to run in parallel from temporary table. Please find code snippet below. Thanks in advance for your cooperation. HiveContext hiveContext = new HiveContext(sparkContext); String mainQueryHql = getFileContent(mainQueryFilePath); DataFrame df = hiveContext.sql(mainQueryHql).persist(StorageLevel.MEMORY_AND_DISK_SER()); df.show(); System.out.println("Total Records in Main Query " + df.count()); df.registerTempTable(tempTable); ArrayList<DataFrame> dataFrameList = new ArrayList(); DataFrame dfSubQuery = null; String subQuires = getFileContent(subQueryFilePath); String[] alQuires = subQuires.split(";"); for(int i=0; i<alQuires.length; i++) { System.out.println("Query no " + i +" is : " + alQuires[i]); logger.debug("Query no " + i +" is : " + alQuires[i]); dfSubQuery = hiveContext.sql(alQuires[i]); dfSubQuery.show(); dataFrameList.add(dfSubQuery); }
... View more
Labels:
- Labels:
-
Apache Spark