@Ramya Grandhi Yes you can, however, this will spike your CPU and RAM usage, thereby affecting the overall response time of your system. You might notice sluggish performance, system might occasionally hang. To minimize these effects, when you are using the Sandbox, ensure all other unnecessary applications/browsers are closed. ... View more

Thank you! ... View more

This works for me! Thank you. ... View more

You're welcome. Could you kindly accept my answer if I have answered your question adequately? ... View more

@bhaskaran periasamy You must use the correct package in you Pig script. data = LOAD 'db.table' using org.apache.hive.hcatalog.pig.HCatLoader(); ... View more

@Adnan Alvee You could use the wholetextfiles() in SparkContext provided by Scala. Here is a simple outline that will help you avoid the spark-submit for each file and thereby save you the 15-30 seconds per file by iterating over multiple files within the same job. val data = sc.wholeTextFiles("HDFS_PATH") val files = data.map { case (filename, content) => filename} def doSomething(file: String) = { println (file); // your logic of processing a single file comes here val logData = sc.textFile(file); val numAs = logData.filter(line => line.contains("a")).count(); println("Lines with a: %s".format(numAs)); // save rdd of single file processed data to hdfs comes here } files.collect.foreach( filename => { doSomething(filename) }) where : data - org.apache.spark.rdd.RDD[(String, String)] files - org.apache.spark.rdd.RDD[String]- filenames doSomething(filename) - your requirement/logic HDFS_PATH - hdfs path to your source directory (you could even restrict to import certain kind of files by specifying path as "/hdfspath/*.csv" sc - SparkContext instance ... View more

@Rkg Grg 'last_ddltime' gets updated everytime a table is modified. How are you replacing the data file of the table ? I have always used LOAD DATA to replace the data from a new file and it updates last_ddltime on every occassion. LOAD DATA [LOCAL] INPATH '/pathToNewFile' OVERWRITE INTO TABLE tablename; Converting unixtime to timestamp select cast(from_unixtime(1483631785) AS timestamp); ... View more