I was able to create hive table on top of json files. below is the syntax i used to create external table..so i donot have to move data, all i need is to add partition CREATE EXTERNAL TABLE hdfs_audit( access string, agenthost string, cliip string, enforcer string, event_count bigint, event_dur_ms bigint, evttime timestamp, id string, logtype string, policy bigint, reason string, repo string, repotype bigint, requser string, restype string, resource string, result bigint, seq_num bigint) PARTITIONED BY ( evt_time string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://CLUSTERNAME/ranger/database/hdfs'; Add partition: ALTER TABLE ranger_audit.hdfs_audit ADD PARTITION (evt_time='20160601') LOCATION '/ranger/audit/hdfs/20160601/hdfs/20160601'; ... View more

it is an issue with meta data..we just dropped it and recreated the table and all is well... ... View more

@Raja Sekhar Chintalapati There is no Spark authentication against LDAP in a non kerberized environment. If a Spark job reads from HDFS and the user running the job does not have sufficient HDFS permission, Spark will fail to read data. Spark HiveContext does not connect to HiveServer2. It connects to Hive metastore once you provide the Hive configuration (hive-site.xml) to Spark, else it creates its own metastore in it's working directory I don't know a way to suppress the info in sparl-sql The Spark Master UI is typically on the node with Driver running on port 4040. You can define ports for the Driver, File Server, Executor, UI etc. See doc here See also setting Spark Configuratin here: https://spark.apache.org/docs/1.1.0/configuration.html See also for YARN Mode: http://spark.apache.org/docs/latest/security.html Example SPARK_MASTER_OPTS="-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" SPARK_WORKER_OPTS="-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" Programmatic Example import org.apache.spark.SparkConf import org,apache.spark.SparkContext val conf = new SparkConf() .setMaster(master) .setAppName("namexxx") .set("spark.driver.port", "7001") .set("spark.fileserver.port", "7002") .set("spark.broadcast.port", "7003") .set("spark.replClassServer.port", "7004") .set("spark.blockManager.port", "7005") .set("spark.executor.port", "7006") val sc= new SparkContext(conf) ... View more