You are life saver, I have been struggling with this for 7-8 hours and my deadline to submit a case study was close. Thanks alot!!! ... View more

@PriyalPotnis as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

I found other simple solution for this issue, Simply find faulty partition from partition list by using command. show partitions table table_name;   then rename the faulty partition to some other name in correct format of your partition. In my case, I used ALTER table table_name partition (date_flag='2021-11-25_bak') rename to partition (date_flag='2021-01-01'); ... View more

echo "scan 'emp'" | $HBASE_HOME/bin/hbase shell | awk -F'=' '{print $2}' | awk -F ':' '{print $2}'|awk -F ',' '{print $1}' ... View more

Yes, metadata does store the columns name in its database. hive > show columns in table_name: hive> set hive.cli.print.header=true; To view the column names of your table. ... View more

Thanks for sharing the code of your solution. I've also found that just making HiveContext variable lazy works: val sparkConf = new SparkConf().setAppName("StreamHDFSdata") sparkConf.set("spark.dynamicAllocation.enabled","false") val ssc = new StreamingContext(sparkConf, Seconds(5)) ssc.checkpoint("/user/hdpuser/checkpoint") val sc = ssc.sparkContext val smDStream = ssc.textFileStream("/user/hdpuser/data") val smSplitted = smDStream.map( x => x.split(";") ).map( x => Row.fromSeq( x ) ) ... lazy val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) smSplitted.foreachRDD( rdd => { // use sqlContext here } ) ... View more

How did u solved it ??? Which things one has to check ? ... View more

@DWinters   How were you able to overcome this issue? Could you please post a sample solution ... View more

Want to point out couple of things.. date is a hive key word. To have column called date you need to say `date` string,. If you specify that a table is partitioned ( in your cased partitioned by (`date` string) ), that partition is implicitly accepted as the last column of the table. No need to specify it in the schema again. CREATE EXTERNAL TABLE user ( userId BIGINT, type INT, level TINYINT )PARTITIONED BY (`date` String)LOCATION '/external_table_path'; As @Steven O'Neill mentioned, when ingesting data into HDFS, you should create sub-directories like date=2016-01-01 under your table location. Then your alter table statement will work. ... View more