Hi,  how can I access the position of the element (in my case of the MAP) in the array column?   TABLE DDL: CREATE TABLE mytable (     id int,     location ARRAY< MAP< string, string> > )   When I query the table with this SQL I got a position of the KEY-VALUE pair INSIDE the MAP, but what I really want to access is the position in the location ARRAY, e.g. 1st location, 2nd location and so on.   select c.id, a.pos, a.key, a.value from mytable c left join c.locations a order by c.id, a.pos;   Thanks     ... View more

Hello,   I tried to make a simple application in Spark Streaming which reads every 5s new data from HDFS and simply inserts into a Hive table. On the official Spark web site I have found an example, how to perform SQL operations on DStream data, via foreachRDD function, but the catch is, that the example used sqlContext and transformed the data from RDD to DataFrame.  The problem is, that with this DF, the data cannot be saved (appended) to an existing permanent Hive table. HiveContext has to be created.   So I tried this program, it works, but fails after a while, because runs out of memory, because it creates every time a new HiveContext object.   I tried to create the HiveContext BEFORE the map, and broadcast it, but it failed. I tried to call getOrCreate, which works fine with sqlContext but not with hiveContext.   Any ideas? Thanks   Tomas         val sparkConf = new SparkConf().setAppName("StreamHDFSdata") sparkConf.set("spark.dynamicAllocation.enabled","false") val ssc = new StreamingContext(sparkConf, Seconds(5)) ssc.checkpoint("/user/hdpuser/checkpoint") val sc = ssc.sparkContext val smDStream = ssc.textFileStream("/user/hdpuser/data") val smSplitted = smDStream.map( x => x.split(";") ).map( x => Row.fromSeq( x ) ) val smStruct = StructType( (0 to 10).toList.map( x => "col"+x.toString).map( y => StructField( y , StringType, true ) ) ) //val hiveCx = new org.apache.spark.sql.hive.HiveContext(sc) //val sqlBc = sc.broadcast( hiveCx ) smSplitted.foreachRDD( rdd => { //val sqlContext = SQLContext.getOrCreate(rdd.sparkContext) --> sqlContext cannot be used for permanent table create val sqlContext = new org.apache.spark.sql.hive.HiveContext(rdd.sparkContext) //val sqlContext = sqlBc.value --> THIS DOES NOT WORK: fail during runtime //val sqlContext = new HiveContext.getOrCreate(rdd.sparkContext) --> THIS DOES NOT WORK EITHER: fail during runtime //import hiveCx.implicits._ val smDF = sqlContext.createDataFrame( rdd, smStruct ) //val smDF = rdd.toDF smDF.registerTempTable("sm") val smTrgPart = sqlContext.sql("insert into table onlinetblsm select * from sm") smTrgPart.write.mode(SaveMode.Append).saveAsTable("onlinetblsm") } )   ... View more

Hbase has a nice feature called counter increment, where you can atomicly increment a value and get back the result. I want to create ta simple external table over this hbase table, but I dont know how to choose the correct data type for Hive/Impala.   The value of colfam:FL_Lock is this: 20160512_000006 column=colfam:FL_Lock, timestamp=1464120550634, value=\x00\x00\x00\x00\x00\x00\x00\x00   If I create external table with string, the query returns nothing, no error If I create external table with bigint/decimal/int, the query returns NULL and ERROR from Impala:   Error converting column colfam:FL_Lock: '' TO INT     Any ideas how to map correctly this Hbase column? Thanks Tomas   ... View more

Hi,   is it a bug, or a desired feature that the create external table (or change location of external table) is allowed only for serveradmin roles? Based on the documentation the database ALL permission should be sufficient, but there is a statement that also the URI should be accessible. However when I change my test user permission and remove serveradmin, he cannot create an external table pointing to his home directory such like this: create table part ( i int, s string ) stored as textfile location '/user/testuser/part';   ERROR: AuthorizationException: User 'testuser@MYREALM.LOCAL' does not have privileges to access: hdfs://hdfscluster/user/testuser/part   After enabling the serveradmin right for testuser the command executes correctly and the table is created.   Any hints on this? Thanks ... View more