Member since
05-11-2017
11
Posts
0
Kudos Received
0
Solutions
11-01-2017
05:36 AM
We need run Create table and alter table statements to hive from within a spark streaming application. The spark version is 2.1.X and it is HDP 2.6.2. spark.sqlContext.sql("CREATE ...")
spark.sqlContext.sql("ALTER ...") The create statement works but alter fails with Spark error "Operation not allowed". We are stuck and not able to proceed further. Is there any way by which we can run Alter table commands to hive with HDP 2.6.2 from within a spark application. From the SparkSession we cannot get hold of HiveContext hence that approach is also not possible. Any help in this matter is much appreciated.
... View more
Labels:
- Labels:
-
Apache Spark
08-17-2017
08:24 AM
@Eugene Koifman We have tested with Hive JDBC and Hive streaming. The behavior seems to be same when we do compaction along with Hive JDBC. If we do compaction then we don't see much difference between these two. It would be of great help if you could share more details of the advantages of Hive streaming compared to Hive JDBC.
... View more
08-03-2017
04:31 AM
Is streaming API integrated with Spark. When we tried to use HiveEndPoint classes within a spark context, many weird class loader issues have come up.
... View more
08-01-2017
08:47 AM
I am using spark session to save a data frame to hive table. The code is as below.
df.write.mode(SaveMode.Append).format("orc").insertInto("table")
The data comes to spark from kafka. This can be huge amount of data coming throughout the day. Does , spark dataframe save internally does hive compaction ?. If not what is the best way to do compaction at regular intervals without affecting the table insertions.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
05-23-2017
11:03 AM
What is the advantage of hive-streaming over hive-jdbc. If we do batch in jdbc what extra advantage hive-streaming has over hive jdbc
... View more
Labels:
- Labels:
-
Apache Hive
05-14-2017
12:06 PM
Anybody has used hive-streaming inside spark and deployed in a cluster ? Is this something correct or wrong usage. Is there any url that shows using hive-streaming inside spark program in cluster mode.
... View more
05-14-2017
12:05 PM
Anybody has used hive-streaming inside spark and deployed in a cluster ? Is this something correct or wrong usage
... View more
05-11-2017
05:11 PM
Is it a valid scenario to use hive-streaming inside a spark program. I have seen examples of hive streaming as standalone program and spark streaming for writing to hive. Never seen any program where hive-streaming is used inside a spark application and submitted to cluster. Does hive streaming work inside a spark application or is this a totally wrong usage. Please share your thoughts.
... View more
Labels:
- Labels:
-
Apache HCatalog
-
Apache Hive
-
Apache Spark