Member since
11-22-2016
50
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3181 | 01-17-2017 02:54 PM |
01-11-2017
05:12 AM
@Steven O'Neill i'm not touching the data in the HDFS. it has created a seperate folder for the date column on using alter statement.
... View more
01-11-2017
02:58 AM
@DWinters can you elaborate the way you have solved it, cause i just ran in to this issue. I have 11 columns in data frame(added a timestamp column to 10 columned RDD). And i have a hive table with complete 11 columns one is partitioned by timestamp.
... View more
01-10-2017
09:15 AM
@Neeraj Sabharwal
Hi all,
Im trying to create an external hive partitioned table which location points to an HDFS location.This HDFS location get appended every time i run my spark streaming application, so my hive table appends too. Kafka >> Spark Streaming >> HDFS >> Hive External Table. I could do the above flow smoothly with a non partitioned table, but when i want to add partitions to my external table i'm not able to get data in to my hive external table whatsoever. I have tried the below after creating the external table, but still the problem persists. <code>CREATE EXTERNAL TABLE user (
userId BIGINT,
type INT,
level TINYINT,
date String
)
PARTITIONED BY (date String)
LOCATION '/external_table_path'; <code>ALTER TABLE user ADD PARTITION(date='2010-02-22');
I tried the above fix but the things haven't changed. What is the best work around to get my appending HDFS data in to an external Partition table??
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
01-04-2017
11:51 AM
you can add the spark assembly jar in the global location like hdfs:/// . And set the spark.yarn.jar value in spark-defaults.conf to that spark assembly jar in hdfs path.
... View more
01-04-2017
11:47 AM
Can you check your, spark-env.sh file. Make sure you set the HADOOP_CONF_DIR and JAVA_HOME in the spark-env.sh too
... View more
01-03-2017
11:52 PM
i was pointinn to a wrong broker so the zk was unable to find the topics
... View more
01-03-2017
11:47 PM
1 Kudo
Still the issue is persisting, What else can we do to make it work other than hive-site.xml
... View more
12-29-2016
02:26 AM
i'm using sbt, should i use spark-submit everytime we need to run a project? SBT run, is catering my needs for now, as im using it in local mode.
... View more
12-29-2016
01:14 AM
i'm running my spark job using sbt,i.e., "sbt run". You think this can be a problem? Or should we stick to the spark-submit all the time?
... View more
12-29-2016
01:10 AM
i'm running my spark job using sbt,i.e., "sbt run". You think this can be a problem? Or should we stick to the spark-submit all the time?
... View more