Hi, I am running: HDP-2.6.0.3 Spark2 2.1.0 Hive 1.2.1000 HDFS 2.7.3 This is driving me mad so any help is appreciated. I am trying to load a Hive table. I have tried in Hive View 2.0 and also in Spark. I get no errors and it works if I run it quickly, but my hdfs data keeps disappearing! hc.sql("SET hive.support.sql11.reserved.keywords=false;")
hc.sql("add jar /usr/hdp/2.6.0.3-8/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar;")
hc.sql("DROP TABLE tweets11")
hc.sql("create table tweets15 ( racist boolean, contributors string, coordinates string, created_at string, entities struct < hashtags: array <string>, symbols: array <string>, urls: array <struct <display_url: string, expanded_url: string, indices: array <int>, url: string>>, user_mentions: array <string>>, favorite_count int, favorited boolean, filter_level string, geo string, id bigint, id_str string, in_reply_to_screen_name string, in_reply_to_status_id string, in_reply_to_status_id_str string, in_reply_to_user_id string, in_reply_to_user_id_str string, is_quote_status boolean, lang string, place string, possibly_sensitive boolean, retweet_count int, retweeted boolean, source string, text string, timestamp_ms string, truncated boolean, `user` struct < contributors_enabled: boolean, created_at: string, default_profile: boolean, default_profile_image: boolean, description: string, favourites_count: int, follow_request_sent: string, followers_count: int, `following`: string, friends_count: int, geo_enabled: boolean, id: bigint, id_str: string, is_translator: boolean, lang: string, listed_count: int, location: string, name: string, notifications: string, profile_background_color: string, profile_background_image_url: string, profile_background_image_url_https: string, profile_background_tile: boolean, profile_image_url: string, profile_image_url_https: string, profile_link_color: string, profile_sidebar_border_color: string, profile_sidebar_fill_color: string, profile_text_color: string, profile_use_background_image: boolean, protected: boolean, screen_name: string, statuses_count: int, time_zone: string, url: string, utc_offset: string, verified: boolean>) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE")
hc.sql("LOAD DATA INPATH '/data' OVERWRITE INTO TABLE tweets15") /data is on hdfs and contains a couple of text files. But when I run this I get: res59: org.apache.spark.sql.DataFrame = [key: string, value: string]
res60: org.apache.spark.sql.DataFrame = [result: int]
res61: org.apache.spark.sql.DataFrame = []
res62: org.apache.spark.sql.DataFrame = []
res63: org.apache.spark.sql.DataFrame = [] and /data is gone or disappears after a while! Same thing happens in Hive: add jar /usr/hdp/2.6.0.3-8/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar;
DROP TABLE tweets0;
create table tweets0
( racist boolean, contributors string, coordinates string, created_at string, entities struct < hashtags: array <string>, symbols: array <string>, urls: array <struct <display_url: string, expanded_url: string, indices: array <int>, url: string>>, user_mentions: array <string>>, favorite_count int, favorited boolean, filter_level string, geo string, id bigint, id_str string, in_reply_to_screen_name string, in_reply_to_status_id string, in_reply_to_status_id_str string, in_reply_to_user_id string, in_reply_to_user_id_str string, is_quote_status boolean, lang string, place string, possibly_sensitive boolean, retweet_count int, retweeted boolean, source string, text string, timestamp_ms string, truncated boolean, `user` struct < contributors_enabled: boolean, created_at: string, default_profile: boolean, default_profile_image: boolean, description: string, favourites_count: int, follow_request_sent: string, followers_count: int, `following`: string, friends_count: int, geo_enabled: boolean, id: bigint, id_str: string, is_translator: boolean, lang: string, listed_count: int, location: string, name: string, notifications: string, profile_background_color: string, profile_background_image_url: string, profile_background_image_url_https: string, profile_background_tile: boolean, profile_image_url: string, profile_image_url_https: string, profile_link_color: string, profile_sidebar_border_color: string, profile_sidebar_fill_color: string, profile_text_color: string, profile_use_background_image: boolean, protected: boolean, screen_name: string, statuses_count: int, time_zone: string, url: string, utc_offset: string, verified: boolean>)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS TEXTFILE; LOAD DATA INPATH '/data' OVERWRITE INTO TABLE tweets0;
It is not necessarily immediately after running this code (code may have nothing to do with it) but nothing else is happening on the cluster. TIA!!
... View more
