Anyone have a basic guide in moving my hDP 2.6 cluster to aws? I need to move the data which seems to be a fairly straightforward copy, right? But then how do I use Amazon processing that matches my hive, spark etc installation from my cluster? TIA! ... View more

EDIT: I forgot to say I am trying to run Spark 2.2 as an independent service that uses HDP2.6. Please help I am running out of time! I run: ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue examples/jars/spark-examples*.jar 10 --executor-cores 4 --num-executors 11 --driver-java-options="-Dhdp.version=2.6.0.3-8" --conf "spark.executor.extraJavaOptions=-Dhdp.version=2.6.0.3-8" I get this error: Spark YARN Cluster mode get this error “Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster” I have tried all the fixes I can find except: 1. Classpath issues. Wher do I set this and to waht? 2. This question suggests it may be due to missing jars. Which jars do I need and what do I do with them? TIA! ... View more

I am getting desperate here! My Spark2 jobs take hours then get stuck! I have a 4 node cluster each with 16GB RAM and 8 cores. I run HDP 2.6, Spark 2.1 and Zeppelin 0.7. I have: spark.executor.instances 11 spark.executor.cores 2 spark.executor.memory 4G yarn.nodemanager.resource.memory-mb=14336 yarn.nodemanager.resource.cpu-vcores =7 Via Zeppelin (same notebook) I do an INSERT into a Hive table:: dfPredictions.write.mode(SaveMode.Append).insertInto("default.predictions") for a 50 column table with about 12 million records. This gets split into 3 stages of 75, 75 and 200 tasks. The 75 and 75 get stuck at stages 73 and 74 and the garbage collection lasts for hours. Any idea what I can try? EDIT: I have not looked at tweaking partitions, can anyone give me pointers on how to do that, please? ... View more

I have a 4 node cluster each with 16GB RAM and 8 cores. I run HDP 2.6, Spark 2.1 and Zeppelin 0.7. I have: spark.executor.instances 11 spark.executor.cores 2 spark.executor.memory 4G yarn.nodemanager.resource.memory-mb= 14336 yarn.nodemanager.resource.cpu-vcores = 7 In an earlier Zeppelin paragraph I did: spark.sql("set hive.execution.engine=tez;") although I do not know if Tez is doing the job, how do I tell? (Tez does not work if I do a job via the Hive View 2.0). Via Zeppelin (same notebook) I do an INSERT into a Hive table:: dfPredictions.write.mode(SaveMode.Append).insertInto("default.predictions") for a 50 column table with about 12 million records. This gets split into 3 stages of 75, 75 and 200 tasks. The first stage is running and has already taken 3.2 hours to do 45 out of the 75 tasks. Does this seem right with this size cluster? UPDATE: nearly finished! Some tasks have hours of garbage collection: ... View more

Hi, I am running: HDP-2.6.0.3 Spark2 2.1.0 Hive 1.2.1000 HDFS 2.7.3 This is driving me mad so any help is appreciated. I am trying to load a Hive table. I have tried in Hive View 2.0 and also in Spark. I get no errors and it works if I run it quickly, but my hdfs data keeps disappearing! hc.sql("SET hive.support.sql11.reserved.keywords=false;") hc.sql("add jar /usr/hdp/2.6.0.3-8/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar;") hc.sql("DROP TABLE tweets11") hc.sql("create table tweets15 ( racist boolean, contributors string, coordinates string, created_at string, entities struct < hashtags: array <string>, symbols: array <string>, urls: array <struct <display_url: string, expanded_url: string, indices: array <int>, url: string>>, user_mentions: array <string>>, favorite_count int, favorited boolean, filter_level string, geo string, id bigint, id_str string, in_reply_to_screen_name string, in_reply_to_status_id string, in_reply_to_status_id_str string, in_reply_to_user_id string, in_reply_to_user_id_str string, is_quote_status boolean, lang string, place string, possibly_sensitive boolean, retweet_count int, retweeted boolean, source string, text string, timestamp_ms string, truncated boolean, `user` struct < contributors_enabled: boolean, created_at: string, default_profile: boolean, default_profile_image: boolean, description: string, favourites_count: int, follow_request_sent: string, followers_count: int, `following`: string, friends_count: int, geo_enabled: boolean, id: bigint, id_str: string, is_translator: boolean, lang: string, listed_count: int, location: string, name: string, notifications: string, profile_background_color: string, profile_background_image_url: string, profile_background_image_url_https: string, profile_background_tile: boolean, profile_image_url: string, profile_image_url_https: string, profile_link_color: string, profile_sidebar_border_color: string, profile_sidebar_fill_color: string, profile_text_color: string, profile_use_background_image: boolean, protected: boolean, screen_name: string, statuses_count: int, time_zone: string, url: string, utc_offset: string, verified: boolean>) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE") hc.sql("LOAD DATA INPATH '/data' OVERWRITE INTO TABLE tweets15") /data is on hdfs and contains a couple of text files. But when I run this I get: res59: org.apache.spark.sql.DataFrame = [key: string, value: string] res60: org.apache.spark.sql.DataFrame = [result: int] res61: org.apache.spark.sql.DataFrame = [] res62: org.apache.spark.sql.DataFrame = [] res63: org.apache.spark.sql.DataFrame = [] and /data is gone or disappears after a while! Same thing happens in Hive: add jar /usr/hdp/2.6.0.3-8/hive/lib/json-serde-1.3.7-jar-with-dependencies.jar; DROP TABLE tweets0; create table tweets0 ( racist boolean, contributors string, coordinates string, created_at string, entities struct < hashtags: array <string>, symbols: array <string>, urls: array <struct <display_url: string, expanded_url: string, indices: array <int>, url: string>>, user_mentions: array <string>>, favorite_count int, favorited boolean, filter_level string, geo string, id bigint, id_str string, in_reply_to_screen_name string, in_reply_to_status_id string, in_reply_to_status_id_str string, in_reply_to_user_id string, in_reply_to_user_id_str string, is_quote_status boolean, lang string, place string, possibly_sensitive boolean, retweet_count int, retweeted boolean, source string, text string, timestamp_ms string, truncated boolean, `user` struct < contributors_enabled: boolean, created_at: string, default_profile: boolean, default_profile_image: boolean, description: string, favourites_count: int, follow_request_sent: string, followers_count: int, `following`: string, friends_count: int, geo_enabled: boolean, id: bigint, id_str: string, is_translator: boolean, lang: string, listed_count: int, location: string, name: string, notifications: string, profile_background_color: string, profile_background_image_url: string, profile_background_image_url_https: string, profile_background_tile: boolean, profile_image_url: string, profile_image_url_https: string, profile_link_color: string, profile_sidebar_border_color: string, profile_sidebar_fill_color: string, profile_text_color: string, profile_use_background_image: boolean, protected: boolean, screen_name: string, statuses_count: int, time_zone: string, url: string, utc_offset: string, verified: boolean>) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' STORED AS TEXTFILE; LOAD DATA INPATH '/data' OVERWRITE INTO TABLE tweets0; It is not necessarily immediately after running this code (code may have nothing to do with it) but nothing else is happening on the cluster. TIA!! ... View more

I'm writing Spark code in Zeppelin, and I wondered what is the best way of structuring it. For example, currently in a paragraph I want to call some UDFs which fit nicely into a scala object, like this. I could put them all inline, but it will be a lot of code in a paragraph. So questions: 1. How do I use external objects in Zeppelin? 2. Is this the best way to structure a Zeppelin/Spark app? TIA! FYI: HDP-2.6.0.3, Spark 2.1.0, Zeppelin 0.7.0, Ambari 2.5.0.3 ... View more