Running in HDP 2.4 hive> SELECT COUNT(*) from tweets; I get Error: Failure while running task:java.lang.IllegalArgumentException: tez.runtime.io.sort.mb 859 should be larger than 0 and should be less than the available task memory (MB):789 I have tried setting tez.runtime.io.sort.mb to 789 and also to 600. I have tried running hive thus: hive --hiveconf hive.tez.container.size=1024 Always the same error. My config is: Using cores=16 memory=64GB disks=4 hbase=True Profile: cores=16 memory=49152MB reserved=16GB usableMem=48GB disks=4 Num Container=8 Container Ram=6144MB Used Ram=48GB Unused Ram=16GB ***** mapred-site.xml ***** mapreduce.map.memory.mb=6144 mapreduce.map.java.opts=-Xmx4096m mapreduce.reduce.memory.mb=6144 mapreduce.reduce.java.opts=-Xmx4096m mapreduce.task.io.sort.mb=1792 ***** yarn-site.xml ***** yarn.scheduler.minimum-allocation-mb=6144 yarn.scheduler.maximum-allocation-mb=49152 yarn.nodemanager.resource.memory-mb=49152 yarn.app.mapreduce.am.resource.mb=6144 yarn.app.mapreduce.am.command-opts=-Xmx4096m ***** tez-site.xml ***** tez.am.resource.memory.mb=6144 tez.am.java.opts=-Xmx4096m ***** hive-site.xml ***** hive.tez.container.size=6144 hive.tez.java.opts=-Xmx4096m hive.auto.convert.join.noconditionaltask.size=1342177000 Any help is much appreciated. TIA!! ... View more

From this Hive script (submitted in Ambari 2.2.1.0): set hive.execution.engine=tez; SET hive.support.sql11.reserved.keywords=false; create table tweets ( contributors string, coordinates string, created_at string, entities struct < hashtags: array <string>, symbols: array <string>, urls: array <struct < display_url: string, expanded_url: string, indices: array <tinyint>, url: string>>, user_mentions: array <string>>, favorite_count tinyint, favorited boolean, filter_level string, geo string, id bigint, id_str binary, in_reply_to_screen_name string, in_reply_to_status_id string, in_reply_to_status_id_str string, in_reply_to_user_id string, in_reply_to_user_id_str string, is_quote_status boolean, lang string, place string, possibly_sensitive boolean, retweet_count tinyint, retweeted boolean, source string, text string, timestamp_ms string, truncated boolean, user struct < contributors_enabled: boolean, created_at: string, default_profile: boolean, default_profile_image: boolean, description: string, favourites_count: tinyint, follow_request_sent: string, followers_count: tinyint, following: string, friends_count: tinyint, geo_enabled: boolean, id: bigint, id_str: binary, is_translator: boolean, lang: string, listed_count: tinyint, location: string, name: string, notifications: string, profile_background_color: binary, profile_background_image_url: string, profile_background_image_url_https: string, profile_background_tile: boolean, profile_image_url: string, profile_image_url_https: string, profile_link_color: binary, profile_sidebar_border_color: binary, profile_sidebar_fill_color: binary, profile_text_color: binary, profile_use_background_image: boolean, protected: boolean, screen_name: string, statuses_count: smallint, time_zone: string, url: string, utc_offset: string, verified: boolean> ) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' AS ORC tblproperties ("orc.compress"="ZLIB"); LOAD DATA LOCAL INPATH '/home/ed/Downloads/hive-json-master/1abbo.txt' OVERWRITE INTO TABLE tweets; SELECT COUNT(*) from tweets; I am getting: Error while compiling statement: FAILED: ParseException line 4:4 missing KW_ROLE at 'hive' near 'hive' line 4:8 missing EOF at '.' near 'hive' [ERROR_STATUS] I use Hdp 2.4.2.0-258. I found the following but don't know if it any help in fixing this. Any help is much appreciated: https://www.mail-archive.com/issues@ambari.apache.org/msg00327.html ... View more