Hi @suresh krish Sorry about the late response. You can increase HADOOP_HEAPSIZE and restarts HiveServer2 via Ambari. If your cluster is 23, I suggest it to increase to around 6GB. ... View more

Hi, @suresh krish How big is your cluster and what is your current HADOOP_HEAPSIZE in hive-env.sh at hiveserver2 host? One possible solution is to increase it. ... View more

@k v vittal hebbar This may be a dumb advice, but the error suggests that 'volumes/disk1/test1.txt' does not exist. You can try to give a full path to see if that resolves an issue. ... View more

Hi @kavitha velaga What is your select query? If you are executing a simple select query on orc table, it may be just using stats and not triggering any yarn job. That's why you don't see it in application log. ... View more

Hi @mike harding I don't know the way of find time remaining, but you can certainly see the progress via beeline. Let's say you are going to run 'select count(*)' from some table and explain show one mapper and one reducer: 0: jdbc:hive2://nat-d7-0629-hs2-12-rangeroff-> explain select count(*) from olympic_avro; +-------------------------------------------------------------------------------------------------------------+--+ | Explain | +-------------------------------------------------------------------------------------------------------------+--+ | STAGE DEPENDENCIES: | | Stage-1 is a root stage | | Stage-0 depends on stages: Stage-1 | | | | STAGE PLANS: | | Stage: Stage-1 | | Tez | | DagId: hive_20160706123306_5e633ebb-9c9c-43df-8fb2-dc16c59ccf72:332 | | Edges: | | Reducer 2 <- Map 1 (SIMPLE_EDGE) | | Vertices: | | Map 1 | | Map Operator Tree: | | TableScan | | alias: olympic_avro | | Statistics: Num rows: 1 Data size: 545870 Basic stats: COMPLETE Column stats: COMPLETE | | Select Operator | | Statistics: Num rows: 1 Data size: 545870 Basic stats: COMPLETE Column stats: COMPLETE | | Group By Operator | | aggregations: count() | | mode: hash | | outputColumnNames: _col0 | | Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE | | Reduce Output Operator | | sort order: | | Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE | | value expressions: _col0 (type: bigint) | | Reducer 2 | | Reduce Operator Tree: | | Group By Operator | | aggregations: count(VALUE._col0) | | mode: mergepartial | | outputColumnNames: _col0 | | Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE | | File Output Operator | | compressed: false | | Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE | | table: | | input format: org.apache.hadoop.mapred.TextInputFormat | | output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | | serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | | | | Stage: Stage-0 | | Fetch Operator | | limit: -1 | | Processor Tree: | | ListSink | | | +-------------------------------------------------------------------------------------------------------------+--+ 48 rows selected (0.262 seconds) When you run this via beeline, you can see the progress of the mapper and reducer as shown in the below: 0: jdbc:hive2://nat-d7-0629-hs2-12-rangeroff-> select count(*) from olympic_avro; INFO : Session is already open INFO : Dag name: select count(*) from olympic_avro(Stage-1) INFO : INFO : Status: Running (Executing on YARN cluster with App id application_1467208466253_0312) INFO : Map 1: 0/1Reducer 2: 0/1 INFO : Map 1: 0/1Reducer 2: 0/1 INFO : Map 1: 0/1Reducer 2: 0/1 INFO : Map 1: 0/1Reducer 2: 0/1 INFO : Map 1: 0(+1)/1Reducer 2: 0/1 INFO : Map 1: 0(+1)/1Reducer 2: 0/1 INFO : Map 1: 1/1Reducer 2: 0/1 INFO : Map 1: 1/1Reducer 2: 0(+1)/1 INFO : Map 1: 1/1Reducer 2: 1/1 +-------+--+ | _c0 | +-------+--+ | 8618 | +-------+--+ 1 row selected (16.531 seconds) ... View more

@soumyabrata kole, Once you confirm the issue with the older version, I appreciate if you can mark my above answer as the best answer. Thanks! ... View more

Hi @soumyabrata kole In order to go back to Hive ver 0.11, I have to go back to HDP 1.3 while the latest version of HDP is 2.4. Since I don't have access to a cluster with HDP 1.3, I cannot say for sure, but your issue could be only with older version of Hive. At least, I did not face your issue with Hive 1.2.1. ... View more

Hi @soumyabrata kole, Sorry about misunderstanding your question. I tried with hive ver 1.2.1 and could not reproduce your issue as shown in the below. Can you tell us what version of hive or HDP are you using? hive> create table olympic_avro > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' > STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' > OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' > tblproperties ('avro.schema.literal'='{ > "name": "my_record", > "type": "record", > "fields": [ > {"name":"athelete", "type":"string"}, > {"name":"age", "type":"int"}, > {"name":"country", "type":"string"}, > {"name":"year", "type":"string"}, > {"name":"closing", "type":"string"}, > {"name":"sport", "type":"string"}, > {"name":"gold", "type":"int"}, > {"name":"silver", "type":"int"}, > {"name":"bronze", "type":"int"}, > {"name":"total", "type":"int"} > ]}'); OK Time taken: 0.499 seconds hive> select count(*) from olympic_avro; Query ID = hrt_qa_20160704004020_5a1590ac-63cb-40bc-b1a8-92e28a709cb3 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1467208466253_0272) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 SUCCEEDED 0 0 0 0 0 0 Reducer 2 ...... SUCCEEDED 1 1 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 14.15 s -------------------------------------------------------------------------------- OK 0 Time taken: 18.618 seconds, Fetched: 1 row(s) hive> load data local inpath '/home/hrt_qa/olympic.avro' into table olympic_avro; Loading data to table default.olympic_avro Table default.olympic_avro stats: [numFiles=1, totalSize=545870] OK Time taken: 0.688 seconds hive> select * from olympic_avro limit 5; OK Michael Phelps23United States200808-24-08Swimming8008 Michael Phelps19United States200408-29-04Swimming6028 Michael Phelps27United States201208-12-12Swimming4206 Natalie Coughlin25United States200808-24-08Swimming1236 Aleksey Nemov24Russia200010-01-00Gymnastics2136 Time taken: 0.234 seconds, Fetched: 5 row(s) ... View more

Hi @soumyabrata kole, I see you place user.avsc in hdfs file, but do you also have that file in local file system since you run 'LOAD DATA LOCAL INPATH'? Instead of 'LOCAL DATA LOCAL INPATH', you could run 'LOCAL DATA INPATH'. But the better solution would be switch your Step 1 and Step 2. If you place a source file first and then create a table, then data should be automatically loaded into a table. Or if you don't want to lose your source file ever when you drop a table, you can create 'external' table. Then if even if you drop a table, user.avsc won't get deleted. ... View more

Hi @suresh krish As @Rajkumar Singh advised, that parameter cannot be modified at runtime. That means you have to modify that in hive-site.xml and restart HiveServer2. Then the change should be in effect when you connect to HS2 via beeline. ... View more

Hi @khaja pasha shaik The first thing I noticed in your connection string is that it contains 'hostname'. Please replace that with whatever `hostname -f` returns. ... View more

@Aidan, The file is on Linux, not HDFS. hive-site.xml should be located on /etc/hive/conf/hive-site.xml. If hive.query.log.location is undefined in hive-site.xml. You can run 'set hive.query.log.location' via CLI to get the value of hive.query.log.location. Hope this helps. ... View more

@Aidan, Hive query log is located at the value of hive.query.log.location in hive-site.xml. By default it should be at /tmp/[user_name]. In that log, you can see hive queries. ... View more

@suresh, Can you try with '\u0001' instead? ... View more

@Saleema, You can move files to some hdfs dir where hive has write permission, for example, /user/[usrname] and then create external table via hive CLI like the below: create external table (i int) row formate delimited fields terminated by ',' stored as textfile location '/user/[usrname]'; More details: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/moving_data_from_hdfs_to_hive_external_table_method.html ... View more

There was a known issue with empty partitions/files/tables: https://issues.apache.org/jira/browse/HIVE-13394 Is your case such a case? ... View more

The error suggests that your YARN Application Timeline Server may be down. Can you check to see if it is running? ... View more

Hi Antonio Arfe Can you connect to HiveServer2 via beeline with your connection string? If you can, the error may suggest that an yarn app is not starting for some reason. You can verify that by visiting resource manager UI. ... View more

I see you are using constant value for partition column. You might be hitting this issue: https://issues.apache.org/jira/browse/HIVE-12893 ... View more

An example of jdbc connection string is the below: jdbc:hive2:/[your_zookeeper_quorum]/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@EXAMPLE.COM;transportMode=http;httpPath=cliservice You might be specifying httpPath to something other than 'cliservice' in jdbc connection string. ... View more