@Paul Hernandez Hey Paul - did you find a solution to this? It looks like its only parquet thats affected..csv doesnt have this problem. I too have data in subdirectories and spark sql returns null ... View more

@Gunther Hagleitner @Scott Shaw any ideas? ... View more

@Gunther Hagleitner See error below: 0: jdbc:hive2://localhost:10000> select count(*) from db1.table1; Waiting to acquire compile lock. Acquired the compile lock. ---------------------------------------------------------------------------------------------- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED ---------------------------------------------------------------------------------------------- Map 1 ......... container RUNNING 1034 1010 0 24 39 0 Reducer 2 container INITED 1 0 0 1 0 0 ---------------------------------------------------------------------------------------------- VERTICES: 00/02 [=========================>>-] 97% ELAPSED TIME: 53.36 s ---------------------------------------------------------------------------------------------- Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1525980884509_57715_1_00, diagnostics=[Task failed, taskId=task_1525980884509_57715_1_00_000565, diagnostics=[TaskAttempt 0 failed, info=[Container container_1525980884509_57715_01_000003 finished with diagnostics set to [Container failed, exitCode=-104. Container [pid=14308,containerID=container_1525980884509_57715_01_000003] is running beyond physical memory limits. Current usage: 1.6 GB of 1.5 GB physical memory used; 3.5 GB of 7.5 GB virtual memory used. Killing container. Dump of the process-tree for container_1525980884509_57715_01_000003 : I too agree that its weird a simple count(*) is getting OOM. Interestingly, if i do something like the below, it counts fine: 0: jdbc:hive2://localhost:10000> select sum(cnt) from (select count(*) cnt, run_detail_id from db1.table1 group by run_detail_id) a; Waiting to acquire compile lock. Acquired the compile lock. ---------------------------------------------------------------------------------------------- VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED ---------------------------------------------------------------------------------------------- Map 1 .......... container SUCCEEDED 1034 1034 0 0 0 0 Reducer 2 ...... container SUCCEEDED 211 211 0 0 0 0 Reducer 3 ...... container SUCCEEDED 1 1 0 0 0 0 ---------------------------------------------------------------------------------------------- VERTICES: 03/03 [==========================>>] 100% ELAPSED TIME: 47.64 s ---------------------------------------------------------------------------------------------- +------------+ | _c0 | +------------+ | 227394274 | +------------+ ... View more

@Scott Shaw Thanks Scott. That is what I had assumed but whats interesting is i have 2 tables: Table2 is much larger than 1 based off TableScan. Table 1: |TableScan[TS_0](rows=1 width=53922865152)| and Table 2: |TableScan[TS_0](rows=1 width=83922865152)| When I do a count(*) group by col0 on table 1. There is only one ID. When I do a count(*) group by col0 on table 2 there are about 5-6 ID's but still one that is heavily skewed. Table 1 gets OOM error while table 2 doesnt. Any ideas on why this would be happening? ... View more