Thanks! because i have null values in my data set as well, i used colaesce, and it worked! Your query was the basis though, so thanks again! Query by @nramanaiah that worked for me, as I have null records in the dataset: select Currency, (coalesce(spend_a,0)) + (colaesce(spend_b,0)) + coalesce(spend_c,0)) + coalesce(spend_d,0)) as total_spend from test   ... View more

If its a tez application, AM logs will show how much memory is currently allocated/consumed by the application & how much free resources available in the queue at that specific time. eg., 2024-04-22 23:27:20,636 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:843776, vCores:206> Free: <memory:2048, vCores:306> pendingRequests: 0 delayedContainers: 205 heartbeats: 101 lastPreemptionHeartbeat: 100 2024-04-22 23:27:30,660 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:155648, vCores:38> Free: <memory:495616, vCores:356> pendingRequests: 0 delayedContainers: 38 heartbeats: 151 lastPreemptionHeartbeat: 150 This allocation details will be logged frequently in Tez AM logs. ... View more

Thank you, guys, for your answers. ... View more

@nramanaiah have been able to run further testing and confirm that my partitions are purging as expected! thanks again for the assistance! ... View more

@ho_ddeok, Has any of the replies helped resolve your issue? If so, can you please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future?  ... View more

@Hafiz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  ... View more

Not Yet ... View more

@dmharshit  Yarn queue where the Tez application gets submitted don't have enough capacity to start a new application.   You should be able to see tez application in yarn rm -> accepted applications. Tune NM capacity + reduce following configs to let yarn accept & launch new application. -- Setting AM size to 512mb set tez.am.resource.memory.mb=512; -- Setting Task container size to 512mb set hive.tez.container.size=512; ... View more

I know this post was a long time ago, but does the script below add text qualifiers if a field contains one or more  '|'?   INSERT OVERWRITE DIRECTORY '/tmp/text-pipe' ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE SELECT * FROM textcomma; Thanks!  ... View more

I suspect the cause for "alter table" slowness is same as described here HIVE-23959 & HIVE-23806 might help your situation.     ... View more