Created on 05-16-2016 10:15 AM - edited 08-18-2019 06:03 AM
Stack : Installed HDP-2.3.2.0-2950 using Ambari 2.1.
Tables from a SQL Server schema have been imported on HDFS using Sqoop.
The Hive config in Ambari is as follows(Tez is the execution engine) :
There is a huge table with 24187713416 rows but it is just an ordinary external table with no partitioning etc., when I execute a select query, following is the output(I couldn't figure out if Tez is in action) :
hive (odp_dw_may2016)> > > select * from FactSampleValue where Snapshot_Id = 643067 and SmapiName_ver = 'Abnormal_ShutDown_ct_1' and SmapiColName = 'Abnormal_ShutDown_ct_1_000'; Query ID = hive_20160516092942_4b1c90ab-77bd-4ea8-bf3e-9af611a63143 Total jobs = 1 Launching Job 1 out of 1 Tez session was closed. Reopening... Session re-established. Status: Running (Executing on YARN cluster with App id application_1446726117927_0076) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 .......... SUCCEEDED 3098 3098 0 0 0 0 -------------------------------------------------------------------------------- VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 1341.80 s -------------------------------------------------------------------------------- OK factsamplevalue.chassino factsamplevalue.snapshot_id factsamplevalue.ecu_id factsamplevalue.smapiname_ver factsamplevalue.smapicolname factsamplevalue.indexet factsamplevalue.value factsamplevalue.samplefrequency factsamplevalue.vectorperc factsamplevalue.smapiname factsamplevalue.diffvalue factsamplevalue.commvalue 1872663 643067 28 Abnormal_ShutDown_ct_1 Abnormal_ShutDown_ct_1_000 0 1.0 NULL 100.0 Abnormal_ShutDown_ct NULL NULL Time taken: 1350.861 seconds, Fetched: 1 row(s)
I have several questions pertaining only to the usage of Tez :
Created 05-16-2016 11:14 AM
1) yes you can see the "Tez session was closed ...
2) In anything after HDP2 tez is enabled by default. MapReduce might be going away as an option anyway
3) You can still use set execution engine in queries set hive.execution.engine=mr or tez
4) Not sure what you mean with utiliy. The Tez view in ambari would provide the functionality I am not completely sure about the out of the box integration with resource manager
Created 05-16-2016 11:14 AM
1) yes you can see the "Tez session was closed ...
2) In anything after HDP2 tez is enabled by default. MapReduce might be going away as an option anyway
3) You can still use set execution engine in queries set hive.execution.engine=mr or tez
4) Not sure what you mean with utiliy. The Tez view in ambari would provide the functionality I am not completely sure about the out of the box integration with resource manager