question Re: Tez session hasn't been created yet. Opening session in Archives of Support Questions (Read Only)

Tez session hasn't been created yet. Opening session

dnboamo — Tue, 05 Jul 2016 13:42:29 GMT

Hello,

I'm trying to do an export table to a file using beeline.

jdbc:hive2://servername> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/custorder' select * from custorder;

But the operation is stop with : Tez session hasn't been created yet. Opening session for a very long time.

Actually, nothing happend even after one hour.

Can someone help me on this issue?

Thank very much.

Best regards,

Daniela

Re: Tez session hasn't been created yet. Opening session

ssubhas — Tue, 05 Jul 2016 14:12:57 GMT

@Daniela Boamba

Can you try running the insert query after changing execution engine to mr?

set hive.execution.engine=mr;

Thanks and Regards,

Sindhu

Re: Tez session hasn't been created yet. Opening session

dnboamo — Tue, 05 Jul 2016 14:18:55 GMT

Thank you very much Sindhu!

It is working!

Re: Tez session hasn't been created yet. Opening session

ssubhas — Tue, 05 Jul 2016 14:25:25 GMT

@Daniela Boamba

That means the issue is with Tez alone not Resource Manager. Can you please check if the Tez client is installed or not?

Thanks and Regards,

Sindhu

Re: Tez session hasn't been created yet. Opening session

dnboamo — Tue, 05 Jul 2016 14:34:33 GMT

Yes, it is.

Re: Tez session hasn't been created yet. Opening session

ssubhas — Tue, 05 Jul 2016 14:56:20 GMT

@Daniela Boamba

Can you share Resource manager log and Hive server 2log for analysis?

Re: Tez session hasn't been created yet. Opening session

dnboamo — Tue, 05 Jul 2016 18:47:03 GMT

Hi Sidhu,

hiveserver2.log

2016-07-05 07:16:51,503 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.7.0.2.3.0.0-2557, revision=ab9a0295b8de3f8711cf2bd4d837cd0b66be7cf a, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150714-0943 ] 2016-07-05 07:16:51,503 INFO [HiveServer2-Background-Pool: Thread-42]: tez.TezSessionState (TezSessionState.java:open(180)) - Opening new Tez Session (id: a549f4eb-ab4c-4f57-a180-3e573160b9d9, scratch dir: hdfs://dewdftzld058.dhcp.pgdev .sap.corp:8020/tmp/hive/hive2/_tez_session_dir/a549f4eb-ab4c-4f57-a180-3e573160b9d9) 2016-07-05 07:16:51,591 INFO [HiveServer2-Background-Pool: Thread-42]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(296)) - Timeline service address: http://servername:8188/ws/v1/timeline/ 2016-07-05 07:16:51,591 INFO [HiveServer2-Background-Pool: Thread-42]: client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at dewdftzld058.dhcp.pgdev.sap.corp/10.208.159.170:8050 2016-07-05 07:16:51,592 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(315)) - Using org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage Timeline ACLs 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(296)) - Timeline service address: http://servername:8188/ws/v1/timeline/ 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(333)) - Session mode. Starting session. 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(172)) - Using tez.lib.uris value from configuration: /hdp/apps/2.3.0.0-2557/tez/tez.tar.gz 2016-07-05 07:16:51,663 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory hdfs://dewdftzld058.dhcp.pgdev.sap.corp:8020/tmp/hive/hive2/_tez_ session_dir/a549f4eb-ab4c-4f57-a180-3e573160b9d9/.tez/application_1467702596252_0002 doesn't exist and is created 2016-07-05 07:16:51,694 INFO [HiveServer2-Background-Pool: Thread-42]: acls.ATSHistoryACLPolicyManager (ATSHistoryACLPolicyManager.java:createSessionDomain(168)) - Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1 467702596252_0002 2016-07-05 07:16:52,370 INFO [HiveServer2-Background-Pool: Thread-42]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted application application_1467702596252_0002 2016-07-05 07:16:52,372 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: http://servername:8088/proxy/application_1467702596252_0002/ 2016-07-05 07:16:58,918 INFO [HiveServer2-Handler-Pool: Thread-34]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2016-07-05 07:16:58,921 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.ObjectStore (ObjectStore.java:initialize(289)) - ObjectStore, initialize called 2016-07-05 07:16:58,926 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is MYSQL 2016-07-05 07:16:58,926 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.ObjectStore (ObjectStore.java:setConf(272)) - Initialized ObjectStore 2016-07-05 07:16:58,932 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created local directory: /tmp/ade9eb71-bf8e-48f0-9f60-1052457f2c53_resources 2016-07-05 07:16:58,935 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created HDFS directory: /tmp/hive/hive2/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:16:58,936 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created local directory: /tmp/hive/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:16:58,938 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created HDFS directory: /tmp/hive/hive2/ade9eb71-bf8e-48f0-9f60-1052457f2c53/_tmp_space.db 2016-07-05 07:16:58,940 INFO [HiveServer2-Handler-Pool: Thread-34]: session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(236)) - Operation log session directory is created: /home/hive/${system:java.io.tmpdir}/${system :user.name}/operation_logs/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:17:01,383 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use default 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse start=1467703021384 end=1467703021385 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: get_database: default 2016-07-05 07:17:01,386 INFO [HiveServer2-Handler-Pool: Thread-34]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hive2 ip=unknown-ip-addr cmd=get_database: default 2016-07-05 07:17:01,392 INFO [HiveServer2-Handler-Pool: Thread-34]: ql.Driver (Driver.java:compile(436)) - Semantic Analysis Completed 2016-07-05 07:17:01,392 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=semanticAnalyze start=1467703021385 end=1467703021392 duration=7 from=org.apache.hadoop.hive.ql.Driv er>

Re: Tez session hasn't been created yet. Opening session

colton_rodgers — Wed, 06 Jul 2016 01:18:31 GMT

The problem could lie with not enough resources available for a Tez session to start. Could you run

yarn application -list

on a YARN client host and see if there are any processes hogging up resources or possibly check the configs and logs to make sure that Tez isn't just hanging and waiting for resources to become available.

Re: Tez session hasn't been created yet. Opening session

mkumar13 — Wed, 06 Jul 2016 17:56:24 GMT

If YARN cluster dont have capacity to launch AM then Hive CLI waits indefinitely to launch an AM. Tez have its own optimization step where it pre-allocate resources saving the user the overhead of launching a AM everytime a new query launch from same session. But when it lanches using mr it dont have any condition called preallocate and works successfully.

Re: Tez session hasn't been created yet. Opening session

zyang — Fri, 08 Jul 2016 04:14:45 GMT

This log is incomplete and doesn't help. Would you mind sharing the hive log when you trigger the same error? Your can share it via pastebin or collabedit.

Re: Tez session hasn't been created yet. Opening session

dnboamo — Fri, 08 Jul 2016 14:58:05 GMT

HI Rodgers,

Please find bellow the result of the command :

[spark@servername ~]$ yarn application -list 16/07/08 07:04:34 INFO impl.TimelineClientImpl: Timeline service address: http://servername:8188/ws/v1/timeline/ 16/07/08 07:04:34 INFO client.RMProxy: Connecting to ResourceManager at servername/ip:8050 Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):5 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1467377616814_0001 SparkSQL::servername SPARK spark default RUNNING UNDEFINED 10% http://servername:4040 application_1467377616814_0007 HIVE-701d4281-08f4-4bc4-aab3-724c931fc606 TEZ spark default ACCEPTED UNDEFINED 0% N/A application_1467377616814_0005 HIVE-55924703-c794-413f-8df0-fbb464b48885 TEZ spark default ACCEPTED UNDEFINED 0% N/A application_1467377616814_0010 HIVE-9ef8ceda-b31f-4311-8b4a-288b5f476e93 TEZ hive2 default ACCEPTED UNDEFINED 0% N/A application_1467702596252_0002 HIVE-a549f4eb-ab4c-4f57-a180-3e573160b9d9 TEZ hive2 default ACCEPTED UNDEFINED 0% N/A

thank you very much.

Best regards,

daniela

Hi Rodgers,

please find bellow the result of the yarn application -list

[spark@servername ~]$ yarn application -list 16/07/08 07:04:34 INFO impl.TimelineClientImpl: Timeline service address: http://servername:8188/ws/v1/timeline/ 16/07/08 07:04:34 INFO client.RMProxy: Connecting to ResourceManager at servername/10.208.159.170:8050 Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):5 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1467377616814_0001 SparkSQL::servername SPARK spark default RUNNING UNDEFINED 10% http://servername:4040 application_1467377616814_0007 HIVE-701d4281-08f4-4bc4-aab3-724c931fc606 TEZ spark default ACCEPTED UNDEFINED 0% N/A application_1467377616814_0005 HIVE-55924703-c794-413f-8df0-fbb464b48885 TEZ spark default ACCEPTED UNDEFINED 0% N/A application_1467377616814_0010 HIVE-9ef8ceda-b31f-4311-8b4a-288b5f476e93 TEZ hive2 default ACCEPTED UNDEFINED 0% N/A application_1467702596252_0002 HIVE-a549f4eb-ab4c-4f57-a180-3e573160b9d9 TEZ hive2 default ACCEPTED UNDEFINED 0% N/A

Thank you,

Daniela

Re: Tez session hasn't been created yet. Opening session

colton_rodgers — Fri, 08 Jul 2016 21:21:13 GMT

If you could kill all of those processes (if your cluster will allow) and try running again. It looks like a lot of Tez resources are busy and thus could be why you are experiencing the hanging. If not you may need to look into YARN Queue Manager

Re: Tez session hasn't been created yet. Opening session

zhouyf_zz — Wed, 01 Aug 2018 16:57:55 GMT

I have the same issue with you.

I solved it by modifying the following..

reduce Tez config : tez.am.resource.memory.mb =256

default 2048 .

and Increase Yarn ： Memory allocated for all YARN containers on a node 。

I guess on hive cli use Tez need {tez.am.resource.memory.mb} memory . when {Memory allocated for all YARN containers on a node} no have enough memory . Then Application-id is hang.

Re: Tez session hasn't been created yet. Opening session

dnboamo — Thu, 02 Aug 2018 13:41:10 GMT

Already done this. 😞