Tez session hasn't been created yet. Opening session

[#1]

Hello,

I'm trying to do an export table to a file using beeline.

jdbc:hive2://servername> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/custorder' select * from custorder;

But the operation is stop with : Tez session hasn't been created yet. Opening session for a very long time.

Actually, nothing happend even after one hour.

Can someone help me on this issue?

Thank very much.

Best regards,

Daniela

[#2 · Accepted solution]

@Daniela Boamba

Can you try running the insert query after changing execution engine to mr?

set hive.execution.engine=mr;

Thanks and Regards,

Sindhu

[#3]

Already done this. 😞

[#4]

Thank you very much Sindhu!

It is working!

[#5]

@Daniela Boamba

That means the issue is with Tez alone not Resource Manager. Can you please check if the Tez client is installed or not?

Thanks and Regards,

Sindhu

[#6]

Yes, it is.

[#7]

@Daniela Boamba

Can you share Resource manager log and Hive server 2log for analysis?

[#8]

Hi Sidhu,

hiveserver2.log

2016-07-05 07:16:51,503 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.7.0.2.3.0.0-2557, revision=ab9a0295b8de3f8711cf2bd4d837cd0b66be7cf a, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150714-0943 ] 2016-07-05 07:16:51,503 INFO [HiveServer2-Background-Pool: Thread-42]: tez.TezSessionState (TezSessionState.java:open(180)) - Opening new Tez Session (id: a549f4eb-ab4c-4f57-a180-3e573160b9d9, scratch dir: hdfs://dewdftzld058.dhcp.pgdev .sap.corp:8020/tmp/hive/hive2/_tez_session_dir/a549f4eb-ab4c-4f57-a180-3e573160b9d9) 2016-07-05 07:16:51,591 INFO [HiveServer2-Background-Pool: Thread-42]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(296)) - Timeline service address: http://servername:8188/ws/v1/timeline/ 2016-07-05 07:16:51,591 INFO [HiveServer2-Background-Pool: Thread-42]: client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at dewdftzld058.dhcp.pgdev.sap.corp/10.208.159.170:8050 2016-07-05 07:16:51,592 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(315)) - Using org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager to manage Timeline ACLs 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(296)) - Timeline service address: http://servername:8188/ws/v1/timeline/ 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(333)) - Session mode. Starting session. 2016-07-05 07:16:51,649 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(172)) - Using tez.lib.uris value from configuration: /hdp/apps/2.3.0.0-2557/tez/tez.tar.gz 2016-07-05 07:16:51,663 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory hdfs://dewdftzld058.dhcp.pgdev.sap.corp:8020/tmp/hive/hive2/_tez_ session_dir/a549f4eb-ab4c-4f57-a180-3e573160b9d9/.tez/application_1467702596252_0002 doesn't exist and is created 2016-07-05 07:16:51,694 INFO [HiveServer2-Background-Pool: Thread-42]: acls.ATSHistoryACLPolicyManager (ATSHistoryACLPolicyManager.java:createSessionDomain(168)) - Created Timeline Domain for History ACLs, domainId=Tez_ATS_application_1 467702596252_0002 2016-07-05 07:16:52,370 INFO [HiveServer2-Background-Pool: Thread-42]: impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted application application_1467702596252_0002 2016-07-05 07:16:52,372 INFO [HiveServer2-Background-Pool: Thread-42]: client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: http://servername:8088/proxy/application_1467702596252_0002/ 2016-07-05 07:16:58,918 INFO [HiveServer2-Handler-Pool: Thread-34]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2016-07-05 07:16:58,921 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.ObjectStore (ObjectStore.java:initialize(289)) - ObjectStore, initialize called 2016-07-05 07:16:58,926 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(139)) - Using direct SQL, underlying DB is MYSQL 2016-07-05 07:16:58,926 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.ObjectStore (ObjectStore.java:setConf(272)) - Initialized ObjectStore 2016-07-05 07:16:58,932 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created local directory: /tmp/ade9eb71-bf8e-48f0-9f60-1052457f2c53_resources 2016-07-05 07:16:58,935 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created HDFS directory: /tmp/hive/hive2/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:16:58,936 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created local directory: /tmp/hive/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:16:58,938 INFO [HiveServer2-Handler-Pool: Thread-34]: session.SessionState (SessionState.java:createPath(641)) - Created HDFS directory: /tmp/hive/hive2/ade9eb71-bf8e-48f0-9f60-1052457f2c53/_tmp_space.db 2016-07-05 07:16:58,940 INFO [HiveServer2-Handler-Pool: Thread-34]: session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(236)) - Operation log session directory is created: /home/hive/${system:java.io.tmpdir}/${system :user.name}/operation_logs/ade9eb71-bf8e-48f0-9f60-1052457f2c53 2016-07-05 07:17:01,383 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: use default 2016-07-05 07:17:01,384 INFO [HiveServer2-Handler-Pool: Thread-34]: parse.ParseDriver (ParseDriver.java:parse(209)) - Parse Completed 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=parse start=1467703021384 end=1467703021385 duration=1 from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 2016-07-05 07:17:01,385 INFO [HiveServer2-Handler-Pool: Thread-34]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(746)) - 1: get_database: default 2016-07-05 07:17:01,386 INFO [HiveServer2-Handler-Pool: Thread-34]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(371)) - ugi=hive2 ip=unknown-ip-addr cmd=get_database: default 2016-07-05 07:17:01,392 INFO [HiveServer2-Handler-Pool: Thread-34]: ql.Driver (Driver.java:compile(436)) - Semantic Analysis Completed 2016-07-05 07:17:01,392 INFO [HiveServer2-Handler-Pool: Thread-34]: log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG method=semanticAnalyze start=1467703021385 end=1467703021392 duration=7 from=org.apache.hadoop.hive.ql.Driv er>

[#9]

This log is incomplete and doesn't help. Would you mind sharing the hive log when you trigger the same error? Your can share it via pastebin or collabedit.

[#10]

The problem could lie with not enough resources available for a Tez session to start. Could you run

yarn application -list

on a YARN client host and see if there are any processes hogging up resources or possibly check the configs and logs to make sure that Tez isn't just hanging and waiting for resources to become available.