question Re: TezSessionManager - Exception while waiting for Tez client to be ready in Support Questions

TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 18:03:31 GMT

Hi folks,

I'm following HDP Hello World tutorial. In Lab 3 when I execute the PIG script by checking "Execute on Tez" checkbox I got the following error. I think I have a configuration issue.

Have an idea?

I've attached the log file.

Thanks

Özgür

[PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager - Exception while waiting for Tez client to be ready

 org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration

 	at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:168)

 	at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:841)

 	at org.apache.tez.client.TezClient.start(TezClient.java:380)

 	at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:96)

 	at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:211)

 	at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:158)

 	at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:195)

 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)

 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

 	at java.lang.Thread.run(Thread.java:745)

 2016-01-25 11:06:57,500 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG

tezerror.txt

Re: TezSessionManager - Exception while waiting for Tez client to be ready

bleonhardi — Mon, 25 Jan 2016 18:20:57 GMT

There are different possibilities. Normally this means the tez libraries are not present in HDFS. Are you using the sandbox?

You should check if the tez client is installed on your pig client, if the tez-site.xml contains the tez.lib.uris property and if the tez libraries are actually in HDFS and valid ( download them and untar to check ) /hdp/apps/<hdp_version>/tez/tez.tar.gz

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-ffec9e6b-41f4-47de-b5cd-1403b4c4a7c8.1.html

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 19:27:25 GMT

Thanks @Benjamin Leonhardi 🙂

No not sandbox, I have 6 node vm cluster, 3 data, 2 master and 1 ambari server. 2.3.4 is HDP version, 2.2 is Ambari version.

I have no error on Ambari, I have a TEZ 0.7.0.2.3.4.0-1310 instance.

I'll use hadoof fs -ls when checking right? if you can give a hint, it'll be great...

I'll check the link you sent.

Appreciate your help.

Regards.

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 25 Jan 2016 19:34:08 GMT

@Özgür Akdemirci

Plesse see this

2016-01-25 11:33:01,324 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: /hdp/apps/2.3.2.0-2950/tez/tez.tar.gz

job-1453721160224-0004-logs.txt

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 19 Aug 2019 11:18:52 GMT

Make sure you have the above entry under Tez

Re: TezSessionManager - Exception while waiting for Tez client to be ready

bleonhardi — Mon, 25 Jan 2016 19:36:46 GMT

wrote as an answer because of the character limit:

yes first go into ambari or perhaps better the OS and search for the tez.lib.uris property in the properties file

less /etc/tez/conf/tez-site.xml

You should find something like this:

<value>/hdp/apps/${hdp.version}/tez/tez.tar.gz</value>

if this is not available you may have a different problem. ( Tez client not installed some configuration issue)

You can then check if these files exist in HDFS with

hadoop fs -ls /hdp/apps/

find the version number for example 2.3.2.0-2950

[root@sandbox ~]# hadoop fs -ls /hdp/apps/2.3.2.0-2950/tez

Found 1 items

-r--r--r-- 3 hdfs hadoop 56926645 2015-10-27 14:40 /hdp/apps/2.3.2.0-2950/tez/tez.tar.gz

You can check if this file is corrupted somehow with

hadoop fs -get /hdp/apps/2.3.2.0-2950/tez/tez.tar.gz

You can then try to untar it to see if that works.

If the file doesn't exist in HDFS you can find it in the installation directory of HDP (/usr/hdp/2.3.2.0-2950/tez/lib/tez.tar.gz on the local filesystem ) You could then put it into hdfs

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 25 Jan 2016 19:38:27 GMT

location /usr/hdp/2.3.4.0-3276/tez/lib/tez.tar.gz

[root@phdns02 conf]# hdfs dfs -ls /hdp/apps/2.3.4.0-3276/tez

Found 1 items

-r--r--r-- 1 hdfs hadoop 58357866 2015-11-19 11:36 /hdp/apps/2.3.4.0-3276/tez/tez.tar.gz

[root@phdns02 conf]#

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 19:52:29 GMT

Thanks @Benjamin Leonhardi

ssh-tez.png I get the tar, and untar it without an error.

Thanks @Neeraj Sabharwal I have that setting.

tez-ok.png

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 25 Jan 2016 20:06:44 GMT

@Özgür Akdemirci Did you see the job log that I attached?

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 20:22:17 GMT

@Neeraj SabharwalI've checked, but I didnot understand ;),

It's a successful job log for your cluster I think...

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 25 Jan 2016 20:28:54 GMT

@Özgür Akdemirci It made the call to tez uri. In your case, it cannot fine files related to tez.lib.uri

2016-01-25 11:33:01,324 [PigTezLauncher-0] INFO org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from configuration: /hdp/apps/2.3.2.0-2950/tez/tez.tar.gz

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 20:35:11 GMT

That's ok, but I've sent images, as you can see there files exist.

Thanks anyway...

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 23:03:55 GMT

The last row gives the error, If it's commented out, no error.

It must be about hive configuration. I'm using mysql jdbc jars. I'll check for it...

a = LOAD 'geolocation' using org.apache.hive.hcatalog.pig.HCatLoader();

b = filter a by event != 'normal';

c = foreach b generate driverid, event, (int) '1' as occurance;

d = group c by driverid;

e = foreach d generate group as driverid, SUM(c.occurance) as t_occ;

g = LOAD 'drivermileage' using org.apache.hive.hcatalog.pig.HCatLoader();

h = join e by driverid, g by driverid; final_data = foreach h generate $0 as driverid, $1 as events, $3 as totmiles, (float) $3/$1 as riskfactor;

/*store final_data into 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();*/

Re: TezSessionManager - Exception while waiting for Tez client to be ready

nsabharwal — Mon, 25 Jan 2016 23:07:45 GMT

@Özgür Akdemirci I am following this http://hortonworks.com/hadoop-tutorial/faster-pig-tez/

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 23:16:28 GMT

@Neeraj Sabharwal

and this is mine, I'll check yours. Thanks.

http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#section_5

Re: TezSessionManager - Exception while waiting for Tez client to be ready

aervits — Mon, 25 Jan 2016 23:19:03 GMT

@Özgür Akdemirci

you're saving a relation riskfactor and then you're trying to store final_data into it. Riskfactor is already used. Try giving the output a new name.

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Mon, 25 Jan 2016 23:33:23 GMT

The pig script is from helloworld tutorial indeed,just copied and pasted. The first one is a column name, the second is an orc table i created in hive. But i'll change the values and try. Hope its the reason. Thanks @Artem Ervits

But tomorrow ;(

also tried to out to an hdfs file. Same error.

u may be right ;)ihope

Re: TezSessionManager - Exception while waiting for Tez client to be ready

aervits — Mon, 25 Jan 2016 23:39:54 GMT

@Özgür Akdemirci there are a lot of comments on the tutorial similar to your issues. I'm actually wrong and leading you on a wrong path. Try to read the comments under the tutorial, did you try to run with -useHCatalog? It is case sensitive.

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Tue, 26 Jan 2016 00:59:35 GMT

@Artem Ervits I've already tried -useHCatalog. I'll read other comments, hope thereis a solution. there were 2 pig clients 3 tez clients on my servers, i installed one more pig client. But that did not solve either. I read a bug fix about that.

. Thanks..

Re: TezSessionManager - Exception while waiting for Tez client to be ready

oakdemirci — Tue, 26 Jan 2016 16:29:17 GMT

Hi guys, have a great day,

Is there any place on ambari, that I can check environment variables like PIG_CLASSPATH, CLASSPATH etc, do they exist?

I validate my pig installation by this Pig validation

and tez by tez validation both are ok, when checked alone they are working. 😄

but my pig script does not work on tez, After reading Pig on Tez Mode link I come to this point, check these environment variables.

@Artem Ervits I read the comments, they are different than mine... 😞