question Re: How To Process Data with Apache Pig tutorial SLOW in Archives of Support Questions (Read Only)

How To Process Data with Apache Pig tutorial SLOW

mvogt — Mon, 08 Feb 2016 04:49:10 GMT

Hello all -

Just a quick LOW PRIORITY question for anyone who has run the tutorial "How To Process Data with Apache Pig".

I created the script, and running the job as I write this. It has been running for 2 hours. Does this seem SLOW to anyone else?

I am running on a machine with an i7 processor, have 16 Gb of RAM, of which the Ambari Sandbox is utilizing 8 Gb. Are there other configuration options that should be set? Although - this seems like a massive amount of resources in use already.

Re: How To Process Data with Apache Pig tutorial SLOW

nsabharwal — Mon, 08 Feb 2016 04:51:32 GMT

@Mike Vogt

Have you configured yarn queues?

There is high probability that some other job is consuming all the resources

Check RM ui from ambari

Re: How To Process Data with Apache Pig tutorial SLOW

nsabharwal — Mon, 08 Feb 2016 04:54:53 GMT

@Mike Vogt

Make sure core components are up

Hdfs

Yarn

Mapreduce

Re: How To Process Data with Apache Pig tutorial SLOW

mvogt — Tue, 09 Feb 2016 09:29:38 GMT

Your genius level skills shine through once again! Thanks very much!

Re: How To Process Data with Apache Pig tutorial SLOW

LesterMartin — Wed, 10 Feb 2016 06:26:17 GMT

I just ran this tutorial on my 16GB i7 MBPro (gave the VM 8GB just as you) and could get it to run in 100 secs with MR and about 65 secs using Tez. I then ran the same script from the CLI and got those times down to about 60 and 25 secs on MR and Tez, respectively. I'm using the 2.3.2 Sandbox and the only thing I had to do was start the History Server was showing up red in Ambari.

Re: How To Process Data with Apache Pig tutorial SLOW

aervits — Wed, 10 Feb 2016 06:28:13 GMT

Tez benefits from warm containers so consecutive execution of same scripts should be better. Didn't know MR was performing better in CLI, can't explain that 🙂 @Lester Martin

Re: How To Process Data with Apache Pig tutorial SLOW

LesterMartin — Wed, 10 Feb 2016 06:39:40 GMT

It ~seems~ that the Ambari Views were adding about 30 seconds to the run times. Here's some of my notes around timings; notice the actual log-reported job times are pretty consistent from CLI and View runs.

Ran From	Exec Eng	Job Time	Clock Time
Ambari View	MR	64 sec	103 sec
Ambari View	Tez	25 sec	63 sec
CLI	MR	59 sec	61 sec
CLI	Tez	25 sec	27 sec

Actual job times were consistent for each execution engine (Tez twice as fast), but Ambari View ~seemed~ to add 30+ secs overall. I'm sure my the extremely constrained HDP stack on a tiny little psuedo-cluster (aka the Sandbox) is a big factor in this (understandable).

Re: How To Process Data with Apache Pig tutorial SLOW

LesterMartin — Wed, 10 Feb 2016 06:41:44 GMT

Yep, my History Server was down and had to be manually started.

Re: How To Process Data with Apache Pig tutorial SLOW

nsabharwal — Wed, 10 Feb 2016 06:46:09 GMT

@Lester Martin Thanks for testing and confirming. I think you should publish article based on your comments

Re: How To Process Data with Apache Pig tutorial SLOW

LesterMartin — Wed, 10 Feb 2016 06:48:39 GMT

I'm working with @Rafael Coss to make sure the instructions are extremely crisp as I think there are a few things that could easily trip up a novice which is who we are targeting with these tutorials.

Re: How To Process Data with Apache Pig tutorial SLOW

kabadou_rawia — Wed, 06 Dec 2017 19:56:08 GMT

@Mike Vogt @Lester Martin @Rafael Coss

Hello, I'm facing the same issue but by following the tutorial mentioned in:

https://hortonworks.com/tutorial/hadoop-tutorial-getting-started-with-hdp/section/4/.

Once i execute my pig script, it is stuck in running status as mentioned in status.png.

From RM UI, my application is also stuck in Running status as shown in rm-application.png and i attached the launched job in MapReduce in mr-job.png.

From pig view log, i got hive-log.png.

How can i resolve my issue? I'll be really grateful if you could help me.

Re: How To Process Data with Apache Pig tutorial SLOW

LesterMartin — Sun, 10 Dec 2017 04:45:24 GMT

From looking at your RM UI it sure looks like both of these jobs are basically fighting each other to get running. Meaning, the AppMaster containers are running, but they can't get anymore more containers to be run from YARN. My recommendation would be to give the VM 10GB of memory (that's how I run it on my 16GB laptop) when you restart it. I'd also try to run it from the command line just to take the Ambari View out of the picture, but if you want to run it in Ambari then kill any application via the RM UI that is around should it hang again. Good luck and happy Hadooping!