Support Questions
Find answers, ask questions, and share your expertise

Workflow queue assignments issues & Pig failing to run in workflow

Explorer

Running HPD 2.6.4

Ambari 2.6.1

I've inherited an issue. So the workflow co-ordinator has been deployed using Hue, first I wanted to move away from Hue and keep things hortonworks by creating the workflow and co-ordinator in the workflow manager in ambari.

I've installed the view into ambari and created the workflow and set up all the actions in the job as they were in Hue. When I try and run the pig portion it fails to run though. I'm not sure what logs I'm supposed to be looking at to find out why this fails but all I'm seeing is info messages and no reason as to why it failed. It failes in under 20 seconds so it looks like it is having issues with creating the environment for pig rather than something in the job itself. It does however run happly from the co-ordinator that has been created in Hue though.

Thinking I am struggling with getting pig to run in the workflow manager view I tried to tweek the job in Hue, I've set the properties on the workflow of "mapred.job.queue.name"="dataprocessing" which worked for all the Mapreduce type jobs but not those that ran in TEZ (for hive) which when I stoped and thought about it made sence so I found the variable "tez.queue.name" and set that to "dataprocessing" to but the TEZ jobs are still going to default.

I want this running in data processing so that everyone else can happily muck about in default till we get some better practices set up but this job will have it's own queue with reserved resource and will run happily in the background regardless of what large jobs anyone else runs in the rest of the cluster. I'm a infrastructure guy so not dumb but workflows and pig are all a bit new to me so this may be something glaringly obvious I'm just not seeing so any advice would be greatly appreceated both on getting the TEZ stuff to run in the right queue and getting the whole thing migrated to work flow manager so I can remove all this Hue stuff out of the system.