Oozie Stuck in PREP - HDP 2.5.3

[#1]

When running any jobs, that were previously working, the job goes to PREP state (after the Start). Jobs were previously working with no issue.

When running an Oozie Service check, the following is received.

We recently upgraded from 2.3.0 to 2.5.3 and the jobs ran for a couple of days with no issue then in the middle of the day the jobs started acting like this.

When on the phone with Hortonworks Tech Support, we did set oozie.service.AuthorizationService.security.enabled to false and it temporarily started working and then quit again.

We do have HA enabled, but any of the posts that I have found didn't fix the issue.

[#2]

@Jesse Passwaters please share job.properties and workflow.xml files

[#3]

Here are the 2 files. This is what is used when I run the service check which fails.

workflow.xml

jobproperties.txt

[#4 · Accepted solution]

After further troubleshooting and the changing of properties (with no success), I took the last resort of deleting the Oozie service and setting up on a new DB.

For now everything is back to normal. Root cause I suspect is db corruption.

This StackOverflow post kind of led me to do the inevitable.