question Re: Oozie server fails to start on Cloudera Express 5.6.0 in Support Questions

Oozie server fails to start on Cloudera Express 5.6.0

idodds — Wed, 17 Apr 2024 12:31:35 GMT

Hi All,

After starting up a Cluster that has been shutdown for some time I find that one of the Oozie servers will not stay up. It initially starts then goes down. There are two errors reported in the log file. I assume that one of them is responsible but I'm not sure which. Any help would be appreciated. The errors are:

First Error:
Source: PurgeXCommand
Message: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: The column index is out of range: 3, number of columns: 2.
FailedObject: select w.id, w.parentId from WorkflowJobBean w where w.endTimestamp < :endTime and w.parentId like '%C@%' [java.lang.String]

Second Error:
Source: CoordStatusTransitXCommand
Message: SERVER[<redacted>] USER[<redacted>] GROUP[-] TOKEN[] APP[<redacted>] JOB[0000068-201217031325641-oozie-oozi-C] ACTION[-] XException,
org.apache.oozie.command.CommandException: E0606: Could not get lock [coord_status_transit_b60ff85c-fe55-4bf5-8003-ee73935ca076], timed out [0]ms

Any guidance would be appreciated.

Best regards,

Ian.

Re: Oozie server fails to start on Cloudera Express 5.6.0

Scharan — Sat, 20 Apr 2024 14:06:07 GMT

Hello @idodds

Can you try setting oozie.wf.validate.ForkJoin=false in the oozie-site

Re: Oozie server fails to start on Cloudera Express 5.6.0

idodds — Mon, 22 Apr 2024 10:46:29 GMT

Hello @Scharan,

thanks for the advice. I've tried setting that property in the jobs.properties file. After the service restart I'm no longer getting the E0606. However, the PurgeXCommand error remains and the service still fails to stay up.

I'm beginning to suspect that I have an incorrect component version installed as it looks like PurgeXCommand is the issue. The error is being raised by openjpa-2.2.2-r422266. I'm wondering if this version is incompatible with Cloudera Express 5.6.0.

Re: Oozie server fails to start on Cloudera Express 5.6.0

Scharan — Mon, 22 Apr 2024 12:50:54 GMT

Hello @idodds

It seems to be a known issue. OPENJPA-2482, Can you try replacing OpenJPA 2.2.2 jars under /opt/cloudera/parcels/CDH/jars/ with the 2.4.2 version? You can download using the below command.

wget http://www-eu.apache.org/dist/openjpa/2.4.2/apache-openjpa-2.4.2-binary.zip

I suggest you take a backup before replacing the jars.

Re: Oozie server fails to start on Cloudera Express 5.6.0

idodds — Tue, 23 Apr 2024 15:54:10 GMT

Hello @Scharan , thanks again for the lead. I'll look into this and report back.

Thanks,

Ian.

Re: Oozie server fails to start on Cloudera Express 5.6.0

idodds — Wed, 15 May 2024 14:08:33 GMT

Hello @Scharan,

I resolved this by setting a number of oozie jobs to a status of "FAILED" directly in the oozie database. They were rogue jobs at a status of RUNNING in the database.

Thanks for your input and replies!

Ian.