Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Highlighted

403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

New Contributor
 
9 REPLIES 9
Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Mentor

please provide logs

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

New Contributor

haproxy load balancer logs.

Mar 21 06:34:23 localhost haproxy[28217]: 10.2.0.19:43610 [21/Mar/2016:06:34:23.401] oozie oozie_servers/app2 4/0/0/1/5 403 1389 - - ---- 1/1/0/0/0 0/0 "GET /oozie/?user.name=oozie HTTP/1.1"

Mar 21 06:34:57 localhost haproxy[28217]: 10.2.0.25:59788 [21/Mar/2016:06:34:57.697] oozie oozie_servers/app1 0/0/0/1/1 401 1252 - - ---- 1/1/0/0/0 0/0 "GET /oozie/?user.name=oozie HTTP/1.1"

Mar 21 06:34:57 localhost haproxy[28217]: 10.2.0.25:59788 [21/Mar/2016:06:34:57.698] oozie oozie_servers/app2 1/0/0/2/3 403 1389 - - ---- 1/1/0/0/0 0/0 "GET /oozie/?user.name=oozie HTTP/1.1"

Mar 21 06:35:23 localhost haproxy[28217]: 10.2.0.19:43622 [21/Mar/2016:06:35:23.149] oozie oozie_servers/app1 0/0/0/1/1 401 1252 - - ---- 1/1/0/0/0 0/0 "GET /oozie/?user.name=oozie HTTP/1.1"

Mar 21 06:35:23 localhost haproxy[28217]: 10.2.0.19:43622 [21/Mar/2016:06:35:23.150] oozie oozie_servers/app2 1/0/0/1/2 403 1389 - - ---- 1/1/0/0/0 0/0 "GET /oozie/?user.name=oozie HTTP/1.1"

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Contributor

When using Oozie HA in secure mode, the keytab must have the HTTP/_HOST principals for the Oozie servers as well as the oozie load balancer.

And the property oozie.authentication.kerberos.principal should be set to a value '*' in oozie-site.xml.

If the problem still exists , can you provide your load balancer configuration.

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

New Contributor

Thanks for reply Murali as you mentioned i did those steps load balancer principal to spnego servcie keytab on both oozie hosts.do i need to add it to oozie service keytab as well?

my load balancer config.

#---------------------------------------------------------------------

# Example configuration for a possible web application. See the

# full configuration options online.

#

# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#

#---------------------------------------------------------------------

#---------------------------------------------------------------------

# Global settings

#---------------------------------------------------------------------

global

# to have these messages end up in /var/log/haproxy.log you will

# need to:

#

# 1) configure syslog to accept network log events. This is done

# by adding the '-r' option to the SYSLOGD_OPTIONS in

# /etc/sysconfig/syslog

#

# 2) configure local2 events to go to the /var/log/haproxy.log

# file. A line like the following can be added to

# /etc/sysconfig/syslog

#

# local2.* /var/log/haproxy.log

#

log 127.0.0.1 local2

chroot /var/lib/haproxy

pidfile /var/run/haproxy.pid

maxconn 4000

user haproxy

group haproxy

daemon

# turn on stats unix socket

stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

mode http

log global

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout http-keep-alive 10s

timeout check 10s

maxconn 3000

#---------------------------------------------------------------------

# main frontend which proxys to the backends

#---------------------------------------------------------------------

frontend oozie

bind *:11000

mode http

default_backend oozie_servers

#acl url_static path_beg -i /static /images /javascript /stylesheets

#acl url_static path_end -i .jpg .gif .png .css .js

#use_backend static if url_static

#default_backend app

#---------------------------------------------------------------------

# static backend for serving up images, stylesheets and such

#---------------------------------------------------------------------

backend static

balance roundrobin

server static 127.0.0.1:4331 check

#---------------------------------------------------------------------

# round robin balancing between the various backends

#---------------------------------------------------------------------

backend oozie_servers

balance roundrobin

server app1 ha1.cloud:11000 check

server app2 ha2.cloud:11000 check

#server app3 127.0.0.1:5003 check

#server app4 127.0.0.1:5004 check

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Contributor

It is better to create a another keytab file instead of spengo like oozie.ha.keytab and make sure that it contains both the oozie server and load balancer principal and set the file to oozie.authentication.kerberos.keytab.

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Rising Star

I had similar issue. I used keepalived service between two servers where oozie servers are running. So there is no additional server for load balancer. I had one virtual-IP and FQDN bind to virtual-IP. Lets assume that FQDN is failover.example.com. I just created new kerberos principal HTTP/failover.example.com@REALM and added it into keytab that is marked on oozie's configuration field - oozie.authentication.kerberos.keytab. And now I was able to use oozie service via http. But the question is - is it right solution? Or should I create a new keytab file instead to add new principal end of the existing file. The problem of my solution is in example if I regenerate keytabs again via Ambari. Then I will lost failover principal rows from spnego.service.keytab.

Br, Margus

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Cloudera Employee

I have exactly the same issue :

Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 403, message: Forbidden

I created principal and appended or put it separately but could not generate ticket for this keytab , now my ticket for oozie keytab is like that, I have problem just with loadbalancer I could access the 2 oozie instances.

Default principal: oozie/hc1.domain.com@TCAD.domain.COM Valid starting Expires Service principal krbtgt/TCAD.domain.COM@TCAD.domain.COM renew until 05/19/2016 12:20:50 HTTP/hc1.bmeu.com@TCAD.domain.COM renew until 05/19/2016 12:20:50 22:20:50 HTTP/hc2.domain.com@TCAD.domain.COM renew until 05/19/2016 12:20:50 HTTP/hc3.domain.com@TCAD.domain.COM renew until 05/19/2016 12:20:50

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Cloudera Employee

HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 WWW-Authenticate: Negotiate Set-Cookie: hadoop.auth=; Path=/; Domain= ; HttpOnly Content-Type: text/html;charset=utf-8 Content-Length: 997 Date: Thu, 12 May 2016 11:51:53 GMT HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Set-Cookie: hadoop.auth=; Path=/; Domain= ; HttpOnly Content-Type: text/html;charset=utf-8 Content-Length: 1165 Date: Thu, 12 May 2016 11:51:53 GMT <html><head><title>Apache Tomcat/6.0.44 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 403 - GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.44</h3></body></html>[root@hc3 keytabs]#

Highlighted

Re: 403 forbidden error while accessing oozie server through load balancer in kerberized cluster.

Cloudera Employee

@Narasimha Muvva Could you please tell us how did you solve that, I'm facing the same issue. Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here