Reply
New Contributor
Posts: 3
Registered: ‎02-27-2017
Accepted Solution

401 Unauthorized Error Livy+ Hue

Hi Team, I recently configured Hue 3.11 with all backend configuration for Hive / WebHDFS etc. However while configuring Spaky/ Livy connectivity on Hue, I am getting 401 unauthorized under "Notebook" section on Hue and also while fetching session data through curl command. Below is the error that I am getting on command line: # curl  http://localhost:8998/sessions <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1" /> Error 401 HTTP ERROR: 401

Problem accessing /sessions. Reason:

    Authentication required

Powered by Jetty:// Below is the error from Hue: <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1" /> Error 401 HTTP ERROR: 401

Problem accessing /sessions. Reason:

 Authentication required

Powered by Jetty:// (error 401) Note : Cluster is kerberized.            Livy is configured with spnego kerberos principal /keytab as well as livy kerberos princial /keytab           Can someone help on this ? Regards Ajay Rathod

Cloudera Employee
Posts: 723
Registered: ‎07-30-2013

Re: 401 Unauthorized Error Livy Hue

Could you confirm on http://groups.google.com/a/cloudera.org/group/livy-user
if Livy supports Kerberos?
Cloudera Employee
Posts: 723
Registered: ‎07-30-2013

Re: 401 Unauthorized Error Livy Hue

New Contributor
Posts: 3
Registered: ‎02-27-2017

Re: 401 Unauthorized Error Livy Hue

Thanks Romain,

This helped me in fixing the issue.

Regards
Ajay Rathod
Explorer
Posts: 18
Registered: ‎01-29-2018

Re: 401 Unauthorized Error Livy Hue

Hi @Romainr, in which section/configuration file I should set this parameter ( self._client.set_kerberos_auth() ) into Hue Service? Many thanks! Alex
Posts: 1,043
Topics: 1
Kudos: 262
Solutions: 130
Registered: ‎04-22-2014

Re: 401 Unauthorized Error Livy Hue

@AlexPQ,

 

Based on the referenced Jira, kerberos auth is now configurable.

 

In CDH 5.11.0 and higher, you can enable kerberos auth in hue.ini:

 

[spark]

security_enabled=true

 

 

Explorer
Posts: 18
Registered: ‎01-29-2018

Re: 401 Unauthorized Error Livy Hue

Hi @bgooley,

Thanks a lot for the information!

Regards,

Alex
Expert Contributor
Posts: 73
Registered: ‎09-14-2017

Re: 401 Unauthorized Error Livy Hue

[ Edited ]

The parameter:
[spark]
security_enabled=true

 

need to be added in Cloudera Manager Hue configuration to the below section:
Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

 

However it still gave an error: Unable to authenticate . The Hue Admin screen showed a misconfiguration in Spark: The app wont work without a running Livy Spark Server. 

Also in  /var/log/hue/error.log below message was present:

GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server HTTP/localhost@EXAMPLE.COM not found in Kerberos database', -1765328377))

 

These errors were resolved by adding the line to the to the Hue Safety valve above:

[spark]

security_enabled=true

livy_server_host=yourlivyhostfqdn

 

Highlighted
New Contributor
Posts: 4
Registered: ‎04-08-2019

Re: 401 Unauthorized Error Livy Hue

Dear Romainr:

    I follow your advise to modify the srcCode and made the hue3.9 notebook worked on my CDH5.7.1 Kerberized cluster with livy0.5.0.Now i can using spark-shell on notebook and its run well.

    but on yarn i saw the spark job user was always livy(i set livy be a haddop.proxyuser), seems its bind on the user which the keytab of livy-server-luancher. so i can't control the notebook  authority by hue.

    i can see the hue set proxyuser on a the token, but the spark job's user was not be reset.

 

 

livy-conf

livy.impersonation.enabled=true
livy.repl.enable-hive-context=true
livy.spark.deploy-mode=client
livy.spark.master=yarn
livy.superusers=hue
livy.server.auth.type=kerberos
livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.keytab
livy.server.auth.kerberos.principal=HTTP/xxx.com@xxx.COM
livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.keytab
livy.server.launch.kerberos.principal=livy/xxx.com@xxx.COM

livy-log

19/04/12 14:48:14 INFO InteractiveSession$: Creating Interactive session 3: [owner: hue, request: [kind: pyspark, proxyUser: Some(baoyong), heartbeatTimeoutInSecond: 0]]
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 diagnostics: N/A
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 ApplicationMaster host: 192.168.103.166
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 ApplicationMaster RPC port: 0
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 queue: root.livy
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 start time: 1555051701595
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 final status: UNDEFINED
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 tracking URL: http://bigdata166.xxx.com:8088/proxy/application_1555044848792_0063/
19/04/12 14:48:25 INFO LineBufferedStream: stdout: 	 user: livy
19/04/12 14:48:28 INFO InteractiveSession: Interactive session 3 created [appid: application_1555044848792_0063, owner: hue, proxyUser: None, state: idle, kind: pyspark, info: {driverLogUrl=null, sparkUiUrl=null}]