Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

HUE fails with saml

Labels:
avatar
author-rank Tomas79
Guru

Created on ‎11-19-2018 01:16 AM - edited ‎09-16-2022 06:54 AM

Hi, does anybody have a similar experience with python errors when using HUE with SAML?

My error after switching to saml authentication is :

 

[root@ip-10-85-150-28 env]# pwd
/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env
./bin/pip list
pysaml2 (4.4.0)

-> HUE ERROR log:
[07/Nov/2018 00:19:47 -0800] access       INFO     10.85.239.102 -anon- - "POST / HTTP/1.1" -- login redirection
[07/Nov/2018 00:19:47 -0800] access       INFO     10.85.239.102 -anon- - "POST / HTTP/1.1" returned in 0ms
[07/Nov/2018 00:19:47 -0800] client       INFO     destination to provider: https://10.85.150.28:8888
[07/Nov/2018 00:19:47 -0800] entity       INFO     REQUEST: <?xml version='1.0' encoding='UTF-8'?>
<samlp:AuthnRequest xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://10.85.150.28:8888/saml2/acs/" Destination="https://10.85.150.28:8888" ID="id-nLFNl6R57kIYXjA0m" IssueInstant="2018-11-07T08:19:47Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0"><saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">https://10.85.150.28:8888/saml2/metadata/</saml:Issuer><samlp:NameIDPolicy AllowCreate="false" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" /></samlp:AuthnRequest>
[07/Nov/2018 00:19:47 -0800] client       INFO     AuthNReq: <?xml version='1.0' encoding='UTF-8'?>
<samlp:AuthnRequest xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://10.85.150.28:8888/saml2/acs/" Destination="https://10.85.150.28:8888" ID="id-nLFNl6R57kIYXjA0m" IssueInstant="2018-11-07T08:19:47Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0"><saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">https://10.85.150.28:8888/saml2/metadata/</saml:Issuer><samlp:NameIDPolicy AllowCreate="false" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" /></samlp:AuthnRequest>
[07/Nov/2018 00:19:47 -0800] entity       INFO     HTTP REDIRECT
[07/Nov/2018 00:19:47 -0800] middleware   INFO     Processing exception: 'NoneType' object has no attribute 'get_signer': Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/views.py", line 175, in login
    binding=binding, sign=False, sigalg=sigalg)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/client.py", line 76, in prepare_for_authenticate
    **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/client.py", line 129, in prepare_for_negotiated_authenticate
    relay_state, **args)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/entity.py", line 231, in apply_binding
    signer = self.sec.sec_backend.get_signer(kwargs['sigalg'])
AttributeError: 'NoneType' object has no attribute 'get_signer'

I tried to upgrade pysaml but it broked the whole setup.

 

Followed the recommendation from the docs and installed additional packages (running on Centos 7.5)

 

install git gcc python-devel swig openssl

sudo vi /usr/java/jdk1.8.0_191-amd64/jre/lib/security/java.security

/jdk.certpath.disabledAlgorithms=MD2, MD5, RC4, DH, SHA1 jdkCA & usage TLSServer,..../

wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-6.noarch.rpm

rpm -ivh epel-release-7-6.noarch.rpm

yum install xmlsec1 xmlsec1-openssl

Thanks

11,573 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank bgooley author-rank
Master Guru

Created ‎11-21-2018 11:56 AM

@Tomas79,

 

As for your "duplicate entry" issue, it makes sense if your email address is longer than the username column width (which is 30 characters, I think).  You log in the first and the username obtained from the SAML response nameID is truncated when added to the database.  Then, the next time you log in, a search for the full email address is done (and not found due to the username truncation) since no rows are returned, Hue considers this a new user and attempts to add it.  This fails since the truncated name already exists.

 

In order to get around that problem, I suppose you could expand the "username" column to 40 or 50 characters, but I think you wanted to use attribute-based usernames instead.

 

Getting back to the attributes, I do see you have:

 

Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"

 

This seems good, but there is some back-end computation that results in Hue not being able to see your attribute value.

 

To understand this better, note this flow of mapping to get from the SAML response to a Hue username:

 

SAML response attribute/value ====> pysaml attribute/value

pysaml attribute/value               ====> djangosaml "username" attribute

djangosaml user                        ====> Hue user

 

For the SAML response to pysaml attribute mapping, there is a built-in mapping in hue/desktop/libs/libsaml/attribute-maps/SAML2.py for urn:oasis:names:tc:SAML:2.0:attrname-format:uri

 

It maps response attributes to pysaml attributes and stores the value in memory.  There is no "uid" mapping by default.  Rather, the SAML2.py mapping file looks for the OID for "uid" which is urn:oid:0.9.2342.19200300.100.1.1

 

Since urn:oid:0.9.2342.19200300.100.1.1 is not found in the response, your attribute and value is not seen.

 

That is a long explanation to come to a couple options you have:

 

(1)

 

Configure your IDP to emit the uid attribute with the format urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified instead of urn:oasis:names:tc:SAML:2.0:attrname-format:uri

 

 

This will tell the client that the attribute does not conform to standard OID formatting so the attribute name should be obtained literally (as is) and the value retrieved.

 

(2)

 

Create a custom mapping for the "uid" attribute in a mapping attribute for 

urn:oasis:names:tc:SAML:2.0:attrname-format:uri

To do so, you can follow these instructions:

 

[A]

 

Create a directory that will house your attribute mapping file.


For example:

# mkdir /opt/cloudera/saml/attribute_mapping

# chown hue:hue /opt/cloudera/saml/attribute_mapping

 

[B]

 

Place the custom attribute mapping file in the directory created in step A

 

"saml_uri.py" file contents:

 

MAP = {
"identifier": "urn:oasis:names:tc:SAML:2.0:attrname-format:uri",
"fro": {
  'uid': 'uid',
},
"to": {
  'uid': 'uid',
}

NOTE: I believe the above should work as the "fro" section maps assertion attribute name to pysaml name and the "to" section does the reverse.  I think it should be OK to have both sides the same.

 

NOTE2: make sure hue can read from the saml_uri.py file.
For example:

# chown hue:hue saml_uri.py

 

There should now be an attribute mapping file named saml_uri.py with the following location:

/opt/cloudera/saml/attribute_mapping/saml_uri.py

 

[C]

 

Configure Hue's Service-Wide safety valve with the following addition in the [libsaml] section:

 

attribute_map_dir=/opt/cloudera/saml/attribute_mapping

user_attribute_mapping='{"uid":"username"}'

 

Make sure to restart Hue after the change to the safety valve.

 

NOTE: the only attribute that Hue really needs or cares about in this case is whatever maps to "username"

 

Let me know if you have any questions.

View solution in original post

11,465 Views
0 Kudos
15 REPLIES 15

avatar
author-rank Tomas79
Guru

Created ‎11-21-2018 12:38 AM

@bgooley, thanks for the helpfull insights, I was able to move forward a little bit. But to keep the chronology of those test attempts clear, I reply first for the first suggestion to change the mapping in HUE and send a response uid. What I did:

In AAD I created in Attributes and Claims a new uid and saml_user attributes and mapped a first name to it (let pretend that this is the user name I want to use).

saml_user = user.givenname
uid = user.givenname

In HUE I have added this

+user_attribute_mapping='{"uid":"username"}'

 Now I expected that the hue will find the user name either in uid or in saml_user. But the logs shows the issue is the same:

 

[20/Nov/2018 23:57:55 -0800] response     INFO     Subject NameID: <?xml version='1.0' encoding='UTF-8'?><saml:NameID xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"><REDACTED-MY-USER-NAME></saml:NameID>
[20/Nov/2018 23:57:55 -0800] response     DEBUG    Attribute Statement: <?xml version='1.0' encoding='UTF-8'?><saml:AttributeStatement xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><saml:Attribute Name="http://schemas.microsoft.com/identity/claims/tenantid" NameFormat=".....
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="http://schemas.microsoft.com/identity/claims/tenantid" NameFo...
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="http://schemas.microsoft.com/identity/claims/objectidentifier" NameFo...
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="http://schemas.microsoft.com/identity/claims/identityprovider" NameFo...
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="http://schemas.microsoft.com/claims/authnmethodsreferences" NameFor...
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="saml_user" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"><saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">Tomas</saml:AttributeValue></saml:Attribute>
[20/Nov/2018 23:57:55 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?><saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"><saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">Tomas</saml:AttributeValue></saml:Attribute>
[20/Nov/2018 23:57:55 -0800] response     ERROR    Missing Attribute Statement
[20/Nov/2018 23:57:55 -0800] response     DEBUG    --- AVA: {}
[20/Nov/2018 23:57:55 -0800] client_base  INFO     --- ADDED person info ----
[20/Nov/2018 23:57:55 -0800] backends     ERROR    The attributes dictionary is empty
[20/Nov/2018 23:57:55 -0800] backends     ERROR    Could not find saml_user value
[20/Nov/2018 23:57:55 -0800] views        WARNING  Could not authenticate user received in SAML Assertion. Session info: {'authn_info': [('urn:oasis:names:tc:SAML:2.0:ac:classes:Password', [], '2018-11-07T07:43:32.657Z')], 'name_id': <saml2.saml.NameID object at 0x7faf037550d0>, 'not_on_or_after': 1542790374, 'session_index': '_70892f13-d4ad-4fe2-808c-38e9d85a408b', 'came_from': None, 'ava': {}, 'issuer': 'https://sts.windows.net/f0ba4ef9-b8da-..../'}
[20/Nov/2018 23:57:55 -0800] middleware   INFO     Processing exception: : Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/views.py", line 296, in assertion_consumer_service
    raise PermissionDenied
PermissionDenied

In the error log above I think the important message is that hue does not see any attribute list at all! And you can see, that the response contain couple of attribute elements and two of them are the uid and saml_user, mapping to "Tomas" as a value.  So how to configure saml settings in hue to work with this response structure?

 

Ok second part of my answer, your suggestion to map the user_name with a name_id_format. This actually worked nicely, but as you said, and I agree, I dont want to use email address as user name in HUE. 

So changing the settings in HUE:

username_source=nameid
name_id_format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"

resulted in this:

 

[21/Nov/2018 00:06:57 -0800] attribute_converter INFO     Unknown attribute name: <?xml version='1.0' encoding='UTF-8'?>
<saml:Attribute xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"><saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xsi:type="xs:string">Tomas</saml:AttributeValue></saml:Attribute>
[21/Nov/2018 00:06:57 -0800] response     ERROR    Missing Attribute Statement
[21/Nov/2018 00:06:57 -0800] response     DEBUG    --- AVA: {}
[21/Nov/2018 00:06:57 -0800] client_base  INFO     --- ADDED person info ----
[21/Nov/2018 00:06:57 -0800] backends     ERROR    The attributes dictionary is empty
[21/Nov/2018 00:06:57 -0800] cursors      WARNING  /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/mysql/base.py:124: Warning: Data truncated for column 'username' at row 1
  return self.cursor.execute(query, args)

[21/Nov/2018 00:06:57 -0800] models       WARNING  /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/backends.py:221: DeprecationWarning: The use of AUTH_PROFILE_MODULE to define user profiles has been deprecated.
  profile = user.get_profile()

[21/Nov/2018 00:06:57 -0800] backend      INFO     Augmenting users with class: <class 'desktop.auth.backend.DefaultUserAugmentor'>
[21/Nov/2018 00:06:57 -0800] access       INFO     10.85.xxx.xxx <REDACTED-MY-USER-NAME> - "POST /saml2/acs/ HTTP/1.1" returned in 113ms
[21/Nov/2018 00:06:59 -0800] access       INFO     10.85.xxx.xxx <REDACTED-MY-USER-NAME_TRUNCATED> - "GET / HTTP/1.1" returned in 926ms
[21/Nov/2018 00:07:00 -0800] connectionpool INFO     Starting new HTTPS connection (1): ip-10-85-yyy-yy.eu-west-1.compute.internal:14000
[21/Nov/2018 00:07:00 -0800] connectionpool DEBUG    "ip-10-85-yyy-yy.eu-west-1.compute.internal:14000 GET /webhdfs/v1/user/<REDACTED-MY-USER-NAME_TRUNCATED>m?op=GETFILESTATUS&user.name=hue&doas=<REDACTED-MY-USER-NAME_TRUNCATED> HTTP/1.1" 401 997
[21/Nov/2018 00:07:00 -0800] kerberos_    DEBUG    handle_401(): Handling: 401
[21/Nov/2018 00:07:00 -0800] kerberos_    DEBUG    generate_request_header(): host_port_thread: ip-10-85-yyy-yy.eu-west-1.compute.internal_14000_140122880198400
[21/Nov/2018 00:07:00 -0800] kerberos_    DEBUG    authenticate_user(): Authorization header: NegotiawnwMi.....
[21/Nov/2018 00:07:01 -0800] connectionpool DEBUG    "ip-10-85-yyy-yy.eu-west-1.compute.internal:14000 GET /webhdfs/v1/user/<REDACTED-MY-USER-NAME_TRUNCATED>m?op=GETFILESTATUS&user.name=hue&doas=<REDACTED-MY-USER-NAME_TRUNCATED> HTTP/1.1" 404 None
[21/Nov/2018 00:07:01 -0800] kerberos_    DEBUG    authenticate_user(): returning <Response [404]>

As you can see the application picked correctly the username from nameid, and trucnated it (because it is a long email address). But the error regarding attributes are still there. What I would like to achieve is to map the user name to those "uid" or "saml_user" attributes instead.

(Btw the two last errors I assume are from the NameNode when HUE asks the NN to fetch the user's data from HDFS, but this user apparently does not exists)

 

And my last question is this is working as an IDP initiated login (when I test it from AAD), or it is working when I have logged in and in another tab I open hue URL. But in different window, opening the HUE does not redirect me to the windows login page. I thought this information  - where to redirect - should be in the idp.xml file and the HUE application should redirect the user to the login page.

 

Thanks

Tomas

 

 

6,661 Views
0 Kudos

avatar
author-rank Tomas79
Guru

Created ‎11-21-2018 01:24 AM

One more issue I observed with sign in with email address:
- I sign in first time, it is ok, Hue creates home directory and user entry
- I sign out, and then sign in again, and it failes on duplicity:

 

[21/Nov/2018 01:11:01 -0800] middleware   INFO     Processing exception: (1062, "Duplicate entry '<REDACTED>onmicrosoft.co' for key 'username'"): Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/views.py", line 293, in assertion_consumer_service
    create_unknown_user=create_unknown_user)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/contrib/auth/__init__.py", line 49, in authenticate
    user = backend.authenticate(**credentials)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/core/src/desktop/lib/metrics/registry.py", line 388, in wrapper
    return fn(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/desktop/libs/libsaml/src/libsaml/backend.py", line 53, in authenticate
    return super(SAML2Backend, self).authenticate(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/backends.py", line 115, in authenticate
    create_unknown_user, main_attribute, attributes, attribute_mapping)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/backends.py", line 152, in get_saml2_user
    return self._get_or_create_saml2_user(main_attribute, attributes, attribute_mapping)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/backends.py", line 167, in _get_or_create_saml2_user
    defaults=user_create_defaults, **user_query_args)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 391, in get_or_create
    six.reraise(*exc_info)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 383, in get_or_create
    obj.save(force_insert=True, using=self.db)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/base.py", line 545, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/base.py", line 573, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/base.py", line 654, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/base.py", line 687, in _do_insert
    using=using, raw=raw)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/manager.py", line 232, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 1514, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/sql/compiler.py", line 903, in execute_sql
    cursor.execute(sql, params)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '<REDACTED>onmicrosoft.co' for key 'username'")

As this error comes from the database, it looks like Hue is looking into a database with a non-truncated username, and then tries to insert the truncated user name.

It looks to me as a bug, but not important, as I want to use other field from the Attribute list.

6,658 Views
0 Kudos

avatar
author-rank bgooley author-rank
Master Guru

Created ‎11-21-2018 11:56 AM

@Tomas79,

 

As for your "duplicate entry" issue, it makes sense if your email address is longer than the username column width (which is 30 characters, I think).  You log in the first and the username obtained from the SAML response nameID is truncated when added to the database.  Then, the next time you log in, a search for the full email address is done (and not found due to the username truncation) since no rows are returned, Hue considers this a new user and attempts to add it.  This fails since the truncated name already exists.

 

In order to get around that problem, I suppose you could expand the "username" column to 40 or 50 characters, but I think you wanted to use attribute-based usernames instead.

 

Getting back to the attributes, I do see you have:

 

Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"

 

This seems good, but there is some back-end computation that results in Hue not being able to see your attribute value.

 

To understand this better, note this flow of mapping to get from the SAML response to a Hue username:

 

SAML response attribute/value ====> pysaml attribute/value

pysaml attribute/value               ====> djangosaml "username" attribute

djangosaml user                        ====> Hue user

 

For the SAML response to pysaml attribute mapping, there is a built-in mapping in hue/desktop/libs/libsaml/attribute-maps/SAML2.py for urn:oasis:names:tc:SAML:2.0:attrname-format:uri

 

It maps response attributes to pysaml attributes and stores the value in memory.  There is no "uid" mapping by default.  Rather, the SAML2.py mapping file looks for the OID for "uid" which is urn:oid:0.9.2342.19200300.100.1.1

 

Since urn:oid:0.9.2342.19200300.100.1.1 is not found in the response, your attribute and value is not seen.

 

That is a long explanation to come to a couple options you have:

 

(1)

 

Configure your IDP to emit the uid attribute with the format urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified instead of urn:oasis:names:tc:SAML:2.0:attrname-format:uri

 

 

This will tell the client that the attribute does not conform to standard OID formatting so the attribute name should be obtained literally (as is) and the value retrieved.

 

(2)

 

Create a custom mapping for the "uid" attribute in a mapping attribute for 

urn:oasis:names:tc:SAML:2.0:attrname-format:uri

To do so, you can follow these instructions:

 

[A]

 

Create a directory that will house your attribute mapping file.


For example:

# mkdir /opt/cloudera/saml/attribute_mapping

# chown hue:hue /opt/cloudera/saml/attribute_mapping

 

[B]

 

Place the custom attribute mapping file in the directory created in step A

 

"saml_uri.py" file contents:

 

MAP = {
"identifier": "urn:oasis:names:tc:SAML:2.0:attrname-format:uri",
"fro": {
  'uid': 'uid',
},
"to": {
  'uid': 'uid',
}

NOTE: I believe the above should work as the "fro" section maps assertion attribute name to pysaml name and the "to" section does the reverse.  I think it should be OK to have both sides the same.

 

NOTE2: make sure hue can read from the saml_uri.py file.
For example:

# chown hue:hue saml_uri.py

 

There should now be an attribute mapping file named saml_uri.py with the following location:

/opt/cloudera/saml/attribute_mapping/saml_uri.py

 

[C]

 

Configure Hue's Service-Wide safety valve with the following addition in the [libsaml] section:

 

attribute_map_dir=/opt/cloudera/saml/attribute_mapping

user_attribute_mapping='{"uid":"username"}'

 

Make sure to restart Hue after the change to the safety valve.

 

NOTE: the only attribute that Hue really needs or cares about in this case is whatever maps to "username"

 

Let me know if you have any questions.

11,466 Views
0 Kudos

avatar
author-rank Tomas79
Guru

Created ‎11-22-2018 12:32 AM

I was not able to set Name format, so opted for the second, and it resolved the issue. Thanks @bgooley!

 

6,638 Views
0 Kudos

avatar
author-rank MahendraDevu
Explorer

Created on ‎04-05-2019 02:04 PM - edited ‎04-05-2019 02:07 PM

we are facing this issue, and trying to see if anybody else saw the same..   we have the key in the right locaiton, and we have tried encrypted and un-encrypted version, and still see the same problem.

We have tested DER, des3 version of the key file and no progres.. 

Do we need stick to a certain format of the key.. 

 

[05/Apr/2019 16:37:03 -0400] entity       ERROR    Signature Error: None

[05/Apr/2019 16:37:03 -0400] client_base  ERROR    XML parse error: None

[05/Apr/2019 16:37:03 -0400] views        ERROR    SAML Identity Provider is not configured correctly: certificate key is missing!

Traceback (most recent call last):

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/views.py", line 254, in assertion_consumer_service

    response = client.parse_authn_request_response(xmlstr, BINDING_HTTP_POST, outstanding_queries)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/client_base.py", line 597, in parse_authn_request_response

    binding, **kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/entity.py", line 1140, in _parse_response

    response = response.loads(xmlstr, False, origxml=origxml)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/response.py", line 512, in loads

    self._loads(xmldata, decode, origxml)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/response.py", line 337, in _loads

    **args)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/sigver.py", line 1840, in correctly_signed_response

    class_name(response), origdoc)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/pysaml2-4.4.0-py2.7.egg/saml2/sigver.py", line 1614, in _check_signature

    raise MissingKey("%s" % issuer)

MissingKey: None

[05/Apr/2019 16:37:03 -0400] middleware   INFO     Processing exception: : Traceback (most recent call last):

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 112, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py", line 371, in inner

    return func(*args, **kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/http.py", line 41, in inner

    return func(request, *args, **kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/views/decorators/csrf.py", line 57, in wrapped_view

    return view_func(*args, **kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/views.py", line 272, in assertion_consumer_service

    return fail_acs_response(request)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/utils.py", line 85, in fail_acs_response

    return failure_function(request, *args, **kwargs)

  File "/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hue/build/env/lib/python2.7/site-packages/djangosaml2-0.16.4-py2.7.egg/djangosaml2/acs_failures.py", line 22, in exception_failure

    raise exc_class

PermissionDenied

 

[05/Apr/2019 16:37:03 -0400] access       WARNING  162.44.245.32 -anon- - "POST /saml2/acs/ HTTP/1.1" -- 403 access forbidden

6,048 Views
0 Kudos

avatar
author-rank ebeb
Expert Contributor

Created on ‎11-24-2021 10:26 AM - edited ‎11-24-2021 01:00 PM

Hello @MahendraDevu

Did you resolve the error in HUE SAML we are getting this in CDP 7.1.7 after upgrade. SAML was working in CDH5.16 HUE before upgrade:
[05/Apr/2019 16:37:03 -0400] views ERROR SAML Identity Provider is not configured correctly: certificate key is missing!

 

UPDATE: Resolved this issue by making the IDP <md:EntityDescriptor entityID same as that on the metadata.xml we specified in HUE Advanced Configuration snippet hue_safety_valve.ini metadata_file . There was a mismatch between IDP value and what was in the metadata file. 

4,831 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements