Created on
11-19-2019
07:28 AM
- last edited on
11-19-2019
07:51 AM
by
VidyaSargur
Hi
Created a cluster via cloudbreak from the blueprint. Trying to access Knox Admin UI from Ambari with no luck. URI points to https://<master_public_ip>:8443/<cluster_name>/manager/admin-ui, so I got 404. Investigated topologies and there is no topology named `manager`, only `admin.xml`, `default.xml`, `dp-proxy.xml` and `knoxsso.xml`. I'd like to secure Apache Livy Server with Knox. I Can successfully auth to YARNUIV2 or to WEBHDFS with curl, for example:
curl -iku admin:password -X GET 'https://<master_public_ip>:8443/<cluster_name>/dp-proxy/webhdfs/v1/?op=LISTSTATUS'
forks fine. How I could access Knox Admin UI or secure Livy Server with Knox?
I'm using:
Cloudbreak: 2.9.1
Ambari: 2.7.4.0
HPD: 3.1.4.0-315
Blueprint: HDP 3.1 - Data Science: Apache Spark 2, Apache Zeppelin
Created 11-19-2019 07:49 AM
For internet searchers:
Livy can be accessed via
curl -iku admin:password -X GET 'https://<master_public_ip>:8443/<cluster_name>/dp-proxy/livy/v1/sessions'
Username, password and cluster_name are set when cluster is created by cloudbreak. Master_public_ip is ip of ther VM which serves as ambari
Here is a python snippet to test it working:
import json, pprint, requests, textwrap
session_url = "https://<master_public_ip>:8443/<cluster_name>/dp-proxy/livy/v1/sessions"
headers = { 'X-Requested-By': 'livy'}
data = {'kind': 'spark'}
basic_auth = ('admin', 'password')
response = requests.post(session_url, headers=headers, auth=basic_auth, data=json.dumps(data), verify=False)
print(response.status_code)
print(response.text)
response.json()
Created on 11-21-2019 08:03 AM - edited 11-21-2019 08:04 AM
I will answer to myself and for those who will search.
Start Demo LDAP from Ambari UI as manual said.
It looks like admin-ui doesn't enabled by default. So I created manager.xml in
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
<a href="<a href="<a href="<a href="http://www.apache.org/licenses/LICENSE-2.0" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a>" target="_blank"><a href="http://www.apache.org/licenses/LICENSE-2.0</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a>>" target="_blank"><a href="<a href="http://www.apache.org/licenses/LICENSE-2.0</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a>" target="_blank"><a href="http://www.apache.org/licenses/LICENSE-2.0</a</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a</a>>>" target="_blank"><a href="<a href="<a href="http://www.apache.org/licenses/LICENSE-2.0</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a>" target="_blank"><a href="http://www.apache.org/licenses/LICENSE-2.0</a</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a</a>>" target="_blank"><a href="<a href="http://www.apache.org/licenses/LICENSE-2.0</a</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a</a>" target="_blank"><a href="http://www.apache.org/licenses/LICENSE-2.0</a</a</a" target="_blank">http://www.apache.org/licenses/LICENSE-2.0</a</a</a</a>>>>
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<topology>
<name>manager</name>
<gateway>
<provider>
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param>
<name>csrf.enabled</name>
<value>true</value>
</param>
<param>
<name>csrf.customHeader</name>
<value>X-XSRF-Header</value>
</param>
<param>
<name>csrf.methodsToIgnore</name>
<value>GET,OPTIONS,HEAD</value>
</param>
<param>
<name>xframe.options.enabled</name>
<value>true</value>
</param>
<param>
<name>xss.protection.enabled</name>
<value>true</value>
</param>
<param>
<name>strict.transport.enabled</name>
<value>false</value>
</param>
</provider>
<provider>
<role>federation</role>
<name>SSOCookieProvider</name>
<enabled>true</enabled>
</provider>
<provider>
<role>identity-assertion</role>
<name>HadoopGroupProvider</name>
<enabled>true</enabled>
<param>
<name>CENTRAL_GROUP_CONFIG_PREFIX</name>
<value>gateway.group.config.</value>
</param>
</provider>
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
<param>
<name>knox.acl.mode</name>
<value>OR</value>
</param>
<param>
<name>knox.acl</name>
<value>KNOX_ADMIN_USERS;KNOX_ADMIN_GROUPS;*</value>
</param>
</provider>
</gateway>
<service>
<role>KNOX</role>
</service>
<application>
<name>admin-ui</name>
</application>
</topology>
Don't forget to change the owner:
chown knox:knox /etc/knox/conf/topologies/manager.xml
curl -iku admin:admin-password -X GET 'https://<master_public_ip>:8443/<clustername>/dp-proxy/livy/v1/sessions'
Test admin UI:
https://<master_public_ip>:8443/<clustername>/manager/admin-ui/
PS. Somehow if I query LDAP with ldapseach at master node and localhost:33389 it returns me users with randomly generated passwords. But users.ldif file has simple passwords as it configured in Ambari. And simple passwords work, so what's wrong with ldap at localhost:33389 and which LDAP queried by Knox?