This article list down the steps involved to enable SSL for Livy Server in HDP 2.6.1.0
This setup instruction is for example purpose. Production cluster setup may be different.
Prerequisite:
Firstly, prepare the keystore and truststore files for enabling SSL in Livy server. The steps followed to generate keystore and truststores are for testing purpose only.
1) create keystore files, certificates, and truststore files for each host in cluster.
- Generate Keystore file for Livy server
| keytool -genkey -alias <host> -keyalg RSA -keysize 1024 –dname CN=<host>,OU=hw,O=hw,L=paloalto,ST=ca,C=us –keypass <keyPassword> -keystore <keystore_file> -storepass <storePassword> |
- Create a certificate
| keytool -export -alias <host> -keystore <keystore_file> -rfc –file <cert_file> -storepass <StorePassword> |
- Create Truststore file
| keytool -import -noprompt -alias <host> -file <cert_file> -keystore <truststore_file> -storepass <truststorePassword> |
Update Livy configuration:
As next step, livy.conf should be updated to set keystore details as below.
Steps to follow to update livy.conf using Ambari UI:
- Go to the config page for Spark ( https://<ambari-host>:<ambari-port>/#/main/services/SPARK/configs)
- Click on custom livy-conf tab
- Click on Add Property link. Add below configurations.
| livy.keystore = <keystore_file> livy.keystore.password = <storePassword> livy.key-password = <keyPassword> |
- Save the configuration group.
- Restart Livy Server
In future version, Livy configuration will be updated as per LIVY-287 .
Access Livy Server:
After enabling SSL over Livy server. Livy server should be accessible over https protocol.
Example: https://<livy host>:<livy port>
Open Python Interactive shell and run below code snippet to test SSL enabled Livy server.
- Create Session:
livy_url = “https://<livy host>:<livy port>/sessions”
data = {'kind': 'spark', 'numExecutors': 1}
headers = {‘Content-Type’: ‘application/json’}
r = requests.post(livy_url, data=json.dumps(data), headers=headers, auth=HTTPKerberosAuth(mutual_authentication=REQUIRED, sanitize_mutual_error_response=False), verify=False)
r.json()| {u’state’: u’starting’, u’id’: 0, u’kind’: u’spark’} |
- Get status of Running Session:
session_url = “https://<livy host>:<livy port>/sessions/0”
headers = {‘Content-Type’: ‘application/json’}
r = requests.get(session_url, headers=headers, auth=HTTPKerberosAuth(mutual_authentication=REQUIRED, sanitize_mutual_error_response=False), verify=False)
r.json()| {u'kind': u'spark', u'log': [u'YARN Diagnostics:', u'AM container is launched, waiting for AM container to Register with RM'], u'proxyUser': u'hrt_qa', u'appInfo': {u'driverLogUrl': u'https://xxx:54321/node/containerlogs/container_e01_1496699604100_0008_01_000001/hrt_qa', u'sparkUiUrl': u'https://xxx:8088/proxy/application_1496699604100_0008/'}, u'state': u'starting', u'appId': u'application_1496699604100_0008', u'owner': u'hrt_qa', u'id': 0} |
- Submit Statement:
session_url = “https://<livy host>:<livy port>/sessions/0/statements”
data ={"code": "sc.parallelize(1 to 10).count()"}
headers = {‘Content-Type’: ‘application/json’}
r = requests.get(session_url, headers=headers, auth=HTTPKerberosAuth(mutual_authentication=REQUIRED, sanitize_mutual_error_response=False), verify=False)
r.json()| {u'output': {u'status': u'ok', u'execution_count': 0, u'data': {u'text/plain': u'res0: Long = 10'}}, u'state': u'available', u'id': 0} |
In HDP-2.6.1.0, Zeppelin-Livy interpreter does not work with SSL enabled Livy server. (ZEPPELIN-2584)
