Member since
12-03-2016
91
Posts
22
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
8506 | 08-27-2019 10:45 AM | |
1947 | 12-24-2018 01:08 PM | |
4587 | 09-16-2018 06:45 PM | |
1123 | 12-12-2016 01:44 AM |
07-08-2022
08:44 AM
I thought it was something like this, but it was hard to believe! After 2 full installation of CDP base, it seems clear that CDP may have been a big step for the final user, but still has a lot of room for improvement in the sysadmin and devops side of the platform, specially in the way-back or recovery of many central configuration changes (kerberos, TLS) where it really sucks, even when compared with the now ancient HDP3.
... View more
06-20-2022
05:27 PM
The solution above has been the accepted answer for a long time (since CM 6.x), but the procedure is not complete, at least for Cloudera Manager 7.6 with CDP Base 7.1.6 or above. After using Auto-TLS and needing to disable it because I needed to rename some nodes, I changed the web_tls and agent_tls settings in the Database, and changed the config.ini to disable TLS in all the agents' config and restarted all the cloudera-scm-* daemons (server/agents) and I was able to log to the UI and reach the hosts, but in "Management >> Security" the Cloudera Manager UI keeps showing "Auto-TLS is Enabled" in the menu bar, and if I try to add a new host it will fail with an error like "Unable to deploy host certificates" ... so it seems CM still has Auto-TLS enable somewhere/somehow. I have made a dump and double checked there isn't any TLS config parameter enabled in the DB, but there should be some other place from where CM assumes this is enabled.
... View more
05-05-2022
04:36 PM
As a general statement this is not right by any means. LDAP provides secure and encrypted authentication (encrypted user password and SSL/TLS communication) , together with user/group management. It's only the Hadoop stack does not support this and the two only autentication methods implemented for all the CDP components are the dummy simple auth (described above) and the Kerberos authentication (used in combination with PAM or LDAP for user/group mappings). As an example, nothing less than Knox (the security gateway to HDP or CDP) implements full authenticacion using only LDAP (with TLS), and it only relies on Kerberos to authenticate a single service/proxy user to communicate with the rest of the cluster.
... View more
12-01-2020
12:55 PM
The following map rule is wrong: RULE:[2:\$1@\$0](rm@MY_REALM)s/.*/rm/ the user for the ResourceManager is not "rm" but "yarn" and this should be the replacement value. This is the same as for the hadoop.security.auth_to_local in Hadoop/HDFS configuration.
... View more
11-12-2020
07:10 PM
To give some extra information (in case somebody guess about it) I have also tested adding the -b and -c options to curl, in order to use the received cookies in subsequent calls. Something like this: curl -i -k -u "admin:*****" -b cookies.txt -c cookies.txt --config headers-post.conf ... and the options included in headers-post.conf are: -X POST
-H "X-Requested-By:admin"
-H "Content-Type: application/json"
-H "X-XSRF-HEADER:valid" But the problem is the same and I still receive "HTTP/1.1 403 Forbidden" when trying to execute any statement with LIvy over the session. Best regards
... View more
11-12-2020
06:21 PM
I have a kerberized HDP 3.1.4 authenticating against an IPA server (Kerberos/LDAP). Most of the supported web services are working via Knox (WEBHDFS, HDFSUI, YARNUI, YARNUIV2, SPARKHISTORYUI, OOZIEUI, HDFSUI, etc) , in some cases after fixing or replacing the service definitions with the ones from Knox 1.4.0 (some of ones included with HDP 3 have erros o missing redirects). Now I trying to expose the Livy Rest API via Knox and have tried both the LIVY service definitions in HDP 3 (it includes 3 versions 0.4.0, 0.4.1, 0.4.2) and with the one from Knox 1.4.0. A side note is that the service definition from upstream Knox 1.4.0 has a lower version number (0.4.0) but seems to be more recent including many additional rewrite rules and not using the "/v1" hack from the preliminary versions. The service definition for Knox in the ui.xml topology I'm using is: <service>
<role>LIVYSERVER</role>
<url>http://hdp3-livy:8999</url>
</service> I'm using curl for testing and I'm able to initialize the "pyspark" session via Knox (authenticated with admin) to Livy as "knox" user and with the user authenticated to Knox (in this case "admin") as the "proxyUser": $ curl -k -u admin:${SECRET} --config headers-post.conf --data '{"kind": "pyspark"}' $LIVY_URL/sessions
HTTP/1.1 200 OK {
"appId": null, "appInfo": {"driverLogUrl": null, "sparkUiUrl": null},
"id": 19, "kind": "pyspark", "log": [],
"owner": "knox", "proxyUser": "admin", "state": "starting"
} However when in the next step I try to execute some code I get an empty response with code "403 Forbidden": $ curl -k -u admin:${SECRET} --config livy-headers-post.conf $LIVY_URL/sessions/$SESSIONID/statements -d '{"code":"2 + 2"}'
HTTP/1.1 403 Forbidden If I do the same from Knox server, authenticated with kerberos keytab as user knox and including "proxyUser": "admin" in the JSON request (as Knox does) , I get the same response in the session setup but the subsequent code statement works as expected and executes the code: [knox@server ~]$ curl --negotiate -u : --config headers-post.conf $LIVY_URL/sessions/$SESSIONID/statements -d '{"code":"2 + 2"}'
HTTP/1.1 200 OK {
"code": "2 + 2", "id": 0, "output": null, "progress": 0.0, "state": "waiting"
} As an extra diagnostic I have made a capture of traffic for both executions (via Knox and directly with knox using kinit + proxyUser) and the authentication headers received by Livy from the Knox server are clearly different with each method: Authencated as admin with Knox: POST /sessions/19/statements?doAs=admin HTTP/1.1
X-Forwarded-Proto: https
X-Forwarded-Port: 10443
...
X-Forwarded-Context: /gateway/ui
X-Requested-By: admin
Accept: */*
User-Agent: curl/7.29.0
X-XSRF-HEADER: valid
Content-Type: application/json
Transfer-Encoding: chunked
Host: hdp3-livy:8999
Connection: Keep-Alive
Cookie: hadoop.auth="u=knox&p=knox/hdp3-knox@EXAMPLE.COM&t=kerberos&e=1605153545612s=xD7N1bfFduRqQPQ/qtOkg0OVVs6sXC2C2MnTDlUDrSo="
10
{"code":"2 + 2"}
0 Authenticated with kerberos as knox user (--negotiate): POST /sessions/18/statements HTTP/1.1
User-Agent: curl/7.29.0
Host: hdp3-dtlk-mn01.dtlk.in.iantel.com.uy:8999
Accept: */*
X-Requested-By: admin
Content-Type: application/json
X-XSRF-HEADER:valid
Content-Length: 16
{"code":"2 + 2"}
@
HTTP/1.1 401 Authentication required
Date: Wed, 11 Nov 2020 19:18:50 GMT
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
...
Content-Length: 353
Server: Jetty(9.3.24.v20180605)
POST /sessions/18/statements HTTP/1.1
Authorization: Negotiate YIIC1AYJKoZIhvcSAQICAQBuggLD
......
@
HTTP/1.1 201 Created
Date: Wed, 11 Nov 2020 19:18:50 GMT
WWW-Authenticate: Negotiate YGoGCSqGSIb....
...
Content-Type: application/json;charset=utf-8
Location: /sessions/18/statements/0
Content-Length: 70
Server: Jetty(9.3.24.v20180605)
{"id":0,"code":"2 + 2","state":"waiting","output":null,"progress":0.0} Does anybody knows how to fix this and make Livy work with HTTPS and basic authentication via the Apache Knox Gateway server using Kerberos?
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache Spark
-
Kerberos
08-19-2020
07:42 AM
If you use the $ZK_HOST defined in infra-solr-env.sh you should not need to include the /infra-solr prefix when getting the solrconfig.xml: source /etc/ambari-infra-solr/conf/infra-solr-env.sh /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh -z $ZK_HOST \ -cmd getfile /configs/ranger_audits/solrconfig.xml solrconfig.xml The same applies when uploading the edited config.
... View more
07-24-2020
08:09 AM
May be it's because of a diferente version (I'm using HDP and Hadoop 3) but this doesn't work as described here. In first place if you try to set a variable in the "hiveConf:" namespace you will get an error like this: Error while processing statement: Cannot modify hiveConf:test at runtime You have to use the "hivevar:" namespace for this like: :2> set hivevar:test=date_sub(current_date, 5); But more importante, Hive won't expand the variable value definition as shown here: :2> set hivevar:test; +-----------------------------------------+ | hivevar:test=date_sub(current_date, 5) | +-----------------------------------------+ So the INSERT will not interpreted as you stated but instead as: INSERT INTO TABLE am_temp2 PARTITION (search_date=date_sub(current_date, 5)) and this for some reason is not supported in Hive and gives a compilation error: FAILED: ParseException line 1:50 cannot recognize input near 'date_sub' '(' It would be very useful to insert data into static partitioning using pre-calculated variable values like this, from functions or select queries, but I still haven't found how to do this in HiveQL. As a reference, this seems to be (at least partially) related with this: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/hive-overview/content/hive_use_variables.html
... View more
08-27-2019
10:45 AM
2 Kudos
This feature is not (well) documented anywere and the intructions for user impersonation in Zeppelin's manual only works for shell or for spark when not using Kerberos, so I will respond myself showing how I was able to make this work with Kerberos and HDP 3.1 (zeppelin 0.8.0). First you DON'T have to change/uncoment ZEPPELIN_IMPERSONATE_SPARK_PROXY_USER in zeppelin-env and should leave this with the default value "true" (meaning that Zeppelin will use --proxy-user option when impersonation is enabled in the spark2 interpreter). The after you kerberized the cluster you will have to edit the Spark2 interpreter and change the following: Set the interpeter to be instantiated "Per User | Isolated" and select the "User Impersonate" checkbox. Remove the following properties from the interpreter configuration: spark.yarn.keytab=/etc/security/keytabs/zeppelin.server.kerberos.keytab
spark.yarn.principal=zeppelin-mycluster@MYREALM.COM Add the following properties to the interpreter (fix domain and kerberos realm's names) : zeppelin.spark.keytab=/etc/security/keytabs/zeppelin.server.kerberos.keytab
zeppelin.spark.principal=zeppelin-mycluster@MYREALM.COM Save and restart the interpreter After you should be able to run your %spark2.* jobs from Zeppelin as the logger user.
... View more
08-24-2019
05:20 PM
Hello, I would like to learn about the meaning of the "priority-utilization" ordering policy in my root queue, but at least after the Hortonworks-Cloudera fusion the referenced article is not publicly available. Do you have any other source to get such information?
... View more
08-24-2019
08:08 AM
After checking the official Apache Hadoop docs and some other Internet sources it seems to me that the description of the behavior for the property
yarn.scheduler.capacity.queue-mappings-override.enable=true/false
of the YARN Capacity Scheduler is wrong (and actually the opposite) in the official documentation of HDP 3.1.0 at the "Managing Data Operating System | Enable Override of Default Queue Mappings" section.
It states that when this property is set to true "it allow the user to override the default user to queue mappings defined with the yarn.scheduler.capacity.queue-mappings property" , and indeed gives an example showing this (wrongly) expected behavior.
However, as clearly explained in the responses by @gnovak and @vasubramanian for the question " Yarn queue Mapping" in this same forum, the purpose of that property is exactly the opposite:
if yarn.scheduler.capacity.queue-mappings-override.enable is set to false then user's have more control on which queue they want to submit the application either by setting the queue name or from default mapping, in this case user specified queue name takes the precedence , but if yarn.scheduler.capacity.queue-mappings-override.enable set to true then admin's have more control and admin can override user's explicit settings.
In other words, what the property enables is the possibility for the administrator to use the property "yarn.scheduler.capacity.queue-mappings" to override what the user or application configuration states, and not the other way.
May please some of the most experienced contributors in this forum confirm this? and if that's the case, if it were possible that someone from the company could correct such and important error in the documentation?
Best regards.
... View more
Labels:
- Labels:
-
Apache YARN
05-24-2019
02:55 PM
@jingyong zou you should not use AUTH_LDAP_APPEND_DOMAIN unless your users are authenticating using the fully qualified principal name as in "username@mydomain.com" instead of simply "username". If you use uid or samAccountName as AUTH_LDAP_UID_FIELD (as is the case with OpenLDAP, IPA or AD) then this is not needed, Also check the values for the parameters AUTH_USER_REGISTRATION=True and AUTH_USER_REGISTRATION_ROLE which should be set to a valid role in Superset (Public, Gamma, Alpha o Admin). Another not very documented parameter which may be important depending on your LDAP setup is AUTH_LDAP_USERNAME_FORMAT, check this also. With the previous advises in mind, check carefully the following documentation articles and you may be able to find your appropiate options combination to make LDAP work with Superset: https://flask-appbuilder.readthedocs.io/en/latest/config.html https://superset.incubator.apache.org/security.html https://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-ldap A tcpdump capture in your Superset server + wireshark analysis may be also of much help to debug what is your current Superset config sending to the LDAP server. In my case this was the "final step" to fit all the pieces.
... View more
05-08-2019
06:35 PM
I have succesfully deployed HiveServer2 Interactive (Hive LLAP) in a kerberized HDP 3.1 cluster, and I'm able to access the HiveServer2 Interactive UI autenticated as admin user after making a kinit to the Kerberos admin's principal as show here: However when I try to access the "Local logs" tab I get an "Unauthorized user" error as shown bellow: I'm testing with latest Firefox and Chrome versions, and the YARN RM UI (v1 and v2) and Hadoop consoles are working with SPNEGO authentication for the admin user side by side (on different tabs) with this very same user and browser. Does anybody know some fix to make this work for the admin or any other kerberos authenticated user? Any help will be greatly appreciated. Best regards Luis
... View more
Labels:
- Labels:
-
Apache Hive
04-03-2019
07:56 PM
According tto the documentation in this link: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/data-operating-system/content/partition_a_cluster_using_node_labels.html the information in this articulo is wrong because if you assing all the nodes to 3 different exclusive Node Label partitions, AND you do not set the default Node Label in the default Queue then you will not have resources available to run any Job sent to the default queue without an explicit label.
... View more
04-01-2019
03:23 PM
If you set these properties by hand, take into account that in HDP 3.x the route for the Hive Warehouse has been changed to: /warehouse/tablespace/managed/hive Also in the Ambari configuration for Druid 3.1 the property is set to: hive.druid.storage.storageDirectory = {{druid_storage_dir}} which is expanded to /apps/druid/warehouse, a different path from the Hive Warehouse.
... View more
03-27-2019
04:02 PM
At least in HDP 3 and above the extjs package is not in the default HDP yum repositories. This must be downloaded and installed manually as described here: https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/managing-and-monitoring-ambari/content/amb_enable_the_oozie_ui.html wget http://public-repo-1.hortonworks.com/HDP-UTILS-GPL-1.1.0.22/repos/centos7/extjs/extjs-2.2-1.noarch.rpm
rpm -ivh extjs-2.2-1.noarch.rpm
... View more
03-27-2019
03:50 PM
This doesnt work in HDP 3+ . The package is not found in the repos: [root@agesic-mn02 ~]# yum search extjs
Warning: No matches found for: extjs
No matches found Also the oozie and oozie-webapp packages ar already installed. Any suggestion?
... View more
03-01-2019
02:33 PM
We have tested disabling user impersontation in Livy interpreter and HWC also works in this case. So the problem seems to be we are not able to use HWC with Spark from Zeppeling and using impersonation at the same time. This seem a HUGE drawback for using Zeppelin with HDP 3.x in a secured multi user environment with policy based access restrictions.
... View more
02-26-2019
03:23 AM
You where able to make Hive Warehouse Connector work with Kerberos in the Spark Zeppelin interpreter when using USER IMPERSONATION, or only running with the "zeppelin" user?? In my case al the other Spark client interfaces (spark-shell, pyspark, spark-submit, etc) are working and I'm experiencing the same problem as you when trying to use Livy with HWC. But in my case, the Spark Zeppelin interpreter is also not working when I enable impersonation, which is needed to impose authorization restrictions to Hive data access with Ranger. If you were able to make HWC work with Spark Interpreter in Zeppelin AND IMPERSONATION enabled; I would be very grateful if you could share the changes you have made in the interpreter's configuration to make this work. The Zeppelin Spark Interpreter with impersonation disabled is working with HWC, but I NEED data access authorization, so this is not an option for me. Best Regards
... View more
02-03-2019
12:33 AM
2 Kudos
Has anybody been able to make Spark-Hive integration using Hive Warehouse Connector work in Zeppelin with impersonation in a Kerberized cluster? I have followed the steps in this article
https://community.hortonworks.com/articles/223626/integrating-apache-hive-with-apache-spark-hive-war.html and Spark works with the HWC using spark-shell or pyspark from the console, but Spark in Zeppelin will not work when using impersonation with the given instructions, neither with %spark2 or %livy2 interpreters. In the case of %spark2 interpreters it works ONLY if I disable impersonation and explicitly configure the Zeppelin's kerberos principal in the interpreter via the "spark.yarn.principal/keytab" properties. But in this case all the jobs are run as "zeppelin" user, but this is not an option for me in a multiuser environment. If I don't use HWC Spark is working perfectly with user impersonation in both interpreters. But with the %livy interpreter or when I enable impersonation in the %spark2 interpreter I get an error related to the spark not being able to create an SQL connection pool because of Kerberos GSS initialization failed: java.lang.RuntimeException: java.sql.SQLException: Cannot create
PoolableConnectionFactory (Could not open client transport for any of
the Server URI's in ZooKeeper: GSS initiate failed)...Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: GSS initiate failed
at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:333) <br>... 59 more
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) I have tried all the workarounds, checked the proxyusers configuration for both the zeppelin and livy principal in Hadoop/HDFS core-site, and also defined both users as Livy superusers but the problem persist. I suspect maybe it's not posible to use Spark/Hive integration with user impersonation in the case of the %spark2 interpreter. But this has to be possible at least with the %livy interpeter. Does anyone has an idea of or some suggestion on which extra configurations need to be done in order to make HWC work inside Zeppelin?? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
-
Apache Zeppelin
01-16-2019
12:19 PM
2 Kudos
Just in case this help someone else, I did a search for the missing JAR and found it in the ambari-server package. So I was able to "bypass" this problem by copying this file to the required path for ambari-agent in all the cluster nodes by using pscp: pscp -h cluster/hdpall /var/lib/ambari-server/resources/stack-hooks/before-START/files/fast-hdfs-resource.jar /var/lib/ambari-agent/lib/ This solved the error but I consider this a "hack" and expect maybe there is a better and more general answer.
... View more
01-16-2019
12:09 PM
2 Kudos
After a fresh and kerberized install of HDP 3.1 I get the following error when restarting Knox from Ambari: 2019-01-16 08:27:58,350 - Execute['hadoop --config /usr/hdp/3.1.0.0-78/hadoop/conf jar /var/lib/ambari-agent/lib/fast-hdfs-resource.jar /var/lib/ambari-agent/tmp/hdfs_resources_1547638078.26.json'] {'logoutput': None, 'path': ['/usr/hdp/3.1.0.0-78/hadoop/bin'], 'user': 'hdfs'}
2019-01-16 08:27:58,898 - Audit directory creation in HDFS for KNOX Ranger plugin failed with error:
Execution of 'hadoop --config /usr/hdp/3.1.0.0-78/hadoop/conf jar /var/lib/ambari-agent/lib/fast-hdfs-resource.jar /var/lib/ambari-agent/tmp/hdfs_resources_1547638078.26.json' returned 255. JAR does not exist or is not a normal file: /var/lib/ambari-agent/lib/fast-hdfs-resource.jar
Traceback (most recent call last):
.... The error seems to be a mising JAR "fast-hdfs-resource.jar" inside the Ambari Agent folder /var/lib/ambari-agent. Just in case I have checked both the files and the post install scripts in the ambari-agent RPM package and this file doesn't seems to be present or created during install.
... View more
Labels:
12-24-2018
05:38 PM
I followed this with HDP 2.6.5 and the HBaseUI became accessible in the given URL but has many errors and links not working inside. I posted a question on how to fix this and then the answer resolving most of these issues here: https://community.hortonworks.com/questions/231948/how-to-fix-knox-hbase-ui.html You are welcome to test this and include these fixes in your article if you find it appropriate. Best regards
... View more
12-24-2018
05:29 PM
@Isaac Arnault I invested (and spent) 6 hours trying to fix this and finally found a solutions. So because
I'm still not able to create an article (because of the reputation system at this site) this was the only way I found to share this with the community to help others and avoid them to repeat the unsavory path 😉 My best wishes to you as well and I hope this may help you and someone else. Best regards
... View more
12-24-2018
04:00 PM
As with other Knox UI services (as HbaseUI or SparkHistoryUI) this error once again is related with some missing (and/or wrong) rules in the YARNUI service definitions instaled with HDP 2.6.5 under /var/lib/knox/data-2.6.5.0-292/services/yarnui/2.7.0/ To handle correctly these link URLs including arguments in the proxy pages patch the service's rewrite.xml file as follows: --- rewrite.xml.hdp
+++ rewrite.xml@@ -35,6 +35,9 @@
<rule dir="IN" name="YARNUI/yarn/inbound/cluster" pattern="*://*:*/**/yarn/cluster/{**}">
<rewrite template="{$serviceUrl[YARNUI]}/cluster/{**}"/>
</rule>
+<rule dir="IN" name="YARNUI/yarn/inbound/proxy1" pattern="*://*:*/**/yarn/proxy/{**}?{**}">
+ <rewrite template="{$serviceUrl[YARNUI]}/proxy/{**}?{**}"/>
+</rule>
<rule dir="IN" name="YARNUI/yarn/inbound/proxy" pattern="*://*:*/**/yarn/proxy/{**}">
<rewrite template="{$serviceUrl[YARNUI]}/proxy/{**}"/>
</rule>
@@ -250,9 +253,13 @@
<match pattern="*://*:*/proxy/{**}"/>
<rewrite template="{$frontend[url]}/yarn/proxy/{**}"/>
</rule>
-<rule dir="OUT" name="YARNUI/yarn/outbound/apps/history1">
- <match pattern="/proxy/{**}"/>
- <rewrite template="{$frontend[url]}/yarn/proxy/{**}"/>
+<rule dir="OUT" name="YARNUI/yarn/outbound/apps/history1" flow="OR">
+ <match pattern="/proxy/{**}?{**}">
+ <rewrite template="{$frontend[url]}/yarn/proxy/{**}?{**}"/>
+ </match>
+ <match pattern="/proxy/{**}">
+ <rewrite template="{$frontend[url]}/yarn/proxy/{**}"/>
+ </match>
</rule> Then remove the deployed runtime topology folders: # rm -rf /usr/hdp/current/knox-server/data/deployments/*.topo.* and the restart Knox and your links in the Knox/YarnUI proxied Spark UI will include the required parameters and work as expected:
... View more
12-24-2018
03:51 PM
In HDP 2.6.5 manged by Ambari I have a working Knox topology exposing the Resource Manager Yarn UI with the folloing tags in the ui.xml (and Ambari managed default.xm) topology files: <service>
<role>YARNUI</role>
<url>http://{resourcemanager_server}:8088</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{resourcemanager_server}:8088/ws</url>
</service>
This is working and I'm able to access the YARN UI via Knox with the following URL: https://knox.mydomain.org:8443/gateway/ui/yarn/ However when I access to the "ApplicationMaster" for a running Spark task, and I reach the Spark UI via the "/proxy/" subpath under the Resource Manager UI; the main page is shown correctly but most of the links under the "Jobs", "Stages" "SQL" and "JDBC/ODBC Server" (for Thift Server) are not working and give a Not Found error as shown. Based in the displayed error and comparing with the corresponding
page when accessed directly in the Resource Manager UI (port 8088) it
seems these links are missing the URL parameters (id and others) and
this is the reason for the errors How to fix this?
... View more
Labels:
12-24-2018
01:44 PM
WARNING: when doing the previous changes to the installed service.xml and rewrite.xml (under data/services/...) DO NOT create a backup copy (ex. rewrite.xml.orig) or move the original version to a "backup" sub-folder under this path!! Knox will load ALL the xml files it finds under "/usr/hdp/current/knox-server/data/services" (weird but real!) and this will trigger many strange and confusing behaviors!! I've wasted a few hours trying to make the above work, just because of this 😞
... View more
12-24-2018
01:08 PM
Thank you very much @scharan for your response I tried these steps and indeed posted an
answer based on this to my own question because after some minor fixed (some trivial errors as duplicated slashes on rewrite patterns) these service definitions seemed to fix the header links problems. But after some testing I
found out that this version introduced many other problems with the internal links and realized that the new situation was worst than with the original HDP version so I reverted these changes and deleted my answer. Then I decided to patch the service definition in HDP 2.6.5 adding the missing rewrite rules to the rewrite.xml file as follows: <rule dir="OUT" name="HBASEUI/hbase/outbound/tables" pattern="/tablesDetailed.jsp">
<rewrite template="{$frontend[url]}/hbase/webui/tablesDetailed.jsp"/>
</rule>
- <rule dir="OUT" name="HBASEUI/hbase/outbound/logs" pattern="/logs/">
+ <rule dir="OUT" name="HBASEUI/hbase/outbound/procedures" pattern="/procedures.jsp">
+ <rewrite template="{$frontend[url]}/hbase/webui/procedures.jsp"/>
+ </rule>
+ <rule dir="OUT" name="HBASEUI/hbase/outbound/regionserver/nohome" pattern="/rs-status/">
+ <rewrite template="{$frontend[url]}/hbase/webui/"/>
+ </rule>
+
+ <rule dir="OUT" name="HBASEUI/hbase/outbound/logs" pattern="/logs">
<rewrite template="{$frontend[url]}/hbase/webui/logs/"/>
</rule> This solved most of the problems but the links to the Home ("/") and Metric Dump ("/jmx") were still wrong, event when the outgoing rewrite rule for the "/jmx" was present in the rewrite.xml: <rule dir="OUT" name="HBASEUI/hbase/outbound/jmx" pattern="/jmx">
<rewrite template="{$frontend[url]}/hbase/webui/jmx"/>
</rule> Some more testing showed me that Knox has a bug (or feature?) that prevents an outgoing implicit rule for matching when the length of the pattern has less than 4 chars! (ex. jmx) To fix these problems I followed the article Proxying a UI using Knox to create a rewrite filter rule and apply this to the "response.body" to the "text/html" pages. These are the needed changes in service.xml: <routes>
<route path="/hbase/webui/">
<rewrite apply="HBASEUI/hbase/inbound/master/root" to="request.url"/>
+ <rewrite apply="HBASEUI/hbase/outbound/master/filter" to="response.body"/>
</route>
<route path="/hbase/webui/**">
<rewrite apply="HBASEUI/hbase/inbound/master/path" to="request.url"/>
<rewrite apply="HBASEUI/hbase/outbound/headers" to="response.headers"/>
+ <rewrite apply="HBASEUI/hbase/outbound/master/filter" to="response.body"/>
</route>
<route path="/hbase/webui/**?**">
<rewrite apply="HBASEUI/hbase/inbound/master/query" to="request.url"/>
@@ -30,9 +32,11 @@
</route>
<route path="/hbase/webui/regionserver/**?{host}?{port}">
<rewrite apply="HBASEUI/hbase/inbound/regionserver/home" to="request.url"/>
+ <rewrite apply="HBASEUI/hbase/outbound/master/filter" to="response.body"/>
</route>
<route path="/hbase/webui/master/**?{host}?{port}">
<rewrite apply="HBASEUI/hbase/inbound/master/home" to="request.url"/>
+ <rewrite apply="HBASEUI/hbase/outbound/master/filter" to="response.body"/>
</route>
<route path="/hbase/webui/logs?**">
<rewrite apply="HBASEUI/hbase/outbound/headers" to="response.headers"/> An then add these rules to the end of rewrite.xml: <rule dir="OUT" name="HBASEUI/hbase/outbound/master/filter/home">
<rewrite template="{$frontend[path]}/hbase/webui/"/>
</rule>
<rule dir="OUT" name="HBASEUI/hbase/outbound/master/filter/jmx">
<rewrite template="{$frontend[path]}/hbase/webui/jmx"/>
</rule>
<filter name="HBASEUI/hbase/outbound/master/filter">
<content type="text/html">
<apply path="/jmx" rule="HBASEUI/hbase/outbound/master/filter/jmx"/>
<apply path="/" rule="HBASEUI/hbase/outbound/master/filter/home"/>
</content>
</filter>
Now finally all the header links are OK in all the pages (including the internal ones to the RegionServer nodes status) and all the HBaseUI seems to be working as expected. The only remaining problem seems to be with some very internal and rarely used links at the "Regions" section inside the RegionServer nodes subpages (this will require more rewrite tweaks). This worked for me and I hope this will help someone to make the HBaseUI usable through Knox.
... View more
12-22-2018
11:35 PM
In an Ambari HDP 2.6.5 install after configuring a Knox topology to expose Hbase UI with following tags: <service>
<role>HBASEUI</role>
<url>http://{{hbase_master_host}}:16010</url>
</service>
you will able to access the HBase UI in the URL https://knoxserver:8443/gateway/ui/hbase/webui/ but many of the links at the top menu bar, as for example "Procedures" or "Local Logs", will point to the wrong URL without the gateway prefix. For example the "Procedures" menu will point to "https://knoxserver:8443/procedures.jsp" and gives you a "Not Found" error. How to fix this?
... View more
Labels:
12-22-2018
11:04 PM
To enable the HDP UI's you will need these additional service definitions in your XML topology file: <service>
<role>NAMENODE</role><url>hdfs://{namenode-server}:8020</url>
</service>
<service>
<role>HDFSUI</role><url>http://{namenode-server}:50070</url>
</service>
<service>
<role>YARNUI</role><url>http://{resourcemanager-server}:8088</url>
</service>
<service>
<role>RESOURCEMANAGER</role><url>http://{resourcemanager-server}:8088/ws</url>
</service>
<service>
<role>JOBHISTORYUI</role><url>http://{mapreduce-history-server}:19888</url>
</service>
<service>
<role>OOZIE</role><url>http://{oozie-server}:11000/oozie</url>
</service>
<service>
<role>OOZIEUI</role><url>http://{oozie-server}:11000/oozie</url>
</service>
<service>
<role>HBASEUI</role><url>http://{hbase-master-server}:16010</url>
</service>
Take a look to the service definitions under /usr/hdp/current/knox-server/data/services/ and to the official Knox docs section bellow were you will find the URL mapping to access each service UI: http://knox.apache.org/books/knox-0-12-0/user-guide.html#UI+Service+Details
... View more