Hello, we have deployed an HDP 2.6.4 cluster on some VMs, some days ago the masternode has been hot migrated by the VM sys admins, after that we are unable to make any action on that node using Ambari UI and some services have Heartbeat Lost despite the fact that are UP. We also tried to kill and then start again the services but we were unable to start using Ambari. We tried also to restart the ambari-server and all the ambari-agent without success, in the log files there is no ERROR message, how can we troubleshoot this? The screenshot is about services on the same node (same ambari-agent instance) Thank you ... View more

Hello, we want to configure the SSO login for Ambari and Ranger through Knox to an external SSO openid connect service. We followed the guide described here Ranger SSO works well (so I don't think that the problem is the Knox configuration) but Ambari is not working, after a redirect on the external service and the login phase shows the following message: Login Redirect Issue For single sign-on, make sure that Knox Gateway and Ambari Server are located on the same host or subdomain.Alternatively login as an Ambari local user using the local login page. in the ambari-server.log we found this entry: Cannot find user from JWT. Please, ensure LDAP is configured and users are synced. but we are not using LDAP. These are the topologies set in Knox, this is the Advanced knoxsso-topology (we replaced the name of our customer with [our customer]): <topology> <gateway> <provider> <role>webappsec</role> <name>WebAppSec</name> <enabled>true</enabled> <param><name>xframe.options.enabled</name><value>true</value></param> </provider> <provider> <role>federation</role> <name>pac4j</name> <enabled>true</enabled> <param> <name>pac4j.callbackUrl</name> <value>https://[our customer]romlakp01.global.[our customer].org:8443/gateway/knoxsso/api/v1/websso</value> </param> <param> <name>pac4j.id_attribute</name> <value>nickname</value> </param> <param> <name>clientName</name> <value>OidcClient</value> </param> <param> <name>oidc.id</name> <value>493f2182-caaa-4cef-8cf3-644bda0dfaaa</value> </param> <param> <name>oidc.secret</name> <value>NzYT4WVCe53TYCaaasZn5BLuzoRLiqqBDF3VBaaa</value> </param> <param> <name>oidc.discoveryUri</name> <value>https://fs.auth.[our customer].org/adfs/.well-known/openid-configuration/</value> </param> <param> <name>oidc.preferredJwsAlgorithm</name> <value>RS256</value> </param> </provider> </gateway> <application> <name>knoxauth</name> </application> <service> <role>KNOXSSO</role> <param> <name>knoxsso.cookie.secure.only</name> <value>false</value> </param> <param> <name>knoxsso.token.ttl</name> <value>100000</value> </param> <param> <name>knoxsso.redirect.whitelist.regex</name> <value>^https?:\/\/(datalake\.efs\.[our customer]\.org|10\.11\.41\.115|[our customer]romlakp01\.global\.[our customer]\.org|www\.local\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1).*$</value> </param> </service> </topology> and this is the Advanced topology: <topology> <gateway> <provider> <role>webappsec</role> <name>WebAppSec</name> <enabled>true</enabled> <param> <name>cors.enabled</name> <value>true</value> </param> </provider> <provider> <role>federation</role> <name>SSOCookieProvider</name> <enabled>true</enabled> <param> <name>sso.authentication.provider.url</name> <value>https://datalake.efs.[our customer].org:8443/gateway/knoxsso/api/v1/websso</value> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> </gateway> <service> <role>AMBARIUI</role> <url>http://[our customer]romlakp01.global.[our customer].org:8080</url> </service> <service> <role>NAMENODE</role> <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url> </service> <service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service> <service> <role>WEBHDFS</role> {{webhdfs_service_urls}} </service> <service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service> <service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> <service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url> </service> <service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service> <service> <role>DRUID-COORDINATOR-UI</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-COORDINATOR</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-OVERLORD-UI</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-OVERLORD</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-ROUTER</role> {{druid_router_urls}} </service> <service> <role>DRUID-BROKER</role> {{druid_broker_urls}} </service> <service> <role>ZEPPELINUI</role> {{zeppelin_ui_urls}} </service> <service> <role>ZEPPELINWS</role> {{zeppelin_ws_urls}} </service> </topology> Other logs with masquerade IP/HOSTS gateway-audit.log: ... 18/07/12 18:57:54 ||cdeb0a8b-80ef-4010-8afe-799680ce49ed|audit|10.65.41.55|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true|unavailable|Request method: GET 18/07/12 18:57:54 ||cdeb0a8b-80ef-4010-8afe-799680ce49ed|audit|10.65.41.55|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true|success|Response status: 302 18/07/12 18:58:14 ||490861a3-301b-4c91-8379-eed2d8eebee8|audit|10.65.41.55|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=OidcClient&code=AAAAAAAAAAAAAAAAAAAAAA.vkpjqBjo1QhOBs5AjevDlneCYHI.imYm3Se2M5ox5Pz-xIVdeiRBy4o6DbEJys5UWNOw_I1J1qmAbij6jOVz74t2mMpR7X-UYdscncf6qGJxNbhpxXeULgpYE5AY5KV5sbvZ3orkCmGX6YCgFeZHJ28C_FNaSz2ZO-4gSn8oY0x4EOaoERDTu-TCM6qktErx4oU8-e_WIqNWjZQhgSRv3G7fbwkPOFqUZow5ehJyzr988gGxCLw0hBxYjg4M8u4x6nSa6kckeb57j2mwbKU51xiQOOb6XB9ibIUJrRMevi6JojRyoO55-2UQ1rKDBn7Qr48c_735KPx3Tmye1hdBfyx5aV1vu_10qSY-WtKMu_SzvCm12w&state=vnALts3YDByujG_44ZcD_yOiuWdCBO32usY0jv467MM|unavailable|Request method: GET 18/07/12 18:58:14 ||490861a3-301b-4c91-8379-eed2d8eebee8|audit|10.65.41.55|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=OidcClient&code=AAAAAAAAAAAAAAAAAAAAAA.vkpjqBjo1QhOBs5AjevDlneCYHI.imYm3Se2M5ox5Pz-xIVdeiRBy4o6DbEJys5UWNOw_I1J1qmAbij6jOVz74t2mMpR7X-UYdscncf6qGJxNbhpxXeULgpYE5AY5KV5sbvZ3orkCmGX6YCgFeZHJ28C_FNaSz2ZO-4gSn8oY0x4EOaoERDTu-TCM6qktErx4oU8-e_WIqNWjZQhgSRv3G7fbwkPOFqUZow5ehJyzr988gGxCLw0hBxYjg4M8u4x6nSa6kckeb57j2mwbKU51xiQOOb6XB9ibIUJrRMevi6JojRyoO55-2UQ1rKDBn7Qr48c_735KPx3Tmye1hdBfyx5aV1vu_10qSY-WtKMu_SzvCm12w&state=vnALts3YDByujG_44ZcD_yOiuWdCBO32usY0jv467MM|success|Response status: 302 18/07/12 18:58:14 ||8d28b5b6-4e8b-41ca-9753-29f9ff0b8bf5|audit|10.65.41.55|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true|unavailable|Request method: GET 18/07/12 18:58:15 ||8d28b5b6-4e8b-41ca-9753-29f9ff0b8bf5|audit|10.65.41.55|KNOXSSO|{--MY OpenIDc User--}|||authentication|uri|/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true|success| 18/07/12 18:58:15 ||8d28b5b6-4e8b-41ca-9753-29f9ff0b8bf5|audit|10.65.41.55|KNOXSSO|{--MY OpenIDc User--}|||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true|success|Response status: 303 ... gateway.log: ... 2018-07-12 18:57:54,981 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /api/v1/websso 2018-07-12 18:57:54,982 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: pac4jUserProfile = null 2018-07-12 18:57:54,983 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: pac4jRequestedUrl = https://{--MY HOST NAME--}:8443/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true 2018-07-12 18:57:54,987 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: OidcClient$attemptedAuthentication = null 2018-07-12 18:57:54,987 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: oidcStateAttribute = vnALts3YDByujG_44ZcD_yOiuWdCBO32usY0jv467MM 2018-07-12 18:58:14,496 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /api/v1/websso 2018-07-12 18:58:14,687 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: oidcStateAttribute = vnALts3YDByujG_44ZcD_yOiuWdCBO32usY0jv467MM 2018-07-12 18:58:14,688 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: OidcClient$attemptedAuthentication = 2018-07-12 18:58:14,732 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: pac4jUserProfile = <OidcProfile> | id: {--MY OpenIDc User--} | attributes: {sub={--MY OpenIDc User--}} | roles: [] | permissions: [] | isRemembered: false | 2018-07-12 18:58:14,983 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: pac4jRequestedUrl = https://{--MY HOST NAME--}:8443/gateway/knoxsso/api/v1/websso?originalUrl=http%3A%2F%2F{--MY IP--}%3A8080%2F%23%2Flogin?redirected=true 2018-07-12 18:58:14,983 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: pac4jRequestedUrl = null 2018-07-12 18:58:14,993 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /api/v1/websso 2018-07-12 18:58:15,161 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: pac4jUserProfile = <OidcProfile> | id: {--MY OpenIDc User--} | attributes: {sub={--MY OpenIDc User--}} | roles: [] | permissions: [] | isRemembered: false | 2018-07-12 18:58:15,305 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(90)) - Get from session: pac4jUserProfile = <OidcProfile> | id: {--MY OpenIDc User--} | attributes: {sub={--MY OpenIDc User--}} | roles: [] | permissions: [] | isRemembered: false | 2018-07-12 18:58:15,306 DEBUG filter.Pac4jIdentityAdapter (Pac4jIdentityAdapter.java:doFilter(70)) - User authenticated as: <OidcProfile> | id: {--MY OpenIDc User--} | attributes: {sub={--MY OpenIDc User--}} | roles: [] | permissions: [] | isRemembered: false | 2018-07-12 18:58:15,306 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(105)) - Save in session: pac4jUserProfile = 2018-07-12 18:58:15,310 WARN service.knoxsso (WebSSOResource.java:init(102)) - The SSO cookie SecureOnly flag is set to FALSE and is therefore insecure. 2018-07-12 18:58:15,311 INFO service.knoxsso (WebSSOResource.java:getCookieValue(318)) - Unable to find cookie with name: original-url 2018-07-12 18:58:15,316 DEBUG service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(276)) - Adding the following JWT token as a cookie: eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJrOHRYUVN1M284bkY1SWZyQ3F6RlcyQkRcL0RBY2JsQ0xPSWlMSzlOTitiYz0iLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTMxNDE0Nzk1fQ.uepdHwqJvq0ri-tuBEZxpOHfJImbTC8UgDhXIzjOXYrCoTv7jl3_yNZWqwTZiK_hDx4Ni33_3Ao8dNq9fABncjPMEO1b8zip8j4mHCRplAyWdpwt5DHJnuaVlHNIA_ROcMSakUfEZTW7XSGjhbv1KWDCrFCwm0woe2acA2CNPsw 2018-07-12 18:58:15,316 INFO service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(292)) - JWT cookie successfully added. 2018-07-12 18:58:15,316 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(202)) - About to redirect to original URL: http://{--MY IP--}/#/login?redirected=true ... ambari-server.log: ... 12 Jul 2018 19:11:40,571 WARN [ambari-client-thread-37] JwtAuthenticationFilter:173 - JWT authentication failed - Cannot find user from JWT. Please, ensure LDAP is configured and users are synced. ... ambari-alerts.log: ...empty... can you help us? Thank you ... View more