Member since
01-12-2022
28
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2460 | 04-03-2024 12:17 AM |
08-07-2023
08:18 PM
I have a secure hadoop cluster with HDP3.1，I recently tried to interconnect this cluster with the Knox component to implement a secure proxy. This cluster has kerberos, Ldap, and https enabled, I create one config like this: <topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>main.ldapRealm</name>
<value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>cn=admin,dc=datasw,dc=com</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://hdp001.datasw.com:389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>authentication</role>
<name>HadoopAuth</name>
<enabled>true</enabled>
<param>
<name>config.prefix</name>
<value>hadoop.auth.config</value>
</param>
<param>
<name>hadoop.auth.config.type</name>
<value>kerberos</value>
</param>
<param>
<name>hadoop.auth.config.simple.anonymous.allowed</name>
<value>false</value>
</param>
<param>
<name>hadoop.auth.config.token.validity</name>
<value>1800</value>
</param>
<param>
<name>hadoop.auth.config.cookie.domain</name>
<value>datasw.com</value>
</param>
<param>
<name>hadoop.auth.config.cookie.path</name>
<value>gateway/default</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.principal</name>
<value>HTTP/hdp003.datasw@DATASW.COM</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.keytab</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.name.rules</name>
<value>DEFAULT</value>
</param>
<param>
<name>fs.defaultFS</name>
<value>hdfs://hdfsCluster</value>
</param>
<param>
<name>dfs.internal.nameservices</name>
<value>hdfsCluster</value>
</param>
<param>
<name>dfs.ha.namenodes.hdfsCluster</name>
<value>nn1,nn2</value>
</param>
<param>
<name>dfs.nameservices</name>
<value>hdfsCluster</value>
</param>
<param>
<name>dfs.namenode.https-address</name>
<value>hdp001.datasw:50470</value>
</param>
<param>
<name>dfs.namenode.https-address.hdfsCluster.nn1</name>
<value>hdp001.datasw:50470</value>
</param>
<param>
<name>dfs.namenode.https-address.hdfsCluster.nn2</name>
<value>hdp002.datasw:50470</value>
</param>
</provider>
</gateway>
<service>
<role>HDFSUI</role>
<url>https://hdp002.datasw.com:50470</url>
</service>
</topology> and I copy the hadoop cluster's truststore.jks file to the $GATEWAY_HOME/data/security/keystores/ and set gateway.httpclient.truststore.path param in gateway-stie.xml: <property>
<name>gateway.httpclient.truststore.path</name>
<value>/usr/local/knox/data/security/keystores/truststore.jks</value>
</property>
<property>
<name>gateway.httpclient.truststore.type</name>
<value>JKS</value>
</property>
<property>
<name>gateway.httpclient.truststore.password.alias</name>
<value>pthdp</value>
</property> Then I restart the Knox gateway，but when I access the NameNode webUi, I receive the following error message: 2023-08-08 11:14:38,050 58fc3dbf-4c6e-4684-860d-0a4e443f85d2 WARN knox.gateway (DefaultDispatch.java:executeOutboundRequest(183)) - Connection exception dispatching request: https://hdp002.datasw.com:50470/?user.name=admin javax.net.ssl.SSLPeerUnverifiedException: Certificate for <hdp002.datasw.com> doesn't match any of the subject alternative names: []
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <hdp002.datasw.com> doesn't match any of the subject alternative names: []
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:166) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:152) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequestWrapper(DefaultDispatch.java:135) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:300) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:183) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:127) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.access$000(AbstractIdentityAssertionFilter.java:55) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter$1.run(AbstractIdentityAssertionFilter.java:161) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_291]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_291]
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doAs(AbstractIdentityAssertionFilter.java:156) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:146) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:241) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:57) ~[gateway-provider-rewrite-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) ~[gateway-spi-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_291]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_291]
at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:146) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:76) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) ~[shiro-core-1.10.0.jar:1.10.0]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ~[shiro-core-1.10.0.jar:1.10.0]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) ~[shiro-core-1.10.0.jar:1.10.0]
at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:73) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) ~[shiro-web-1.10.0.jar:1.10.0]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) ~[shiro-web-1.10.0.jar:1.10.0] In order to achieve Knox proxy, What else do I need to do?
... View more
Labels:
- Labels:
-
Apache Knox
-
Kerberos
07-27-2023
01:01 AM
Hi, My hadoop cluster use HDP3.1.5-152.0, I start httpfs in cli command, and my webhdfs is work fine. But when i send request to the httpfs service , i got 500 code error like this: 15:50:39,064 WARN ServletHandler:632 - /webhdfs/v1/user
java.lang.IllegalArgumentException: Empty key
at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
at org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
at org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1619)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745) This error occurs regardless of whether my cluster has kerberos and ssl enabled, Please give me some help, thanks!
... View more
Labels:
- Labels:
-
HDFS
12-06-2022
07:34 PM
Hi,recently I have been trying to track my spark application using SparkListener. Now I have a problem: There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application. And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks. Some can help me ?
... View more
Labels:
- Labels:
-
Apache Spark
12-06-2022
07:04 PM
1 Kudo
Thanks,but this api is used to delete the entity,I'm afraid it can't be used to delete a deleted Relationship hardly.
... View more
11-03-2022
02:25 AM
You can try this using the Rest api: # DELETE the Client
curl -k -u admin:admin -H "X-Requested-By: ambari" -X DELETE
"$HTTP_PROTOCOL://$CONSOLE_NODE:$PORT/api/v1/clusters/$CLUSTER/host_components?HostRoles/component_name.in(HDFS_CLIENT)&HostRoles/host_name.in(hostname01,hostname02)"
# INSTAL the Client
curl -v -k -i -u 'admin:admin' -H 'X-Requested-By: ambari' -X POST -d '
{
"RequestInfo":{
"query":"Hosts/host_name.in(hostname01)"
},
"Body":{
"host_components":[
{
"HostRoles":{
"component_name":"HDFS_CLIENT"
}
}
]
}
}' '$HTTP_PROTOCOL://$CONSOLE_NODE:$PORT/api/v1/clusters/$CLUSTER/hosts'
... View more
11-03-2022
02:00 AM
Hi, I'm trying to configure the spark-connector for apache atlas.When I execute this statement "create table dept_three_spark as select * from dept_four_ext_hive;" ,the table created successed ,but I get the following error: And when I visited atlas, a lot of information was missing. I tried to add the number of kafka topic ATLAS_HOOK and ATLAS_ENITIES partitions, but no use. What can I do next to solve this problem? Spark version: 2.3.0 Atlas version: 2.2.0 Kafka version: 2.0.0 HDP version: 3.1.5.0-152 Spark-connector version: 0.1.0.3.1.5.0-152
... View more
Labels:
11-01-2022
12:52 AM
1 Kudo
Hi: Is it currently possible to hard delete (purge) Relationships and RelationshipDefs via the API? I use the api /api/atlas/admin/purge/ to delete a DELETED Relationship but no use. Any help will be useful,Thanks. The Atlas version is 2.2.0
... View more
Labels:
- Labels:
-
Apache Atlas
07-23-2022
08:51 PM
Hi, Recently, in our production environment, there was an abnormal exit of the Hbase Master. In the log, I found that the ZK connection had some problems.According to the sessionid, I found that the sessionid of one of our applications was the same as the sessionid established by the master at the time in question, and then the application side closed the session, and then the master hung. Is this possible? Why does a session with the same ZooKeeper sessionid appear?
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Zookeeper
07-08-2022
11:22 PM
I know.Need FQDN likes testhost.magg.com,Thanks
... View more
07-08-2022
09:37 PM
Thanks.I open the https for nifi.And it works when Knox and Nifi on same server.But when I deploy NIFI and KNOX separately, I am redirected infinitely when SSO logs in to NIFI and I find Cookies have problems. Is this because of cross-domain? Knox gateway.log has "JWT cookie successfully added." but nifi-user.log have " [<anonymous>] GET https://XXX:9443/nifi-api/flow/current-user"
... View more
- « Previous
-
- 1
- 2
- Next »