Member since
12-12-2015
27
Posts
7
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1230 | 12-12-2016 05:53 PM | |
827 | 10-21-2016 08:53 AM |
02-11-2019
08:51 AM
Hello, To fix this issue, I used to stop/start the Flume process on a daily basis. Recently we have migrated from Flume to NiFi; much more stable. Rgds Laurent
... View more
08-01-2018
02:07 PM
Hello, I am trying to retrieve a specific field from my LDAP directory (cn), but it doesn't seem to be taken into account by the Ranger usersync process when I specify the following value "cn" to the variable "Username Attribute" Though this value seems allowed (possible values : uid or cn) it doesn't work on my HDP 2.6.2.0; I still see the field "uid" feeding the Ranger database. Is it a bug or did I miss something ? Thanks in advance for your insights. LC
... View more
Labels:
- Labels:
-
Apache Ranger
08-02-2017
02:39 PM
Hello @rbiswas, Sorry for the delay in getting back to you (I was on holidays). Thanks for your answers. Yes we can close the thread. regards Laurent
... View more
07-20-2017
08:37 AM
Hello @rbiswas, Sorry, I'm a bit confused by your last statement. Could you please confirm that if I define a replication factor of 4, and 4 racks, I will get the following distribution of replicas ? (see diagram below) regards Laurent
... View more
07-17-2017
08:19 AM
thanks @rbiswas for your answer. My concern is regarding the speed of the replication if, let's say one rack is unavailable during 24 / 48hours for maintenance reasons, and in the meantime HDFS is trying to replicate all then data on the remaining rack, thus might saturate the disk space on this rack ! I can't find any documentation mentionning this " HDFS rebalance speed" . Also it looks to me that, if the number of replica factor is equal to the number of racks, there is no guarantee that there will be a replica spread in each rack. Do you confirm it ? Thanks in advance. rgds Laurent
... View more
07-13-2017
09:06 AM
Hello, I am in the process of improving the resilience of our hadoop clusters. We are using a twin-datacenter architecture; the hadoop cluster nodes are located in two different buildings separated by 10 km with Namenode HA activated. We are using a replica factor of 4 + 2 rack awareness (on rack per site). The replica factor of 4 is probably a bit "luxury", but it might protect against the lost of an entire rack (lost of a site) + the lost of some nodes on the remaining site. In case of losing en entire rack, I am wondering if HDFS will try to replicate the data on the remaining rack, thus we will get 4 replica on the same rack and overconsume space on the remaining rack ?...or will it "disable" the replica that is supposed to be located on the failed rack ? Does it make sense to create 4 racks (one for each replica) in order to ensure that the data will be replicated on the both sites in a balanced way (2x2) ? Many thanks in advance for your feedback. Regards Laurent
... View more
Labels:
- Labels:
-
Apache Hadoop
06-16-2017
12:06 PM
Thanks @Vipin Rathor for your elaborated answer. I understand the advantages of using Kerberos, however what I found tedious is that I need to recreate user accounts (Principals) within the Kerberos Database as well as managing new passwords policies. as I said inmy previous message, ideally I would like to configure the cluster to authenticate users against LDAP and retrieve automatically a Kerberos ticket, but I don't know if it's feasible. Regards Laurent
... View more
06-14-2017
11:41 AM
Hello @Vipin Rathor, I don't find any clear documentation about the entire setup of a kerberized cluster synchronized with LDAP (not AD) in order to retrieve kerberos token, and authorize the access via Ranger and Knox. The first stage so far is to secure WebHDFS. I don't want to generate keytabs for our hundred of users; I would like to get authenticated by LDAP, and then retrieve a Kerberos token automatically.. Any clue how to do this ? regards Lau
... View more
06-13-2017
05:18 PM
Hi, I'm starting to get really lost in setting a secured cluster; I found it really complicated to properly configure Ranger and Knox on a kerberized cluster. Is there someone who would be keen in helping me ? I would be very grateful. Thanks in advance. Rgds Laurent
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache Ranger
05-04-2017
01:04 PM
Hello, I'm wondering weither it's possible to simplify the architecture by allowing Knox to retrieve a generic Kerberos token as soon as we get authenticated on the Knox gateway with our LDAP user account ? any clue ? thanks in advance. rgds Laurent
... View more
05-02-2017
01:53 PM
thanks @Eyad Garelnabi for your answer. The thing is that I'm using LDAP, and not AD. rgds
Laurent
... View more
05-02-2017
08:03 AM
Hello @ Vipin Rathor, Please find attached the architecture image : simplified-security-archi.png Regards Laurent
... View more
04-28-2017
12:46 PM
Hello, I'm in a process of securing our 20 hadoop clusters used by hundred of users. Though I have understood (and tested on a Sandbox) the needs of deploying the components Kerberos-Ranger-knox to get a proper securisation, this solution looks really complex when it's about to manage multiple clusters and hundred of users that are referenced in a LDAP central repository. Indeed, as far as I have understood, I need to create as many principals as users, generate keytabs files per user and per Hadoop services, and populate thoses files onto the Hadoop clusters. This solution seems very tedious to setup and maintain for the Ops people.
The global architecture could be as follow :
However, ideally, I would like an user to get authenticated with LDAP (as it is currently the case, through SSH (or Knox)), and obtains automatically a Kerberos token and the proper autorisations (provided by Ranger) to use the Hadoop services and access to its HDFS directories. From the documentation, I don't see such kind of "simple" way of working when dealing with LDAP !?? Did I miss something ?
Is anyone has already setup such kind of infrastructure on Enterprise-grade production clusters ?
I would be glad to get some feedback or any ideas. Thanks in advance. Rgds Laurent
... View more
Labels:
- Labels:
-
Apache Ranger
04-12-2017
07:59 AM
Hello, No solution to this issue, except re-installing the cluster! Before doing so, we have renamed the HDFS directories in order to preserve the data. rgds Laurent
... View more
02-20-2017
03:23 PM
Thanks Predrag ! The authentication works , .....but I endup with another java error. Any clue ? LDAP authentication successful!
java.lang.NullPointerException
at javax.naming.InitialContext.getURLScheme(InitialContext.java:294)
at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:343)
at javax.naming.directory.InitialDirContext.getURLOrDefaultInitDirCtx(InitialDirContext.java:106)
at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:267)
at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.rolesFor(KnoxLdapRealm.java:254)
at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getRoles(KnoxLdapRealm.java:237)
at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.queryForAuthorizationInfo(KnoxLdapRealm.java:223)
at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:313)
at org.apache.shiro.realm.AuthorizingRealm.getAuthorizationInfo(AuthorizingRealm.java:341)
at org.apache.shiro.realm.AuthorizingRealm.hasRole(AuthorizingRealm.java:571)
at org.apache.shiro.authz.ModularRealmAuthorizer.hasRole(ModularRealmAuthorizer.java:374)
at org.apache.shiro.mgt.AuthorizingSecurityManager.hasRole(AuthorizingSecurityManager.java:153)
at org.apache.shiro.subject.support.DelegatingSubject.hasRole(DelegatingSubject.java:224)
at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.getGroups(KnoxCLI.java:1434)
at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execute(KnoxCLI.java:1404)
at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1675)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
at org.apache.hadoop.gateway.launcher.C
at org.apache.hadoop.gateway.launcher.L
at org.apache.hadoop.gateway.launcher.L
... View more
02-16-2017
01:27 PM
Hello, I'm currently setting up Knox (and Ranger) on my cluster, and I'm experiencing issues having Knox to connect to my OpenLDAP servers in order to authenticate users. (For information, I did already setup Ambari and Ranger (+usersync) to sync with my OpenLDAP server and It works fine.) While testing the following command : # /usr/hdp/2.5.0.0-1245/knox/bin/knoxcli.sh user-auth-test --cluster default --u <mylogin> --p <my_password> --g --d I get the following output : org.apache.shiro.authc.AuthenticationException: LDAP authentication failed.
[LDAP: error code 49 - Invalid Credentials] This message makes sense as I don't know how to specify a password to connect to the LDAP server within my Knox configuration (topology). Is there a specific param entries that allow to indicate this password ? Thanks in advance. Kind regards Laurent
... View more
Labels:
- Labels:
-
Apache Knox
12-12-2016
05:53 PM
Hello, We have finally re-installed the cluster (HDP 2.4 + Ambari 2.4.1), and the hosts are Ok now. Rgds Laurent
... View more
10-26-2016
07:38 AM
1 Kudo
@Artem Ervits The problem is not only visible from my PC, but also from all the Ambari users; it's not related to the browser. When accessing the "Host tab", the Web console log output mentions this error : { "status" : 400,
"message" : "The properties [Hosts/rack_i…nts/logging] specified in the request or predicate are not supported for the resource type Host."
}
...while trying to issue the following request : http://<ambari_server>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts/rack_i%E2%80%A6nts/logging&page_size=25&from=0&sortBy=Hosts/host_name.asc&_=1477464892288%20500>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts/rack_i%E2%80%A6nts/logging&page_size=25&from=0&sortBy=Hosts/host_name.asc&_=1477464892288%20500 Is it related to the "rack-aware" configuration of our cluster ?....because if I issue the request : http://:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts ... it works fine. So the hosts are correctly seen in the database. Rgds Laurent
... View more
10-25-2016
10:36 AM
Venkat, The Ambari agents have also been upgraded. As I said earlier, The hosts table seems Ok, Ambari server is able to manage the hosts (for instance, adding new services) , it's just this Web page which doesn't list the hosts !!?
... View more
10-25-2016
09:12 AM
Hello @Artem Ervits The hosts are presents in the Host table, and we can definitely see them via a curl command. It looks like the issue resides on the generation of the Web page. No, we didn't execute the reset command. rgds Laurent
... View more
10-21-2016
08:53 AM
Hello, Actually I've fixed the issue; I have corrected the Source agent in order to re-establish the connection in case of connection cut triggered by the provider of data.
... View more
10-19-2016
02:10 PM
2 Kudos
Hello, After an upgrade to the version HDP 2.4 with Ambari 2.4.1, we are unable to see the list of hosts managed by the cluster on the Hosts tab. is there any way to enable/register again hosts in that view ? Thanks in advance. Kind regards LC
... View more
Labels:
- Labels:
-
Apache Ambari
10-14-2016
11:08 AM
Hello, I have installed and configured a websocket-type source on Flume. It works fine...except that the data is retrieved just once (only when we start the Flume agent), and not continuously ! Is there any particular parameter to setup in order to get Flume continuously retrieving some data feed ? thanks in advance. LC
... View more
Labels:
- Labels:
-
Apache Flume
12-23-2015
03:15 PM
Hello Neeraj, Actually , there is no such network equipment; it's direct access.It's just that the URL address form is not accepted by the jconcole/jstack tools.regards.
... View more
12-15-2015
08:22 AM
3 Kudos
Hello, I try to connect to the JMX console of the ResourceManager via some tools like Jconsole, but it fails establish a proper connection : Exception in thread "main" java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.onnectIOException: non-JRMP server at remote endpoint] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.archive.jmx.Client.execute(Client.java:225) at org.archive.jmx.Client.main(Client.java:154) ....though the JMX data are visible via the URL : http://<ResourceManager>:8088/jmx Any idea on how to enable a correct JMX port for the Yarn ResourceManager ? Thanks in advance. Kind regards LC
... View more
Labels:
- Labels:
-
Apache YARN