About laule75

laule75 · ‎02-11-2019

Hello, To fix this issue, I used to stop/start the Flume process on a daily basis. Recently we have migrated from Flume to NiFi; much more stable. Rgds Laurent

laule75 · ‎08-01-2018

Hello, I am trying to retrieve a specific field from my LDAP directory (cn), but it doesn't seem to be taken into account by the Ranger usersync process when I specify the following value "cn" to the variable "Username Attribute" Though this value seems allowed (possible values : uid or cn) it doesn't work on my HDP 2.6.2.0; I still see the field "uid" feeding the Ranger database. Is it a bug or did I miss something ? Thanks in advance for your insights. LC

laule75 · ‎08-02-2017

Hello @rbiswas, Sorry for the delay in getting back to you (I was on holidays). Thanks for your answers. Yes we can close the thread. regards Laurent

laule75 · ‎07-20-2017

Hello @rbiswas, Sorry, I'm a bit confused by your last statement. Could you please confirm that if I define a replication factor of 4, and 4 racks, I will get the following distribution of replicas ? (see diagram below) regards Laurent

laule75 · ‎07-17-2017

thanks @rbiswas for your answer. My concern is regarding the speed of the replication if, let's say one rack is unavailable during 24 / 48hours for maintenance reasons, and in the meantime HDFS is trying to replicate all then data on the remaining rack, thus might saturate the disk space on this rack ! I can't find any documentation mentionning this " HDFS rebalance speed" . Also it looks to me that, if the number of replica factor is equal to the number of racks, there is no guarantee that there will be a replica spread in each rack. Do you confirm it ? Thanks in advance. rgds Laurent

laule75 · ‎07-13-2017

Hello, I am in the process of improving the resilience of our hadoop clusters. We are using a twin-datacenter architecture; the hadoop cluster nodes are located in two different buildings separated by 10 km with Namenode HA activated. We are using a replica factor of 4 + 2 rack awareness (on rack per site). The replica factor of 4 is probably a bit "luxury", but it might protect against the lost of an entire rack (lost of a site) + the lost of some nodes on the remaining site. In case of losing en entire rack, I am wondering if HDFS will try to replicate the data on the remaining rack, thus we will get 4 replica on the same rack and overconsume space on the remaining rack ?...or will it "disable" the replica that is supposed to be located on the failed rack ? Does it make sense to create 4 racks (one for each replica) in order to ensure that the data will be replicated on the both sites in a balanced way (2x2) ? Many thanks in advance for your feedback. Regards Laurent

laule75 · ‎06-16-2017

Thanks @Vipin Rathor for your elaborated answer. I understand the advantages of using Kerberos, however what I found tedious is that I need to recreate user accounts (Principals) within the Kerberos Database as well as managing new passwords policies. as I said inmy previous message, ideally I would like to configure the cluster to authenticate users against LDAP and retrieve automatically a Kerberos ticket, but I don't know if it's feasible. Regards Laurent

laule75 · ‎06-14-2017

Hello @Vipin Rathor, I don't find any clear documentation about the entire setup of a kerberized cluster synchronized with LDAP (not AD) in order to retrieve kerberos token, and authorize the access via Ranger and Knox. The first stage so far is to secure WebHDFS. I don't want to generate keytabs for our hundred of users; I would like to get authenticated by LDAP, and then retrieve a Kerberos token automatically.. Any clue how to do this ? regards Lau

laule75 · ‎06-13-2017

Hi, I'm starting to get really lost in setting a secured cluster; I found it really complicated to properly configure Ranger and Knox on a kerberized cluster. Is there someone who would be keen in helping me ? I would be very grateful. Thanks in advance. Rgds Laurent

laule75 · ‎05-04-2017

Hello, I'm wondering weither it's possible to simplify the architecture by allowing Knox to retrieve a generic Kerberos token as soon as we get authenticated on the Knox gateway with our LDAP user account ? any clue ? thanks in advance. rgds Laurent

laule75 · ‎05-02-2017

thanks @Eyad Garelnabi for your answer. The thing is that I'm using LDAP, and not AD. rgds Laurent

laule75 · ‎05-02-2017

Hello @ Vipin Rathor, Please find attached the architecture image : simplified-security-archi.png Regards Laurent

laule75 · ‎04-28-2017

Hello, I'm in a process of securing our 20 hadoop clusters used by hundred of users. Though I have understood (and tested on a Sandbox) the needs of deploying the components Kerberos-Ranger-knox to get a proper securisation, this solution looks really complex when it's about to manage multiple clusters and hundred of users that are referenced in a LDAP central repository. Indeed, as far as I have understood, I need to create as many principals as users, generate keytabs files per user and per Hadoop services, and populate thoses files onto the Hadoop clusters. This solution seems very tedious to setup and maintain for the Ops people. The global architecture could be as follow : However, ideally, I would like an user to get authenticated with LDAP (as it is currently the case, through SSH (or Knox)), and obtains automatically a Kerberos token and the proper autorisations (provided by Ranger) to use the Hadoop services and access to its HDFS directories. From the documentation, I don't see such kind of "simple" way of working when dealing with LDAP !?? Did I miss something ? Is anyone has already setup such kind of infrastructure on Enterprise-grade production clusters ? I would be glad to get some feedback or any ideas. Thanks in advance. Rgds Laurent

laule75 · ‎04-12-2017

Hello, No solution to this issue, except re-installing the cluster! Before doing so, we have renamed the HDFS directories in order to preserve the data. rgds Laurent

laule75 · ‎02-20-2017

Thanks Predrag ! The authentication works , .....but I endup with another java error. Any clue ? LDAP authentication successful! java.lang.NullPointerException at javax.naming.InitialContext.getURLScheme(InitialContext.java:294) at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:343) at javax.naming.directory.InitialDirContext.getURLOrDefaultInitDirCtx(InitialDirContext.java:106) at javax.naming.directory.InitialDirContext.search(InitialDirContext.java:267) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.rolesFor(KnoxLdapRealm.java:254) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getRoles(KnoxLdapRealm.java:237) at org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.queryForAuthorizationInfo(KnoxLdapRealm.java:223) at org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthorizationInfo(JndiLdapRealm.java:313) at org.apache.shiro.realm.AuthorizingRealm.getAuthorizationInfo(AuthorizingRealm.java:341) at org.apache.shiro.realm.AuthorizingRealm.hasRole(AuthorizingRealm.java:571) at org.apache.shiro.authz.ModularRealmAuthorizer.hasRole(ModularRealmAuthorizer.java:374) at org.apache.shiro.mgt.AuthorizingSecurityManager.hasRole(AuthorizingSecurityManager.java:153) at org.apache.shiro.subject.support.DelegatingSubject.hasRole(DelegatingSubject.java:224) at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.getGroups(KnoxCLI.java:1434) at org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execute(KnoxCLI.java:1404) at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:138) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1675) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70) at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39) at org.apache.hadoop.gateway.launcher.C at org.apache.hadoop.gateway.launcher.L at org.apache.hadoop.gateway.launcher.L

laule75 · ‎02-16-2017

Hello, I'm currently setting up Knox (and Ranger) on my cluster, and I'm experiencing issues having Knox to connect to my OpenLDAP servers in order to authenticate users. (For information, I did already setup Ambari and Ranger (+usersync) to sync with my OpenLDAP server and It works fine.) While testing the following command : # /usr/hdp/2.5.0.0-1245/knox/bin/knoxcli.sh user-auth-test --cluster default --u <mylogin> --p <my_password> --g --d I get the following output : org.apache.shiro.authc.AuthenticationException: LDAP authentication failed. [LDAP: error code 49 - Invalid Credentials] This message makes sense as I don't know how to specify a password to connect to the LDAP server within my Knox configuration (topology). Is there a specific param entries that allow to indicate this password ? Thanks in advance. Kind regards Laurent

laule75 · ‎12-12-2016

Hello, We have finally re-installed the cluster (HDP 2.4 + Ambari 2.4.1), and the hosts are Ok now. Rgds Laurent

laule75 · ‎10-26-2016

@Artem Ervits The problem is not only visible from my PC, but also from all the Ambari users; it's not related to the browser. When accessing the "Host tab", the Web console log output mentions this error : { "status" : 400, "message" : "The properties [Hosts/rack_iâ€¦nts/logging] specified in the request or predicate are not supported for the resource type Host." } ...while trying to issue the following request : http://<ambari_server>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts/rack_i%E2%80%A6nts/logging&page_size=25&from=0&sortBy=Hosts/host_name.asc&_=1477464892288%20500>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts/rack_i%E2%80%A6nts/logging&page_size=25&from=0&sortBy=Hosts/host_name.asc&_=1477464892288%20500 Is it related to the "rack-aware" configuration of our cluster ?....because if I issue the request : http://:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts>:8080/api/v1/clusters/OCP_prod/hosts?fields=Hosts ... it works fine. So the hosts are correctly seen in the database. Rgds Laurent

laule75 · ‎10-25-2016

Venkat, The Ambari agents have also been upgraded. As I said earlier, The hosts table seems Ok, Ambari server is able to manage the hosts (for instance, adding new services) , it's just this Web page which doesn't list the hosts !!?

laule75 · ‎10-25-2016

Hello @Artem Ervits The hosts are presents in the Host table, and we can definitely see them via a curl command. It looks like the issue resides on the generation of the Web page. No, we didn't execute the reset command. rgds Laurent

laule75 · ‎10-21-2016

Hello, Actually I've fixed the issue; I have corrected the Source agent in order to re-establish the connection in case of connection cut triggered by the provider of data.

laule75 · ‎10-19-2016

Hello, After an upgrade to the version HDP 2.4 with Ambari 2.4.1, we are unable to see the list of hosts managed by the cluster on the Hosts tab. is there any way to enable/register again hosts in that view ? Thanks in advance. Kind regards LC

laule75 · ‎10-14-2016

Hello, I have installed and configured a websocket-type source on Flume. It works fine...except that the data is retrieved just once (only when we start the Flume agent), and not continuously ! Is there any particular parameter to setup in order to get Flume continuously retrieving some data feed ? thanks in advance. LC

laule75 · ‎06-29-2016

Hello, I'm also interested in implementing multi-tenancy on Hbase, for instance being able to isolate table among differents users, and to be able to do some accounting (identify clearly the space used / cpu consummed, etc...) Any idea ? thanks in advance. LC

laule75 · ‎03-09-2016

Many thanks Raul ! It works perfectly.

laule75 · ‎12-23-2015

Hello Neeraj, Actually , there is no such network equipment; it's direct access.It's just that the URL address form is not accepted by the jconcole/jstack tools.regards.

laule75 · ‎12-15-2015

Hello, I try to connect to the JMX console of the ResourceManager via some tools like Jconsole, but it fails establish a proper connection : Exception in thread "main" java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.onnectIOException: non-JRMP server at remote endpoint] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.archive.jmx.Client.execute(Client.java:225) at org.archive.jmx.Client.main(Client.java:154) ....though the JMX data are visible via the URL : http://<ResourceManager>:8088/jmx Any idea on how to enable a correct JMX port for the Yarn ResourceManager ? Thanks in advance. Kind regards LC

Online	Offline
Last Visited	‎03-12-2019 03:47 PM

Date Registered	‎12-12-2015 02:26 PM
Last Visited	‎03-12-2019 03:47 PM
Total Messages Posted	27
Total Kudos Received	7

Re: Hosts not listed anymore after Ambari upgrade ...

Re: Flume - Polling of data continuously

Re: Flume - Polling of data continuously

Ranger USersync - Username Attribute - cn

Re: HDFS resiliency - DR - rack aware

Re: HDFS resiliency - DR - rack aware

Re: HDFS resiliency - DR - rack aware

HDFS resiliency - DR - rack aware

Re: Assistance in setting up Kerberos - Ranger - K...

Re: Assistance in setting up Kerberos - Ranger - K...

Assistance in setting up Kerberos - Ranger - Knox

Re: Secured architecture design

Re: Secured architecture design

Re: Secured architecture design

Secured architecture design

Re: Hosts not listed anymore after Ambari upgrade ...

Re: knox : LDAP setup : How to specify a password ...

knox : LDAP setup : How to specify a password toco...

Re: Hosts not listed anymore after Ambari upgrade ...

Re: Hosts not listed anymore after Ambari upgrade ...

Re: Hosts not listed anymore after Ambari upgrade ...

Re: Hosts not listed anymore after Ambari upgrade ...

Re: Flume - Polling of data continuously

Hosts not listed anymore after Ambari upgrade to 2...

Flume - Polling of data continuously

Re: HBase multi tenancy Best practice

Re: YARN JMX access

Re: YARN JMX access

YARN JMX access