Member since
09-24-2015
178
Posts
113
Kudos Received
28
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3434 | 05-25-2016 02:39 AM | |
3664 | 05-03-2016 01:27 PM | |
857 | 04-26-2016 07:59 PM | |
14590 | 03-24-2016 04:10 PM | |
2112 | 02-02-2016 11:50 PM |
10-23-2015
06:16 PM
1 Kudo
Ulimit does not depend on the size of the cluster but more on the individual node, workloads and user concurrency. I set these values as best practice, which BTW is a large enough value and is probably never reached. * - nofile 32768
* - nproc 65536 Ambari actually changes the limit of the core files created as part of the commands executed like in this case -
... View more
10-23-2015
03:07 PM
What precautions & extra configurations, if any, are needed when adding worker nodes with different capacity to a cluster? My understanding is that YARN will be able to just manage the nodes without anything special. For e.g - Any issues with adding 3 nodes with following config to an existing POC cluster that has similar nodes 8 cores, 32 Gigs, 3 TB DAS for data- node1 - 8 cores, 64GB RAM, NO STORAGE
node2 - 8 cores, 64GB RAM, 2 TB
node3 - 8 cores, 64GB RAM, 2TB Also, how do you configure YARN to utilize different amount of memory on these heterogeneous boxes?
... View more
Labels:
- Labels:
-
Apache YARN
10-22-2015
09:16 PM
Vel, thanks for the answer. Just because Neeraj has the screen snapshots (which will make it clearer for the next guys), I will select his answer as correct but I know you are the source for his answer too 🙂
... View more
10-22-2015
08:24 PM
3 Kudos
Changing Ranger UI password causes Namenode to stop because it seems that the password needs to be updated in HDFS plugin within HDFS config. However, I am not sure which config needs to be updated in HDFS plugin. Please advise.
... View more
Labels:
- Labels:
-
Apache Ranger
10-22-2015
08:20 PM
2 Kudos
In HA environment, we can access HDFS by referring the active namenode directory but I am interested in finding if there is a way to access HDFS using nameservice id such that if and when the HDFS fails over to the passive namenode, then the client can just continue to use HDFS without manually changing the configuration.
... View more
Labels:
- Labels:
-
Apache Hadoop
10-22-2015
07:33 PM
When a certain amount of memory is given to ResourceManager (Memory allocated for all YARN containers on a node), is it immediately blocked or gradually/progressively used on as-needed basis until that capacity is reached?
... View more
Labels:
- Labels:
-
Apache YARN
10-22-2015
07:18 PM
Knox provides solution for perimeter security and like any security component, (encryption or authorization) does add overhead to the processing time. With that said, in order to determine whether a security tool is needed or not, performance is not considered as a deciding factor. We can load balance the traffic across multiple knox instances to distributed the load to avoid too much degradation in performance.
... View more
10-21-2015
04:17 PM
Does that mean Ranger itself cannot be HA but it can work with HA enabled services? And if we the repo has to be changed manually, what is it that Ranger is doing for hA?
... View more
10-21-2015
02:27 PM
1 Kudo
+1 on Nifi. If customer does not want to go that route (which they should because it is probably the most elegant solution) the other option would be to export the data from oracle using export utility to the local file system. Configure Flume agent to listen to the export directory and use Kafka Sink to place the data on Kafka. Depending on the downstream processing, Flume can chunk the data into appropriate sizes (1 row, 10 rows, n rows)
... View more
10-21-2015
02:18 PM
4 Kudos
Hi Wes - It is possible. I used the following knox config in past to do it. For 2 - You would need a different knox topology. <provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<!--
session timeout in minutes, this is really idle timeout,
defaults to 30mins, if the property value is not defined,,
current client authentication would expire if client idles contiuosly for more than this value
-->
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<!-- ADJUST host, port for your AD setup-->
<value>ldap://ad.client.com:389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<!-- Param below is ignored-->
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>cn={0},ou=hadoop,ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
</param>
<!-- Param above is ignored-->
<param>
<name>main.ldapRealm.userSearchAttributeName</name>
<value>sAMAccountName</value>
</param>
<param>
<name>main.ldapRealm.userObjectClass</name>
<value>person</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.systemUsername</name>
<value>cn=hadoop_knox_id,ou=process,ou=accounts,dc=ad,dc=client,dc=com</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.systemPassword</name>
<value>passwd_4_hadoop_knox_id</value>
</param>
<!-- search base used to search for user bind DN and groups -->
<param>
<name>main.ldapRealm.searchBase</name>
<value>ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
</param>
<!-- search base used to search for user bind DN.
Defaults to the value of main.ldapRealm.searchBase.
If main.ldapRealm.userSearchAttributeName is defined,
vlaue for main.ldapRealm.searchBase or main.ldapRealm.userSearchBase
should be defined -->
<param>
<name>main.ldapRealm.userSearchBase</name>
<value>ou=personal,ou=accounts,dc=ad,dc=client,dc=com</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
... View more