About jagadeesan

jagadeesan · ‎01-15-2019

@Michael Bronson Decommissioning is a process that supports removing components and their hosts from the cluster. You must decommission a master or slave running on a host before removing it or its host from service. Decommissioning helps you to prevent potential loss of data or disruption of service. Below HDP documentation for Ambari-2.6.1 help you to decommission a DataNode. When DataNode decommissioning process is finished, the status display changes to Decommissioned. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-operations/content/how_to_decommission_a_component.html I hope that the above answers your questions.

jagadeesan · ‎01-14-2019

@Michael Bronson Below article help you to replacing faulty disks on datanode. https://community.hortonworks.com/articles/3131/replacing-disk-on-datanode-hosts.html Please accept the answer you found most useful.

jagadeesan · ‎01-11-2019

@harish If you made update on that particular queue value then you can ignore it. Because yarn automatically made necessary changes after you updated that particular queue.

jagadeesan · ‎01-11-2019

@harish There are some ordering policies to use different algorithms to decide which leaf queue should be scheduled the first. This ordering policy is set on the parent queue and its value is 'utilization' or 'priority-utilization'. (YARN-5864) You can refer below article for more info https://community.hortonworks.com/content/supportkb/174918/the-ordering-policy-setting-for-root-yarn-queue-ya.html Please accept the answer you found most useful.

jagadeesan · ‎01-07-2019

@Harish More Please can you confirm whether new user having appropriate permission to access all database like old user, you can check all defined policies in Ranger/Sentry depend on your cluster security access configuration.

jagadeesan · ‎01-05-2019

@Raghav Mp HDFS root scratch directory for Hive jobs, which gets created with write all (733) permission. For each connecting user, an HDFS scratch directory ${hive.exec.scratchdir}/ is created with ${hive.scratch.dir.permission}. To fix this issue, you can set below property in hive-site.xml, this is used for setting values for the entire Hive configuration <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>Scratch space for Hive jobs</description> </property> For more detail: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Please accept the answer you found most useful.

jagadeesan · ‎01-05-2019

@Akash Dixit Suppose below answer won't resolve, please share detailed error for troubleshoot.

jagadeesan · ‎01-05-2019

@Akash Dixit This is most probably a permission issue. There are two ways to solve it. You can disable permission in hdfs-site.xml <property> <name>dfs.permissions</name> <value>false</value> </property> Or either you can give write permission to user 'training' in hadoop. Please accept the answer you found most useful.

jagadeesan · ‎01-02-2019

@Suraj Singh Seems it's similar like what we discussed in below thread https://community.hortonworks.com/questions/232093/yarn-jobs-are-getting-stuck-in-accepted-state.html If resubmit jobs will get success ? As discussed earlier this is open bug which fixed in further releases. If you need to apply patch, please involve Hortonworks support. If you are a customer, HWX can release a patch for you if it's technically possible based on specifics of the JIRAs. If you don't have support, you can certainly do it but test it first apply the patch in dev/test and see if it resolves your problem.

jagadeesan · ‎01-01-2019

@Michael Bronson Please can you check the zookeeper logs (/var/log/zookeeper) of master1.sys89.com. This can happen if there are too many open connections. Check where there were any warning messages stating with “Too many connections from {IP address of master1.sys89.com}”. Using netstat command also you can verify netstat -no | grep :2181 | wc -l To fix this issue, kindly clear up all stale connections manually or try increasing the maxClientCnxns setting at /etc/zookeeper/2.6.4.0-91/0/zoo.cfg. From your zoo.cfg file I can see value is maxClientCnxns=60 which is default. You can increase it by adding the maxClientCnxns=4096 and restart respective affected services.

jagadeesan · ‎01-01-2019

@Suraj Singh Actually particular fix is released in following version 3.1.0, 2.10.0, 2.9.1, 3.0.1 . Related JIRA https://issues.apache.org/jira/browse/YARN-7873. Fix you read complete comments then you will get idea about why they revert YARN-6078 and not released on 3.0.0.

jagadeesan · ‎12-28-2018

@Slimani Ibrahim It's not recommended to format the NameNode more than once except when NameNode loses metadata information. The reason could be this property which tells NameNode where to store its metadata information on disk is dfs.namenode.name.dir in your case its points to /tmp, so every time you restart your system the /tmp directory gets flushed and hence you have to format the NameNode again. So, make sure you point the property dfs.namenode.name.dir to a more persistent location (something like /hadoop/hdfs/namenode similar for datanode property/hadoop/hdfs/datanode) which does not get's cleared every time if you restart your system that will resolve this problem. I hope that the above answers your questions. Please accept the answer you found most useful.

jagadeesan · ‎12-28-2018

@Artyom Timofeev As for containers are stuck in localizing phase, seems you are hit on this reported bug on Yarn which is resolved in 3.0.0 version. https://issues.apache.org/jira/browse/YARN-6078

jagadeesan · ‎12-27-2018

@Harish More Please can you confirm whether did you created an HBase keystore and export the certificate ? If not please follow below steps. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/enable_ssl_on_the_hbase_rest_server.html https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/enable_ssl_on_the_hbase_web_ui.html If yes, then follow link to configure ranger ssl connection for hbase. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/configuring_ambari_ranger_ssl_using_public_ca_certificates_configure_ranger_hbase_plugin_for_ssl.html Important note: In the case where multiple servers talking to ranger admin for downloading policies for the same service/repository (e.g. HBase Master and Region servers, be sure to use the repo name or a common string across all of the nodes (i.e., HbasePlugin) https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/ch04s20s02s04s01.htm Please accept the answer you found most useful

jagadeesan · ‎12-27-2018

@Michael Bronson After configuration changes, it's safe to restart required services, those restart will make necessary new changes into the system. In our case, yarn.nodemanager.local-dirs will point out to new location /grid/sdb/hadoop/yarn/local instead of old location /var/hadoop/yarn/local . In short, restart will not cause any issue either after delete old files or after change in YARN configuration. I hope this answered your concerns.

jagadeesan · ‎12-27-2018

@Michael Bronson Sure

jagadeesan · ‎12-27-2018

@Michael Bronson Yes true, similarly it's not a good idea to use /var for yarn.nodemanager.local-dirs which are container local. Typically, you can direct these to all the data mount points (like /grid/sdb/hadoop/yarn/local). Same thing for yarn logs (/grid/sdb/hadoop/yarn/log) yarn.nodemanager.log-dirs. This can help with reducing all your IO going to your OS disk (where you typically have /var). You can take a look at http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/ I hope that the above answers your questions. Please accept the answer you found most useful.

jagadeesan · ‎12-27-2018

@Michael Bronson Yes you can remove old files. Below article help you to with proper steps. https://community.hortonworks.com/articles/92339/how-to-clear-local-file-cache-and-user-cache-for-y.html Please accept the answer you found most useful.

jagadeesan · ‎12-27-2018

Problem Description: Ambari-infra-solr is running fine but using a "ps" command shows a password like below. According to security policy, this is consider as security breach.The issue occurred because the value of property infra_solr_trust_store_password and infra_solr_key_store_password showing cleartext passwords in java Options. $ ps -ef | grep -i 'ambari-infra' 1008 25938 1 21 07:25 ?00:00:11 /usr/jdk64/jdk1.8.0_112/bin/java -server -Xms1024m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/ambari-infra-solr/solr_gc.log -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18886 -Dcom.sun.management.jmxremote.rmi.port=18886 -Djava.rmi.server.hostname=test2.example.com -DzkClientTimeout=60000 -DzkHost=test1.example.com:2181,test2.example.com:2181,test3.example.com:2181/infra-solr -Djetty.port=8886 -DSTOP.PORT=7886 -DSTOP.KEY=solrrocks -Dhost=test2.example.com -Duser.timezone=UTC -Djetty.home=/usr/lib/ambari-infra-solr/server -Dsolr.solr.home=/opt/ambari_infra_solr/data -Dsolr.install.dir=/usr/lib/ambari-infra-solr -Dlog4j.configuration=file:/etc/ambari-infra-solr/conf/log4j.properties -Dsolr.jetty.keystore=/etc/security/serverKeys/infra.solr.keyStore.jks -Dsolr.jetty.keystore.password=bigdata -Dsolr.jetty.truststore=/etc/security/serverKeys/infra.solr.trustStore.jks -Dsolr.jetty.truststore.password=bigdata -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Djavax.net.ssl.keyStore=/etc/security/serverKeys/infra.solr.keyStore.jks -Djavax.net.ssl.keyStorePassword=bigdata -Djavax.net.ssl.trustStore=/etc/security/serverKeys/infra.solr.trustStore.jks -Djavax.net.ssl.trustStorePassword=bigdata -Dsolr.jetty.https.port=8886 -Dsolr.authentication.httpclient.configurer=org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=/etc/ambari-infra-solr/conf/infra_solr_jaas.conf -Dsolr.kerberos.principal=HTTP/test2.example.com@EXAMPLE.COM -Dsolr.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Dsolr.kerberos.cookie.domain=test2.example.com -Dsolr.kerberos.name.rules=DEFAULT -XX:OnOutOfMemoryError=/usr/lib/ambari-infra-solr/bin/oom_solr.sh 8886 /var/log/ambari-infra-solr -jar start.jar --module=https Article: This article help to set hash password instead of showing clearest passwords in java options. Using Ambari inbuilt jetty jar file, we can hash password either OBF or MD5 format and pass those value in infra-solo-env to hide password from ambari-infra solr process. Step-1: Generate encrypt password using jetty jar file, where <password> is the password you used for the keystore/truststore java -cp /usr/lib/ambari-infra-solr/server/lib/jetty-util-9.2.13.v20150730.jar org.eclipse.jetty.util.security.Password <password> java -cp /usr/lib/ambari-infra-solr/server/lib/jetty-util-9.2.13.v20150730.jar org.eclipse.jetty.util.security.Password bigdata 2018-12-27 07:51:13.605:INFO::main: Logging initialized @171ms bigdata OBF:1rpc1wtw1sp11sov1sop1wui1rpa MD5:27819cfe72583a34d13a40bb74154c91 Step-2: Update below properties from Ambari under Ambari Infra Config Tab in Advanced infra-solr-env section (You can mention hashed_password of either OBF or MD5 there) Before: SOLR_SSL_KEY_STORE_PASSWORD={{infra_solr_keystore_hashed_password}} SOLR_SSL_TRUST_STORE_PASSWORD={{infra_solr_truststore_hashed_password}} Now: SOLR_SSL_KEY_STORE_PASSWORD=OBF:1rpc1wtw1sp11sov1sop1wui1rpa SOLR_SSL_TRUST_STORE_PASSWORD=OBF:1rpc1wtw1sp11sov1sop1wui1rpa Step-3: Need to restart required services through ambari and verify with grep process of ambari-infra solr process. $ ps -ef | grep -i 'ambari-infra'1008 17641 17 08:03 ?00:00:10 /usr/jdk64/jdk1.8.0_112/bin/java -server -Xms1024m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/ambari-infra-solr/solr_gc.log -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18886 -Dcom.sun.management.jmxremote.rmi.port=18886 -Djava.rmi.server.hostname=test2.example.com -DzkClientTimeout=60000 -DzkHost=test1.example.com:2181,test2.example.com:2181,test3.example.com:2181/infra-solr -Djetty.port=8886 -DSTOP.PORT=7886 -DSTOP.KEY=solrrocks -Dhost=test2.example.com -Duser.timezone=UTC -Djetty.home=/usr/lib/ambari-infra-solr/server -Dsolr.solr.home=/opt/ambari_infra_solr/data -Dsolr.install.dir=/usr/lib/ambari-infra-solr -Dlog4j.configuration=file:/etc/ambari-infra-solr/conf/log4j.properties -Dsolr.jetty.keystore=/etc/security/serverKeys/infra.solr.keyStore.jks -Dsolr.jetty.keystore.password=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.truststore=/etc/security/serverKeys/infra.solr.trustStore.jks -Dsolr.jetty.truststore.password=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Djavax.net.ssl.keyStore=/etc/security/serverKeys/infra.solr.keyStore.jks -Djavax.net.ssl.keyStorePassword=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Djavax.net.ssl.trustStore=/etc/security/serverKeys/infra.solr.trustStore.jks -Djavax.net.ssl.trustStorePassword=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.https.port=8886 -Dsolr.authentication.httpclient.configurer=org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=/etc/ambari-infra-solr/conf/infra_solr_jaas.conf -Dsolr.kerberos.principal=HTTP/test2.example.com@EXAMPLE.COM -Dsolr.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Dsolr.kerberos.cookie.domain=test2.example.com -Dsolr.kerberos.name.rules=DEFAULT -XX:OnOutOfMemoryError=/usr/lib/ambari-infra-solr/bin/oom_solr.sh 8886 /var/log/ambari-infra-solr -jar start.jar --module=https Ambari will automatically decrypt password with inbuilt jetty jar. For more details of jetty you can refer following link, https://wiki.eclipse.org/Jetty/Howto/Secure_Passwords

jagadeesan · ‎12-27-2018

@Dukool SHarma The MapReduce sort the intermediate data(between mapper and reducer phase) by key by default. If we want the data should be sort based on value, then we need secondary sorting. For more Information you can reference below links: https://www.oreilly.com/library/view/data-algorithms/9781491906170/ch01.html https://www.quora.com/What-is-secondary-sort-in-Hadoop-and-how-does-it-work/answer/Sudarshan-Sreenivasan-1 Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration. yarn.nodemanager.resource.percentage-physical-cpu-limit 100 Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.

jagadeesan · ‎12-26-2018

For example your physical memory size is 10GB you need to allocated for every container request at the RM, is around 9GB (90% of your physical memory) then you can mention in below parameter yarn.scheduler.maximum-allocation-mb 9216 Similarly amount of physical memory allocated for containers in node manager, that can be mention using below parameter. yarn.nodemanager.resource.memory-mb 9216 These all settings are manage by single configuration file I.e., yarn-site.xml which you can modify via ambari under Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site I hope that the above answers your questions. Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

@Anjali Shevadkar Below article help you to configure spark fine grain security. https://community.hortonworks.com/content/kbentry/72454/apache-spark-fine-grain-security-with-llap-test-dr.html Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

If you set 100% ram only for yarn then other services will get crash / GC issue. It's not recommendable to set all physical memory to one component.

jagadeesan · ‎12-26-2018

For changing memory value through ambari configuration manager, login into ambari UI, Yarn —> Configs —> Settings —> memory —> Memory allocated for all YARN containers on a node and Minimum / Maximum Container Size memory. and for advanced turning of yarn modify Yarn —> Configs —> Settings —>Advanced —> Advanced yarn-site

jagadeesan · ‎12-26-2018

@Jack If the table has TBLPROPERTIES("auto.purge"="true") the previous data of the table is not moved to Trash when INSERT OVERWRITE query is run against the table. This functionality is applicable only for managed tables and is turned off when "auto.purge" property is unset or set to false. Related JIRAs HIVE-15880, HIVE-9118 Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

@Jack If the table has TBLPROPERTIES("auto.purge"="true") the previous data of the table is not moved to Trash when INSERT OVERWRITE query is run against the table. This functionality is applicable only for managed tables and is turned off when "auto.purge" property is unset or set to false. Related JIRAs HIVE-15880, HIVE-9118 Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

@Jack From your above output I can see it's MANAGED_TABLE. If the table has TBLPROPERTIES ("auto.purge"="true") the previous data of the table is not moved to Trash when INSERT OVERWRITE query is run against the table. This functionality is applicable only for managed tables and is turned off when "auto.purge" property is unset or set to false. For more detail HIVE-15880

jagadeesan · ‎12-26-2018

@Jack Please can you share output of below command. It is hive 3.x or hive 2.x ? DESCRIBE FORMATTED <table_name>

jagadeesan · ‎12-26-2018

@Ilia K You can manually configure your memory allocation settings based on the hardware specifications. Please refer below links for more tuning up RAM in yarn configuration. https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_command-line-installation/content/determine-hdp-memory-config.htm

Online	Offline
Last Visited	‎08-19-2019 11:53 PM

Date Registered	‎11-12-2018 10:00 AM
Last Visited	‎08-19-2019 11:53 PM
Total Messages Posted	63
Total Kudos Received	66

Re: yarn job stuck in accepted state randomly with...

Re: yarn local dirs + safety moving the local dir ...

Re: YARN LOCAL DIRS + deletion

Re: yarn + usercache + size of local folder increa...

Re: datanode machine + how many datanode we can ad...

Re: what are the steps in order to replace faulty ...

Re: what are the steps in order to replace faulty ...

Re: Yarn queue policy

Re: Yarn queue policy

Re: New Hue user is not able to see database

Re: /tmp/hive permission keep updating as per user...

Re: Hi All, I am trying to execute wordcount progr...

Re: Hi All, I am trying to execute wordcount progr...

Re: yarn job stuck in accepted state randomly with...

Re: zookeeper-client cannot connect to zookeeper ...

Re: Yarn jobs are getting stuck in ACCEPTED state

Re: How can i start Namenode service of Hadoop wit...

Re: Yarn jobs are getting stuck in ACCEPTED state

Re: Ranger ssl connection error for hbase

Re: yarn local dirs + safety moving the local dir ...

Re: YARN LOCAL DIRS + deletion

Re: yarn local dirs + safety moving the local dir ...

Re: YARN LOCAL DIRS + deletion

Hide Password from ambari-infra-solr process

Re: In Mapreduce how to sort intermediate output b...

Re: yarn configure to use all ram

Re: yarn configure to use all ram

Re: How to Apply Ranger policies on Spark

Re: yarn configure to use all ram

Re: yarn configure to use all ram

Re: hive3.0 Incremental directory

Re: My table is a normal table. When insert overwr...

Re: After hive overwrite, the data of HDFS is not ...

Re: After hive overwrite, the data of HDFS is not ...

Re: yarn configure to use all ram