About jagadeesan

jagadeesan · ‎04-03-2020

Hi @Gubbi You can try the below script. source_dir="/user/mft/inbound/" dest_dir="/users/aws/outbound/" for file in "$source_dir"*; do dest=$(echo $file | grep -o -P '(?<=_).*(?=.csv)'); mkdir -p "$dest_dir$dest"; mv $file "$dest_dir$dest"; done Hope this helps. Please accept the answer and vote up if it did.

jagadeesan · ‎04-03-2020

Hi @Mondi The important differences between parcels and packages are: Parcels are self-contained and installed in a versioned directory, which means that multiple versions of a given parcel can be installed side-by-side. You can then designate one of these installed versions as the active one. With packages, only one package can be installed at a time so there is no distinction between what is installed and what is active. You can install parcels at any location in the filesystem. They are installed by default in /opt/cloudera/parcels. In contrast, packages are installed in /usr/lib. When you install from the Parcels page, Cloudera Manager automatically downloads, distributes and activates the correct parcel for the operating system running on each host in the cluster. Note: You cannot install software using both parcels and packages in the same cluster. Because of their unique properties, parcels offer more advantages over packages, for more details please refer here Hope this helps. Please accept the answer and vote up if it did. Regards,

jagadeesan · ‎04-01-2020

Try to clean up some old files from your disk or else add some more space into the disk.

jagadeesan · ‎04-01-2020

Hi @Dharm, What do you mean by bad health space here? Is HDFS Full and getting critical alerts either in Ambari or Cloudera Manager? Are you able to read and write on HDFS? Please can you elaborate as well it will be more helpful if you attach some screenshot to understand the issue? Regards

jagadeesan · ‎03-31-2020

Hi @sppandita85BLR Currently, there is no documented procedure to migrate from HDP. In these cases, it's best to engage with your local Cloudera account rep and professional services. They may help you with the runbook to do the migration or any other feasibility. Hope this helps. Please accept the answer and vote up if it did. Regards,

jagadeesan · ‎03-27-2020

Hi @rajisridhar You can use a command like this to get the start and end time and then store it where you wish to or configure mail accordingly to your requirements. Example: $ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe . .---------------------------------------------------------------------------------------------------------------------------------------------------------------- Workflow Name : map-reduce-wf App Path : hdfs://localhost:8020/user/joe/workflows/map-reduce Status : SUCCEEDED Run : 0 User : joe Group : users Created : 2009-05-26 05:01 +0000 Started : 2009-05-26 05:01 +0000 Ended : 2009-05-26 05:01 +0000 Actions .---------------------------------------------------------------------------------------------------------------------------------------------------------------- Action Name Type Status Transition External Id External Status Error Code Start End .---------------------------------------------------------------------------------------------------------------------------------------------------------------- hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000 .---------------------------------------------------------------------------------------------------------------------------------------------------------------- For detailed information see this: https://oozie.apache.org/docs/3.3.2/DG_CommandLineTool.html#Jobs_Operations View solution in original post Hope this helps. Please accept the answer and vote up if it did.

jagadeesan · ‎03-27-2020

Hi @JasmineD, We might need to consider backing up the following: flow.xml.gz users.xml authorizations.xml All config files in NiFi conf directory NiFi local state from each node NiFi cluster state stored in zookeeper. Please make sure that you have stored the configuration passwords safely. NiFi relies on sensitive.props.key password to decrypt sensitive property values from flow.xml.gz file. If they do not know sensitive props key, they would need to manually clear all encoded values from flow.xml.gz. This action will clear all passwords in all components on the canvas. We need to re-enter all of them once NiFi was recovered. Also, if there are any local files that are required by the DataFlows, that would also need to be backed up as well. (i.e., Custom processor jars, user-built scripts, externally referenced config/jar files used by some processors, etc.). Note: All the repositories in NiFi are backed up by default. Here is a good article to see how backup works in NiFi. https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 Hope this helps. Please accept the answer and vote up if it did.

jagadeesan · ‎11-24-2019

Hi @anshuman Yes, we have Node labels support in the new CDP. For more details, you can check CDP documentations. https://docs.cloudera.com/ -> Cloudera Data Platform -> Runtime -> Cloudera Runtime https://docs.cloudera.com/runtime/7.0.2/yarn-allocate-resources/topics/yarn-configuring-node-labels.html FYI. Cloudera Runtime is the core open-source software distribution within Cloudera Data Platform (CDP) that is maintained, supported, versioned, and packaged as a single entity by Cloudera. Cloudera Runtime includes approximately 50 open source projects that comprise the core distribution of data management tools within CDP, including Cloudera Manager, which is used to configure and monitor clusters managed in CDP.

jagadeesan · ‎11-24-2019

Hi @Tanred, Ideally, it should run every day at 5 pm. Please, can you check the NIFI app logs of 2 consecutive days and see the log entry at that time. Else paste those error trace here for further analysis.

jagadeesan · ‎01-15-2019

@Michael Bronson Decommissioning is a process that supports removing components and their hosts from the cluster. You must decommission a master or slave running on a host before removing it or its host from service. Decommissioning helps you to prevent potential loss of data or disruption of service. Below HDP documentation for Ambari-2.6.1 help you to decommission a DataNode. When DataNode decommissioning process is finished, the status display changes to Decommissioned. https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-operations/content/how_to_decommission_a_component.html I hope that the above answers your questions.

jagadeesan · ‎01-14-2019

@Michael Bronson Below article help you to replacing faulty disks on datanode. https://community.hortonworks.com/articles/3131/replacing-disk-on-datanode-hosts.html Please accept the answer you found most useful.

jagadeesan · ‎01-11-2019

@harish If you made update on that particular queue value then you can ignore it. Because yarn automatically made necessary changes after you updated that particular queue.

jagadeesan · ‎01-11-2019

@harish There are some ordering policies to use different algorithms to decide which leaf queue should be scheduled the first. This ordering policy is set on the parent queue and its value is 'utilization' or 'priority-utilization'. (YARN-5864) You can refer below article for more info https://community.hortonworks.com/content/supportkb/174918/the-ordering-policy-setting-for-root-yarn-queue-ya.html Please accept the answer you found most useful.

jagadeesan · ‎01-07-2019

@Harish More Please can you confirm whether new user having appropriate permission to access all database like old user, you can check all defined policies in Ranger/Sentry depend on your cluster security access configuration.

jagadeesan · ‎01-05-2019

@Raghav Mp HDFS root scratch directory for Hive jobs, which gets created with write all (733) permission. For each connecting user, an HDFS scratch directory ${hive.exec.scratchdir}/ is created with ${hive.scratch.dir.permission}. To fix this issue, you can set below property in hive-site.xml, this is used for setting values for the entire Hive configuration <property> <name>hive.exec.scratchdir</name> <value>/tmp/hive</value> <description>Scratch space for Hive jobs</description> </property> For more detail: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Please accept the answer you found most useful.

jagadeesan · ‎01-05-2019

@Akash Dixit Suppose below answer won't resolve, please share detailed error for troubleshoot.

jagadeesan · ‎01-05-2019

@Akash Dixit This is most probably a permission issue. There are two ways to solve it. You can disable permission in hdfs-site.xml <property> <name>dfs.permissions</name> <value>false</value> </property> Or either you can give write permission to user 'training' in hadoop. Please accept the answer you found most useful.

jagadeesan · ‎01-02-2019

@Suraj Singh Seems it's similar like what we discussed in below thread https://community.hortonworks.com/questions/232093/yarn-jobs-are-getting-stuck-in-accepted-state.html If resubmit jobs will get success ? As discussed earlier this is open bug which fixed in further releases. If you need to apply patch, please involve Hortonworks support. If you are a customer, HWX can release a patch for you if it's technically possible based on specifics of the JIRAs. If you don't have support, you can certainly do it but test it first apply the patch in dev/test and see if it resolves your problem.

jagadeesan · ‎01-01-2019

@Michael Bronson Please can you check the zookeeper logs (/var/log/zookeeper) of master1.sys89.com. This can happen if there are too many open connections. Check where there were any warning messages stating with “Too many connections from {IP address of master1.sys89.com}”. Using netstat command also you can verify netstat -no | grep :2181 | wc -l To fix this issue, kindly clear up all stale connections manually or try increasing the maxClientCnxns setting at /etc/zookeeper/2.6.4.0-91/0/zoo.cfg. From your zoo.cfg file I can see value is maxClientCnxns=60 which is default. You can increase it by adding the maxClientCnxns=4096 and restart respective affected services.

jagadeesan · ‎01-01-2019

@Suraj Singh Actually particular fix is released in following version 3.1.0, 2.10.0, 2.9.1, 3.0.1 . Related JIRA https://issues.apache.org/jira/browse/YARN-7873. Fix you read complete comments then you will get idea about why they revert YARN-6078 and not released on 3.0.0.

jagadeesan · ‎12-28-2018

@Slimani Ibrahim It's not recommended to format the NameNode more than once except when NameNode loses metadata information. The reason could be this property which tells NameNode where to store its metadata information on disk is dfs.namenode.name.dir in your case its points to /tmp, so every time you restart your system the /tmp directory gets flushed and hence you have to format the NameNode again. So, make sure you point the property dfs.namenode.name.dir to a more persistent location (something like /hadoop/hdfs/namenode similar for datanode property/hadoop/hdfs/datanode) which does not get's cleared every time if you restart your system that will resolve this problem. I hope that the above answers your questions. Please accept the answer you found most useful.

jagadeesan · ‎12-28-2018

@Artyom Timofeev As for containers are stuck in localizing phase, seems you are hit on this reported bug on Yarn which is resolved in 3.0.0 version. https://issues.apache.org/jira/browse/YARN-6078

jagadeesan · ‎12-27-2018

@Harish More Please can you confirm whether did you created an HBase keystore and export the certificate ? If not please follow below steps. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/enable_ssl_on_the_hbase_rest_server.html https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/enable_ssl_on_the_hbase_web_ui.html If yes, then follow link to configure ranger ssl connection for hbase. https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/configuring-wire-encryption/content/configuring_ambari_ranger_ssl_using_public_ca_certificates_configure_ranger_hbase_plugin_for_ssl.html Important note: In the case where multiple servers talking to ranger admin for downloading policies for the same service/repository (e.g. HBase Master and Region servers, be sure to use the repo name or a common string across all of the nodes (i.e., HbasePlugin) https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/ch04s20s02s04s01.htm Please accept the answer you found most useful

jagadeesan · ‎12-27-2018

@Michael Bronson After configuration changes, it's safe to restart required services, those restart will make necessary new changes into the system. In our case, yarn.nodemanager.local-dirs will point out to new location /grid/sdb/hadoop/yarn/local instead of old location /var/hadoop/yarn/local . In short, restart will not cause any issue either after delete old files or after change in YARN configuration. I hope this answered your concerns.

jagadeesan · ‎12-27-2018

@Michael Bronson Sure

jagadeesan · ‎12-27-2018

@Michael Bronson Yes true, similarly it's not a good idea to use /var for yarn.nodemanager.local-dirs which are container local. Typically, you can direct these to all the data mount points (like /grid/sdb/hadoop/yarn/local). Same thing for yarn logs (/grid/sdb/hadoop/yarn/log) yarn.nodemanager.log-dirs. This can help with reducing all your IO going to your OS disk (where you typically have /var). You can take a look at http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/ I hope that the above answers your questions. Please accept the answer you found most useful.

jagadeesan · ‎12-27-2018

@Michael Bronson Yes you can remove old files. Below article help you to with proper steps. https://community.hortonworks.com/articles/92339/how-to-clear-local-file-cache-and-user-cache-for-y.html Please accept the answer you found most useful.

jagadeesan · ‎12-27-2018

Problem Description: Ambari-infra-solr is running fine but using a "ps" command shows a password like below. According to security policy, this is consider as security breach.The issue occurred because the value of property infra_solr_trust_store_password and infra_solr_key_store_password showing cleartext passwords in java Options. $ ps -ef | grep -i 'ambari-infra' 1008 25938 1 21 07:25 ?00:00:11 /usr/jdk64/jdk1.8.0_112/bin/java -server -Xms1024m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/ambari-infra-solr/solr_gc.log -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18886 -Dcom.sun.management.jmxremote.rmi.port=18886 -Djava.rmi.server.hostname=test2.example.com -DzkClientTimeout=60000 -DzkHost=test1.example.com:2181,test2.example.com:2181,test3.example.com:2181/infra-solr -Djetty.port=8886 -DSTOP.PORT=7886 -DSTOP.KEY=solrrocks -Dhost=test2.example.com -Duser.timezone=UTC -Djetty.home=/usr/lib/ambari-infra-solr/server -Dsolr.solr.home=/opt/ambari_infra_solr/data -Dsolr.install.dir=/usr/lib/ambari-infra-solr -Dlog4j.configuration=file:/etc/ambari-infra-solr/conf/log4j.properties -Dsolr.jetty.keystore=/etc/security/serverKeys/infra.solr.keyStore.jks -Dsolr.jetty.keystore.password=bigdata -Dsolr.jetty.truststore=/etc/security/serverKeys/infra.solr.trustStore.jks -Dsolr.jetty.truststore.password=bigdata -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Djavax.net.ssl.keyStore=/etc/security/serverKeys/infra.solr.keyStore.jks -Djavax.net.ssl.keyStorePassword=bigdata -Djavax.net.ssl.trustStore=/etc/security/serverKeys/infra.solr.trustStore.jks -Djavax.net.ssl.trustStorePassword=bigdata -Dsolr.jetty.https.port=8886 -Dsolr.authentication.httpclient.configurer=org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=/etc/ambari-infra-solr/conf/infra_solr_jaas.conf -Dsolr.kerberos.principal=HTTP/test2.example.com@EXAMPLE.COM -Dsolr.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Dsolr.kerberos.cookie.domain=test2.example.com -Dsolr.kerberos.name.rules=DEFAULT -XX:OnOutOfMemoryError=/usr/lib/ambari-infra-solr/bin/oom_solr.sh 8886 /var/log/ambari-infra-solr -jar start.jar --module=https Article: This article help to set hash password instead of showing clearest passwords in java options. Using Ambari inbuilt jetty jar file, we can hash password either OBF or MD5 format and pass those value in infra-solo-env to hide password from ambari-infra solr process. Step-1: Generate encrypt password using jetty jar file, where <password> is the password you used for the keystore/truststore java -cp /usr/lib/ambari-infra-solr/server/lib/jetty-util-9.2.13.v20150730.jar org.eclipse.jetty.util.security.Password <password> java -cp /usr/lib/ambari-infra-solr/server/lib/jetty-util-9.2.13.v20150730.jar org.eclipse.jetty.util.security.Password bigdata 2018-12-27 07:51:13.605:INFO::main: Logging initialized @171ms bigdata OBF:1rpc1wtw1sp11sov1sop1wui1rpa MD5:27819cfe72583a34d13a40bb74154c91 Step-2: Update below properties from Ambari under Ambari Infra Config Tab in Advanced infra-solr-env section (You can mention hashed_password of either OBF or MD5 there) Before: SOLR_SSL_KEY_STORE_PASSWORD={{infra_solr_keystore_hashed_password}} SOLR_SSL_TRUST_STORE_PASSWORD={{infra_solr_truststore_hashed_password}} Now: SOLR_SSL_KEY_STORE_PASSWORD=OBF:1rpc1wtw1sp11sov1sop1wui1rpa SOLR_SSL_TRUST_STORE_PASSWORD=OBF:1rpc1wtw1sp11sov1sop1wui1rpa Step-3: Need to restart required services through ambari and verify with grep process of ambari-infra solr process. $ ps -ef | grep -i 'ambari-infra'1008 17641 17 08:03 ?00:00:10 /usr/jdk64/jdk1.8.0_112/bin/java -server -Xms1024m -Xmx2048m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/ambari-infra-solr/solr_gc.log -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18886 -Dcom.sun.management.jmxremote.rmi.port=18886 -Djava.rmi.server.hostname=test2.example.com -DzkClientTimeout=60000 -DzkHost=test1.example.com:2181,test2.example.com:2181,test3.example.com:2181/infra-solr -Djetty.port=8886 -DSTOP.PORT=7886 -DSTOP.KEY=solrrocks -Dhost=test2.example.com -Duser.timezone=UTC -Djetty.home=/usr/lib/ambari-infra-solr/server -Dsolr.solr.home=/opt/ambari_infra_solr/data -Dsolr.install.dir=/usr/lib/ambari-infra-solr -Dlog4j.configuration=file:/etc/ambari-infra-solr/conf/log4j.properties -Dsolr.jetty.keystore=/etc/security/serverKeys/infra.solr.keyStore.jks -Dsolr.jetty.keystore.password=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.truststore=/etc/security/serverKeys/infra.solr.trustStore.jks -Dsolr.jetty.truststore.password=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.ssl.needClientAuth=false -Dsolr.jetty.ssl.wantClientAuth=false -Djavax.net.ssl.keyStore=/etc/security/serverKeys/infra.solr.keyStore.jks -Djavax.net.ssl.keyStorePassword=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Djavax.net.ssl.trustStore=/etc/security/serverKeys/infra.solr.trustStore.jks -Djavax.net.ssl.trustStorePassword=OBF:1rpc1wtw1sp11sov1sop1wui1rpa -Dsolr.jetty.https.port=8886 -Dsolr.authentication.httpclient.configurer=org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer -DauthenticationPlugin=org.apache.solr.security.KerberosPlugin -Djava.security.auth.login.config=/etc/ambari-infra-solr/conf/infra_solr_jaas.conf -Dsolr.kerberos.principal=HTTP/test2.example.com@EXAMPLE.COM -Dsolr.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab -Dsolr.kerberos.cookie.domain=test2.example.com -Dsolr.kerberos.name.rules=DEFAULT -XX:OnOutOfMemoryError=/usr/lib/ambari-infra-solr/bin/oom_solr.sh 8886 /var/log/ambari-infra-solr -jar start.jar --module=https Ambari will automatically decrypt password with inbuilt jetty jar. For more details of jetty you can refer following link, https://wiki.eclipse.org/Jetty/Howto/Secure_Passwords

jagadeesan · ‎12-27-2018

@Dukool SHarma The MapReduce sort the intermediate data(between mapper and reducer phase) by key by default. If we want the data should be sort based on value, then we need secondary sorting. For more Information you can reference below links: https://www.oreilly.com/library/view/data-algorithms/9781491906170/ch01.html https://www.quora.com/What-is-secondary-sort-in-Hadoop-and-how-does-it-work/answer/Sudarshan-Sreenivasan-1 Please accept the answer you found most useful.

jagadeesan · ‎12-26-2018

For your question, answer is No. As of now we can't mange memory of individual node level using percentage. Currently yarn support only CPU, using below configuration. yarn.nodemanager.resource.percentage-physical-cpu-limit 100 Percentage of CPU that can be allocated for containers. This setting allows users to limit the amount of CPU that YARN containers use. Currently functional only on Linux using cgroups. The default is to use 100% of CPU.

Online	Offline
Last Visited	‎04-03-2020 10:26 AM

Date Registered	‎11-12-2018 10:00 AM
Last Visited	‎04-03-2020 10:26 AM
Total Messages Posted	72
Total Kudos Received	115

Re: HDP to Cloudera Migration

Re: get oozie workflow name with start and end tim...

Re: Node Label Support in CDP

Re: yarn job stuck in accepted state randomly with...

Re: yarn local dirs + safety moving the local dir ...

Re: Transfer files based on date present in filena...

Re: Difference between upgrading using Parcels vs ...

Re: hdfs is in critical in stage due to bad health...

Re: hdfs is in critical in stage due to bad healt...

Re: HDP to Cloudera Migration

Re: get oozie workflow name with start and end tim...

Re: Best practice for Nifi env backup

Re: Node Label Support in CDP

Re: Nifi CRON based Scheduling done for every day...

Re: what are the steps in order to replace faulty ...

Re: what are the steps in order to replace faulty ...

Re: Yarn queue policy

Re: Yarn queue policy

Re: New Hue user is not able to see database

Re: /tmp/hive permission keep updating as per user...

Re: Hi All, I am trying to execute wordcount progr...

Re: Hi All, I am trying to execute wordcount progr...

Re: yarn job stuck in accepted state randomly with...

Re: zookeeper-client cannot connect to zookeeper ...

Re: Yarn jobs are getting stuck in ACCEPTED state

Re: How can i start Namenode service of Hadoop wit...

Re: Yarn jobs are getting stuck in ACCEPTED state

Re: Ranger ssl connection error for hbase

Re: yarn local dirs + safety moving the local dir ...

Re: YARN LOCAL DIRS + deletion

Re: yarn local dirs + safety moving the local dir ...

Re: YARN LOCAL DIRS + deletion

Hide Password from ambari-infra-solr process

Re: In Mapreduce how to sort intermediate output b...

Re: yarn configure to use all ram