Re: How to know CPU Utilization of Data Node ?

cdraper · ‎07-13-2017

When using post-user-creation-hook.sh script to create home directories for users we can edit the script to set a quota too. For information on enabling HDFS home dir creation see: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-administration/content/create_user_home_directory.html If you want to set a quota on this dir you can edit: /var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh # the default implementation creates user home folders; the first argument must be the username ambari_sudo "yarn jar /var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/files/fast-hdfs-resource.jar $JSON_INPUT" #ADD THESE LINES while read -r LINE do USR_NAME=$(echo "$LINE" | awk -F, '{print $1}') hdfs dfsadmin -setSpaceQuota 10 /user/$USR_NAME > /tmp/posthook.tmp done <"$CSV_FILE" #END ADD QUOTA if [ "$DEBUG" -gt "0" ]; then echo "Switch debug OFF";set -x;unset DEBUG; else echo "debug: OFF"; fi unset DEBUG } main "$@" Add the lines between the comments and save. Now when a user is added both a home dir is created with a 10G quota set.

cdraper · ‎06-30-2017

When configuring LDAPS in HDP its common to see wrong certificates used or certificates without the correct chain. To ensure the correct chain of certificates is used when configuring LDAPS you can use openssl to read the certificate from the server and save it to a file. This file can them be imported into, for example, the Ambari truststore. echo -n | openssl s_client -connect <ad-server>:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /root/ldaps-cert.pem

cdraper · ‎06-21-2017

Current on version HDP-2.6.x - Ambari-2.5.x if the zookeeper principal name is changed or customized manual changes are required for HDFS, Yarn and Ambari-Infra. In Ambari - config - for yarn and HDFS: yarn-env.sh YARN_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sand-box-zookeeper -Djava.security.auth.login.config=/etc/hadoop/2.6.0.3-8/0/yarn_jaas.conf -Dzookeeper.sasl.clientconfig=Client $YARN_OPTS" hadoop/conf/hadoop-env.sh export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sandbox-zookeeper -Djava.security.auth.login.config=/usr/hdp/current/hadoop-client/conf/secure/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS" For Ambari-Infra/Solr edit /usr/lib/ambari-infra-solr-client/solrCloudCli.sh PATH=$JAVA_HOME/bin:$PATH $JVM -classpath "$sdir:$sdir/libs/*" -Dzookeeper.sasl.client.username=sandbox-zookeeper org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI ${1+"$@"} edit /usr/lib/ambari-infra-solr/bin/solr #add lines below to bottom of script ADDITIONAL_CMD_OPTS="$ADDITIONAL_CMD_OPTS -Dzookeeper.sasl.client.username=hcedhp02-zookeeper" launch_solr "$FG" "$ADDITIONAL_CMD_OPTS" These services will now restart correctly and use your custom zookeeper principal name for the client connection.

cdraper · ‎06-16-2017

I am far from a Yarn "expert" however my understanding of the above is that the jobs run in stages. The job itself has a good understanding of the number of containers that be will required to be launched and complete the entire job. Adding the stage 2 containers to the pending queue allows for pre-warming of containers so as a stage 1 task finishes its resulting data can be past to a pre-warmed container (shaving a small amount of time from execution) for a stage 2 task. A job with several stages cannot start the stage 2 containers until the data set from stage 1 is complete thats why the containers are still pending. I hope someone can allobrate or correct me on this answer if not entirley correct.

cdraper · ‎06-12-2017

Session is different than cache as far as I understand (which is somewhat limited!): Sessions is the amount of time you can be logged in to say hive. During this session it wont try to re-authenticate you. If you quit the hive session and upload a file via knox WebHDFS before the cache timeout you will need to submit your username and password again to authenticate the request but it won't make an AD call and use the cached credentials. Thus limiting AD transactions. Pretty sure this would need some testing to confirm. If I find a moment I will try myself this is an intresting topic.

cdraper · ‎06-12-2017

1) Standby namenode stores a second copy of the fsimage and should be up and ideally hosted on a second node. (this is not HA) 2) No Hbase does not store any information for Namenode or any other service as far as I am aware. We do have an embedded Hbase server for our Metrics system but thats out side the scope of this conversation. 3)Here is something important you need to know. Anything you allocate to heap size on a java program will be allocated at run-time. So if you have 5 apps each assigned 1G ram heap on a 4G system 4 will start but the 5th will fail because it cannot allocate the RAM. Simple example. 4) Check you have the HDP.repo and Ambari.repo in /etc/yum.repos.d/ redhat/centos6 is not a problem at all. I would stick with that in my personal opinion as there is much more OS specific detail of HDP on this platform. Other OS are also fine but for beginners I would stick with Centos6/7 ---- How you should approach this: Stop everything from Ambari. Start Zookeeper 1 or 3 nodes depends on setup but not 2. Namenode usually likes ZK up before it starts. Now start namenode and snamenode. Attach any failure logs here as an attachment. HDFS is the first system that needs to be up. I assume you have not installed ranger at this time? In case you have remove it will complicate things at this point. If I was learning all over again I would just start with HDFS/Zookeeper/YARN/MAPRED get those working on a single node and do some tutorials. Everything else will build off this and can be added a service at a time.

cdraper · ‎06-12-2017

You can find details here: https://shiro.apache.org/session-management.html#SessionManagement-EHCacheSessionDAO There is a default ehcache.xml file included with Knox which sets the TTL for entities to 120 seconds. This is included in the JAR file /usr/hdp/current/knox-server/dep/shiro-ehcache-1.2.3.jar. You can override this by extracting the file ('unzip /usr/hdp/current/knox-server/dep/shiro-ehcache-1.2.3.jar org/apache/shiro/cache/ehcache/ehcache.xml') and copying the extracted file to /etc/knox/conf, then editing it. See the documentation here for details: http://knox.apache.org/books/knox-0-9-1/user-guide.html#Advanced+Caching+Config You'll then need to add the following to your Knox topology with the other cache settings provided in the link you already have: https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.0/bk_dataflow-security/content/ldap_authentication_caching.html Add to your Knox topology: <param name="main.cacheManager.cacheManagerConfigFile" value="classpath:ehcache.xml" />

cdraper · ‎06-12-2017

One possible solution would reconfigure the LVM for /home. If I understand you have 950GB assigned to home. HDP will use little of this so you can reduce to say 50G using: lvreduce This should free up some more space to create a new LVM say: /usr_tmp Create (100G+), format and mount this LVM. Shut everything down and switch to run level 1 (init 1) This should now allow you copy all the data in /usr to usr_tmp. Delete the original /usr dir and remount your new /usr_tmp LVM as /usr A reboot might be a wise move at this point. You should now see /usr/hdp has a lot more space. You might want to repeat this for hdfs.datanode.data.dir location. NOTE: You might find by default that Ambari found all disk mounts and assigned a entry for this config option. On a development VM with just one virtual disk you should assign just one directory or LVM. EXAMPLE: /hdp_data/

cdraper · ‎06-12-2017

You might need to provide a little further information. Do you have namenode HA. Is there a port listening on 50070 on that node (netstat -plant | grep 50070) Is there any information in the namenode logs> (/var/log/hadoop/hdfs/)

cdraper · ‎06-09-2017

After restarting any datanode it should automatically register with the namenode again. If this is not happening you might want to upload the datanode logs from the period it started so we can see if its registering.

cdraper · ‎06-09-2017

You can use syntax like: keytool -list -v -keystore /etc/security/ssl/hdp_keystore.jks -storepass Hadoop123 Replacing the path and password to that configured in Ranger to test the keystore and ensure the path/password and certificated included are correct

cdraper · ‎06-09-2017

It's not about the number of active zookeeper nodes but more about have the majority of "known" ZK nodes up. As an example the loss of one node in a 3-member ZooKeeper quorum is tolerable, because 2 out of 3 remaining machines still count as a majority (out of the fully identified quorum of 3). No loss of functionality will be experienced with the loss of only one of the peers (in three), two peers (in five), etc. One of the remaining two will automatically be assigned the leader role, in case of a leader failure. Writes can continue to happen, and no special mode is invoked. Clients will continue to see the same behaviour they expect out ofZooKeeper, even in such a situation. No manual intervention is required for this procedure, ZK is automatically HA. The administrator's guide of ZooKeeper covers this: http://archive.cloudera.com/cdh5/cdh/5/zookeeper/zookeeperAdmin.html

cdraper · ‎06-09-2017

I think you can do this using the WebHDFS REST API. The call Get_Trash_Root will tell you where the current users trash folder is. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Trash_Root Then rename the files to change its path to the .trash folder https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Rename_a_FileDirectory

cdraper · ‎06-09-2017

If you are using Ambari we already collect this information. If you go to the hosts section you can see some graphs on the right hand side. If you have Ambari Metrics and Grafana installed you will see we already have a very robust monitoring platform intergrated within Ambari.

cdraper · ‎06-08-2017

I guess you are using some type of cloud instance? Normally the images used for provision of these instances are based on small partitions for root even if you specifically requested a 800G disk. Fortunately these are using LVM disk partitions. The command lsblk will show you what "physical" disks you have: lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk ├─sda1 8:1 0 4G 0 part │ └─md1 9:1 0 4G 0 raid1 / ├─sda2 8:2 0 2G 0 part [SWAP] └─sda3 8:3 0 925.5G 0 part └─md3 9:3 0 925.5G 0 raid1 ├─vg00-usr 252:0 0 5G 0 lvm /usr ├─vg00-var 252:1 0 105G 0 lvm /var └─vg00-home 252:2 0 165G 0 lvm /home sdb 8:16 0 931.5G 0 disk ├─sdb1 8:17 0 4G 0 part │ └─md1 9:1 0 4G 0 raid1 / ├─sdb2 8:18 0 2G 0 part [SWAP] └─sdb3 8:19 0 925.5G 0 part └─md3 9:3 0 925.5G 0 raid1 ├─vg00-usr 252:0 0 5G 0 lvm /usr ├─vg00-var 252:1 0 105G 0 lvm /var └─vg00-home 252:2 0 165G 0 lvm /home lvscan will show logical volumes and the sizes they are. # lvscan ACTIVE '/dev/vg00/usr' [5.00 GiB] inherit ACTIVE '/dev/vg00/var' [105.00 GiB] inherit ACTIVE '/dev/vg00/home' [165.00 GiB] inherit In HDP the two directories you should assign space to are /usr and /var and if they are already defined as an LVM partition you can resize with the resize2fs command. sudo resize2fs /dev/vg00/usr

cdraper · ‎06-08-2017

You might find this article interesting written by one of our technical support engineers: https://community.hortonworks.com/articles/30653/openldap-setup.html It covers setup of OpenLdap and give an example ldif.

cdraper · ‎06-08-2017

You have the UI set to SSL mode. Please check to ensure you have configured the trust store correctly and have correctly loaded in the SSL certificate you wish to server with the Ranger UI. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/configure_ambari_ranger_ssl_self_signed_cert.html

cdraper · ‎06-08-2017

In centos 7 set your hostname in /etc/sysconfig/network something like master1.abc.xyz. Get your ip and add this and your host name to /etc/hosts. Make sure you have entries for each server in your hosts file like so: 172.26.15.16 master1.abc.xyz 72.26.98.253 master4.abc.xyz Both nodes should now be able to talk to one another using their fully qualified hostnames. You could create sub domains via your hosting provider but this is not required for testing or learning. Hosts file will work like a local DNS server to each host and they will use this as a primary source for looking up DNS before using the server listed in /etc/resolv.conf for domain name resolution.

cdraper · ‎05-17-2017

Ambari user sync will fail to map or import users when trying to pull from groups with 1,500+ members. What we see when we use ldapsearch to query a large group is: <snip> member;range=0-1499: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net member;range=0-1499: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net </snip> This is seems to come from the LDAP Policy value: MaxValRange "MaxValueRange controls the number of values that are returned on a single attribute on a single object. Default"1500 Hard Limit: 5000" -- http://ldapwiki.com/wiki/MaxValRange To fix this: Go to the domain controller that we’re connecting to for the sync Find the file ntdsutil.exe (most likely under c:\windows\system32 or c:\winnt\system32 Run the ntdsutil.exe Type “ldap policies” and enter Type "connections" and enter Type "Connect to server [YourDCName]" and enter Type "q" and enter Type "Show Values" to see the current settings Type “Set MaxValRange to 2500” and enter Type “Commit Changes” and enter Type “Show Values” and enter -- https://support.intranetconnections.com/hc/en-us/articles/214747288-Changing-LDAP-Settings-Increasing-MaxPageSize member: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net member: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net The group should now successfully sync with Ambari

cdraper · ‎03-31-2017

After a few hours of debugging SSSD and mapping users/groups I wanted to make a post here to try and save someone the pain. I had SSD configured correctly using the following document: https://github.com/HortonworksUniversity/Security_Labs#lab-1 What I found by adding debug_level=7 to the sssd.conf file was this cryptic message: Trying to resolve service 'AD_GC' I realized at some point I was firewall'd off to the Active Directory Global Catalog port 3286, once I opened this I can now get the correct groups mapped to my SSSD users. Hope this saves someone some time in the future!

cdraper · ‎03-29-2017

Its basic local auth using the pg_hba.conf file. Default username and password for the Ambari DB is: ambari/bigdata Read more here: https://www.postgresql.org/docs/9.4/static/auth-pg-hba-conf.html

cdraper · ‎03-29-2017

When using kerberos with HDP it's not uncommon to find the odd strange encryption type floating around, possibly from a badly configured AD server. By adding the following to the Ambari -> Kerberos config section under supported encryption types its possible to isolate this issue for diagnostic. While its probably not a wise idea to run with all these enabled in production having a full list of supported types can be useful for diagnostic or reference. des-cbc-crc des-cbc-md4 des-cbc-md5 des3-cbc-sha1 arcfour-hmac arcfour-hmac-exp aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha384-192 camellia128-cts-cmac camellia256-cts-cmac

cdraper · ‎03-29-2017

A quick easy way to export users from LDAP to a CSV for use in importing to Ambari. ldapsearch -h ldapserver.mycompany.com -p 389 -D bind-user -w bind-password -b OU=London\ Users,OU=my\ dept,DC=mycompany,DC=com -s sub "(objectClass=user)" | grep sAMAccountName | sed 's/sAMAccountName://g' | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'

cdraper · ‎03-28-2017

I think this seems to be very much work in progress and not officially support yet by Hortonworks though I do see active development on this from our team. https://issues.apache.org/jira/browse/YARN-3611 Below are some steps recorded from one of our staff testing this. They themselves had mixed results. Steps required on each node with a running node manager, before we can start testing -------- a. Ensure that we are NOT running a secure cluster - docker support does not work with secure clusters. b. yum install docker-io - installs docker/dependencies c. service docker start - starts the docker daemon d. docker pull sequenceiq/hadoop-docker:2.5.1 ( ~ 1.5 GB, I think ) - this has to be done in advance before launching containers since its very slow. e. add user yarn to group docker (or whichever user yarn is running as). Add group docker in case it doesn’t exist already. f. modify yarn-site.xml to add yarn.nodemanager.docker-container-executor.exec-name (point to docker binary) and yarn.nodemanager.container-executor.class (to use docker container executor) g. restart node manager Running a test MR job I would suggest that maybe this is not mature enough for production but still might be worth while for experimentation.

cdraper · ‎03-28-2017

Is there anything in the xa_portal.log for ranger that might give further clues? You might attach it here for review.

cdraper · ‎03-02-2017

Ambari Infra Solr Fails to start with "Unrecognized option: --secure-solr-znode" check the versions of ambari-infra like below, notice the version mis-match between client and server. [root@master1 zookeeper]# rpm -qa | grep ambari ambari-infra-solr-client-2.4.2.0-136.x86_64 ambari-infra-solr-2.5.0.0-771.x86_64 Upgrade ambari-infra-solr-client

cdraper · ‎02-08-2017

After trying to disable kerberos via the UI the following message is seen in the ambari-server logs and the service fails to start. ...INFO [main] KerberosChecker:57 - Checking Ambari Server Kerberos credentials. WARN [main] KerberosChecker:81 - /etc/security/keytabs/ambari.server.keytab doesn't exist ERROR [main] KerberosChecker:115 - Configuration Error - either doNotPrompt should be false or at least one of useTicketCache, useKeyTab, tryFirstPass and useFirstPass should be true Edit: /etc/ambari-server/conf/ambari.properties Ensure the following is set: kerberos.check.jaas.configuration=false Further disable kerberos for ambari server by removing: "-Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false" from AMBARI_JVM_ARGS value in /var/lib/ambari-server/ambari-env.sh If you decide to enable kerberos again, you have to manually revert changes in /var/lib/ambari-server/ambari-env.sh

cdraper · ‎02-07-2017

Very nice guide Sagar thank you!

cdraper · ‎12-28-2016

Great guide found this very useful thanks!

cdraper · ‎12-22-2016

You FSImage can be exported to XML for diagnostics using the following hadoop oiv -i fsimage -p XML -o output.xml

Online	Offline
Last Visited	‎06-20-2019 04:30 PM

Date Registered	‎09-21-2015 08:16 AM
Last Visited	‎06-20-2019 04:30 PM
Total Messages Posted	38
Total Kudos Received	30

Re: How to know CPU Utilization of Data Node ?

Re: I have assigned 800 gb on all my instance when...

post-user-creation-hook.sh - Adding HDFS quotas

Extracting Active Directory LDAPS SSL Ccertificate

HDFS - YARN - InfraSolr: ZK Client when using cust...

Re: Interpreting Number of Pending containers in A...

Re: How to set Cache expiry for Knox LDAP Authent...

Re: HDFS NameNode Web UI connection faile to http:...

Re: How to set Cache expiry for Knox LDAP Authent...

Re: Issue In Installing Ambari. CentOS6

Re: HDFS NameNode Web UI connection faile to http:...

Re: How to add datanode in running cluster

Re: Ranger KMS/Admin: "the trustAnchors parameter ...

Re: Zookeeper Setup for 2 Rack Data Center

Re: About hadoop api safely delete

Re: How to know CPU Utilization of Data Node ?

Re: I have assigned 800 gb on all my instance when...

Re: ranger ldap intergation with openldap

Re: Ranger KMS/Admin: "the trustAnchors parameter ...

Re: Super Noob DNS routing and setup for Ambari Se...

Ambari problem and solution when trying to sync AD...

SSSD

Re: How is PostgreS DB authentication secured on a...

Kerberos: List of supported encryption types

Ambari: LDAP to CSV - Quick one liner

Re: Hdp2.5.x support docker executor container?

Re: Ranger Policy Report

Ambari Infra Solr Failts to start with "Unrecogniz...

Ambari: Manually disable kerberos authentication f...

Re: Ranger Ldap Integration

Re: Manual Keytab / Principal creation for IPA to ...

HDFS: Save FSImage to XML