Member since
09-21-2015
38
Posts
31
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1258 | 06-09-2017 09:18 AM | |
646 | 06-08-2017 03:01 PM |
07-13-2017
01:48 PM
When using post-user-creation-hook.sh script to create home directories for users we can edit the script to set a quota too. For information on enabling HDFS home dir creation see: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-administration/content/create_user_home_directory.html If you want to set a quota on this dir you can edit: /var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh # the default implementation creates user home folders; the first argument must be the username
ambari_sudo "yarn jar /var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/files/fast-hdfs-resource.jar $JSON_INPUT"
#ADD THESE LINES
while read -r LINE
do
USR_NAME=$(echo "$LINE" | awk -F, '{print $1}')
hdfs dfsadmin -setSpaceQuota 10 /user/$USR_NAME > /tmp/posthook.tmp
done <"$CSV_FILE"
#END ADD QUOTA
if [ "$DEBUG" -gt "0" ]; then echo "Switch debug OFF";set -x;unset DEBUG; else echo "debug: OFF"; fi
unset DEBUG
}
main "$@"
Add the lines between the comments and save. Now when a user is added both a home dir is created with a 10G quota set.
... View more
- Find more articles tagged with:
- ambari-server
- Design & Architecture
- HDFS
- How-ToTutorial
Labels:
06-30-2017
10:59 AM
2 Kudos
When configuring LDAPS in HDP its common to see wrong certificates used or certificates without the correct chain. To ensure the correct chain of certificates is used when configuring LDAPS you can use openssl to read the certificate from the server and save it to a file. This file can them be imported into, for example, the Ambari truststore. echo -n | openssl s_client -connect <ad-server>:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /root/ldaps-cert.pem
... View more
- Find more articles tagged with:
- active-directory
- FAQ
- keystore
- LDAP
- Security
06-21-2017
02:23 PM
Current on version HDP-2.6.x - Ambari-2.5.x if the zookeeper principal name is changed or customized manual changes are required for HDFS, Yarn and Ambari-Infra. In Ambari - config - for yarn and HDFS:
yarn-env.sh YARN_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sand-box-zookeeper -Djava.security.auth.login.config=/etc/hadoop/2.6.0.3-8/0/yarn_jaas.conf -Dzookeeper.sasl.clientconfig=Client $YARN_OPTS"
hadoop/conf/hadoop-env.sh export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sandbox-zookeeper -Djava.security.auth.login.config=/usr/hdp/current/hadoop-client/conf/secure/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS"
For Ambari-Infra/Solr
edit /usr/lib/ambari-infra-solr-client/solrCloudCli.sh PATH=$JAVA_HOME/bin:$PATH $JVM -classpath "$sdir:$sdir/libs/*" -Dzookeeper.sasl.client.username=sandbox-zookeeper org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI ${1+"$@"}
edit /usr/lib/ambari-infra-solr/bin/solr
#add lines below to bottom of script
ADDITIONAL_CMD_OPTS="$ADDITIONAL_CMD_OPTS -Dzookeeper.sasl.client.username=hcedhp02-zookeeper"
launch_solr "$FG" "$ADDITIONAL_CMD_OPTS" These services will now restart correctly and use your custom zookeeper principal name for the client connection.
... View more
06-16-2017
08:55 AM
I am far from a Yarn "expert" however my understanding of the above is that the jobs run in stages. The job itself has a good understanding of the number of containers that be will required to be launched and complete the entire job. Adding the stage 2 containers to the pending queue allows for pre-warming of containers so as a stage 1 task finishes its resulting data can be past to a pre-warmed container (shaving a small amount of time from execution) for a stage 2 task. A job with several stages cannot start the stage 2 containers until the data set from stage 1 is complete thats why the containers are still pending. I hope someone can allobrate or correct me on this answer if not entirley correct.
... View more
06-12-2017
06:49 PM
Session is different than cache as far as I understand (which is somewhat limited!): Sessions is the amount of time you can be logged in to say hive. During this session it wont try to re-authenticate you. If you quit the hive session and upload a file via knox WebHDFS before the cache timeout you will need to submit your username and password again to authenticate the request but it won't make an AD call and use the cached credentials. Thus limiting AD transactions. Pretty sure this would need some testing to confirm. If I find a moment I will try myself this is an intresting topic.
... View more
06-12-2017
06:18 PM
1) Standby namenode stores a second copy of the fsimage and should be up and ideally hosted on a second node. (this is not HA) 2) No Hbase does not store any information for Namenode or any other service as far as I am aware. We do have an embedded Hbase server for our Metrics system but thats out side the scope of this conversation. 3)Here is something important you need to know. Anything you allocate to heap size on a java program will be allocated at run-time. So if you have 5 apps each assigned 1G ram heap on a 4G system 4 will start but the 5th will fail because it cannot allocate the RAM. Simple example. 4) Check you have the HDP.repo and Ambari.repo in /etc/yum.repos.d/ redhat/centos6 is not a problem at all. I would stick with that in my personal opinion as there is much more OS specific detail of HDP on this platform. Other OS are also fine but for beginners I would stick with Centos6/7 ---- How you should approach this: Stop everything from Ambari. Start Zookeeper 1 or 3 nodes depends on setup but not 2. Namenode usually likes ZK up before it starts. Now start namenode and snamenode. Attach any failure logs here as an attachment.
HDFS is the first system that needs to be up. I assume you have not installed ranger at this time? In case you have remove it will complicate things at this point.
If I was learning all over again I would just start with HDFS/Zookeeper/YARN/MAPRED get those working on a single node and do some tutorials. Everything else will build off this and can be added a service at a time.
... View more
06-12-2017
12:13 PM
2 Kudos
You can find details here: https://shiro.apache.org/session-management.html#SessionManagement-EHCacheSessionDAO There is a default ehcache.xml file included with Knox which sets the TTL for entities to 120 seconds. This is included in the JAR file /usr/hdp/current/knox-server/dep/shiro-ehcache-1.2.3.jar.
You can override this by extracting the file ('unzip /usr/hdp/current/knox-server/dep/shiro-ehcache-1.2.3.jar org/apache/shiro/cache/ehcache/ehcache.xml') and copying the extracted file to /etc/knox/conf, then editing it. See the documentation here for details:
http://knox.apache.org/books/knox-0-9-1/user-guide.html#Advanced+Caching+Config
You'll then need to add the following to your Knox topology with the other cache settings provided in the link you already have: https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.0/bk_dataflow-security/content/ldap_authentication_caching.html Add to your Knox topology: <param name="main.cacheManager.cacheManagerConfigFile" value="classpath:ehcache.xml" />
... View more
06-12-2017
11:26 AM
One possible solution would reconfigure the LVM for /home. If I understand you have 950GB assigned to home. HDP will use little of this so you can reduce to say 50G using: lvreduce This should free up some more space to create a new LVM say: /usr_tmp Create (100G+), format and mount this LVM. Shut everything down and switch to run level 1 (init 1) This should now allow you copy all the data in /usr to usr_tmp. Delete the original /usr dir and remount your new /usr_tmp LVM as /usr A reboot might be a wise move at this point. You should now see /usr/hdp has a lot more space. You might want to repeat this for hdfs.datanode.data.dir location. NOTE: You might find by default that Ambari found all disk mounts and assigned a entry for this config option. On a development VM with just one virtual disk you should assign just one directory or LVM. EXAMPLE: /hdp_data/
... View more
06-12-2017
11:03 AM
You might need to provide a little further information. Do you have namenode HA. Is there a port listening on 50070 on that node (netstat -plant | grep 50070)
Is there any information in the namenode logs> (/var/log/hadoop/hdfs/)
... View more
06-09-2017
12:57 PM
After restarting any datanode it should automatically register with the namenode again. If this is not happening you might want to upload the datanode logs from the period it started so we can see if its registering.
... View more
06-09-2017
12:51 PM
You can use syntax like: keytool -list -v -keystore /etc/security/ssl/hdp_keystore.jks -storepass Hadoop123
Replacing the path and password to that configured in Ranger to test the keystore and ensure the path/password and certificated included are correct
... View more
06-09-2017
09:31 AM
It's not about the number of active zookeeper nodes but more about have the majority of "known" ZK nodes up. As an example the loss of one node in a 3-member ZooKeeper quorum is tolerable, because 2 out of 3 remaining machines still count as a majority (out of the fully identified quorum of 3).
No loss of functionality will be experienced with the loss of only one of the peers (in three), two peers (in five), etc. One of the remaining two will automatically be assigned the leader role, in case of a leader failure. Writes can continue to happen, and no special mode is invoked. Clients will continue to see the same behaviour they expect out ofZooKeeper, even in such a situation. No manual intervention is required for this procedure, ZK is automatically HA.
The administrator's guide of ZooKeeper covers this:
http://archive.cloudera.com/cdh5/cdh/5/zookeeper/zookeeperAdmin.html
... View more
06-09-2017
09:26 AM
I think you can do this using the WebHDFS REST API. The call Get_Trash_Root will tell you where the current users trash folder is. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Trash_Root Then rename the files to change its path to the .trash folder https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Rename_a_FileDirectory
... View more
06-09-2017
09:18 AM
1 Kudo
If you are using Ambari we already collect this information. If you go to the hosts section you can see some graphs on the right hand side. If you have Ambari Metrics and Grafana installed you will see we already have a very robust monitoring platform intergrated within Ambari.
... View more
06-08-2017
03:01 PM
I guess you are using some type of cloud instance? Normally the images used for provision of these instances are based on small partitions for root even if you specifically requested a 800G disk. Fortunately these are using LVM disk partitions. The command lsblk will show you what "physical" disks you have: lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 4G 0 part
│ └─md1 9:1 0 4G 0 raid1 /
├─sda2 8:2 0 2G 0 part [SWAP]
└─sda3 8:3 0 925.5G 0 part
└─md3 9:3 0 925.5G 0 raid1
├─vg00-usr 252:0 0 5G 0 lvm /usr
├─vg00-var 252:1 0 105G 0 lvm /var
└─vg00-home 252:2 0 165G 0 lvm /home
sdb 8:16 0 931.5G 0 disk
├─sdb1 8:17 0 4G 0 part
│ └─md1 9:1 0 4G 0 raid1 /
├─sdb2 8:18 0 2G 0 part [SWAP]
└─sdb3 8:19 0 925.5G 0 part
└─md3 9:3 0 925.5G 0 raid1
├─vg00-usr 252:0 0 5G 0 lvm /usr
├─vg00-var 252:1 0 105G 0 lvm /var
└─vg00-home 252:2 0 165G 0 lvm /home
lvscan will show logical volumes and the sizes they are. # lvscan
ACTIVE '/dev/vg00/usr' [5.00 GiB] inherit
ACTIVE '/dev/vg00/var' [105.00 GiB] inherit
ACTIVE '/dev/vg00/home' [165.00 GiB] inherit
In HDP the two directories you should assign space to are /usr and /var and if they are already defined as an LVM partition you can resize with the resize2fs command. sudo resize2fs /dev/vg00/usr
... View more
06-08-2017
02:53 PM
You might find this article interesting written by one of our technical support engineers: https://community.hortonworks.com/articles/30653/openldap-setup.html It covers setup of OpenLdap and give an example ldif.
... View more
06-08-2017
02:48 PM
You have the UI set to SSL mode. Please check to ensure you have configured the trust store correctly and have correctly loaded in the SSL certificate you wish to server with the Ranger UI. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/configure_ambari_ranger_ssl_self_signed_cert.html
... View more
06-08-2017
02:44 PM
1 Kudo
In centos 7 set your hostname in /etc/sysconfig/network something like master1.abc.xyz. Get your ip and add this and your host name to /etc/hosts. Make sure you have entries for each server in your hosts file like so: 172.26.15.16 master1.abc.xyz 72.26.98.253 master4.abc.xyz Both nodes should now be able to talk to one another using their fully qualified hostnames. You could create sub domains via your hosting provider but this is not required for testing or learning. Hosts file will work like a local DNS server to each host and they will use this as a primary source for looking up DNS before using the server listed in /etc/resolv.conf for domain name resolution.
... View more
05-17-2017
12:55 PM
5 Kudos
Ambari user sync will fail to map or import users when trying to pull from groups with 1,500+ members. What we see when we use ldapsearch to query a large group is: <snip>
member;range=0-1499: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net
member;range=0-1499: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net
</snip> This is seems to come from the LDAP Policy value: MaxValRange "MaxValueRange controls the number of values that are returned on a single attribute on a single object.
Default"1500
Hard Limit: 5000"
-- http://ldapwiki.com/wiki/MaxValRange To fix this: Go to the domain controller that we’re connecting to for the sync Find the file ntdsutil.exe (most likely under c:\windows\system32 or c:\winnt\system32 Run the ntdsutil.exe Type “ldap policies” and enter Type "connections" and enter Type "Connect to server [YourDCName]" and enter Type "q" and enter Type "Show Values" to see the current settings Type “Set MaxValRange to 2500” and enter Type “Commit Changes” and enter Type “Show Values” and enter -- https://support.intranetconnections.com/hc/en-us/articles/214747288-Changing-LDAP-Settings-Increasing-MaxPageSize member: CN=Elgine Metzger,OU=users,OU=test,DC=j4ck3l,DC=net
member: CN=Friedolf Welter,OU=users,OU=test,DC=j4ck3l,DC=net The group should now successfully sync with Ambari
... View more
- Find more articles tagged with:
- active-directory
- Ambari
- ambari-server
- FAQ
- Security
- user-groups
Labels:
03-31-2017
06:45 PM
2 Kudos
After a few hours of debugging SSSD and mapping users/groups I wanted to make a post here to try and save someone the pain. I had SSD configured correctly using the following document: https://github.com/HortonworksUniversity/Security_Labs#lab-1 What I found by adding debug_level=7 to the sssd.conf file was this cryptic message: Trying to resolve service 'AD_GC' I realized at some point I was firewall'd off to the Active Directory Global Catalog port 3286, once I opened this I can now get the correct groups mapped to my SSSD users. Hope this saves someone some time in the future!
... View more
- Find more articles tagged with:
- active-directory
- FAQ
- LDAP
- Security
- sssd
03-29-2017
09:12 AM
Its basic local auth using the pg_hba.conf file. Default username and password for the Ambari DB is: ambari/bigdata Read more here: https://www.postgresql.org/docs/9.4/static/auth-pg-hba-conf.html
... View more
03-29-2017
08:38 AM
2 Kudos
When using kerberos with HDP it's not uncommon to find the odd strange encryption type floating around, possibly from a badly configured AD server. By adding the following to the Ambari -> Kerberos config section under supported encryption types its possible to isolate this issue for diagnostic. While its probably not a wise idea to run with all these enabled in production having a full list of supported types can be useful for diagnostic or reference. des-cbc-crc des-cbc-md4 des-cbc-md5 des3-cbc-sha1 arcfour-hmac arcfour-hmac-exp aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha384-192 camellia128-cts-cmac camellia256-cts-cmac
... View more
- Find more articles tagged with:
- Ambari
- Encryption
- FAQ
- Kerberos
- Security
03-29-2017
08:34 AM
2 Kudos
A quick easy way to export users from LDAP to a CSV for use in importing to Ambari. ldapsearch -h ldapserver.mycompany.com -p 389 -D bind-user -w bind-password -b OU=London\ Users,OU=my\ dept,DC=mycompany,DC=com -s sub "(objectClass=user)" | grep sAMAccountName | sed 's/sAMAccountName://g' | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'
... View more
Labels:
03-28-2017
11:09 AM
1 Kudo
I think this seems to be very much work in progress and not officially support yet by Hortonworks though I do see active development on this from our team. https://issues.apache.org/jira/browse/YARN-3611 Below are some steps recorded from one of our staff testing this. They themselves had mixed results. Steps required on each node with a running node manager, before we can start testing
--------
a. Ensure that we are NOT running a secure cluster - docker support does not work with secure clusters.
b. yum install docker-io - installs docker/dependencies
c. service docker start - starts the docker daemon
d. docker pull sequenceiq/hadoop-docker:2.5.1 ( ~ 1.5 GB, I think ) - this has to be done in advance before launching containers since its very slow.
e. add user yarn to group docker (or whichever user yarn is running as). Add group docker in case it doesn’t exist already.
f. modify yarn-site.xml to add yarn.nodemanager.docker-container-executor.exec-name (point to docker binary) and yarn.nodemanager.container-executor.class (to use docker container executor)
g. restart node manager Running a test MR job I would suggest that maybe this is not mature enough for production but still might be worth while for experimentation.
... View more
03-28-2017
11:02 AM
Is there anything in the xa_portal.log for ranger that might give further clues? You might attach it here for review.
... View more
03-02-2017
12:01 PM
3 Kudos
Ambari Infra Solr Fails to start with "Unrecognized option: --secure-solr-znode" check the versions of ambari-infra like below, notice the version mis-match between client and server. [root@master1 zookeeper]# rpm -qa | grep ambari
ambari-infra-solr-client-2.4.2.0-136.x86_64
ambari-infra-solr-2.5.0.0-771.x86_64 Upgrade ambari-infra-solr-client
... View more
- Find more articles tagged with:
- ambari-infra
- Cloud & Operations
- FAQ
Labels:
02-08-2017
12:19 PM
2 Kudos
After trying to disable kerberos via the UI the following message is seen in the ambari-server logs and the service fails to start. ...INFO [main] KerberosChecker:57 - Checking Ambari Server Kerberos credentials.
WARN [main] KerberosChecker:81 - /etc/security/keytabs/ambari.server.keytab doesn't exist
ERROR [main] KerberosChecker:115 - Configuration Error - either doNotPrompt should be false or at least one of useTicketCache, useKeyTab, tryFirstPass and useFirstPass should be true
Edit: /etc/ambari-server/conf/ambari.properties
Ensure the following is set: kerberos.check.jaas.configuration=false
Further disable kerberos for ambari server by removing: "-Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf
-Djava.security.krb5.conf=/etc/krb5.conf
-Djavax.security.auth.useSubjectCredsOnly=false" from AMBARI_JVM_ARGS value in
/var/lib/ambari-server/ambari-env.sh
If you decide to enable kerberos again, you have to manually revert changes in
/var/lib/ambari-server/ambari-env.sh
... View more
- Find more articles tagged with:
- ambari-server
- FAQ
- Kerberos
- Security
Labels:
12-28-2016
12:00 PM
Great guide found this very useful thanks!
... View more
12-22-2016
11:42 AM
You FSImage can be exported to XML for diagnostics using the following hadoop oiv -i fsimage -p XML -o output.xml
... View more
- Find more articles tagged with:
- HDFS
- How-ToTutorial
- Sandbox & Learning
Labels: