Member since
09-18-2015
191
Posts
81
Kudos Received
40
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2045 | 08-04-2017 08:40 AM | |
5418 | 05-02-2017 01:18 PM | |
1101 | 04-24-2017 08:35 AM | |
1107 | 04-24-2017 08:21 AM | |
1329 | 06-01-2016 08:54 AM |
03-20-2018
12:41 PM
This is very helpful. Do you have any rough estimates around CPU and Memory requirements?
... View more
06-01-2016
06:15 PM
You are correct, use LVM for OS disks, but not data disks. In the end, the filesystem choice doesn't make a huge difference. ext4 everywhere would simply the overall design and allow for the ability to resize filesystems online in the future. Allocating a larger amount of storage to the OS filesystems does simplify the install. Otherwise, during the Ambari install wizard, you need to go through each of the service's configurations and change "/var/log" to one of the data disk mount points (i.e. /opt/dev/sdb as an example above). If you allocated more storage to the OS (and subsequently made /usr say 30GB and /var/log 200GB), you would not have to change as much during the Ambari install. Either approach is viable, so I would suggest discussing with your OS admin team to see if they have a preference. Also note that I'm referring to daemon logs (namenode, resource manager, etc) that end up in /var/log, versus application logs. The yarn settings you show above are for the yarn application logs and local scratch space. You want to follow that same pattern in production.
... View more
06-07-2016
07:09 AM
1 Kudo
If you want to see dates and update history for tutorials then I would suggest looking at the source in github. https://github.com/hortonworks/tutorials Tutorials are updated on Sandbox update release schedule which tend to correspond to major HDP releases. Here you can see the latest version of the HDP tutorials for HDP 2.4: https://github.com/hortonworks/tutorials/tree/hdp/tutorials/hortonworks For example, I believe earlier you had a question on Ipython Notebook with Spar tutorial and here you can see the history of updates for this tutorial: https://github.com/hortonworks/tutorials/commits/hdp/tutorials/hortonworks/ipython-notebook-with-spark/tutorial.md The tutorials list what is the prerequisite to do the tutorial. If you want to learn more about the tutorials or make a contribution then at the bottom of each tutorial there is paragraph that talks about the github repo and contribution guide. I am happy to chat with you to see how we can make this template more descriptive.
... View more
05-27-2016
08:18 AM
Fixed it. I removed the atlas.pid file and restarted the atlas-server. rm /var/run/atlas/atlas.pid
python /usr/hdp/current/atlas-server/bin/atlas_start.py
... View more
05-25-2016
01:15 PM
1 Kudo
Hi @Smart Solutions. Right now... there isn't really a good, quick, clean, easy way to achieve this. You've already identified the thread that I would otherwise point you towards for some ideas. You're just too good! There are two main approaches that I would recommend thinking about, the first is ceasing to make any changes via the web-ui, and only making changes via the API, that way you can just call both your clusters one after the other to make the configuration changes. The second is to use some of the ideas from the thread you linked to, where you would continue maintaining the configs on your "master" cluster, but then extract the configs from your "master" cluster on a regular basis (or on a trigger of config changed?), diff them between the previous "master" cluster config version, and then push the resulting deltas to your "slave" cluster, again via API calls. Either way, there's quite a bit of automation that would be required there. I'd strongly suggest if you want to go down this path, doing your work out in the open, this is something I see come up now and again, so I think you may well get others who would be interested in working with you on this. Longer term, Ambari will no doubt support multi cluster, and this functionality would be a natural extension of that, but progress on those public JIRA's has been slow, with other more important items taking priority. Happy to hear if you have other ideas too, sorry I couldn't be more direct help, but let me know if you plan on cutting some code moving forward, I'm sure it'd be an interesting project. Many thanks.
... View more
05-24-2016
12:48 PM
1 Kudo
Alternatively use kerberos and kerberize the HDFS UI. In this case only SPNEGO enabled browsers will be able to access the ui and you will have the same filesystem access restrictions as users have when directly accessing hdfs.
... View more
05-12-2016
07:25 AM
Hi @Anandha L Ranganathan All the steps are covered in the ambari wizard for the downgrade, but you'll need to pay close attention to the various databases etc that require backing up as you go through. Both roll-back and full downgrade are possible. Make sure you have satisfied all of the prerequisites before beginning as well, and preferably have read through the upgrade document several times to ensure you have accounted for everything. http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_upgrading_Ambari/content/_upgrading_HDP_prerequisites.html I've performed a number of upgrades myself and while there is always the odd thing that crops up, its usually something minor that is easily fixed.
Good luck and I hope everything goes smoothly.
... View more
01-22-2019
10:01 AM
Hi, In order to find the adequate space in any directory during installation or upgrade procedures, for example while doing HDP upgrade you should verify about the availability of adequate space on /usr/hdp for the target HDP version. For that use below format: df -h <Path_of_interest> Example : [alex@machine1]# df -h /usr/hdp/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 528G 22G 506G 5% / [alex@machine1]# You can all parameters like Size of disk, used space, available spave and percentage of usage.
... View more