Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Configuring HDP for a moderate memory

Highlighted

Configuring HDP for a moderate memory

Rising Star

AT this point we are running HDP cluster managed by Ambari for purely educational purposes - like going through Hadoop tutorial examples. We use one node cluster running on computer with 6Gb of memory.

This is constantly causing numerous problems because of lack of memory. Tasks are failing, overtiming, running extremely slow because of intensive memory swap to a hard drive.

The default memory settings in HDP are requesting gigabytes of memory for almost everything. Which is probably good in a production environment but does not make much sense in education environment where we deal with tiny amount of data.

Can please anybody recommend some optimal HDP memory settings for a computer with moderate memory? There are something like 100+ memory related settings in various services configurations and it is really difficult for a Hadoop beginner to understand which ones are most important and which are secondary.

9 REPLIES 9
Highlighted

Re: Configuring HDP for a moderate memory

@Dmitry Otblesk I will suggest you to shutdown few services which you are not using as part of tutorial like HBASE,STORM,ATLAS etc an many more, you can start them once you are covering the tutorial related to these.

Highlighted

Re: Configuring HDP for a moderate memory

Rising Star

@Rajkumar Singh

That did not help so far. At this point I am using Pig only, but it fails all the time because of lack of memory

Highlighted

Re: Configuring HDP for a moderate memory

@Dmitry Otblesk what are the services installed/running on your node?

Highlighted

Re: Configuring HDP for a moderate memory

Rising Star

@Rajkumar Singh

Running services are: HDFS, YARN, MapReduce2, Hive, ZooKeeper

Highlighted

Re: Configuring HDP for a moderate memory

@Dmitry Otblesk

If you want to use HDP for learning I'd suggest you download a copy HDP sandbox from this link http://hortonworks.com/downloads/#sandbox and play with to begin.

Re: Configuring HDP for a moderate memory

Rising Star

@knarendran

We wanted to learn how to work with a real cluster (more than one node), including how to set it up and configure it. Sandbox is not very useful for this goal.

Highlighted

Re: Configuring HDP for a moderate memory

@Dmitry Otblesk

The memory requirement is not driven by little data in the POC environment. It is driven by so many services running in the ecosystem. Even the sandbox requires 8 GB. When you installed this single node cluster, most likely Ambari report some memory limits.

It is not a surprise that such a complex ecosystem requires 8 GB. Even Eclipse IDE requires this days 8 GB to run nicely and that is a development tool.

My 2c is to reset your expectations and as suggested below. Stop some of the services and keep up only those absolutely needed. Sandbox with 8 GB is enough to run through tutorials with small data. If you want to do more than that 16 GB is the minimum.

Highlighted

Re: Configuring HDP for a moderate memory

Rising Star

@Constantin Stanca

Ok, thanks!

Is there any document, which outlines minimal hardware requirements for HDP?

So far I was only able to find document with recommended requirements. These are very different from minimal requirements as recommended requirements have production environment in mind.

Highlighted

Re: Configuring HDP for a moderate memory

Rising Star

@Constantin Stanca

>most likely Ambari report some memory limits

Where exactly Ambari reports this? If you are talking about Dasboard then no, it does not currently show any memory complaints. There are no alerts shown either. Is there any particular place in Ambari where I can check this?

Don't have an account?
Coming from Hortonworks? Activate your account here