Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Contributor
1. Install Java 8. Mac should have the Java 6 installed with the OS. We recommend you switch to Java 8 as Hadoop trunk branch does.
2. Install Xcode from the App Store
3. Install homebrew, the missing package manager for Mac OS X
4. Install all necessary system tools/libraries (mainly maven, make)
brew install maven autoconf automake cmake libtool 
5. Install the protobuffer package which is heavily used in Hadoop IPC layer.
Install homebrew versions
brew tap homebrew/versions
Install the protobuf package version 2.5
brew install homebrew/versions/protobuf250 
6. Clone Hadoop git repository
git clone git://git.apache.org/hadoop.git 
7. Build the Hadoop project using maven
mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true 
8. Install IntelliJ IDEA IDE. This is much better than Eclipse.
9. Create a new project using the existing source. Do use the Maven option if asked.
10. Start coding! Bonus followup: there is a known bug in the TestGetTrimmedStringCollection and TestGetStringCollection methods. Can you find them?
10,049 Views
Comments
avatar
Cloudera Employee

I would like to add that you might have to edit the IntelliJ VM settings, if not done already. This is to avoid out of memory errors and also improve performance. This can be done by editing the idea.vmoptions file in one of the following ways:

  • Use the main menu command Help | Edit Custom VM Options to create a copy of the idea.vmoptions file in the user home.
  • Copy the existing file from the IntelliJ IDEA installation folder somewhere and save the path to this location in the environment variable IDEA_VM_OPTIONS.
  • Edit (or create) idea.vmoptions file in /Users/<username>/Library/Preferences/IdeaIC2016.1 directory.

Change the VM properties as follows.

-Xms4096m
-Xmx4096m
-XX:MaxPermSize=2048m
-XX:ReservedCodeCacheSize=1024m
-XX:+UseCompressedOops

Note that these are just sample values and you can have your own custom set of values for these parameters.

Also note that there is a limit on what theses values can be set to. Setting the ReservedCodeCacheSize to 4096M in a 16GB RAM machine will prevent IntelliJ from booting up (It can at most be 2048M).

avatar
Contributor

Thanks for your tips!

avatar

Hello @Mingliang Liu. Nice article! I'd like to add that in step 7, when doing a distro build, I often like to speed it up a little more by passing the argument -Dmaven.javadoc.skip=true. As long as I don't need to inspect JavaDoc changes, this can make the build complete faster.

Good writeup @Mingliang Liu. In addition to what @Chris Nauroth said, I also add -Dmaven.site.skip=true.

mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true
avatar
Contributor

Thank you @Chris Nauroth and @Arpit Agarwal for your helpful comments. I updated the article, and my MAVEN_OPTS env variable 🙂

avatar
New Contributor

In MacOS Sierra you need to setup openSSL manually before running mvn package command.

brew update
brew install openssl

This will install the openssl + development libraries / headers to /usr/local/opt/openssl but wont link them, even if you try brew link --force. But this will put the development headers and libraries in /usr/local/opt/openssl. So you should to the following before building it.

export OPENSSL_ROOT_DIR=/usr/local/opt/openssl

Then run the mvn package command

avatar
@Mingliang Liu

Great article!
I faced the following error when I tried to build using maven(step 7):
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

Here is how I resolved this by creating a symlink:

ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc /usr/local/bin

avatar

This saved me a good couple hours! Thanks!

avatar
New Contributor

The step 5 is stale. The latest one is run command `brew install protobuf@2.5`

avatar
New Contributor

@Dinesh Chitlangia

You are right. The current homebrew has deprecated the `homebrew/versions` and the new migrated protobuf@2.5 installation does not change the default bin in PATH. As you suggested, you can link it in your PATH, or follow the instruction of `brew info protobuf@2.5`

Version history
Last update:
‎06-01-2016 01:54 PM
Updated by:
Contributors