Setting Hadoop development environment on Mac OS X 10

[#1]

1. Install Java 8. Mac should have the Java 6 installed with the OS. We recommend you switch to Java 8 as Hadoop trunk branch does.

2. Install Xcode from the App Store

3. Install homebrew, the missing package manager for Mac OS X

4. Install all necessary system tools/libraries (mainly maven, make)

brew install maven autoconf automake cmake libtool 

5. Install the protobuffer package which is heavily used in Hadoop IPC layer.

Install homebrew versions

brew tap homebrew/versions

Install the protobuf package version 2.5

brew install homebrew/versions/protobuf250 

6. Clone Hadoop git repository

git clone git://git.apache.org/hadoop.git 

7. Build the Hadoop project using maven

mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true 

8. Install IntelliJ IDEA IDE. This is much better than Eclipse.

9. Create a new project using the existing source. Do use the Maven option if asked.

10. Start coding! Bonus followup: there is a known bug in the TestGetTrimmedStringCollection and TestGetStringCollection methods. Can you find them?

[#2]

I would like to add that you might have to edit the IntelliJ VM settings, if not done already. This is to avoid out of memory errors and also improve performance. This can be done by editing the idea.vmoptions file in one of the following ways:

• Use the main menu command Help | Edit Custom VM Options to create a copy of the idea.vmoptions file in the user home.
• Copy the existing file from the IntelliJ IDEA installation folder somewhere and save the path to this location in the environment variable IDEA_VM_OPTIONS.
• Edit (or create) idea.vmoptions file in /Users/<username>/Library/Preferences/IdeaIC2016.1 directory.

Change the VM properties as follows.

-Xms4096m
-Xmx4096m
-XX:MaxPermSize=2048m
-XX:ReservedCodeCacheSize=1024m
-XX:+UseCompressedOops

Note that these are just sample values and you can have your own custom set of values for these parameters.

Also note that there is a limit on what theses values can be set to. Setting the ReservedCodeCacheSize to 4096M in a 16GB RAM machine will prevent IntelliJ from booting up (It can at most be 2048M).

[#3]

Thanks for your tips!

[#4]

Hello @Mingliang Liu. Nice article! I'd like to add that in step 7, when doing a distro build, I often like to speed it up a little more by passing the argument -Dmaven.javadoc.skip=true. As long as I don't need to inspect JavaDoc changes, this can make the build complete faster.

[#5]

Good writeup @Mingliang Liu. In addition to what @Chris Nauroth said, I also add -Dmaven.site.skip=true.

mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true

[#6]

Thank you @Chris Nauroth and @Arpit Agarwal for your helpful comments. I updated the article, and my MAVEN_OPTS env variable 🙂

[#7]

In MacOS Sierra you need to setup openSSL manually before running mvn package command.

brew update
brew install openssl

This will install the openssl + development libraries / headers to /usr/local/opt/openssl but wont link them, even if you try brew link --force. But this will put the development headers and libraries in /usr/local/opt/openssl. So you should to the following before building it.

export OPENSSL_ROOT_DIR=/usr/local/opt/openssl

Then run the mvn package command

[#8]

@Mingliang Liu

Great article!
I faced the following error when I tried to build using maven(step 7):
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

Here is how I resolved this by creating a symlink:

ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc /usr/local/bin

[#9]

This saved me a good couple hours! Thanks!

[#10]

The step 5 is stale. The latest one is run command `brew install protobuf@2.5`

[#11]

@Dinesh Chitlangia

You are right. The current homebrew has deprecated the `homebrew/versions` and the new migrated protobuf@2.5 installation does not change the default bin in PATH. As you suggested, you can link it in your PATH, or follow the instruction of `brew info protobuf@2.5`