Skip to main content
mliu
mliu
Gold ⭐️
June 1, 2016

Setting Hadoop development environment on Mac OS X 10

  • October 5, 2026 at 9:10 AM
  • 10 replies
  • 16082 views
  • Translate All
1. Install Java 8. Mac should have the Java 6 installed with the OS. We recommend you switch to Java 8 as Hadoop trunk branch does.
2. Install Xcode from the App Store
3. Install homebrew, the missing package manager for Mac OS X
4. Install all necessary system tools/libraries (mainly maven, make)

brew install maven autoconf automake cmake libtool

5. Install the protobuffer package which is heavily used in Hadoop IPC layer.
Install homebrew versions

brew tap homebrew/versions

Install the protobuf package version 2.5

brew install homebrew/versions/protobuf250

6. Clone Hadoop git repository

git clone git://git.apache.org/hadoop.git

7. Build the Hadoop project using maven

mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true

8. Install IntelliJ IDEA IDE. This is much better than Eclipse.
9. Create a new project using the existing source. Do use the Maven option if asked.
10. Start coding! Bonus followup: there is a known bug in the TestGetTrimmedStringCollection and TestGetStringCollection methods. Can you find them?

    10 replies

    hanishakoneru
    Silver ⭐️
    hanishakoneru
    Silver ⭐️
    June 3, 2016

    I would like to add that you might have to edit the IntelliJ VM settings, if not done already. This is to avoid out of memory errors and also improve performance. This can be done by editing the idea.vmoptions file in one of the following ways:

    • Use the main menu command Help | Edit Custom VM Options to create a copy of the idea.vmoptions file in the user home.
    • Copy the existing file from the IntelliJ IDEA installation folder somewhere and save the path to this location in the environment variable IDEA_VM_OPTIONS.
    • Edit (or create) idea.vmoptions file in /Users/<username>/Library/Preferences/IdeaIC2016.1 directory.

    Change the VM properties as follows.

    -Xms4096m
    -Xmx4096m
    -XX:MaxPermSize=2048m
    -XX:ReservedCodeCacheSize=1024m
    -XX:+UseCompressedOops

    Note that these are just sample values and you can have your own custom set of values for these parameters.

    Also note that there is a limit on what theses values can be set to. Setting the ReservedCodeCacheSize to 4096M in a 16GB RAM machine will prevent IntelliJ from booting up (It can at most be 2048M).

    Translate
      mliu
      mliuAuthor
      Gold ⭐️
      June 3, 2016

      Thanks for your tips!

      Translate
      cnauroth
      cnauroth
      Gold ⭐️
      June 9, 2016

      Hello @Mingliang Liu. Nice article! I'd like to add that in step 7, when doing a distro build, I often like to speed it up a little more by passing the argument -Dmaven.javadoc.skip=true. As long as I don't need to inspect JavaDoc changes, this can make the build complete faster.

      Translate
        ArpitAgarwal
        Gold ⭐️
        ArpitAgarwal
        Gold ⭐️
        June 10, 2016

        Good writeup @Mingliang Liu. In addition to what @Chris Nauroth said, I also add -Dmaven.site.skip=true.

        mvn clean package -Pdist,native -Dtar -DskipTests=true -Dmaven.site.skip=true -Dmaven.javadoc.skip=true

        Translate
          mliu
          mliuAuthor
          Gold ⭐️
          June 14, 2016

          Thank you @Chris Nauroth and @Arpit Agarwal for your helpful comments. I updated the article, and my MAVEN_OPTS env variable 🙂

          Translate
            er_soundararaja
            er_soundararaja
            Silver ⭐️
            April 17, 2017

            In MacOS Sierra you need to setup openSSL manually before running mvn package command.

            brew update
            brew install openssl

            This will install the openssl + development libraries / headers to /usr/local/opt/openssl but wont link them, even if you try brew link --force. But this will put the development headers and libraries in /usr/local/opt/openssl. So you should to the following before building it.

            export OPENSSL_ROOT_DIR=/usr/local/opt/openssl

            Then run the mvn package command

            Translate
              dineshc
              Gold ⭐️
              dineshc
              Gold ⭐️
              October 8, 2017

              This saved me a good couple hours! Thanks!

              Translate
              dineshc
              Gold ⭐️
              dineshc
              Gold ⭐️
              October 8, 2017
              @Mingliang Liu

              Great article!
              I faced the following error when I tried to build using maven(step 7):
              Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

              Here is how I resolved this by creating a symlink:

              ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc /usr/local/bin

              Translate
                liuml07
                liuml07
                Silver ⭐️⭐️
                May 13, 2018

                @Dinesh Chitlangia

                You are right. The current homebrew has deprecated the `homebrew/versions` and the new migrated protobuf@2.5 installation does not change the default bin in PATH. As you suggested, you can link it in your PATH, or follow the instruction of `brew info protobuf@2.5`

                Translate
                liuml07
                liuml07
                Silver ⭐️⭐️
                May 13, 2018

                The step 5 is stale. The latest one is run command `brew install protobuf@2.5`

                Translate
                Terms & ConditionsAccessibility statement