Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Building Hadoop on MacOS : An Ant BuildException occurred in project hadoop-hdfs-native-client

Labels:
avatar
author-rank dineshc author-rank
Guru

Created ‎05-11-2018 04:45 AM

I am trying to compile and build hadoop source on MacOS.

Here is the complete error trace.

[exec] Scanning dependencies of target rpc_obj
     [exec] [ 32%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/rpc_connection_impl.cc.o
     [exec] [ 32%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/rpc_engine.cc.o
     [exec] [ 32%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/namenode_tracker.cc.o
     [exec] [ 33%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/request.cc.o
     [exec] [ 33%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/sasl_protocol.cc.o
     [exec] [ 34%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/sasl_engine.cc.o
     [exec] [ 34%] Building CXX object main/native/libhdfspp/lib/rpc/CMakeFiles/rpc_obj.dir/cyrus_sasl_engine.cc.o
     [exec] [ 34%] Built target rpc_obj
     [exec] Scanning dependencies of target rpc
     [exec] [ 35%] Linking CXX static library librpc.a
     [exec] [ 35%] Built target rpc
     [exec] [ 35%] Building CXX object main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/ioservice_impl.cc.o
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:19:
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.h:22:
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:109:30: error: no type named 'function' in namespace 'std'
     [exec]   virtual void PostTask(std::function<void(void)> asyncTask) = 0;
     [exec]                         ~~~~~^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:109:38: error: expected ')'
     [exec]   virtual void PostTask(std::function<void(void)> asyncTask) = 0;
     [exec]                                      ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:109:24: note: to match this '('
     [exec]   virtual void PostTask(std::function<void(void)> asyncTask) = 0;
     [exec]                        ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:117:10: error: no member named 'function' in namespace 'std'
     [exec]     std::function<void(void)> typeEraser = func;
     [exec]     ~~~~~^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:117:28: error: expected '(' for function-style cast or type construction
     [exec]     std::function<void(void)> typeEraser = func;
     [exec]                        ~~~~^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:117:31: error: use of undeclared identifier 'typeEraser'
     [exec]     std::function<void(void)> typeEraser = func;
     [exec]                               ^
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:19:
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.h:45:54: error: non-virtual member function marked 'override' hides virtual member function
     [exec]   void PostTask(std::function<void(void)> asyncTask) override;
     [exec]                                                      ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:109:16: note: hidden overloaded virtual function 'hdfs::IoService::PostTask' declared here: type mismatch at 1st parameter ('int' vs 'std::function<void ()>')
     [exec]   virtual void PostTask(std::function<void(void)> asyncTask) = 0;
     [exec]                ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:34:14: error: allocating an object of abstract class type 'hdfs::IoServiceImpl'
     [exec]   return new IoServiceImpl();
     [exec]              ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:109:16: note: unimplemented pure virtual method 'PostTask' in 'IoServiceImpl'
     [exec]   virtual void PostTask(std::function<void(void)> asyncTask) = 0;
     [exec]                ^
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:19:
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.h:22:
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:61:
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:2143:9: error: field type 'hdfs::IoServiceImpl' is an abstract class
     [exec]     _T2 __second_;
     [exec]         ^
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:2302:15: note: in instantiation of template class 'std::__1::__libcpp_compressed_pair_imp<std::__1::allocator<hdfs::IoServiceImpl>, hdfs::IoServiceImpl, 1>' requested here
     [exec]     : private __libcpp_compressed_pair_imp<_T1, _T2>
     [exec]               ^
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:3816:36: note: in instantiation of template class 'std::__1::__compressed_pair<std::__1::allocator<hdfs::IoServiceImpl>, hdfs::IoServiceImpl>' requested here
     [exec]     __compressed_pair<_Alloc, _Tp> __data_;
     [exec]                                    ^
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:4444:26: note: in instantiation of template class 'std::__1::__shared_ptr_emplace<hdfs::IoServiceImpl, std::__1::allocator<hdfs::IoServiceImpl> >' requested here
     [exec]     ::new(__hold2.get()) _CntrlBlk(__a2, _VSTD::forward<_Args>(__args)...);
     [exec]                          ^
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:4810:29: note: in instantiation of function template specialization 'std::__1::shared_ptr<hdfs::IoServiceImpl>::make_shared<>' requested here
     [exec]     return shared_ptr<_Tp>::make_shared(_VSTD::forward<_Args>(__args)...);
     [exec]                             ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:38:15: note: in instantiation of function template specialization 'std::__1::make_shared<hdfs::IoServiceImpl>' requested here
     [exec]   return std::make_shared<IoServiceImpl>();
     [exec]               ^
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:19:
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.h:22:
     [exec] In file included from /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/ioservice.h:61:
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:4447:28: error: assigning to 'std::__1::__shared_weak_count *' from incompatible type 'pointer' (aka 'std::__1::__shared_ptr_emplace<hdfs::IoServiceImpl, std::__1::allocator<hdfs::IoServiceImpl> > *')
     [exec]     __r.__cntrl_ = __hold2.release();
     [exec]                    ~~~~~~~~^~~~~~~~~
     [exec] /Library/Developer/CommandLineTools/usr/include/c++/v1/memory:4810:29: note: in instantiation of function template specialization 'std::__1::shared_ptr<hdfs::IoServiceImpl>::make_shared<>' requested here
     [exec]     return shared_ptr<_Tp>::make_shared(_VSTD::forward<_Args>(__args)...);
     [exec]                             ^
     [exec] /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/ioservice_impl.cc:38:15: note: in instantiation of function template specialization 'std::__1::make_shared<hdfs::IoServiceImpl>' requested here
     [exec]   return std::make_shared<IoServiceImpl>();
     [exec]               ^
     [exec] 9 errors generated.
     [exec] make[2]: *** [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/ioservice_impl.cc.o] Error 1
     [exec] make[1]: *** [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/all] Error 2
     [exec] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [ 29.737 s]
[INFO] Apache Hadoop HttpFS ............................... SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
[INFO] Apache Hadoop HDFS-RBF ............................. SKIPPED
[INFO] Apache Hadoop HDFS Project ......................... SKIPPED
[INFO] Apache Hadoop YARN ................................. SKIPPED
[INFO] Apache Hadoop YARN API ............................. SKIPPED
[INFO] Apache Hadoop YARN Common .......................... SKIPPED
[INFO] Apache Hadoop YARN Registry ........................ SKIPPED
[INFO] Apache Hadoop YARN Server .......................... SKIPPED
[INFO] Apache Hadoop YARN Server Common ................... SKIPPED
[INFO] Apache Hadoop YARN NodeManager ..................... SKIPPED
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31.310 s
[INFO] Finished at: 2018-05-11T00:27:30-04:00
[INFO] Final Memory: 67M/557M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec failonerror="true" dir="/Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target" executable="make">... @ 9:131 in /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec failonerror="true" dir="/Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target" executable="make">... @ 9:131 in /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part ...<exec failonerror="true" dir="/Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target" executable="make">... @ 9:131 in /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
	at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	... 20 more
Caused by: /Users/dc/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml:9: exec returned: 2
	at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
	at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
	at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
	at org.apache.tools.ant.Task.perform(Task.java:348)
	at org.apache.tools.ant.Target.execute(Target.java:390)
	at org.apache.tools.ant.Target.performTasks(Target.java:411)
	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
	at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
	at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
	... 22 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

I have done the following steps:

Installed XCode and Command Line Developer tools

Installed Protobuf 2.5.0

Followed this HCC article for all other installations: https://community.hortonworks.com/articles/36832/setting-hadoop-development-environment-on-mac-os-x....

Clone github.com/apache/hadoop

cd /.../hadoop

mvn clean package -DskipTests

mvn package -Pdist -Pnative -Dtar -DskipTests

It appears like the C++ code is failing to compile.

Appreciate any help

3,099 Views
0 Kudos
1 REPLY 1

avatar
author-rank gnovak author-rank
Expert Contributor

Created ‎05-15-2018 07:49 AM

@Dinesh Chitlangia

Unfortunately the native build on OS X is broken by HDFS-13403 at this moment on trunk. You have two options:

  1. If you don't need native build, you can build hadoop without the -Pnative option successfully.
  2. The build issue is fixed by HDFS-13534, but it's not merged yet (at the time of writing this answer). You can either wait until it gets merged, or apply it manually:
wget https://issues.apache.org/jira/secure/attachment/12922534/HDFS-13534.001.patch
git apply HDFS-13534.001.patch
2,636 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements