Support Questions
Find answers, ask questions, and share your expertise

How can i install CDH5 on mac, Yosemite?

New Contributor

How can i install CDH5 on mac, yosemite? Is there any way where i can get one click installer to install all modules? 

3 REPLIES 3

Re: How can i install CDH5 on mac, Yosemite?

There is no one-click installer for OS X, so you're best off installing
using the tarballs. Read the section titled "Unmanaged Deployment" in this
article

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_installati...

Regards,
Gautam Gopalakrishnan

Re: How can i install CDH5 on mac, Yosemite?

New Contributor

1. Check the box for Remote Login to enable the service.
2. Allow access for: “Only these users: Administrators”
3. Enable password-less ssh login to localhost for MRv1 and HBase.
4. Open your terminal.
5. Generate an rsa or dsa key.
6. ssh-keygen -t rsa -P ""
7. Continue through the key generator prompts (use default options).
8. Test: ssh localhost
9. Homebrew:
10. Mac OSX and ease of install of MySQL for Hive. Snappy is commonly used within HBase, HDFS, and MapReduce for compression and decompression.
11. CDH:
12. Finally, the easy part: The CDH tarballs are very nicely packaged and easily downloadable from Cloudera’s repository. I’ve downloaded tarballs for CDH 5.1.0.
13. Although Mac OSX’s “Make Alias” feature is bi-directional, do not use it, but instead use your command-line ln -s command, such as ln -s source_file target_file.
14. /Users/jordanh/cloudera/
15. cdh5.1/
16. hadoop -> /Users/jordanh/cloudera/lib/hadoop-2.3.0-cdh5.1.0
17. hbase -> /Users/jordanh/cloudera/lib/hbase-0.98.1-cdh5.1.0
18. hive -> /Users/jordanh/cloudera/lib/hive-0.12.0-cdh5.1.0
19. zookeeper -> /Users/jordanh/cloudera/lib/zookeeper-3.4.5-cdh4.7.0
20. ops/
21. dn
22. logs/hadoop, logs/hbase, logs/yarn
23. nn/
24. pids
25. tmp/
26. zk/
27. You’ll notice above that you’ve created a handful of directories under a folder named ops. You’ll use them later to customize the configuration of the essential components for running Hadoop. Set your environment properties according to the paths where you’ve exploded your tarballs.
28. <configuration>
29. <property>
30. <name>dfs.namenode.name.dir</name>
31. <value>/Users/jordanh/cloudera/ops/nn</value>
32. <description>Determines where on the local filesystem the DFS name node
33. should store the name table(fsimage). If this is a comma-delimited list
34. of directories then the name table is replicated in all of the
35. directories, for redundancy. </description>
36. </property>
37. <property>
38. <name>dfs.datanode.data.dir</name>
39. <value>/Users/jordanh/cloudera/ops/dn/</value>
40. <description>Determines where on the local filesystem an DFS data node
41. should store its blocks. If this is a comma-delimited
42. list of directories, then data will be stored in all named
43. directories, typically on different devices.
44. Directories that do not exist are ignored.
45. </description>
46. </property>
47. <property>
48. <name>dfs.datanode.http.address</name>
49. <value>localhost:50075</value>
50. <description>
51. The datanode http server address and port.
52. If the port is 0 then the server will start on a free port.
53. </description>
54. </property>
55. <property>
56. <name>dfs.replication</name>
57. <value>1</value>
58. <description>Default block replication.
59. The actual number of replications can be specified when the file is created.
60. The default is used if replication is not specified in create time.
61. The primary difference with configuration between running HBase in distributed or standalone mode is with the hbase.cluster.distributed property in hbase-site.xml. Note regarding hbase-site.xml: Property hbase.cluster.distributed is set to false by default and will launch in standalone mode. Also, hbase.zookeeper.quorum is set to localhost by default and does not need to be overridden in our scenario.
62. Pulling it All Together:
63. few steps, we will start/stop the services and provide examples to ensure each service is operating correctly. The steps are listed in the specific order for initialization in order to adhere to dependencies. The order could be reversed for halting the services.
64. Service HDFS
65. NameNode
66. format: hdfs namenode -format
67. start: hdfs namenode
68. stop: Ctrl-C
69. url: http://localhost:50070/dfshealth.html
70. DataNode
71. start: hdfs datanode
72. stop: Ctrl-C
73. url: http://localhost:50075/browseDirectory.jsp?dir=%2F&nnaddr=127.0.0.1:8020
74. Test
75. hadoop fs -mkdir /tmp
76. hadoop fs -put /path/to/local/file.txt /tmp/
77. hadoop fs -cat /tmp/file.txt
78. Service YARN
79. ResourceManager
80. start: yarn resourcemanager
81. stop: Ctrl-C
82. url: http://localhost:8088/cluster
83. NodeManager
84. start: yarn nodemanager
85. stop: Ctrl-C
86. url: http://localhost:8042/node
87. MapReduce Job History Server
88. start: mapred historyserver, mr-jobhistory-daemon.sh start historyserver
89. stop: Ctrl-C, mr-jobhistory-daemon.sh stop historyserver
90. url: http://localhost:19888/jobhistory/app
91. Test Vanilla YARN Application
92. Shell
93.
94. 1 95. hadoop jar $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.1.0.jar -appname DistributedShell -jar $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0-cdh5.1.0.jar -shell_command "ps wwaxr -o pid,stat,%cpu,time,command | head -10" -num_containers 2 -master_memory 1024
96. Test MRv2 YARN TestDFSIO
97. Shell
98.
99. 1100. 2 101. hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -size 1GB
102. hadoop org.apache.hadoop.fs.TestDFSIO -read -nrFiles 5 -size 1GB
103. Test MRv2 YARN Terasort/Teragen
104. Shell
105.
106. 1107. 2 108. hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar teragen 100000000 /tmp/eval/teragen
109. hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar terasort /tmp/eval/teragen /tmp/eval/terasort
110. Test MRv2 YARN Pi
111. Shell
112.
113. 1 114. hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0-cdh5.1.0.jar pi 100 100
115. Service HBase
116. HBase Master/RegionServer/ZooKeeper
117. start: start-hbase.sh
118. stop: stop-hbase.sh
119. logs: /Users/jordanh/cloudera/ops/logs/hbase/
120. url: http://localhost:60010/master-status
121. Test
122. Shell
123.
124. 1
125. 2
126. 3
127. 4
128. 5
129. 6
130. 7
131. 8
132. 9
133. 10
134. 11 135. hbase shell
136. create 'URL_HITS', {NAME=>'HOURLY'},{NAME=>'DAILY'},{NAME=>'YEARLY'}
137. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'HOURLY:2014090110', '10'
138. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'HOURLY:2014090111', '5'
139. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'HOURLY:2014090112', '30'
140. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'HOURLY:2014090113', '80'
141. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'HOURLY:2014090114', '7'
142. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'DAILY:20140901', '10012'
143. put 'URL_HITS', 'com.cloudera.blog.osx.localinstall', 'YEARLY:2014', '93310101'

144. scan 'URL_HITS'
145. Kite SDK Test
146. Get familiar with the Kite SDK by trying out this example that loads data to both HDFS and then HBase. Note that there are a few common issues on your OSX that may surface when running through the Kite SDK example. They can be easily resolved with additional setup/config as specified below.
147. Problem: NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException
148. Resolution: Fix your classpath by making sure to set HIVE_HOME and HCAT_HOME in your environment.
149. Shell
150.
151. 1152. 2 153. export HIVE_HOME="/Users/jordanh/cloudera/${CDH}/hive"
154. export HCAT_HOME="/Users/jordanh/cloudera/${CDH}/hive/hcatalog"
155. Problem: InvocationTargetException Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
156. Resolution: Snappy libraries are not compiled for Mac OSX out of the box. A Snappy Java port was introduced in CDH 5 and likely will require to be recompiled on your machine.
157. Shell
158.
159. 1
160. 2
161. 3162. 4163. 5 164. git clone https://github.com/xerial/snappy-java.git
165. cd snappy-java
166. make

167. cp target/snappy-java-1.1.1.3.jar $HADOOP_HOME/share/hadoop/common/lib/asnappy-java-1.1.1.3.jar
168. Landing Page
169. Creating a landing page will help consolidate all the HTTP addresses of the services that you’re running. Please note that localhost can be replaced with your local hostname (such as jakuza-mbp.local).
170. Service Apache HTTPD
171. start: sudo -s launchctl load -w /System/Library/LaunchDaemons/org.apache.httpd.plist
172. stop: sudo -s launchctl unload -w /System/Library/LaunchDaemons/org.apache.httpd.plist
173. logs: /var/log/apache2/

 

 macos-f31.png

 

Re: How can i install CDH5 on mac, Yosemite?

New Contributor

Is SSH Key gen mandatory while installing hadoop.

 

I'm not able to install ssh.Its shrowing an error while installtion.