Member since
02-27-2017
171
Posts
9
Kudos Received
0
Solutions
02-24-2017
10:34 AM
I am planning to setup a 4 node Production Cluster on Azure VM's. I am planning to have 1 edge Node, 1 Master Node and 2 Slave Nodes. I wanted to setup below mentioned services on that cluster. 1) Namenode 2) Oozie 3) DataNode 4) Yarn 5) Spark 6) Ranger 7) Atlas 😎 Knox 9) Hbase 10) SAP Hana Vora 11) Zookeeper I am actually looking out for any guidelines on Memory, Cores and Storage to be required for different services of hadoop as mentioned above. I need to buy 4 VM's on Azure but i want to understand from Infrastructure perspective that how much memory, cores, Storage would be optimal for above mentioned hadoop services(service wise) ,keeping in mind more services can also be added in future. Is there any reference documentation/link? Any help would be appreciated. Thanks
... View more
Labels:
- Labels:
-
Apache Ambari
02-23-2017
09:24 AM
@apappu I am also facing the same error. I am trying to add Ranger-KMS service from Ambari and while deploying i am seeing the error as "/var/lib/ambari-server/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py" does not exist. I checked the above mentioned path and script at above path does not exist. I have downloaded HDP 2.5 Sandbox for Virtualbox and both ambari-server and ambari-agent is running on localhost and both are working fine. I think this script should come as part of HDP 2.5. It seems to be a issue in Sandbox image. Ambari Version 2.4.0. Could you please suggest the solution? Thanks
... View more
02-22-2017
12:02 PM
@Jay SenSharma I was going through below mentioned link if i need to setup CA signed certificate. I am going to mention list of steps to be performed on 3 node cluster(excluding edge node). Please validate keytool -genkey -keyalg RSA -alias c6401 -keystore /tmp/keystore.jks -storepass bigdata -validity 360-keysize 2048 (To be generated for each Node(master, slave1 and slave2)) keytool -certreq -alias c6401 -keyalg RSA -file /tmp/c6401.csr -keystore /tmp/keystore.jks -storepass bigdata( csr file to be generated for each node keystore.jks file) Nowget the singed cert from CA - file name is/tmp/c6401.crt( How to get this certificate for each node/or single node from CA)? Import the root cert to JKS first.(Ignoreif it already present) keytool -import-alias root -file /tmp/ca.crt -keystore /tmp/keystore.jks (How to get root cert)? Repeat step4 for intermediate cert if there is any.? Import signed cert into JKS. keytool -import-alias c6401 -file /tmp/c6401.crt -keystore /tmp/keystore.jks -storepass bigdata ( to be done for each node)? keytool -import-alias root -file /tmp/ca.crt -keystore /tmp/truststore.jks -storepass bigdata (To be done for each node)? Kindly let me know if this is correct approach? Or is there any other link for multi node? Thanks for great help!! https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html
... View more
02-22-2017
11:36 AM
@Jay SenSharma yeah its a self signed certificate. And now i am seeing errors in installing Ranger as well because of below error. I assume this is due to ssl. right? java.lang.IllegalStateException: Can't get secure connection to master:50470/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState. Truststore path or password is not set
... View more
02-22-2017
11:10 AM
@Jay SenSharma I am seeing that at the top of browser address bar. Although the connection is going on port 50470 and 50475 using https but i dont understand why is it showing as connection insecure at top of browser address bar? Does that make any significance here? Thanks not-secure.png
... View more
02-22-2017
09:36 AM
@Jay SenSharma Could you please anwser query mentioned above? Thanks
... View more
02-22-2017
08:46 AM
@Jay SenSharma I am able to resolve the issue. Actually the problem was that i modified ssl-server.xml and ssl-client.xml from hadoop cli on namenode. Now i did it using Ambari. And provide the correct paths of keystore files and all my services are running. 🙂 Thanks for the help. Just another thing how to test this SSL. I am able to open NameNode UI at port 50470 and DataNode UI at 50475. But connection is still showing as insecure. Any reasons why? Also Can we enable SSL for Ranger and Spark? Thanks
... View more
02-22-2017
08:09 AM
@Jay SenSharma Got this error now. Dont know why it is looking for /etc/security/serverkeys/keystore.jks. Although i have already modified this property from ambari. One more thing. Since i have tried changing these properties from hadoop cli. Do you want me to modify them from Ambari dashboard. I think that might be the issue since it is looking at default location? 2017-02-22 08:04:40,697 ERROR namenode.NameNode (NameNode.java:main(1759)) - Failed to start namenode.
java.io.FileNotFoundException: /etc/security/serverKeys/keystore.jks (No such file or directory)
... View more
02-22-2017
08:03 AM
@Sagar Shimpi Thanks for replying. I have checked the ambari-agent.ini file at all hosts and it is same and ambari-server IP is also correct. All my services were running before setting up SSL. I stopped HDFS, Yarm, Mapreduce and made ssl related changes as mentioned up, since then i am unable to bring up the services due to error mentioned in question. Thanks Rahul
... View more
02-22-2017
08:02 AM
@Jay SenSharma I have followed below mentioned link to setup SSL for HDFS and Yarn. http://getthekt.com/2016/02/securing-hadoop-cluster-part-1-ssltls-for-hdfs-and-yarn/ I assume this is almost same as the one you mentioned. After that i stopped HDFS, Yarn, Mapreduce from Ambari. And modified hdfs-site.xml, yarn-site.xml, mapred-site.xml from Ambari itself. But i have modified ssl-server.xml and ssl-client.xml from Namenode CLI at path /etc/hadoop/conf. And i copied ssl files from master node to slave nodes at same path /etc/hadoop/conf. And then i am trying to restart the HDFS and it is giving above error. I am unable to understand why heartbeats are stopped although ambari-agents are running on all nodes.(Slave1, Slave2 and Master)? Please let me know if there is any issue with above mentioned steps. I appreciate your help!! Thanks
... View more