Building out a cluster is a bit of puzzle and gets especially hairy when the cluster is small, say < 12 nodes. For good or bad this is how I tend to generalize my approach: 1. There are master services (NN, RM) and there are client services (Spark, Hive). Think HA and redundancy for master services. Best not to co-locate multiple master services since that could cause a SPOF. Do not co-locate master and worker (HDFS) services. 2. Services such as Storm, HBase, and Solr will do better on dedicated servers because of their high resource requirements. Not required of course, but be cognizant of the trade-offs. 3. Spark is memory bound, Kafka is IO bound, Storm is CPU bound. When looking at co-locating services try to mix and match. Don't put 2 memory bound services on a single server. 4. I prefer to have a small, dedicated Ambari server. Seems cleaner to me but your mileage may vary. 5. Try to use existing database infrastructure for all your metastores, e.g. Oracle. 6. Never use SAN 7. Think about virtualizing master services, edge nodes, and dev. This list is by no means conclusive and every architect will have additional details (e.g. placing the Spark History server on the same server as HiveServer2). When it really comes down to it you can plan for the worst and hope for the best. Your cluster WILL change over time...guaranteed. Of course you could just deploy in Azure HDInsight and be done with it.... 😉 ... View more

Hi @Jon Maestas. Executing the following should resolve the issue. #Set file-max; no. of open files for single user sudo sh -c 'echo "* soft nofile 200000" >> /etc/security/limits.conf' sudo sh -c 'echo "* hard nofile 200000" >> /etc/security/limits.conf' sudo sh -c 'echo "200000" >> /proc/sys/fs/file-max' sudo sh -c 'echo "fs.file-max=65536" >> /etc/sysctl.conf' #Set process-max sudo sh -c 'echo "* soft nproc 8192" >> /etc/security/limits.conf' sudo sh -c 'echo "* hard nproc 16384" >> /etc/security/limits.conf' sudo sh -c 'echo "* soft nproc 16384" >> /etc/security/limits.d/90-nproc.conf' # ULIMITS to be set sudo sh -c 'echo ULIMITS adjustments' sudo sh -c 'echo "hdfs - nofile 32768" >> /etc/security/limits.conf' sudo sh -c 'echo "mapred - nofile 32768" >> /etc/security/limits.conf' sudo sh -c 'echo "hbase - nofile 32768" >> /etc/security/limits.conf' sudo sh -c 'echo "hdfs - nproc 32768" >> /etc/security/limits.conf' sudo sh -c 'echo "mapred - nproc 32768" >> /etc/security/limits.conf' sudo sh -c 'echo "hbase - nproc 32768" >> /etc/security/limits.conf' # ... View more

@Sunit Gupta Here are some good resources: http://thornydev.blogspot.com/2013/07/querying-json-records-via-hive.html http://engineering.skybettingandgaming.com/2015/01/20/parsing-json-in-hive/ ... View more

@mrizvi could you please attach your code? Thanks. ... View more

@Anurag Setia This doesn't directly answer your question but I would advise against installing HDP on Windows. This option will be deprecated. I'd suggest installing the sandbox if you would like to get familiar with the HDP platform. http://hortonworks.com/products/sandbox/#downloads ... View more

@charan tej At one point you could use Microsoft System Center to monitor HDP on Windows. https://cwiki.apache.org/confluence/display/AMBARI/Ambari+SCOM+Management+Pack. It looks like it hasn't been updated in awhile. Because of the lack of Ambari and Kerberos, we recommend not running HDP on Windows. ... View more