@R c You state that your server has 12 cores (2 threads per core) and 32 GB RAM. You did not state whether your server is x86. If it is a Power7 CPU then 12 cores is a lot of CPU power. I'll assume is x86. Then you state that you would like 6 nodes with 4 vCPU per node and 8-16 GB per each. The RAM math does not add-up. You would need 48-96 GB to meet that wish. You did not mention anything about the type of storage that you could use. Is it the internal storage of the server, NAS or SAN? How much can you use? For this type of cluster, you would need 25-30 GB for root and logs and separate storage for data nodes, whatever you can afford, but not less than 50-100 GB/data node if you really want to have a big data capable for development cluster. If your resources are really 12 cores (24 vCPU) and 32 GB RAM, that is a really not a great RAM/vCPU ratio. A good rule of thumb is to have minimum 4 GB RAM for each vCPU. That means in your case, it would have been good if you had 96 GB RAM. My response would have been easily: 6 nodes as you wanted, each 4vCPU and 16 GB RAM. I would never allocate less than 8 GB of RAM per node. If you RAM is really 32 GB then, maybe a 4-node cluster which will be memory bound. HBase, Spark and Hive LLAP may have some limitations. Keep in mind that your resources are a reasonable fit for a very small POC/dev environment, however, useable. +++ If any of the responses was helpful, please vote and accept best answer. ... View more

@kishore sanchina HiveQL is a query language and Hive is an execution engine. BigSQL is just another execution engine which can co-exist with Hive and leverage Hive storage model and metastore. Tables created in Hive are visible to Big SQL and vice versa. The major difference? Until recently, the response would have been that Hive requires MapReduce and BigSQL uses a different approach leveraging memory, however, recently Hive uses Tez and even more recently uses LLAP and the difference between them is just that they are just alternatives provides by Community vs. IBM. If you were looking from the functional point of view, SQL functions, then IBM’s Big SQL provides higher degree of ANSI SQL language compatibility, however, Hive is almost there and at lower cost and larger community support. +++ If any response was helpful, please vote/accept best answer. ... View more

@Leonid Fedotov LLAP is still in Tech Preview and having more than one HiveServer2 Interactive is not currently supported. HiveServer2 Interactive service is deployed to the host that you want to run it on. As-is, it is a single point of failure and it supports a single queue. Let's remember that a Tech Preview feature is meant to demonstrate the concept, but it usually falls short on enterprise level features like HA, Security, Resource Management. Those are to be added. It is a long shot to setup outside of Ambari. There are no documents to document such approach. You would have to create multiple HiveServer2 instances and point HiveServer2 interactive to those. I would not go that path. I would wait for Ambari 2.5.0 to have two instances. If any response helped, please vote/accept best answer. +++ It is likely that others may be confusing HiveServer2 and HiveServer2 Interactive services and it is important to understand the difference. I started to respond your question before you and @jss had another pass at it. I will still keep the content below for other HCC users. HiveServer1 is a server client model service, which allow users to connect using Hive CLI interface and using thrift client. It support for remote client connection but only one client can connect at a time. It does not provide session management support and because of thrift API it does not provide concurrency control. HiveServer 2 is also a client and server model and it allows to connect many different clients like thrift. HiveServer2 gives multi-client support where many clients can connect at the same time. Authentication is much better using kerberos. It provides support for JDBC and ODBC driver connection. Beeline CLI is used for connecting to HiveServer2. Interactive HiveServer2has been around for a while and it does not have anything to do with Hive 2. The only thing that got better is that multiple interactive HiveServer2 instances can be created via Ambari UI using recent versions of HDP, I believe post HDP 2.2. In HDP 2.2 and earlier, interactive queues can be set up at the command-line. In HDP 2.3 and later, you can use Ambari (a GUI) to set up interactive queues. Multiple HiveServer2 instances can be used for: Load-balancing and high availability using Zookeeper Running multiple applications with different settings Because HiveServer2 uses its own settings file, using one for ETL operations and another for interactive queries is a common practice. All HiveServer2 instances can share the same MetastoreDB. Consequently, setting up multiple HiveServer2 instances that have embedded metastores is a simple operation. HiveServer2 Interactive This is still HiveServer2 instance, however, it is dedicated to HiveServer2 Interactive service. Check this reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_hive-performance-tuning/content/ch_hive_llap.html, section 8. ... View more

@Jeeva Jeeva Multithreading Programming Model and MapReduce Programming Model are based on fundamentally different principles and both are meant to solve different kinds of data storage and processing problems. Multithreading is based on Parallelization of Processing where as Hadoop takes power by Parallelization of Data. If you assume that the Hadoop ecosystem is only MapReduce and Spark batch, then your understanding is correct . However, the ecosystem includes also real-time streaming tools like Apache Storm which uses multi-threading. However, modern tools handle all these programmatic needs for multi-threading by architecture/design. Their focus is scalability by architecture and design and less by laborious programming efforts. References: http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/ https://www.safaribooksonline.com/blog/2014/01/06/multi-threading-storm +++ If it helped, pls vote/accept best answer ... View more