Hello @william266455
Thanks for using Cloudera Community. Your Post covers an extremely wide domain yet I would like to share my perspective from working with Customers on CDP Public Cloud:
- From the 3 Cloud Vendors, AWS was the Pioneer to start with & as such, I wish to point to the Link [1], which offers the 6 Major Advantages of Cloud Computing from an AWS WhitePaper.
- Listing them here, We have "Trading Capital Expenses With Operational Expenses", "Scale", "Stop Guessing Capacity", "Agility", "Go Global", "Avoid Running & Maintaining Data Centre". The PDF does an awesome job of explaining the Benefits & I would skip reiterating the same.
As you posted the Question in Cloudera Community, I would like to take a few minutes sharing the Current On-Premise Model Short-Coming from Hadoop Ecosystem perspective (Note that On-Premise has it's Advantages & Stating On-Premise is Defunct would be an Over-Statement):
- Associated "Big Data" with "Hadoop" isn't new (Although the Focus should be moving from Data Size/Type to Analytics & Value from Data). Hadoop Ecosystem offers a wide variety of Services & they fit Use-Cases with different needs. As the Services increase, So does the Data being processed & reviewed. As such, any Capacity Planning to hard to begin with.
- Be it Scaling Up or Scaling Out, Managing Linux or Windows Server isn't Easy. A Simple Task of adding Resources to an Instance isn't performed without Operational Experience & adding New Instances requires getting in touch with Multiple Parties (Admin, Billing, Vendor, Transport, Sponsor etc)
- Other context like Agility, Monitoring, Scaling, Routing are Self Explanatory.
Now, How does Cloudera helps via Cloudera Data Platform (CDP) is as follows:
- By ensuring Users only bother about their Use-Case (Running a Spark Job, Hive-Impala SQL, NoSQL HBase Analytics etc.) like the Users have been doing on their On-Premise, yet with the Flexibility of Cloud.
- The Best Part is a Customer can have 1 Setup on AWS, another Setup on Azure or GCP as well, yet the UI to perform your Hive/Spark/Kafka/HBase Jobs remains Similar.
- When no Jobs are running, the Scale-Down happens implicitly & when Jobs are running, the Scale-Up happens without any User's intervention. This ensure User focus on their Job only & Cost is Saved as well.
- Avoid Noisy Neighbours by ensuring the Jobs runs via Containers with their own resources & avoid any poor Job take away Host level resources, thereby affecting other Jobs.
As you tag "Cloudera Data Engineering", I wish to share a bit of details on the same. CDE allows you to run Spark Job (Just like you run on On-Premise) yet with the Flexibility of running them on Containers as well. And the Best Part is, End-User don't have to know anything about the Kubernetes Cluster running their Spark Job.
In short, Cloud offers Flexibility, Agility which is hard to achieve with On-Premise Setup. Yet, with Power comes Responsibility & here, AWS have a Shared-Responsibility Model as shared via [2]. Finally, I wish to share Cloudera Public Cloud Offering via [3] & some Awesome Free Training Videos on CDP via [4].
The above Opinion is Completely mine from my Experiences & I have barely scratched the Surface of the benefits from Cloud. I am happy to answer any specific query you may have from the above details. Also, Note that each of the AWS Link or PDF are Public Materials.
- Smarak
[1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/aws-overview.pdf#six-advantages-of-cloud...
[2] https://aws.amazon.com/compliance/shared-responsibility-model/?ref=wellarchitected
[3] https://docs.cloudera.com/cdp/latest/index.html
[4] https://www.cloudera.com/about/training.html#?fq=training%3Acomplimentary%2Ffree
