Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Installing kudu and kafka in cloudera quickstart vm docker image

avatar
New Contributor

Hi All,

 

I have recently installed cloudera quick start vm docker image on mac m1 , have noticed in this 6 years old image ican't find kudu /kafka / nifi , below are the details

 

cdh version

Hadoop 2.6.0-cdh5.7.0
Subversion http://github.com/cloudera/hadoop -r c00978c67b0d3fe9f3b896b5030741bd40bf541a
Compiled by jenkins on 2016-03-23T18:36Z
Compiled with protoc 2.5.0
From source with checksum b2eabfa328e763c88cb14168f9b372
This command was run using /usr/jars/hadoop-common-2.6.0-cdh5.7.0.jar

 

Question 

how to install additional applications from already pulled docker image from docker hub is it possible to install additional missed applications in the same docker ? ,if the answer is yes how and leads ?

 

 

 

 

 

4 REPLIES 4

avatar
Master Collaborator

Specific to NiFi : By default NiFi is not provided as docker image for CDH or CDP , Last docker image for NiFi was published though HDF NiFI https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.2/download/content/downloading-your-software.htm... that to be run standalone not managed by Ambari or Cloudera Manager. 

 

Thank you.

avatar

@BigDataAvengers can you tell us a bit about why you're installing this six year old image? Does what you've already installed work with acceptable responsiveness? You didn't say:

  • where you retrieved the docker quickstart image from
  • how much RAM your mac m1 has available
  • whether or not you were actually able to successfully start up the services, beginning with the Cloudera Manager user interface, that were already present in the cloudera quickstart vm image you installed

The docker image you've installed appears to be based on CDH 5.7, which went out of support at the end of August in 2019 and is even older than the most recent version of the CDH Quickstart, which was based on CDH 5.13, which itself is already out of support. Cloudera Enterprise 5.13 reached it's end of support date in October 2020 (open that link and then expand the section labeled "Cloudera Enterprise products" underneath Current End of Support (EoS) Dates). For this reason, Cloudera is no longer distributing the CDH 5-based Quickstart. It is also why this hyperlink:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cloudera_quickstart_vm....

…to the appropriate documentation results in an HTTP 404 error.

And Kafka was never bundled and delivered with the cloudera quickstart; it always required a separate installation.

 

Next, it's not at all certain that your M1 Mac has enough RAM available in total to be able to run the services you are attempting to run within the container (or containers) without consuming so much of the host machine's memory that OS X starts killing processes to free up memory, and Docker is very much one of the processes subject to being killed. IIRC the memory recommendation for the Docker version of the Quickstart VM itself was about 10 GB, so on a 16GB Mac, with the overhead for Docker and other running applications (such as a web browser), things will be running pretty tight as far as memory goes. Even if you were able to install the docker image that @ckumar pointed you to in another container alongside the quickstart's container, that does not mean that it would provide a useable system, because the HDF NiFi container called for at least 8 GB.

 

So while it is possible that someone with the requisite knowledge of the Linux command line and Docker combined with sufficient skill and abilities with the various required development tools could update the last, outdated cloudera quickstart image to add Kudu and then subsequently figure out how to run it alongside containers for some version of NiFi and Kafka (and indeed, some member of the Cloudera Community may have already done so and be willing to share their method in response to your question), it doesn't mean you'd have the necessary hardware resources to run it on your M1 Mac, or that it would produce acceptable results.

 

You should ask yourself whether or not this would be worthwhile.

 

The current Enterprise Data Platform offered by Cloudera as of this writing is Cloudera Data Platform (CDP), which in it's on-premises "form factor" is now called CDP Private Cloud. CDP supersedes CDH and the components are at the appropriate versions that enable them to work together.

 

If you're just looking to evaluate a data platform, you can currently do so without a Cloudera subscription by downloading and installing the Trial Version of CDP Private Cloud Base Edition of Cloudera Data Platform. Cloudera has an extensive tutorial on installing CDP Private Cloud Base in an Infrastructure as a service (IaaS) fashion using AWS on its website. This approach allows you to leverage machines which have hardware resources, such as plentiful RAM, that your laptop might not have available. And CDP Private Cloud Base Edition ships with Kudu.

 

 

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

avatar
Explorer

Hi @ask_bill_brooks 

I have installed the docker image for learning/practice on Docker

 

1) I have retrieved the docker images from docker hub
2) RAM - 16gb
3) I am able to succefully launch all the applications expect for few

I am unable to connect to Hue upon restarting Cloudera Manager Express

Many thanks for a detailed explanation , i have tried accesing CDP ,but again i need to enroll for 60 day trail account and after that it expires.

I have also explored the option to install in Local Mac m1 but i don't messs up the latop with manual installation.

# I have noticed there is latest version which is atleast 3 years old
docker pull hortonworks/sandbox-hdp:3.0.1

any ideas ?

currently i would prefer on premise ,since cloud i am always worried we need to provide the card details

 

avatar
Community Manager

@BigDataAvengers, Has any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: