$ cd ~
$ mkdir cdpvm
$ cd cdpvm
$ wget https://cloud.centos.org/centos/7/vagrant/x86_64/images/CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box
$ wget https://raw.githubusercontent.com/carrossoni/CDPDCTrial/master/scripts/VMSetup.sh
- Go to the folder that you've downloaded your VM file (cd ~/cdpvm) and initialize the Virtual Machine using the following command:
$ vagrant box add CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box --name centos7
$ vagrant plugin install vagrant-disksize
$ vagrant init centos7
- After this step, you should have a file called "Vagrantfile" in the same directory, open the file with an editor (vim for example) and below the line config.vm.box = "centos7" add the following:
config.vm.box = "centos7"
config.vm.network "private_network", ip: "192.168.10.23"
config.vm.network "public_network"
config.vm.network :forwarded_port, guest: 7180, host: 7180
config.vm.hostname = "cloudera"
config.disksize.size = "100GB"
config.vm.provision "shell", path: "VMSetup.sh"
- Also in the same file find the line that begins with "# Example for VirtualBox:" and add/uncomment the lines below:
config.vm.provider "virtualbox" do |vb|
# Display the VirtualBox GUI when booting the machine
vb.gui = true
# Customize the amount of memory on the VM:
vb.memory = "10024"
vb.cpus = "8"
end
- Save the file and now we can init and bring up the VM:
$ vagrant up
- Now it'll ask to bridge to your public network (only for the first time) normally it's the one that you're connected on the internet, in my case is en0:
- After this, the VM will be provisioned and automated CDP process will start, this will take up to one hour depending on your connection since also it'll configure the VM and also install all the components for Cloudera Manager and the Services in an automated process located in https://github.com/carrossoni/CDPDCTrial/
- The template and the cluster created at the end will contain the following services:
HUE
HDFS
Impala
Ranger
Zookeeper
- After the install you can add more services like Nifi, Kafka etc. depending on the number of resources that you've reserved for the VM.
- After the execution you should see the exit below (this will take up about 30 min to one hour depending on your connection since it'll download all the packages and parcels necessary for provisioning CDP Runtime):
- After this the VM will reboot to do a fresh start, wait around 5 minutes for the services spin up and go to the next step.
Step 2: Cloudera Data Platform Access
- After the automated process our CDP Runtime is ready (actually we've provisioned in only one step)! In your machine browser you can connect to the CM with the following URL:
http://localhost:7180
- Password will be admin/admin after the first login you can choose the 60-day trial option and click in "Continue":
- The Welcome page appears, click in the Cloudera Logo on the top left since we've already added a new cluster with the automated process:
- At this point all the services are initiated, some errors may happen since we are working on a resource constraint environment, usually follow the logs that it'll be easy to see in Cloudera Manager what's happening, also you can suppress warning messages if it's not something critical.
We've our environment ready to work and learn more about CDP!
HUE and Data Access
- You can log in in Hue from the URL http://cloudera:8889/hue and for the first time we will use the user admin/admin, this will be the admin user for HUE:
- For example, I'll upload data from the California COVID-19 Testing that I've downloaded to my machine.
- In HUE go on the left panel and choose "Importer" → Type = File, choose /user/admin directory and then click in "Upload a file", choose your file (statewide_testing.csv) and then "Open". Now click in the file that you've uploaded and this will go to the next step:
- Click in Next and HUE will infer the table name, field types etc, you can change or leave as is and click in "Submit":
- At the end you should see the success of the job, close the job status window, and click in the Query button:
- Now that we've hour data we can query and use Impala SQL in the data that we've uploaded!
(Optional) Ranger Security Masking with Impala Example
- To start using/querying the environment with the system user/password that we've created (cloudera/cloudera) first we need to enter in Ranger we need to allow access to this user, click in the Ranger service and then in Ranger Admin WebUI:
- Now we have the initial Ranger screen. Login with the user/password admin/cloudera123:
- In the HADOOP SQL session click in the Hadoop SQL link. We will create a new policy to allow access to the new table but seeing the tested column in masked format with null results. For that click in the Masking tab and then Add New Policy with the following values:
- Click in the Add button and now go back to the Access tab and Add New Policy Button with the following parameters:
- Click in Add button and now our user should be ready to select only the data on this table with the masked values. First we'll configure the user in HUE, in the left panel click in the initial button and then in "Manage Users":
- Click in "Add User" and then in username put cloudera with the password cloudera, you can skip step 2 and 3 clicking directly in Add user.
- Logout from HUE and login with our new create user, go to the query editor and select the data again:
You should see the masked policy in action!
Summary
In this blog we've learned:
- How to Setup a Vagrant Centos 7 machine with Virtualbox and CDP Packages
- Configure CDP-DC for the first run
- Configure data access
- Setup simple security policies with the masking feature
You can play with the services, install other parcels like Kafka/Nifi/Kudu to create a streaming ingestion pipeline, and query in real-time with Spark/Impala. Of course for that, you'll need more resources and this can be changed in the beginning during the VM Configuration.
@carrossoni , when I open Cloudera Manager, it is taking me to the installation page. That means it has not registered the hosts yet.
The error is also confirming that "
Deployment should contain hosts.
It must be something to deal with the below function.
mgmt_api.auto_assign_roles() # needed?
If I comment that and run below via "vagrant ssh",
python ~/CDPDCTrial/scripts/create_cluster.py ~/CDPDCTrial/conf/cdpsandbox.json
{'active': False,
'can_retry': True,
'children': {'items': []},
'cluster_ref': None,
'end_time': '2020-08-08T08:06:09.975Z',
'host_ref': None,
'id': 16.0,
'name': 'GlobalHostInstall',
'parent': None,
'result_data_url': 'http://cloudera:7180/cmf/command/16/download',
'result_message': 'Failed to complete installation.',
'role_ref': None,
'service_ref': None,
'start_time': '2020-08-08T08:04:44.671Z',
'success': False}
Traceback (most recent call last):
File "/root/CDPDCTrial/scripts/create_cluster.py", line 78, in <module>
mgmt_api.setup_cms(body=api_service)
File "/usr/lib/python2.7/site-packages/cm_client/apis/mgmt_service_resource_api.py", line 1013, in setup_cms
(data) = self.setup_cms_with_http_info(**kwargs)
File "/usr/lib/python2.7/site-packages/cm_client/apis/mgmt_service_resource_api.py", line 1091, in setup_cms_with_http_info
collection_formats=collection_formats)
File "/usr/lib/python2.7/site-packages/cm_client/api_client.py", line 326, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/lib/python2.7/site-packages/cm_client/api_client.py", line 153, in __call_api
_request_timeout=_request_timeout)
File "/usr/lib/python2.7/site-packages/cm_client/api_client.py", line 379, in request
body=body)
File "/usr/lib/python2.7/site-packages/cm_client/rest.py", line 273, in PUT
body=body)
File "/usr/lib/python2.7/site-packages/cm_client/rest.py", line 219, in request
raise ApiException(http_resp=r)
cm_client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Transfer-Encoding': 'chunked', 'Set-Cookie': 'SESSION=b8021e6e-91e8-4b18-895e-8cd1f3a287a0;Path=/;HttpOnly', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Date': 'Sat, 08 Aug 2020 08:06:10 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json;charset=utf-8'})
HTTP response body: {
"message" : "Cannot find a suitable default host for cluster 'null'."
}
Hi @AkhilTech ,
The primary problem is that the host isn't in health state, this is evaluated before provisioning the cluster. This can happen if there isn't enough resources to provision CDP.
If there's enough resources you can:
Check if the service cloudera-scm-agent is configured properly:
- Check if the file /etc/cloudera-scm-agent/config.ini has the lines server_host= and listening_ip= with the same results of the commands "hostname" and "host cloudera".
- Check if the cloudera-scm-agent process is running "sudo service cloudera-scm-agent status".
- Restart the cloudera-scm-agent whit "sudo service cloudera-scm-agent restart".
After this you can check the logs for error messages inside the VM:
- /var/log/cloudera-scm-server/cloudera-scm-server.log
- /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Specially in cloudera-scm-agent log look for the last lines that should contain the error messages on why your host isn't healthy.
You can also use Cloudera Manager to identify what's happening and restart the agent services, if you click in the top left Cloudera logo you should see the initial page and you can go to "Hosts" to see what's happening.
Click on the top left Cloudera Manager Icon
Click in Hosts --> All Hosts
Click on host cloudera, if everything is health should appear something like this:
If not first try to restart the Cloudera Management Service (Actions -> Restart), and wait to see the results. If there's errors you can follow the messages to see the logs and what's may be causing the errors. If it goes up you can run the provisioning python again.
Let me know if this helps,
Thanks,
Luiz
