No space left on device while installing Kudu with Impala on CDH5.8 on AWS

[#1]

Hi,

I am trying to install Kudu with Impala on my CDH 5.8 on AWS.

While downloading the service for Impala-Kudu, I get an error that "No space left on device".

I tried deleting log files but it did not work.

Although I had neither installed anything else than CDH(All services) nor loaded any data, still I get this error.

Can someone please suggest a work arounf for this or advice on which directories can I empty to free up the space.

Thank you.

[#2]

To clarify: I was adding Kudu with Impala using Parcels.

[#3]

Ankita,

The no space left error likely indicates that your root partition is too small. You can check the size and usage of your mounted disks with the 'df' command.

Are you using Cloudera Director to create your CM or CDH instances? Cloudera Director by default will create a 50GB EBS drive as the root partition and will automatically resize the filesystem to use the whole disk.

Can you provide more information about what instance type and ami you are using as well as the the disk size and usage information?

David

[#4]

Hi David,

Thanks for your reply.

I am using Centos 6- With updates and instance type is General purpose
m1.xlarge.

Details: m1.xlarge (8 ECUs, 4 vCPUs, 15 GiB memory, 4 x 420 GiB Storage
Capacity)

I checked the space using df -k command and it shows 100% utilized.

I tried to delete the CDH parcel from the directory /opt/cloudera/parcels
and then download the Impala-Kudu parcel. It gets downloaded (along with
CDH. CDH parcel again gets distributed and activated and fills up the space
upto 100%) but stops while activating due to space issue.

[#5]

Ankita,

As you've found out, CM will re-download the parcels if you delete them so you'll have to figure out why you are running out of disk space.

Can you provide the following information?

- Was this CM provisioned through Cloudera Director? I wouldn't expect this problem on instances that Director provisions unless there is a configuration issue or some error resizing the root partition on this ami.

- What is the ami id?

- Can you provide the output of your df command? I'd like to see the size of the root partition.

David

[#6]

Hi David,

I terminated that particular cluster which was Centos AMI and now trying to
configure on Ubuntu 12.04 which is Cloudera Provisioned AMI.
Now, on this AMI, I am not able to activate my CDH5.8 parcel. It fails
activation on one node and says no space left on device, although it has
84% space.

Please find below the answers to your questions:

Was this CM provisioned through Cloudera Director?
Answer: Yes, Ubuntu 12.04.
AMI ID: ami-13471c23

Output of df command:
[image: Inline image 1]

[#7]

Ankita,

The solution to running out of disk space is to provision an instance with a larger root partition.

What version of Cloudera Director are you running?

Director does not support creation of CM or CDH instances on Ubuntu. See http://www.cloudera.com/documentation/director/latest/topics/director_deployment_requirements.html for supported versions.

Are you provisioning your instances through Director's UI or conf file?

If using Director's UI, can you provide the instance template you are using?

If using conf file, can you provide the conf file you are using (with sensitive information removed)?

Also, the output of your df command did not appear to come through in your last post. Can you include that information also?

David

[#8]

Hi David,

I am not using Cloudera Director. Instead, I am using Cloudera Manager and following the instructions from the website:

http://www.cloudera.com/documentation/archive/manager/4-x/4-7-2/Cloudera-Manager-Installation-Guide/...

Instead of cm4, I use cm5 for getting the latest Cloudera manager.

The output from command line:

Filesystem     1K-blocks    Used Available Use% Mounted on

/dev/xvda1       8256952 6634076   1203448  85% /

udev             7661552       8   7661544   1% /dev

tmpfs            1533964     984   1532980   1% /run

none                5120       0      5120   0% /run/lock

none             7669808       0   7669808   0% /run/shm

/dev/xvdb      433455904  203012 411234588   1% /mnt

/dev/xvdc      433455904  203012 411234588   1% /data0

/dev/xvdd      433455904  203012 411234588   1% /data1

/dev/xvde      433455904  203012 411234588   1% /data2

cm_processes     7669808       0   7669808   0% /run/cloudera-scm-agent/process

Thank you.

[#9]

When you create your instance, the default root drive is 8GB. You should change this to a larger size, maybe 50GB.

If you are creating this manually through the AWS Console, this is on "Step 4. Add Storage" in the launch instance wizard.

[#10]

Hi David,

I tried to increase the size of the root drive from 8GB to more than 50GB and still I get the same error for no space left on device.

Below is the result of the df command:

[root@ip-172-31-25-0 ~]# df

Filesystem     1K-blocks    Used Available Use% Mounted on

/dev/xvde        8256952 7821988     15536 100% /

tmpfs            7685696       0   7685696   0% /dev/shm

/dev/xvdf      433455904  350280 411087320   1% /data0

/dev/xvdg      433455904  361636 411075964   1% /data1

/dev/xvdh      433455904  344808 411092792   1% /data2

/dev/xvdi      433455904  332788 411104812   1% /data3

cm_processes     7685696     544   7685152   1% /var/run/cloudera-scm-agent/process

Thank you.