question Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS in Support Questions

No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Fri, 16 Sep 2022 10:44:51 GMT

Hi,

I am trying to install Kudu with Impala on my CDH 5.8 on AWS.

While downloading the service for Impala-Kudu, I get an error that "No space left on device".

I tried deleting log files but it did not work.

Although I had neither installed anything else than CDH(All services) nor loaded any data, still I get this error.

Can someone please suggest a work arounf for this or advice on which directories can I empty to free up the space.

Thank you.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Wed, 19 Oct 2016 17:57:05 GMT

To clarify: I was adding Kudu with Impala using Parcels.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

dhan — Mon, 24 Oct 2016 16:30:07 GMT

Ankita,

The no space left error likely indicates that your root partition is too small. You can check the size and usage of your mounted disks with the 'df' command.

Are you using Cloudera Director to create your CM or CDH instances? Cloudera Director by default will create a 50GB EBS drive as the root partition and will automatically resize the filesystem to use the whole disk.

Can you provide more information about what instance type and ami you are using as well as the the disk size and usage information?

David

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Mon, 24 Oct 2016 17:52:42 GMT

Hi David,

Thanks for your reply.

I am using Centos 6- With updates and instance type is General purpose
m1.xlarge.

Details: m1.xlarge (8 ECUs, 4 vCPUs, 15 GiB memory, 4 x 420 GiB Storage
Capacity)

I checked the space using df -k command and it shows 100% utilized.

I tried to delete the CDH parcel from the directory /opt/cloudera/parcels
and then download the Impala-Kudu parcel. It gets downloaded (along with
CDH. CDH parcel again gets distributed and activated and fills up the space
upto 100%) but stops while activating due to space issue.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

dhan — Tue, 25 Oct 2016 14:45:58 GMT

Ankita,

As you've found out, CM will re-download the parcels if you delete them so you'll have to figure out why you are running out of disk space.

Can you provide the following information?

- Was this CM provisioned through Cloudera Director? I wouldn't expect this problem on instances that Director provisions unless there is a configuration issue or some error resizing the root partition on this ami.

- What is the ami id?

- Can you provide the output of your df command? I'd like to see the size of the root partition.

David

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Tue, 25 Oct 2016 20:01:42 GMT

Hi David,

I terminated that particular cluster which was Centos AMI and now trying to
configure on Ubuntu 12.04 which is Cloudera Provisioned AMI.
Now, on this AMI, I am not able to activate my CDH5.8 parcel. It fails
activation on one node and says no space left on device, although it has
84% space.

Please find below the answers to your questions:

Was this CM provisioned through Cloudera Director?
Answer: Yes, Ubuntu 12.04.
AMI ID: ami-13471c23

Output of df command:
[image: Inline image 1]

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

dhan — Wed, 26 Oct 2016 20:12:38 GMT

Ankita,

The solution to running out of disk space is to provision an instance with a larger root partition.

What version of Cloudera Director are you running?

Director does not support creation of CM or CDH instances on Ubuntu. See http://www.cloudera.com/documentation/director/latest/topics/director_deployment_requirements.html for supported versions.

Are you provisioning your instances through Director's UI or conf file?

If using Director's UI, can you provide the instance template you are using?

If using conf file, can you provide the conf file you are using (with sensitive information removed)?

Also, the output of your df command did not appear to come through in your last post. Can you include that information also?

David

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Wed, 26 Oct 2016 20:30:35 GMT

Hi David,

I am not using Cloudera Director. Instead, I am using Cloudera Manager and following the instructions from the website:

http://www.cloudera.com/documentation/archive/manager/4-x/4-7-2/Cloudera-Manager-Installation-Guide/cmig_install_on_EC2.html

Instead of cm4, I use cm5 for getting the latest Cloudera manager.

The output from command line:

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvda1 8256952 6634076 1203448 85% /

udev 7661552 8 7661544 1% /dev

tmpfs 1533964 984 1532980 1% /run

none 5120 0 5120 0% /run/lock

none 7669808 0 7669808 0% /run/shm

/dev/xvdb 433455904 203012 411234588 1% /mnt

/dev/xvdc 433455904 203012 411234588 1% /data0

/dev/xvdd 433455904 203012 411234588 1% /data1

/dev/xvde 433455904 203012 411234588 1% /data2

cm_processes 7669808 0 7669808 0% /run/cloudera-scm-agent/process

Thank you.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

dhan — Fri, 28 Oct 2016 15:10:03 GMT

When you create your instance, the default root drive is 8GB. You should change this to a larger size, maybe 50GB.

If you are creating this manually through the AWS Console, this is on "Step 4. Add Storage" in the launch instance wizard.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Wed, 02 Nov 2016 21:03:24 GMT

Hi David,

I tried to increase the size of the root drive from 8GB to more than 50GB and still I get the same error for no space left on device.

Below is the result of the df command:

[root@ip-172-31-25-0 ~]# df

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/xvde 8256952 7821988 15536 100% /

tmpfs 7685696 0 7685696 0% /dev/shm

/dev/xvdf 433455904 350280 411087320 1% /data0

/dev/xvdg 433455904 361636 411075964 1% /data1

/dev/xvdh 433455904 344808 411092792 1% /data2

/dev/xvdi 433455904 332788 411104812 1% /data3

cm_processes 7685696 544 7685152 1% /var/run/cloudera-scm-agent/process

Thank you.

Re: No space left on device while installing Kudu with Impala on CDH5.8 on AWS

AnkitaK — Wed, 02 Nov 2016 21:15:03 GMT

Fixed the issue.

I resized the disk space and was able to download and activate the parcel.

Thanks for your time and help.