Member since
05-06-2019
3
Posts
0
Kudos Received
0
Solutions
05-08-2019
06:56 AM
We are testing our first cluster and appear to have failing Spark jobs due to lack of disk space while the job is running. Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_usnj1cldrmst1-root 50G 49G 1.5G 98% / devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 9.6M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/mapper/centos_usnj1cldrmst1-home 198G 33M 197G 1% /home /dev/sda1 1014M 232M 783M 23% /boot cm_processes 24G 21M 24G 1% /run/cloudera-scm-agent/process tmpfs 4.8G 12K 4.8G 1% /run/user/42 tmpfs 4.8G 0 4.8G 0% /run/user/1000 tmpfs 4.8G 0 4.8G 0% /run/user/0 This is is what it looks like before the job starts: Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_usnj1cldrmst1-root 50G 36G 15G 71% / devtmpfs 24G 0 24G 0% /dev tmpfs 24G 0 24G 0% /dev/shm tmpfs 24G 9.6M 24G 1% /run tmpfs 24G 0 24G 0% /sys/fs/cgroup /dev/mapper/centos_usnj1cldrmst1-home 198G 33M 197G 1% /home /dev/sda1 1014M 232M 783M 23% /boot cm_processes 24G 21M 24G 1% /run/cloudera-scm-agent/process tmpfs 4.8G 12K 4.8G 1% /run/user/42 tmpfs 4.8G 0 4.8G 0% /run/user/1000 tmpfs 4.8G 0 4.8G 0% /run/user/0 [root@usnj1cldrmst1 ~]# df -i /tmp Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos_usnj1cldrmst1-root 26214400 206658 26007742 1% / This is running on a VM so we can easily expand the (single) disk but /dev/mapper/centos_usnj1cldrmst1-root is in the middle of the disk so expanding it may not help. It may be easier to add a new volume and point the temporary work directory target (wherever that is?) to a new location. I'm not sure we we are just configured incorrectly but am looking for suggestions. Thank you
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
05-08-2019
06:01 AM
I did a few more manual installs and uninstalls of the agent package on the target host and then ran the Wizard again from the CDH console and finally the system accounts and agent appear properly. We may be adding more nodes in the near future and expect to see this problem again, but until then I won't be able to duplicate the problem and report back.
... View more
05-06-2019
04:28 PM
I have gone through the installation procedure several times trying to create 3 hosts at once but went back and decided to get the first host up. This first host is up and I am trying to add additonal hosts. On the wizard, Heardbeat is working, on the Install Parcels screen it shows CDH 6.2.0-1.cdh6.2.0.p0.967373 being downloaded/distributed/upacked/activated.
The issue is on the Host Inspector.
1 hosts are reporting with NONE CDH version
Also, none of the cloudera system accounts have been created on the new host. i.e. The user hdfs is missing on the following host
The agent seems to be running and it shows up as Managed when you show all hosts in the Manager. [root@usnj1cldrawrkr2 ~]# systemctl | grep cloudera cloudera-scm-agent.service loaded active running Cloudera Manager Agent Service
I was able to get one other host configured correctly by manually installing and then removing the cloudera services and then pushing out again with the host wizard, but no luck again. Any ideas?
... View more
Labels:
- Labels:
-
Cloudera Manager
-
HDFS