Community Articles
Find and share helpful community-sourced technical articles.
Labels (1)
Guru

Motivation:

In past, there have been many instances here in Community as well as outside, where HDP 2.5 Sandbox users wouldn't be able to enable Kerberos / install Yum packages like SmartSense, KMS, Kerberos client / Access Zeppelin UI etc. like these:

1. https://community.hortonworks.com/questions/82455/zeppelin-ui-returns-error-503-hdp-25-sandbox.html

2. https://community.hortonworks.com/questions/68361/fail-to-install-kerberos-client-in-sandbox-25.html

3. https://community.hortonworks.com/questions/60349/enabling-kerberos-in-hdp-25-kerberos-clientpy-does...

4. https://community.hortonworks.com/questions/68234/fail-to-install-kerberos-in-hdp-sandbox-25.html

5. https://community.hortonworks.com/questions/78464/ranger-kms-cannot-install-successfully-with-ambari...

This article is aimed at resolving all these issues for once & all !

Target Audience:

All the users who are using HDP 2.5 Sandbox image for VirtualBox / VMWare.

Problem:

There is a known bug in Docker OverlayFS, Linux Kernel 3.10 & ext4 filesystem. Due to this bug, the HDP 2.5.0 Sandbox can have problems installing/deleting some services and files from the ext4 filesystem on the Docker ‘sandbox’ container.

The behavior usually is a failure on the install/delete/removal with the following low level error message:

[Error 22]Invalid argument

Reference Bugs:

https://github.com/docker/docker/issues/10294

https://github.com/docker/docker/issues/12488

https://github.com/docker/docker/issues/20640

Solution:

As of this writing, the available workaround is to - Upgrade the Linux Kernel on the "host" Docker system (which is actually running your HDP 2.5 Docker image).

Steps:

1. Login to HDP 2.5 VirtualBox "console" (please note, not regular SSH login !) using root user credentials.

14282-sandbox-console-to-be-used-for-login.png

2. Stop the ‘sandbox’ docker process:

# docker stop sandbox

3. Confirm the sandbox image is stopped:

# docker ps -a

14283-sandbox-stop-docker-image.png

4. Purge the both rescue files in /boot partition because the /boot partition is too small (this is required for next step)

# rm /boot/vmlinuz-0-rescue-1fd5af1a38de4420b2f283cdbbc38136
rm: remove regular file ‘/boot/vmlinuz-0-rescue-1fd5af1a38de4420b2f283cdbbc38136’? y
# rm /boot/initramfs-0-rescue-1fd5af1a38de4420b2f283cdbbc38136.img
rm: remove regular file ‘/boot/initramfs-0-rescue-1fd5af1a38de4420b2f283cdbbc38136.img’? Y

14284-sandbox-cleanup-boot-partition-before-kernel-upgra.png

5. Upgrade the Linux Kernel of the ‘host’ docker system:

Old: Kernel Version: 3.10.0-327.el7.x86_64

New: Kernel Version: 3.10.0-514.6.1.el7.x86_64

# yum update kernel
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-514.6.1.el7 will be installed
--> Processing Dependency: linux-firmware >= 20160830-49 for package: kernel-3.10.0-514.6.1.el7.x86_64
--> Running transaction check
---> Package linux-firmware.noarch 0:20150904-43.git6ebf5d5.el7 will be updated
---> Package linux-firmware.noarch 0:20160830-49.git7534e19.el7 will be an update
--> Processing Conflict: kernel-3.10.0-514.6.1.el7.x86_64 conflicts xfsprogs < 4.3.0
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package xfsprogs.x86_64 0:3.2.2-2.el7 will be updated
---> Package xfsprogs.x86_64 0:4.5.0-9.el7_3 will be an update
--> Processing Conflict: kernel-3.10.0-514.6.1.el7.x86_64 conflicts kmod < 20-9
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package kmod.x86_64 0:20-5.el7 will be updated
---> Package kmod.x86_64 0:20-9.el7 will be an update
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=====================================================================================================================================================================================================
Package                                         Arch                                    Version                                                      Repository                                Size
=====================================================================================================================================================================================================
Installing:
kernel                                          x86_64                                  3.10.0-514.6.1.el7                                           updates                                   37 M
Updating:
kmod                                            x86_64                                  20-9.el7                                                     base                                     115 k
xfsprogs                                        x86_64                                  4.5.0-9.el7_3                                                updates                                  895 k
Updating for dependencies:
linux-firmware                                  noarch                                  20160830-49.git7534e19.el7                                   base                                      31 M
 
Transaction Summary
=====================================================================================================================================================================================================
Install  1 Package
Upgrade  2 Packages (+1 Dependent package)
 
Total size: 70 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : linux-firmware-20160830-49.git7534e19.el7.noarch                                                                                                                                  1/7
  Updating   : kmod-20-9.el7.x86_64                                                                                                                                                              2/7
  Installing : kernel-3.10.0-514.6.1.el7.x86_64                                                                                                                                                  3/7
  Updating   : xfsprogs-4.5.0-9.el7_3.x86_64                                                                                                                                                     4/7
  Cleanup    : linux-firmware-20150904-43.git6ebf5d5.el7.noarch                                                                                                                                  5/7
  Cleanup    : xfsprogs-3.2.2-2.el7.x86_64                                                                                                                                                       6/7
  Cleanup    : kmod-20-5.el7.x86_64                                                                                                                                                              7/7
  Verifying  : kmod-20-9.el7.x86_64                                                                                                                                                              1/7
  Verifying  : kernel-3.10.0-514.6.1.el7.x86_64                                                                                                                                                  2/7
  Verifying  : xfsprogs-4.5.0-9.el7_3.x86_64                                                                                                                                                     3/7
  Verifying  : linux-firmware-20160830-49.git7534e19.el7.noarch                                                                                                                                  4/7
  Verifying  : linux-firmware-20150904-43.git6ebf5d5.el7.noarch                                                                                                                                  5/7
  Verifying  : xfsprogs-3.2.2-2.el7.x86_64                                                                                                                                                       6/7
  Verifying  : kmod-20-5.el7.x86_64                                                                                                                                                              7/7
 
Installed:
  kernel.x86_64 0:3.10.0-514.6.1.el7
 
Updated:
  kmod.x86_64 0:20-9.el7                                                                       
  xfsprogs.x86_64 0:4.5.0-9.el7_3
 
Dependency Updated:
  linux-firmware.noarch 0:20160830-49.git7534e19.el7
 
Complete!

6. Update the grub menu config to remove rescue boot option

# grub2-mkconfig -o /boot/grub2/grub.cfg

14288-sandbox-grub-step-after-kernel-upgrade.png

7. Now restart the entire HDP Sandbox using the 'reboot' command and check that everything comes up without any issue. Notice the new Linux Kernel option now being shown during startup.

Before Kernel upgrade:

14285-sandbox-startup-screen-before-kernel-upgrade.png

After Kernel Upgrade:

14287-sandbox-startup-screen-after-kernel-upgrade.png

8. Once HDP services are back, you are ready to retry the operation which was earlier failing.

Hope this helps!

P.S. - A BIG thanks & shout out to my colleague "Darwin Traver" @dtraver for finding the missing pieces in this puzzle & putting them together, wouldn't be possible without his efforts !!!


sandbox-grub-step-after-kernel-upgrade.png
3,612 Views
Comments
Super Mentor

@Vipin Rathor

Wonderful article!!! I was facing this issue for quite sometime. I will implement the fix to see how it goes.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎08-17-2019 01:27 PM
Updated by:
Contributors
Top Kudoed Authors