Motivation:
In past, there have been many instances here in Community as well as outside, where HDP 2.5 Sandbox users wouldn't be able to enable Kerberos / install Yum packages like SmartSense, KMS, Kerberos client / Access Zeppelin UI etc. like these:
1. https://community.hortonworks.com/questions/82455/zeppelin-ui-returns-error-503-hdp-25-sandbox.html
2. https://community.hortonworks.com/questions/68361/fail-to-install-kerberos-client-in-sandbox-25.html
4. https://community.hortonworks.com/questions/68234/fail-to-install-kerberos-in-hdp-sandbox-25.html
This article is aimed at resolving all these issues for once & all !
Target Audience:
All the users who are using HDP 2.5 Sandbox image for VirtualBox / VMWare.
Problem:
There is a known bug in Docker OverlayFS, Linux Kernel 3.10 & ext4 filesystem. Due to this bug, the HDP 2.5.0 Sandbox can have problems installing/deleting some services and files from the ext4 filesystem on the Docker ‘sandbox’ container.
The behavior usually is a failure on the install/delete/removal with the following low level error message:
[Error 22]Invalid argument
Reference Bugs:
https://github.com/docker/docker/issues/10294
https://github.com/docker/docker/issues/12488
https://github.com/docker/docker/issues/20640
Solution:
As of this writing, the available workaround is to - Upgrade the Linux Kernel on the "host" Docker system (which is actually running your HDP 2.5 Docker image).
Steps:
1. Login to HDP 2.5 VirtualBox "console" (please note, not regular SSH login !) using root user credentials.
2. Stop the ‘sandbox’ docker process:
# docker stop sandbox
3. Confirm the sandbox image is stopped:
# docker ps -a
4. Purge the both rescue files in /boot partition because the /boot partition is too small (this is required for next step)
# rm /boot/vmlinuz-0-rescue-1fd5af1a38de4420b2f283cdbbc38136 rm: remove regular file ‘/boot/vmlinuz-0-rescue-1fd5af1a38de4420b2f283cdbbc38136’? y # rm /boot/initramfs-0-rescue-1fd5af1a38de4420b2f283cdbbc38136.img rm: remove regular file ‘/boot/initramfs-0-rescue-1fd5af1a38de4420b2f283cdbbc38136.img’? Y
5. Upgrade the Linux Kernel of the ‘host’ docker system:
Old: Kernel Version: 3.10.0-327.el7.x86_64
New: Kernel Version: 3.10.0-514.6.1.el7.x86_64
# yum update kernel Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:3.10.0-514.6.1.el7 will be installed --> Processing Dependency: linux-firmware >= 20160830-49 for package: kernel-3.10.0-514.6.1.el7.x86_64 --> Running transaction check ---> Package linux-firmware.noarch 0:20150904-43.git6ebf5d5.el7 will be updated ---> Package linux-firmware.noarch 0:20160830-49.git7534e19.el7 will be an update --> Processing Conflict: kernel-3.10.0-514.6.1.el7.x86_64 conflicts xfsprogs < 4.3.0 --> Restarting Dependency Resolution with new changes. --> Running transaction check ---> Package xfsprogs.x86_64 0:3.2.2-2.el7 will be updated ---> Package xfsprogs.x86_64 0:4.5.0-9.el7_3 will be an update --> Processing Conflict: kernel-3.10.0-514.6.1.el7.x86_64 conflicts kmod < 20-9 --> Restarting Dependency Resolution with new changes. --> Running transaction check ---> Package kmod.x86_64 0:20-5.el7 will be updated ---> Package kmod.x86_64 0:20-9.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================================================================================== Package Arch Version Repository Size ===================================================================================================================================================================================================== Installing: kernel x86_64 3.10.0-514.6.1.el7 updates 37 M Updating: kmod x86_64 20-9.el7 base 115 k xfsprogs x86_64 4.5.0-9.el7_3 updates 895 k Updating for dependencies: linux-firmware noarch 20160830-49.git7534e19.el7 base 31 M Transaction Summary ===================================================================================================================================================================================================== Install 1 Package Upgrade 2 Packages (+1 Dependent package) Total size: 70 M Is this ok [y/d/N]: y Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : linux-firmware-20160830-49.git7534e19.el7.noarch 1/7 Updating : kmod-20-9.el7.x86_64 2/7 Installing : kernel-3.10.0-514.6.1.el7.x86_64 3/7 Updating : xfsprogs-4.5.0-9.el7_3.x86_64 4/7 Cleanup : linux-firmware-20150904-43.git6ebf5d5.el7.noarch 5/7 Cleanup : xfsprogs-3.2.2-2.el7.x86_64 6/7 Cleanup : kmod-20-5.el7.x86_64 7/7 Verifying : kmod-20-9.el7.x86_64 1/7 Verifying : kernel-3.10.0-514.6.1.el7.x86_64 2/7 Verifying : xfsprogs-4.5.0-9.el7_3.x86_64 3/7 Verifying : linux-firmware-20160830-49.git7534e19.el7.noarch 4/7 Verifying : linux-firmware-20150904-43.git6ebf5d5.el7.noarch 5/7 Verifying : xfsprogs-3.2.2-2.el7.x86_64 6/7 Verifying : kmod-20-5.el7.x86_64 7/7 Installed: kernel.x86_64 0:3.10.0-514.6.1.el7 Updated: kmod.x86_64 0:20-9.el7 xfsprogs.x86_64 0:4.5.0-9.el7_3 Dependency Updated: linux-firmware.noarch 0:20160830-49.git7534e19.el7 Complete!
6. Update the grub menu config to remove rescue boot option
# grub2-mkconfig -o /boot/grub2/grub.cfg
7. Now restart the entire HDP Sandbox using the 'reboot' command and check that everything comes up without any issue. Notice the new Linux Kernel option now being shown during startup.
Before Kernel upgrade:
After Kernel Upgrade:
8. Once HDP services are back, you are ready to retry the operation which was earlier failing.
Hope this helps!
P.S. - A BIG thanks & shout out to my colleague "Darwin Traver" @dtraver for finding the missing pieces in this puzzle & putting them together, wouldn't be possible without his efforts !!!
