Hey everyone! I recently identified a small bug in the deployment script for the Docker images used in version 3.0.1 of the HDP.
I couldn't find a way to officially report it, so I was hoping somebody could direct me to the proper channels here. The details of the issue (and the associated fix) are outlined below.
I'm relatively inexperienced with the Hadoop ecosystem, so earlier this weekend, I started working through some Hadoop online courses to improve my skillset. As I tried to follow the instructor, I downloaded the HDP 3.0.1 sandbox Docker image. I'm using an insider build of Windows with the Windows Subsystem for Linux (version 2) turned on. I planned to run the HDP sandbox with the WSL2 Docker engine.
Unfortunately, after running the deployment script to spin up the sandbox-hdp and sandbox-proxy containers, I was initially unsuccessful at accessing the Ambari console. Even after changing my Hosts file to redirect sandbox-hdp.hortonworks.com requests to localhost, HTTP requests to services on ports 4200, 8080, and 1080 (for the web terminal, Ambari landing page, and Hortonworks landing pages, respectively) all returned a generic Nginx 502 Bad Gateway page.
Despite this, both the proxy container (from which the Nginx errors where emanating) and the the sandbox container were online, and I could access each container through an interactive shell. The web services (like the splash server and the Ambari server) had demonstrably failed from various networking issues (according to the system journal), but I couldn't figure out what the precise issue was. Restarting the individually failing services (and eventually restarting the entire sandbox) was unsuccessful.
After several hours of investigating, I realized that there is a small issue in the container deployment script that was responsible for most of my immediate issues. An abbreviated, annotated version or the deployment script is shown below.
#This script downloads HDP sandbox along with their proxy docker container
# CAN EDIT THESE VALUES
# NO EDITS BEYOND THIS LINE
echo $flavor > sandbox-flavor
# ... [Details omitted]
# This script has a bug by default which can be seen below.
# The interpreter directive at the top of the script says to use the system's
# "/usr/bin/env sh" shell. Many systems simply alias this shell to bash, but
# strictly speaking, I believe the resulting shell is only expected to be POSIX-
# A very limited "sh" shell is used in my WSL Ubuntu 18 instance by default,
# meaning that the "==" operators are unrecognized and report errors. Still,
# the script continues, simply resolving "hostname" to an empty string for the
# remaining commands. This causes 502 responses when the sandbox-proxy
# container tries to pass requests to the appropriate sandbox-hdp server.
# start the docker container and proxy
if [ "$flavor" == "hdf" ]; then
elif [ "$flavor" == "hdp" ]; then
# The fix is to simply replace the Bash-friendly "==" operators with POSIX-
# compliant "=" operators.
# ... [Remaining script omitted]
As mentioned in the annotations above, the double-equals operators appear to be acceptable in Bash, but not in a minimally-compliant POSIX-compliant shell.
Since the script specifies the sh shell in the interpreter directive at the top of the script, I assume that that authors' goal was to ensure that the script was portable across environments. Unless the conditionals in the script are corrected to use POSIX-compliant operators, that directive will ironically cause the script to subtly fail on systems like mine that invoke a slimmed-down Bourne shell that can't execute the double-equal comparison operators.
As far as I can tell, the solution just requires replacing the "double-equal" operators with their "single-equal" counterparts.
# start the docker container and proxy
if [ "$flavor" = "hdf" ]; then
elif [ "$flavor" = "hdp" ]; then
Does everybody agree with this fix, or are there considerations I've overlooked? If this is a legitimate solution, how can I see that this fix is made for future versions of the Docker HDP sandbox download?
Thanks in advance!
... View more