Member since
02-09-2016
559
Posts
422
Kudos Received
98
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2138 | 03-02-2018 01:19 AM | |
3528 | 03-02-2018 01:04 AM | |
2372 | 08-02-2017 05:40 PM | |
2345 | 07-17-2017 05:35 PM | |
1721 | 07-10-2017 02:49 PM |
11-14-2016
08:36 PM
1 Kudo
@Kumar Try using back ticks ` around the column name like `column`.
... View more
11-12-2016
01:25 AM
@Jonas Straub
The solrconfig.xml portion you provide doesn't work when I try to create a collection using the data driven schema configs. The problem seems to be the
<processor> entries. I get null pointer exceptions.
The first processor entry should be <processor class="solr.DefaultValueUpdateProcessorFactory">. The second processor entry should be <processor class="solr.processor.DocExpirationUpdateProcessorFactory">.
... View more
11-11-2016
04:42 PM
4 Kudos
@Timothy Spann Have you run analyze table <tablename> compute statistics for columns? https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables
... View more
11-11-2016
12:19 AM
12 Kudos
Objective
The Hortonworks Sandbox for HDP 2.5 now uses Docker containers, even the VirtualBox version. The process for exposing extra ports on older versions of the sandbox was as simple as setting up additional port forwarding rules in VirtualBox. The new container version of the sandbox requires additional steps; you have to do more than just setup port forwarding rules in VirtualBox.
This tutorial will guide you through the process of adding additional ports to the VirtualBox version of the HDP 2.5 sandbox. Prerequisites
You should have already downloaded and installed the VirtualBox version of the Hortonworks Sandbox for HDP 2.5 Hortonworks Sandbox Scope
Mac OS X 10.11.6 (El Capitan)
VirtualBox 5.1.6
HDP 2.5 VirtualBox Sandbox Steps Startup the Sandbox VM
Startup your HDP sandbox. It should be called Hortonworks Docker Sandbox within VirtualBox. You should see something similar to this:
You can start the sandbox by either double-clicking on the virtual machine or by selecting the virtual machine and clicking on the start icon in the menu. Once the virtual machine is started, you should see something similar to this:
NOTE: It may take several minutes for the virtual machine to start. Login to the Sandbox VM
We need to login to the VM that runs the sandbox container. If you use the standard ssh -p 2222 root@localhost , you will actually login to the sandbox container, not the sandbox VM. You can login to the sandbox VM using ssh -p 2122 root@localhost . You can also login to the sandbox VM directly using the VirtualBox VM window by following the instructions on the VM window. On the Mac you click in the VM window and press the Alt/Option + F5 keys. You should see something like this:
The username is root and the password is hadoop . Disable sandbox.service
The sandbox Docker container is set to autostart when the VM starts. I've run into issues trying to top the Docker container. So we'll use a workaround to temporarily disable the sandbox.service. The sandbox VM is based on CentOS 7 and uses systemd.
systemctl disable sandbox.service
Reboot the VM
Now we need to reboot the VM so we can get it started without the sandbox container running. You can do this easily using the init command.
init 6
The VM should reboot. Modify sandbox start script
After the VM reboots, login again using the steps provided above. The sandbox start script is located at /root/start_scripts/start_sandbox.sh . This script has the docker run command which creates or starts the Docker container. We need to edit this script to add our additional ports. For the purposes of this tutorial, we'll add port 8440 which is used by Ambari agents to talk to Ambari server.
vi /root/start_scripts/start_sandbox.sh
Scroll down until you see:
-p 2222:22 sandbox /usr/sbin/sshd -D
We need to add a line after the last port entry. Modify the script so it now looks like this:
-p 2222:22 -p 8440:8440 sandbox /usr/sbin/sshd -D
Now save the file
Press ESC key
:wq!
Delete existing sandbox container
Since we booted up the VM at least once, the sandbox container will already exist. The startup script will not and cannot update an existing container to add another port. So we need to remove the existing sandbox container. The startup script will create a new container with the added ports. Remember with Docker that a Docker image is like a blueprint to the building and a Docker container is the building created from the blueprint. We will delete the container, not the image.
docker rm sandbox
NOTE: This will remove any changes you've made to your Docker container via Ambari, etc. If you want to save any changes you've made, you can update the base sandbox Docker image using docker commit . To update the base sandbox image with any configuration changes you've done use docker commit sandbox sandbox . Enable sandbox.service
Now we need to enable the sandbox service so it auto starts when the VM boots up.
systemctl enable sandbox.service
Reboot the VM
Now it's time to reboot the VM. As before, we'll use the init command
init 6
Verify new ports
After the VM reboots, login again using the steps provided above. We are going to show the running Docker container using the docker ps command.
docker ps
You should notice the standard Docker output from that command. Look for the port 8440 . You should see it in the list. You should see something similar to this:
NOTE: We added the port to the end of the startup script. However, the port will not be displayed at the end of the list. The port will be somewhere in the middle of the list of ports. Update VirtualBox VM ports
Now we need to update the forwarded ports configuration for our VirtualBox VM. Using the VirtualBox user interface, right-click on the Hortonworks Docker Sandbox to display the menu. You should see something similar to this:
Now click the Settings menu option. You should see something similar to this:
Now click on the Network menu icon. You should see something similar to this:
Now click the Advanced dropdown menu near the bottom. You should see something similar to this:
Now click the Port Forwarding button. You should see something similar to this:
Now click the + icon near the upper right to add another port forwarding rule. You should see something similar to this:
Now add an entry with the following info:
Click the blue OK button. Click the OK button. The settings should be saved.
Now port 8440 should be accessible on your computer which is forwarded to 8440 on the VirtualBox VM which is then passed to the Docker container. Review
If you successfully followed along with this tutorial, we were able modify the Docker startup script for our sandbox container with the VirtualBox VM to add another port. After removing the Docker container and restarting the VM, a new Docker container was created with our additional port.
... View more
Labels:
11-10-2016
09:12 PM
Deleting the VM and not the files won't typically resolve anything as the problem is likely the VM image itself. you can import a new copy of the image you downloaded and they can sit side by side. See if a new version works ok. Your VM may somehow be corrupt.
... View more
11-10-2016
08:10 PM
@Karan Alang Can you try giving the virtual machine 10GB or 12GB of ram to see if that helps? I typically run my sandbox using 12GB (12288MB). How did you shutdown the virtual machine when it was working before?
... View more
11-10-2016
04:53 PM
1 Kudo
@Roger Young Ah, I see what you are trying to do. Ambari agents use port 8440 to communicate with the server (https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_reference_guide/content/_default_network_port_numbers_-_ambari.html). The docker container does not expose port 8440. If you want to add that port, you can follow my tutorial for using the native Docker HDP sandbox to easily add ports to your containers before you create them (https://community.hortonworks.com/content/kbentry/58458/installing-docker-version-of-sandbox-on-mac.html). Look at step four in the process. Unfortunately the process is a bit more difficult when using the VirtualBox + Docker version of the sandbox. You will need to log into the VirtualBox VM directly using the VBox UI. The username is root and the password is hadoop. The startup script for the Docker sandbox is located here: /root/start_scripts/start_sandbox.sh Edit that file and add -p 8440:8440 \ At this point you need to delete the existing sandbox container. However that seems to hang for me. So there are extra steps you need to do. 1. Disable the sandbox service systemctl disable sandbox.service 2. Reboot the VM and log back in via the VBox UI. init 6 3. Delete the sandbox container docker rm sandbox 4. Enable the sandbox service systemctl enable sandbox.service 5. Reboot the VM and log back in via the VBox UI. init 6 6. Verify the sandbox service is running docker ps -a You should see 0.0.0.0:8440->8440/tcp in the list of ports 7. Update the VirtualBox forwarded ports for the VM to include 8440.
... View more
11-10-2016
04:27 PM
@Karan Alang You can try having VirutalBox turn off the VM and then restarting the VM to see if the issue goes away.
... View more
11-10-2016
03:05 PM
@Roger Young Yes the Docker container is accessible via the IP address where VirtualBox is running. My laptop is currently 192.168.1.162. From any computer on the network I can access Ambari on my laptop via http://192.168.1.162:8080. This is because VirtualBox has port 8080 forwarding from my laptop to the VirtualBox VM. The VM is running a Docker container that forwards port 8080 on the VM to the Docker container. I hope this helps. If it does, please accept an answer as it helps the community find solutions to questions.
... View more
11-10-2016
02:17 PM
@Roger Young The HDP 2.5 sandboxes are now all based on Docker containers. When you ssh into the sandbox via port 2222, you are connecting to the VirtualBox VM which is then forwarding your request to the Docker container.
... View more