Created 05-09-2018 03:21 AM
Hello,
I know there are several posts about this problem, but I could not figure out what is the best post to solve my problem.
Here is what I have. I after running:
start_sandbox-hdp.sh
I get: "Ambari server 'start' completed successfully"
I believe my docker has been updated. I see some posts saying to modifying the sandbox startup script, but I am not sure what to change. I have Hortonworks HDP 2.6.1 installed.
Any help would be much appreciated.
Here is other information about my environment
OS:
Distributor ID: Ubuntu Description: Ubuntu 16.04.4 LTS Release: 16.04 Codename: xenial
Could it be because of low disk space? 96% used up space on the file system /dev/sdc1 and /devf/sda1. Would this cause the sandbox to not start?
Filesystem 1K-blocks Used Available Use% Mounted on overlay 64989928 61996332 2977212 96% / tmpfs 65536 0 65536 0% /dev tmpfs 16459536 0 16459536 0% /sys/fs/cgroup /dev/sdc1 67075056 1651732 65423324 3% /data /dev/sda1 64989928 61996332 2977212 96% /etc/resolv.conf /dev/sda1 64989928 61996332 2977212 96% /etc/hostname /dev/sda1 64989928 61996332 2977212 96% /etc/hosts shm 65536 0 65536 0% /dev/shm
The sandbox start HDP sh file:
#!/bin/bash echo "Waiting for docker daemon to start up:" until docker ps 2>&1| grep STATUS>/dev/null; do sleep 1; done; >/dev/null docker ps -a | grep sandbox-hdp if [ $? -eq 0 ]; then docker start sandbox-hdp else docker run --name sandbox-hdp --hostname "sandbox.hortonworks.com" --privileged -v /data:/data -d \ -p 1111:111 \ -p 1000:1000 \ -p 1100:1100 \ -p 1220:1220 \ -p 1988:1988 \ -p 2049:2049 \ -p 2100:2100 \ -p 2181:2181 \ -p 3000:3000 \ -p 4040-4050:4040-4050 \ -p 9999:9999 \ -p 4200:4200 \ -p 4242:4242 \ -p 5007:5007 \ -p 5011:5011 \ -p 6001:6001 \ -p 6003:6003 \ -p 6008:6008 \ -p 6080:6080 \ -p 6188:6188 \ -p 8000:8000 \ -p 8005:8005 \ -p 8020:8020 \ -p 8032:8032 \ -p 8040:8040 \ -p 8042:8042 \ -p 8080:8080 \ -p 8082:8082 \ -p 8086:8086 \ -p 8088:8088 \ -p 8090:8090 \ -p 8091:8091 \ -p 8188:8188 \ -p 8443:8443 \ -p 8744:8744 \ -p 8765:8765 \ -p 8886:8886 \ -p 8888:8888 \ -p 8889:8889 \ -p 8983:8983 \ -p 8993:8993 \ -p 9000:9000 \ -p 9995:9995 \ -p 9996:9996 \ -p 10000:10000 \ -p 10001:10001 \ -p 10015:10015 \ -p 10016:10016 \ bash echo "Waiting for docker daemon to start up:"
In an ambari-agent.out file, this is repeated:
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable javax.jdo.option.ConnectionPassword has been successfully created. org.apache.hadoop.security.alias.JavaKeyStoreProvider has been updated. 2018-04-14 02:07:31,850 - Testing the JVM's JCE policy to see it if supports an unlimited key length. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Apr 14, 2018 2:08:44 AM org.apache.hadoop.util.NativeCodeLoader <clinit> WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable oozie.service.JPAService.jdbc.password has been successfully created. org.apache.hadoop.security.alias.JavaKeyStoreProvider has been updated. 2018-04-14 02:09:33,283 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2018-04-14 02:10:34,443 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2018-04-14 02:11:35,382 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2018-04-14 02:12:35,604 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2018-04-14 02:13:35,659 - Testing the JVM's JCE policy to see it if supports an unlimited key length. 2018-04-14 02:14:35,635 - Testing the JVM's JCE policy to see it if supports an unlimited key length. "ambari-agent.out" 112L, 10714C
Created 05-10-2018 04:40 AM
Hi @Aziz Herch, can you help me out? Looking for some help. Thanks, Paul
Created 05-18-2018 10:04 PM
A few notes on issues you may be facing:
- The 'du' output looks to have been made on the VM - so the 96% disk taken is okay, expected even.
- What problem exactly are you running into? As far as I can tell, the warning in ambari-agent.out shouldn't stop any service from running - are any services not coming up?
I hate to be that guy who asks "Have you tried the latest version?" ... but ... Have you considered checking out one of the newer versions of the HDP Sandbox? Problems may be easier to help debug.
Created 05-21-2018 12:50 AM
Hi Edgar,
True, its an Ubuntu VM in Azure.
The problem is I am trying to solve is the sandbox script stops at the point of after starting the ambari server.
Basically, I am just trying to get to HDFS, and Kafka and Jupyter Notebooks so I can do some data processing with pyspark. I used to start the sandbox and then use those apps/tools from there. Plus, I could use ambari server in the sandbox.
If there is a way to run these without starting the sandbox, that would be great. However, I am not sure how to do that. Kind of new to this.
How would I do an upgrade from 2.6.1 to 2.6.4? I will search on how to do this, but if you have something that would be great too.
Created 05-21-2018 01:20 AM
Heya @David Jones!
Regarding the script stopping after starting the ambari server:
Unfortunately, HDP's 2.6.1 start script is a bit before my time, but I'm pinging @glupu in case he can help.
Regarding using the latest version (lots of sweet improvements/additions, *wink wink*):
If you prefer running on Azure, here's a link to deploying the latest sandbox on your cloud: https://hortonworks.com/tutorial/sandbox-deployment-and-install-guide/section/4/
If you prefer running a VM on your personal machine (at least 8GB+ free to allocate to the sandbox), there are VirtualBox, VMWare, or Docker versions here: https://hortonworks.com/downloads/#sandbox
The latest HDP sandbox doesn't come with Jupyter installed by default, but I whipped up the commands you can run inside the HDP sandbox (once deployed) in order to install Jupyter+Python3: https://gist.github.com/orendain/d4fe83c3c11ce77b146762eb3acac471
Good luck, amigo - let me know if I can help!