Member since
03-22-2018
7
Posts
1
Kudos Received
0
Solutions
06-01-2018
02:39 PM
Thanks @Felix Albani ! I have a question! is spark interpreter is the best in this case? is spark work with mongodb in the same way with hdfs (memory+speed) ??
... View more
04-23-2018
09:38 AM
1 Kudo
Hello , I recently installed hdp-2. with hdp2.6 sandbox in vmawre ! I runned some job with spark to transform csv data and I saved them in mongodb Now I want to visualise some charts dashboard from my Mongodb data base so I added mongodb interpreter to zeeplin but it seems mongodb not good since I have a collection that contain 2 GB so I decide to work with spark ! how can I read data from Mongodb ! I must import some library for example com.mongodb.spark.sql._ and com.mongodb.spark.config.ReadConfig com.mongodb.spark.MongoSpark how I can do this in zeeplin ! in addition spark interpreter is better than mongodb interpreter in this context. Thanks in advance
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
04-16-2018
04:42 PM
How I can configure docker in order to run correctly hdp?
... View more
04-16-2018
04:39 PM
No at the begin there is no errors but after 2 days ssh root@sandbox-hdp.hortonworks.com -p 2222 didi'nt give any thing just waiting to connect but any result
... View more
04-16-2018
03:07 PM
hello , I installed docker 17.12 in my
cento7 , I downloaded the script start-sandbox-hdp-standalone_2-6-4.sh. I
edited /etc/systemd/system/docker.service.d/docker.conf I was putting: [Service] ExecStart=
ExecStart=/usr/bin/dockerd --graph=/data/docker --storage-driver=devicemapper
--storage-opt=dm.basesize=400G and I succefully installed
hortonworks docker image.All service were started normally... I was able to connect to hdp-sandbox
with ssh root@sandbox-hdp.hortonworks.com
-p 2222 ... I was putting data to hdfs and it's worked. After 2 days, hdp-sandbox doesn't work and I can't connect to hdp-sandbox.
systemctl status docker -l
● docker.service - Docker Application Container Engine Loaded:
loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset:
disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─docker.conf
Active: active (running) since Thu 2018-04-12 18:20:44 CEST; 3
days ago
Docs: https://docs.docker.com
Main PID: 13006 (dockerd)
Tasks: 598
Memory: 100.4M
CGroup: /system.slice/docker.servic
├─13006
/usr/bin/dockerd --graph=/data/docker --storage-driver=devicemapper
--storage-opt=dm.basesize=300G o├─22053 docker-containerd --config
/var/run/docker/containerd/containerd.toml o ├─29975
/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 60080 -container-ip
172.17.0.2 -container-port 60080 o ├─29992
/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 60000
-container-ip 172.17.0.2 -container-port 60000 o ├─30009
/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 50111 -container-ip
172.17.0.2 -container-port 50111 o ├─30027
/usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 50095
-container-ip 172.17.0.2 -container-port 50095 o├─30044 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 50079 -container-ip 172.17.0.2 -container-port 50079 o├─30060 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 50075 -container-ip 172.17.0.2 -container-port 50075 o├─30077 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 50070 -container-ip 172.17.0.2 -container-port 50070 o├─30094 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 42111 -container-ip 172.17.0.2 -container-port 42111 o├─30111 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 39419 -container-ip 172.17.0.2 -container-port 39419 o├─30128 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 33553 -container-ip 172.17.0.2 -container-port 33553 o├─30145 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 21000 -container-ip 172.17.0.2 -container-port 21000 o├─30162 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 19888 -container-ip 172.17.0.2 -container-port 19888 o├─30179 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 18081 -container-ip 172.17.0.2 -container-port 18081 o├─30196 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 18080 -container-ip 172.17.0.2 -container-port 18080 o├─30212 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 16030 -container-ip 172.17.0.2 -container-port 16030 o├─30228 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 16020 -container-ip 172.17.0.2 -container-port 16020 o├─30245 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 16010 -container-ip 172.17.0.2 -container-port 16010 o├─30261 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 16000 -container-ip 172.17.0.2 -container-port 16000 o├─30277 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15505 -container-ip 172.17.0.2 -container-port 15505 o├─30294 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15504 -container-ip 172.17.0.2 -container-port 15504 o├─30311 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15503 -container-ip 172.17.0.2 -container-port 15503 o├─30328 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15502 -container-ip 172.17.0.2 -container-port 15502 o├─30345 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15501 -container-ip 172.17.0.2 -container-port 15501 o├─30362 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15500 -container-ip 172.17.0.2 -container-port 15500 o├─30379 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15002 -container-ip 172.17.0.2 -container-port 15002 o├─30396 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 15000 -container-ip 172.17.0.2 -container-port 15000 o├─30413 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 11000 -container-ip 172.17.0.2 -container-port 11000 o├─30429 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10502 -container-ip 172.17.0.2 -container-port 10502 o├─30446 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10500 -container-ip 172.17.0.2 -container-port 10500 o├─30463 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10016 -container-ip 172.17.0.2 -container-port 10016 o├─30479 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10015 -container-ip 172.17.0.2 -container-port 10015 o├─30496 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10001 -container-ip 172.17.0.2 -container-port 10001 o├─30513 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 10000 -container-ip 172.17.0.2 -container-port 10000 o├─30530 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 9996 -container-ip 172.17.0.2 -container-port 9996 o├─30547 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 9995 -container-ip 172.17.0.2 -container-port 9995 o├─30565 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 9090 -container-ip 172.17.0.2 -container-port 9090 o├─30582 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 9000 -container-ip 172.17.0.2 -container-port 9000 o├─30599 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8993 -container-ip 172.17.0.2 -container-port 8993 o├─30616 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8983 -container-ip 172.17.0.2 -container-port 8983 o├─30633 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8889 -container-ip 172.17.0.2 -container-port 8889 o├─30649 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8888 -container-ip 172.17.0.2 -container-port 8888 o├─30666 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8886 -container-ip 172.17.0.2 -container-port 8886 o├─30683 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8765 -container-ip 172.17.0.2 -container-port 8765 o├─30699 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8744 -container-ip 172.17.0.2 -container-port 8744 o├─30716 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8443 -container-ip 172.17.0.2 -container-port 8443 o├─30733 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8188 -container-ip 172.17.0.2 -container-port 8188 o├─30749 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8091 -container-ip 172.17.0.2 -container-port 8091 o├─30765 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8090 -container-ip 172.17.0.2 -container-port 8090 o├─30782 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8088 -container-ip 172.17.0.2 -container-port 8088 o├─30799 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8086 -container-ip 172.17.0.2 -container-port 8086 o├─30816 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8082 -container-ip 172.17.0.2 -container-port 8082 o├─30833 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8080 -container-ip 172.17.0.2 -container-port 8080 o├─30850 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8042 -container-ip 172.17.0.2 -container-port 8042 o├─30866 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8040 -container-ip 172.17.0.2 -container-port 8040 o├─30882 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8032 -container-ip 172.17.0.2 -container-port 8032 o├─30899 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8020 -container-ip 172.17.0.2 -container-port 8020 o├─30916 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8005 -container-ip 172.17.0.2 -container-port 8005 o├─30932 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 8000 -container-ip 172.17.0.2 -container-port 8000 o├─30949 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 6188 -container-ip 172.17.0.2 -container-port 6188 o├─30965 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 6080 -container-ip 172.17.0.2 -container-port 6080 o├─30982 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 6008 -container-ip 172.17.0.2 -container-port 6008 o├─30999 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 6003 -container-ip 172.17.0.2 -container-port 6003 o├─31016 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 6001 -container-ip 172.17.0.2 -container-port 6001 o├─31033 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 5011 -container-ip 172.17.0.2 -container-port 5011 o├─31049 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 5007 -container-ip 172.17.0.2 -container-port 5007 o├─31066 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 4242 -container-ip 172.17.0.2 -container-port 4242 o├─31083 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 4200 -container-ip 172.17.0.2 -container-port 4200 o├─31100 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 4040 -container-ip 172.17.0.2 -container-port 4040 o├─31117 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 3000 -container-ip 172.17.0.2 -container-port 3000 o├─31134 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 2222 -container-ip 172.17.0.2 -container-port 22 o├─31151 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 2181 -container-ip 172.17.0.2 -container-port 2181 o├─31168 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 2100 -container-ip 172.17.0.2 -container-port 2100 o├─31185 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 2049 -container-ip 172.17.0.2 -container-port 2049 o├─31202 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0
-host-port 1988 -container-ip 172.17.0.2 -container-port 1988 o├─31218 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 1220 -container-ip 172.17.0.2 -container-port 1220 o├─31235 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 1100 -container-ip 172.17.0.2 -container-port 1100 o├─31252 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 1111 -container-ip 172.17.0.2 -container-port 111 o├─31269 /usr/bin/docker-proxy -proto tcp -host-ip
0.0.0.0 -host-port 1000 -container-ip 172.17.0.2 -container-port 1000 └─31279 docker-containerd-shim -namespace moby -workdir
/data/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/d563b4eb23fbaead69e0fa036be451a852dffe3e974f42135fedeb849b417cce
-address /var/run/docker/containerd/docker-containerd.sock -containerd-binary
/usr/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc Apr 13 10:57:45 Chaouki dockerd[13006]:
time="2018-04-13T10:57:45.951553430+02:00" level=warning
msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/:
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)" Apr 13 10:57:45 Chaouki dockerd[13006]: time="2018-04-13T10:57:45.951645956+02:00"
level=info msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/:
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)" Apr 13 10:57:45Chaouki dockerd[13006]:
time="2018-04-13T10:57:45.951725435+02:00" level=error
msg="Handler for POST /v1.35/images/create returned error: Get https://registry-1.docker.io/v2/:
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)" Apr 13 10:57:51Chaouki dockerd[13006]:
time="2018-04-13T10:57:51.049557725+02:00" level=info
msg="ignoring event" module=libcontainerd namespace=moby
topic=/containers/create type="*events.ContainerCreate" Apr 13 10:57:51 Chaouki dockerd[13006]:
time="2018-04-13T10:57:51+02:00" level=info msg="shim
docker-containerd-shim started"
address="/containerd-shim/moby/d563b4eb23fbaead69e0fa036be451a852dffe3e974f42135fedeb849b417cce/shim.sock"
debug=false module="containerd/tasks" pid=31279 Apr 13 10:57:52 Chaouki dockerd[13006]:
time="2018-04-13T10:57:52+02:00" level=error msg="failed to
disable IPv6 forwarding for container's interface all: open
/proc/sys/net/ipv6/conf/all/disable_ipv6: no such file or directory" Apr 13 10:57:52 Chaouki dockerd[13006]:
time="2018-04-13T10:57:52.547256329+02:00" level=warning
msg="Failed to disable IPv6 on all interfaces on network namespace
\"/var/run/docker/netns/38884a5d7bdb\": reexec to set IPv6 failed:
exit status 4" Apr 13 10:57:52 Chaoukidockerd[13006]:
time="2018-04-13T10:57:52.700738120+02:00" level=warning
msg="unknown container"
container=d563b4eb23fbaead69e0fa036be451a852dffe3e974f42135fedeb849b417cce
module=libcontainerd namespace=plugins.moby Apr 13 10:57:52Chaouki
dockerd[13006]: time="2018-04-13T10:57:52.849206520+02:00"
level=warning msg="unknown container"
container=d563b4eb23fbaead69e0fa036be451a852dffe3e974f42135fedeb849b417cce
module=libcontainerd namespace=plugins.moby Apr 13 10:57:53 Chaouki
dockerd[13006]: time="2018-04-13T10:57:53.328358919+02:00"
level=warning msg="unknown container"
container=d563b4eb23fbaead69e0fa036be451a852dffe3e974f42135fedeb849b417cce
module=libcontainerd namespace=plugins.moby Also, docker info givesthis warning: WARNING: devicemapper: usage
of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device Thanks
... View more
Labels:
03-23-2018
12:28 PM
I tried to instestall hdp sandbox docker image in my centos7 machine i installed docker and i downloaded the latest script start-sandbox-hdp-standalone_2.-6-4.sh when i run sh start-sandbox-hdp-standalone_2-6-4.sh
give me
Waiting for docker daemon to start up: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Unable to find image 'hortonworks/sandbox-hdp-standalone:2.6.4' locally docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'. Error response from daemon: No such container: sandbox-hdp
i would know if there is a possibilty to run hortonworks docker image in a offline machine.
Thanks in advance.
... View more
Labels:
03-23-2018
12:28 PM
Hello I have centos7 and I want to run hortonworks docker image on it,I installed docker 17.12, I downloaded the latest script to run hortonworks , the problem that my machine is offline when I run sh start-sandbox-hdp-standalone_{version}.sh Give me this error """"Waiting for docker daemon to start up: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Unable to find image 'hortonworks/sandbox-hdp-standalone:2.6.4' locally docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). See 'docker run --help'. Error response from daemon: No such container: sandbox-hdp"""" Any help
... View more
Labels:
