seemed to be an issue with my environment, today I was able to pull the image successfully: # docker pull hortonworks/sandbox-hdp:3.0.0 Trying to pull repository docker.io/hortonworks/sandbox-hdp ... sha256:3e75083ce3b38475e6825ab24a09adba229b9d67b0be2218f0ce385dcfa0452a: Pulling from docker.io/hortonworks/sandbox-hdp 70799bbf2226: Pulling fs layer [...] 055f548f8d84: Pull complete Digest: sha256:3e75083ce3b38475e6825ab24a09adba229b9d67b0be2218f0ce385dcfa0452a Status: Downloaded newer image for docker.io/hortonworks/sandbox-hdp:3.0.0 ... View more

@Pankaj Kadam the command to pull is fine, you are right that the version is part of the image name to be pulled. It is just that if you ommit the version it defaults to 'latest', which is typically also maintained as a tag in the docker hub. So there is nothing wrong with the image, it's just unusual to not maintain a 'latest' flag. ... View more

With SSH each server has an authentication key. When you connect the first time, you are asked if you really trust the server to be the correct one. If you confirm, the key is stored in ~/..ssh/known_hosts. When you connect again, this key is compared to the key the server provides to authenticate himself. If they differ you get the error message you just see. Typically it means that the host has changed, which can be an attack. I guess you have that issue, as the name/IP association has changed from you OP OP: 10.0.3.68 slave1 10.0.3.70 slave2 Now as per your error message:10.0.3.70 slave1 ? Maybe you will have to correct the IP addresses for the names? If you are sure your server/IP are correct and uncompromised, you can edit the fie /home/saichanda/.ssh/known_hosts with any editor and replace the key, or you simply do as the message proposes: ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1 ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70 Then you can again ssh the host slave1 and slave2 and accept the host key, ... View more

@Felix Albani thanks for the reply. So actually the repository is public and I have to check on my environment why it fails? The failure is from docker, so i don't think it is related to the software on the image itself. ... View more

in pyspark addPyfile should be working in the same way as in python directly. So maybe you can provide more details on your issue? The addPyfile is wokring? But the import fails? Do you get an error message? ... View more

have a look here: https://community.hortonworks.com/questions/88526/how-to-salt-row-key-in-hbase-table.html Basically it says that your prefix definition should be made in a way that you can calculate it during the query as well. In your (but perhaps simplified) example it might be even numbers prefix 000, odd numbers prefix 001. ... View more

You have created the key for root@saichanda-Optiplex-9020, but you are starting start-hbase.sh as saichanda@saichanda-OptiPlex-9020. I got this wrong from your original post, sorry. Now you have enabled root@saichanda-Optiplex-9020 to login with dmacs@10.0.3.68, but you want to enable saichanda to login as root. You will have to create the key for saichanda@saichanda-OptiPlex-9020 and copy (authorize) it to the root user of your target machines. Basically skip the sudo su - as the first command. First, if not already there, create a pair of keys as saichanda@saichanda-OptiPlex-9020, so that the public key is in /home/saichanda/.ssh/id_rsa.pub. Then still as saichanda, copy the key to the root user of the installation machines. <code>ssh-copy-id -i /root/.ssh/id_rsa root@slave1<br>ssh-copy-id -i /root/.ssh/id_rsa root@slave2<br>ssh-copy-id -i /root/.ssh/id_rsa root@master<br> In case you don't have the root password for the machines and you can only run sudo, you can copy the file ~/.ssh/id_rsa.pub with any other user to the machines in your home dir (slave1, slave2 and master, which is your current machine i guess), and then copy the key into the root dir with (on each machine): sudo cat ~/id_rsa.pub >> /root/.ssh/authorized_keys<br> ... View more

I think you will have to add the complete path, not only the file name: addPyFile (path)[source] Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. ... View more