:~/HDP# sh docker-deploy-hdp30.sh + registry=hortonworks + name=sandbox-hdp + version=3.0.1 + proxyName=sandbox-proxy + proxyVersion=1.0 + flavor=hdp + echo hdp + mkdir -p sandbox/proxy/conf.d + mkdir -p sandbox/proxy/conf.stream.d + docker pull hortonworks/sandbox-hdp:3.0.1 3.0.1: Pulling from hortonworks/sandbox-hdp 70799bbf2226: Downloading [================================> ] 47.17MB/72.85MB 40963917cdad: Pulling fs layer 70799bbf2226: Downloading [===================================> ] 51.45MB/72.85MB ee3ec4e8cb3d: Downloading [=======> ] 57.28MB/368.8MB ee3ec4e8cb3d: Downloading [=======> ] 57.81MB/368.8MB ee3ec4e8cb3d: Downloading [========> ] 64.24MB/368.8MB 7ea5917732c0: Download complete 2d951411620c: Download complete f4c5e354e7fd: Downloading ca01ba34744d: Waiting 83326dded077: Waiting eb3d71b90b73: Waiting bdd1cab41c81: Waiting 500cc770c4bd: Waiting 0cb1decd5474: Waiting b9591f4b6855: Waiting f28e56086127: Waiting e7de4e7d0bca: Waiting ec77967d2166: Waiting 4fdcae170114: Waiting 6347f5df8ffc: Waiting 6a6ecc232709: Waiting ea845898ff50: Waiting 02135573b1bf: Waiting cb0176867cd8: Waiting 3c08321268fd: Pulling fs layer 82e82a97c465: Waiting 3c08321268fd: Waiting 74b321ac2ac5: Waiting 569da02c0a66: Waiting af40820407ef: Pulling fs layer af40820407ef: Waiting unauthorized: authentication required + docker pull hortonworks/sandbox-proxy:1.0 1.0: Pulling from hortonworks/sandbox-proxy Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e Status: Image is up to date for hortonworks/sandbox-proxy:1.0 + [ hdp = hdf ] + [ hdp = hdp ] + hostname=sandbox-hdp.hortonworks.com + docker images + grep hortonworks/sandbox-hdp + awk {print $2} + version= + docker network create cda + docker run --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp: docker: invalid reference format. See 'docker run --help'. + echo Remove existing postgres run files. Please wait Remove existing postgres run files. Please wait + sleep 2 + docker exec -t sandbox-hdp sh -c rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service; Error: No such container: sandbox-hdp + sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh + mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh + chmod +x assets/generate-proxy-deploy-script.sh + assets/generate-proxy-deploy-script.sh + uname + grep MINGW + chmod +x sandbox/proxy/proxy-deploy.sh + sandbox/proxy/proxy-deploy.sh sandbox-proxy baed363d86a7a73e7712b838df09c2aeda99a6b15f0333e9f8e6f445902a383b docker: Error response from daemon: driver failed programming external connectivity on endpoint sandbox-proxy (d53fe3e3eba2b296b2f6acaa6d8202732308d93cd987157130dfdddef1b82170): Error starting userland proxy: listen tcp 0.0.0.0:8042: bind: address already in use. i have gone through the script,:- it means docker pull hortonworks/sandbox-hdp:3:0:1 cat docker-deploy-hdp30.sh # CAN EDIT THESE VALUES registry="hortonworks" name="sandbox-hdp" version="3.0.1" proxyName="sandbox-proxy" proxyVersion="1.0" flavor="hdp" # NO EDITS BEYOND THIS LINE # housekeeping echo $flavor > sandbox-flavor # create necessary folders for nginx and copy over our rule generation script there mkdir -p sandbox/proxy/conf.d mkdir -p sandbox/proxy/conf.stream.d # pull and tag the sandbox and the proxy container docker pull "$registry/$name:$version"........ ======== but it fails with the error "unauthorized: authentication required". even though if i executed docker pull hortonworks/sandbox-hdp:3:0:1 also i had logged into docker with docker login -u <userid> -p <password> ... View more

@saichand akella That should be ok 🙂 you can accept this answer. ... View more

The solution provided by @Harald via link is good enough as a solution. The only prequesite is that the cluster should be on top of Azure Data Lake and not azure datastorage. If its a cluster then solution should be replicated on all nodes. If it's only Edge node then solution should only be applied only edge node. By solution I mean the solution in https://community.hortonworks.com/content/kbentry/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html. Thanks ... View more

@Sim kaur Can you share the latest version of these 2 files /var/log/hive/*.err and /var/log/hive/*.log ... View more

hi Shu you wrote and you are right ..its importing only one record with m1 on each run . I tried 4 sqoop loads and got 4 versions. that means sqoop import to hbase table will not store all the versions but from hbase shell it will store all the versions) is it a bug or feature ? is Hortonworks aware of this and what is their comment ? ... View more

The input file is in the text/html format, and the output file must be in csv format, because it will feed the database. ... View more

Thanks for the Answer Herald, your understanding of my question is correct. All I'm trying is to get the format correct for my hive table column names. I mean I just put the formatted file on the hdfs and read using the external hive table. ... View more