Member since
01-02-2022
8
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3668 | 03-16-2022 12:58 PM |
03-27-2022
11:07 PM
Hi @araujo Many thanks for the explanation, please note M1 doesn't have Virtual box that is the reason i have chosen Docker from the 2nd link you provided do you have any other alternate solution instead of Virtual box ? 1st option - no since its trail 60 days
... View more
03-27-2022
09:47 AM
Hi All,
I am trying all means to find out a latest Cloudera sandbox for practicing purpose on mac m1 , yes i know and experienced many issues due to M1 not supporting most of the softwares with out any tweaks.
I have found few options
Option 1 - Virtual box -> not compatible for M1
Option 2 - Parallels -> not open source
Option 3 - VM ware fusion -> not open source
Option 4 - UTM - again some issues keyboard no sync
Option 5 - Docker desktop -> This options works with tweaks
I have picked up Docker desktop ,since there is a software available for M1 , i have then explored the options to find a suitable cloudera/hortonwork sandbox images
1. Docker - Hortonworks sandbox 3.0.1 -> failed
Hortonworks 3.0.1 -> I have tried with some tweaks but not able to configure in spite of spending a lot of time fixing the issues ,no luck
Issue/error - Failed to get D-Bus connection: No such file or directory
my explanation about the error including the log
https://stackoverflow.com/questions/71489739/deploying-hortonworks-sandbox-on-docker-on-mac-m1-installation-error-arm64/71521909?noredirect=1#comment126482374_71521909
2. Docker - Cloudera sandbox CDH 6.3.0 --> failed
docker run -m 8G --hostname=cloudera.sandbox-cdh:6.3.0 --privileged=true -t -i -v /Users/appleworld/Documents/Docker:/Src --publish-all=true -p 8888 -p 7180:7180 39f9547152c6 /usr/sbin/int docker-clouderasandbox-cdh:6.3.0 --platform amd64
the above docker run command i suspect is incorrect ? , ccan you please correct
issue/errors
jps -> no service information available
Failed to mount tmpfs at /run: Operation not permitted Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted [!!!!!!] Failed to mount API filesystems, freezing
Docker hub Cloudera sandbox image
https://hub.docker.com/r/cloudera/sandbox-cdh/tags?page=1&ordering=-last_updated
3. Docker - Cloudera quickstart vm docker image --> works fine on mac m1
https://hub.docker.com/r/cloudera/quickstart
docker run -m 8G --hostname=quickstart.cloudera --privileged=true -t -i -v /Users/appleworld/Documents/Docker:/Src --publish-all=true -p 8888 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart --platform amd64
# I have also prepared step by step installation
https://medium.com/@rakeshgopidi/installing-cloudera-quickstart-vm-through-docker-hub-on-mac-m1-879f4a3d0fd4
I really want to know why 1. Docker - Hortonworks sandbox 3.0.1 -> failed & 2. Docker - Cloudera sandbox CDH 6.3.0 is not working or not able to fix
can ideas to fix ?
... View more
Labels:
- Labels:
-
Cloudera Essentials
-
Docker
03-18-2022
04:19 AM
Hi All , After some research googling/stackoverflow/blogs/medium i got a way to try to run the Docker image on Mac m1 Ram - i have set 10 gb macbook air M1 Official Cloudera Installation directory https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html https://hub.docker.com/r/hortonworks/sandbox-hdp/tags before running the step by step installation i have tweaked the script in order to fit for Mac M1 step 0 First validate if Rosetta2 is configured correctly. You can test with alpine - very small linux docker container. docker run --rm -ti --platform linux/amd64 alpine:latest uname -a The output should be something like this: Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
Digest: sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4
Status: Downloaded newer image for alpine:latest
Linux 35d1145cff16 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 Linux Step 1 i have updated the docker-deploy-hdp30.sh in a editor and modified following line: docker run --rm -ti --platform linux/amd64 alpine:latest uname -a Step 2 navigated to assets/generate-proxy-deploy-script.sh in an editor & scrolled to the bottom. There's docker run in line 204, added a platform switch after it like this docker run --name sandbox-proxy --network=cda \\
--platform linux/amd64 \\
-v $absPath/assets/nginx.conf:/etc/nginx/nginx.conf \\ Step 3 After doing the changes triggered the command from Terminal sh docker-deploy-hdp30.sh Step 4 i see during step 3 execution the log has triggered an error , below the detailed info sh docker-deploy-hdp30.sh
+ registry=hortonworks
+ name=sandbox-hdp
+ version=3.0.1
+ proxyName=sandbox-proxy
+ proxyVersion=1.0
+ flavor=hdp
+ echo hdp
+ mkdir -p sandbox/proxy/conf.d
+ mkdir -p sandbox/proxy/conf.stream.d
+ docker pull hortonworks/sandbox-hdp:3.0.1
3.0.1: Pulling from hortonworks/sandbox-hdp
70799bbf2226: Pull complete
40963917cdad: Pull complete
3fe9adbb8d7e: Pull complete
ee3ec4e8cb3d: Pull complete
7ea5917732c0: Pull complete
2d951411620c: Pull complete
f4c5e354e7fd: Pull complete
22ffa6ef360f: Pull complete
2060aa0f3751: Pull complete
ca01ba34744d: Pull complete
83326dded077: Pull complete
eb3d71b90b73: Pull complete
bdd1cab41c81: Pull complete
500cc770c4bd: Pull complete
0cb1decd5474: Pull complete
b9591f4b6855: Pull complete
f28e56086127: Pull complete
e7de4e7d0bca: Pull complete
ec77967d2166: Pull complete
4fdcae170114: Pull complete
6347f5df8ffc: Pull complete
6a6ecc232709: Pull complete
ea845898ff50: Pull complete
02135573b1bf: Pull complete
cb0176867cd8: Pull complete
3c08321268fd: Pull complete
82e82a97c465: Pull complete
8aaaa48ed101: Pull complete
74b321ac2ac5: Pull complete
569da02c0a66: Pull complete
af40820407ef: Pull complete
Digest: sha256:7b767af7b42030fb1dd0f672b801199241e6bef1258e3ce57361edb779d95921
Status: Downloaded newer image for hortonworks/sandbox-hdp:3.0.1
docker.io/hortonworks/sandbox-hdp:3.0.1
+ docker pull hortonworks/sandbox-proxy:1.0
1.0: Pulling from hortonworks/sandbox-proxy
951bdea65c93: Pull complete
4b9047c5fbbb: Pull complete
773156407aae: Pull complete
d8524176841d: Pull complete
Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e
Status: Downloaded newer image for hortonworks/sandbox-proxy:1.0
docker.io/hortonworks/sandbox-proxy:1.0
+ '[' hdp == hdf ']'
+ '[' hdp == hdp ']'
+ hostname=sandbox-hdp.hortonworks.com
++ docker images
++ grep hortonworks/sandbox-hdp
++ awk '{print $2}'
+ version=3.0.1
+ docker network create cda
bec5a9e5019dc309591b013444b1e07154c62f3c504d8fcfcae080acfd89cf78
+ docker run --platform linux/amd64 --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp:3.0.1
9b6776f780589eabfed07022627332f964e33444309a564b8df4a8207b27e512
+ echo ' Remove existing postgres run files. Please wait'
Remove existing postgres run files. Please wait
+ sleep 2
+ docker exec -t sandbox-hdp sh -c 'rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;'
Failed to get D-Bus connection: No such file or directory
+ sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh
+ mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
+ chmod +x assets/generate-proxy-deploy-script.sh
+ assets/generate-proxy-deploy-script.sh
+ uname
+ grep MINGW
+ chmod +x sandbox/proxy/proxy-deploy.sh
+ sandbox/proxy/proxy-deploy.sh
07a314bfbc25482a05f622af9f5aabe6642332652696270c6673d740a12f0c7a Failed to get D-Bus connection: No such file or directory how do i validate the status of all the applications and more over i am unable to open the localhost:1080 1) how to check the status of all the services ? 2) whether my installation is incomplete ? 3) do i have to enable something ? can some one help ? # Additional Reference https://javamana.com/2021/02/20210206191356735v.html https://stackoverflow.com/questions/71489739/deploying-hortonworks-sandbox-on-docker-on-mac-m1-installation-error-arm64/71521909#71521909
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
03-16-2022
12:58 PM
@steven-matison Thanks for the response i have setup CDP is working fine ,since the image which i have installed is 6 year old image and it is missing KUDU/Kafka/Nifi and so...on i thought of using the HDP sandbox since it is having the latest necessary applications compared to the once i have tested the image https://hub.docker.com/r/cloudera/quickstart/ which is a 6 years old image and i have installed successfully ,however during the installation i did noticed encountered a warning but the installation went fine and i have tested some examples working fine as expected in MAC M1 , So my confusion over here is how come Horton works sandbox images is not working in MAC m1 in docker ,what i also noticed is the script which i mentioned is some thing different when compared agains the 6 year old images which i have tested. anyways things are getting complicated ,to keep it short i will stick to the 6 year old images ,sicne it is open source (free) , i don't want to touch anything which is limited trail/ asking me to provide card details (cloud(aws/gcp/cloudera cloud version).....
... View more
03-15-2022
03:43 PM
I am trying to setup "Deploying Hortonworks Sandbox on Docker" on MAC M1 for practice , unfortunately i am not able to find a solution in spite of google search /stock overflow
Official Cloudera Installation directory
https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html
Reference installation
https://javamana.com/2021/02/20210206191356735v.html
below is my configuration
Ram - 16gb - i have allocated 12 gb
Rosetta2 Installed
Docker installed Mac M1
Docker hub image link
https://hub.docker.com/r/hortonworks/sandbox-hdp/tags
docker pull hortonworks/sandbox-hdp:3.0.1 --platform linux/amd64
Script
#!/usr/bin/env sh
#This script downloads HDP sandbox along with their proxy docker container
set -x
# CAN EDIT THESE VALUES
registry="hortonworks"
name="sandbox-hdp"
version="3.0.1"
proxyName="sandbox-proxy"
proxyVersion="1.0"
flavor="hdp"
# NO EDITS BEYOND THIS LINE
# housekeeping
echo $flavor > sandbox-flavor
# create necessary folders for nginx and copy over our rule generation script there
mkdir -p sandbox/proxy/conf.d
mkdir -p sandbox/proxy/conf.stream.d
# pull and tag the sandbox and the proxy container
docker pull "$registry/$name:$version"
docker pull "$registry/$proxyName:$proxyVersion"
# start the docker container and proxy
if [ "$flavor" == "hdf" ]; then
hostname="sandbox-hdf.hortonworks.com"
elif [ "$flavor" == "hdp" ]; then
hostname="sandbox-hdp.hortonworks.com"
fi
version=$(docker images | grep $registry/$name | awk '{print $2}');
# Create cda docker network
docker network create cda 2>/dev/null
# Deploy the sandbox into the cda docker network
docker run --privileged --name $name -h $hostname --network=cda --network-alias=$hostname -d "$registry/$name:$version"
echo " Remove existing postgres run files. Please wait"
sleep 2
docker exec -t "$name" sh -c "rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;"
#Deploy the proxy container.
sed 's/sandbox-hdp-security/sandbox-hdp/g' assets/generate-proxy-deploy-script.sh > assets/generate-proxy-deploy-script.sh.new
mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
chmod +x assets/generate-proxy-deploy-script.sh
assets/generate-proxy-deploy-script.sh 2>/dev/null
#check to see if it's windows
if uname | grep MINGW; then
sed -i -e 's/\( \/[a-z]\)/\U\1:/g' sandbox/proxy/proxy-deploy.sh
fi
chmod +x sandbox/proxy/proxy-deploy.sh 2>/dev/null
sandbox/proxy/proxy-deploy.sh
LOG
➜ Documents sh docker-deploy-hdp30.sh
+ registry=hortonworks
+ name=sandbox-hdp
+ version=3.0.1
+ proxyName=sandbox-proxy
+ proxyVersion=1.0
+ flavor=hdp
+ echo hdp
+ mkdir -p sandbox/proxy/conf.d
+ mkdir -p sandbox/proxy/conf.stream.d
+ docker pull hortonworks/sandbox-hdp:3.0.1
3.0.1: Pulling from hortonworks/sandbox-hdp
70799bbf2226: Pull complete
40963917cdad: Pull complete
3fe9adbb8d7e: Pull complete
ee3ec4e8cb3d: Pull complete
7ea5917732c0: Pull complete
2d951411620c: Pull complete
f4c5e354e7fd: Pull complete
22ffa6ef360f: Pull complete
2060aa0f3751: Pull complete
ca01ba34744d: Pull complete
83326dded077: Pull complete
eb3d71b90b73: Pull complete
bdd1cab41c81: Pull complete
500cc770c4bd: Pull complete
0cb1decd5474: Pull complete
b9591f4b6855: Pull complete
f28e56086127: Pull complete
e7de4e7d0bca: Pull complete
ec77967d2166: Pull complete
4fdcae170114: Pull complete
6347f5df8ffc: Pull complete
6a6ecc232709: Pull complete
ea845898ff50: Pull complete
02135573b1bf: Pull complete
cb0176867cd8: Pull complete
3c08321268fd: Pull complete
82e82a97c465: Pull complete
8aaaa48ed101: Pull complete
74b321ac2ac5: Pull complete
569da02c0a66: Pull complete
af40820407ef: Pull complete
Digest: sha256:7b767af7b42030fb1dd0f672b801199241e6bef1258e3ce57361edb779d95921
Status: Downloaded newer image for hortonworks/sandbox-hdp:3.0.1
docker.io/hortonworks/sandbox-hdp:3.0.1
+ docker pull hortonworks/sandbox-proxy:1.0
1.0: Pulling from hortonworks/sandbox-proxy
951bdea65c93: Pull complete
4b9047c5fbbb: Pull complete
773156407aae: Pull complete
d8524176841d: Pull complete
Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e
Status: Downloaded newer image for hortonworks/sandbox-proxy:1.0
docker.io/hortonworks/sandbox-proxy:1.0
+ '[' hdp == hdf ']'
+ '[' hdp == hdp ']'
+ hostname=sandbox-hdp.hortonworks.com
++ docker images
++ grep hortonworks/sandbox-hdp
++ awk '{print $2}'
+ version=3.0.1
+ docker network create cda
ce52e1d04aa49c2da70dfd7405b95fc2e2184fed9625feaea0ddafb9ab5c361d
+ docker run --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp:3.0.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
3ab0f0281918fc10d008c93aa7a844302e974b696e00788df76cde779fdec46d
+ echo ' Remove existing postgres run files. Please wait'
Remove existing postgres run files. Please wait
+ sleep 2
+ docker exec -t sandbox-hdp sh -c 'rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;'
Failed to get D-Bus connection: No such file or directory
+ sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh
+ mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
+ chmod +x assets/generate-proxy-deploy-script.sh
+ assets/generate-proxy-deploy-script.sh
+ uname
+ grep MINGW
+ chmod +x sandbox/proxy/proxy-deploy.sh
+ sandbox/proxy/proxy-deploy.sh
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
fbe0c31a680f51db9c91367df65493ea1cf874cc900466dd943824dc19bd9e74
# References
https://www.cloudera.com/tutorials/learning-the-ropes-of-the-hdp-sandbox.htmlhttps://javamana.com/2021/02/20210206191356735v.html
Is this issue have any solution or not possible to fix ?
... View more
Labels:
03-15-2022
03:59 AM
Hi @ask_bill_brooks I have installed the docker image for learning/practice on Docker 1) I have retrieved the docker images from docker hub 2) RAM - 16gb 3) I am able to succefully launch all the applications expect for few I am unable to connect to Hue upon restarting Cloudera Manager Express Many thanks for a detailed explanation , i have tried accesing CDP ,but again i need to enroll for 60 day trail account and after that it expires. I have also explored the option to install in Local Mac m1 but i don't messs up the latop with manual installation. # I have noticed there is latest version which is atleast 3 years old docker pull hortonworks/sandbox-hdp:3.0.1 any ideas ? currently i would prefer on premise ,since cloud i am always worried we need to provide the card details
... View more
03-02-2022
07:20 AM
I am trying to fetch a count of total columns for a list of individual tables/views from Impala from the same schema.
however i wanted to scan through all the tables from that schema to capture the counts of columns of all the tables in a single query ?
i have already performed a similar exercise from Oracle Exa data ,however since i a new to Impala is there a way to capture all the tables in a single query ?
Exadata query for reference
# Oracle Exadata query i used
select owner, table_name as view_name, count(*) as counts
from dba_tab_cols /*DBA_TABLES_COLUMNS*/
where (owner, table_name) in
(
select owner, view_name
from dba_views /*DBA_VIEWS*/
where 1=1
and owner='DESIRED_SCHEMA_NAME'
)
group by owner ,table_name
order by counts desc;
impala /hive
DESCRIBE schemaName.tableName;
# Impala /Hive
how to find out ? if i need to run a single query check i was following the below
for multiple tables/view how to find out the total column counts ?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
01-02-2022
01:12 PM
Dear Experts,
I have setup Cloudera Docker quick start on macos.
As part of the Big Data Book reading and following some excercises.
Cloudera Version
rpm -qa | grep hadoop
hadoop-mapreduce-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-yarn-nodemanager-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-journalnode-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-yarn-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-0.20-mapreduce-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-libhdfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-yarn-resourcemanager-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-namenode-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-datanode-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-conf-pseudo-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-libhdfs-devel-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-kms-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-httpfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-doc-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-client-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-mapreduce-historyserver-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-secondarynamenode-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-hdfs-fuse-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
hadoop-yarn-proxyserver-2.6.0+cdh5.7.0+1280-1.cdh5.7.0.p0.92.el6.x86_64
unfortunately i am not able to setup Mavern and java on Cloudera docker image quick start
https://github.com/tomwhite/hadoop-book
Building and Running
To build the code, you will first need to have installed Maven and Java.
Question - how to install Mavern and Java in Cloudera Docker Started Image ?
% mvn package -DskipTests
Then type
can some suggest a way forward
... View more
Labels: