Member since
01-02-2022
8
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3985 | 03-16-2022 12:58 PM |
11-21-2023
06:10 AM
In Hive, metadata related to tables and columns is typically stored in the 'hive' database, specifically within the 'TBLS' and 'COLUMNS_V2' tables in the 'metastore' database. It is not recommended for users to query the metadata directly. Instead, users can leverage the 'sys' database tables. Here is a modified query that utilizes the 'hive' database tables: sql USE sys;
-- Get the count of columns for all tables
SELECT
t.tbl_name AS TABLE_NAME,
COUNT(c.column_name) AS COLUMN_COUNT
FROM
tbls t
JOIN
columns_v2 c
ON
t.tbl_id = c.cd_id
GROUP BY
t.tbl_name; Explanation: The 'sys.tbls' table contains information about tables, while the 'sys.columns_v2' table contains information about columns. We join these tables on the 'TBL_ID' and 'CD_ID' columns to retrieve information about columns for each table. The 'COUNT(c.COLUMN_NAME)' expression calculates the count of columns for each table. This query provides a list of tables along with the count of columns for each table, using the 'sys' database tables."
... View more
03-27-2022
11:07 PM
Hi @araujo Many thanks for the explanation, please note M1 doesn't have Virtual box that is the reason i have chosen Docker from the 2nd link you provided do you have any other alternate solution instead of Virtual box ? 1st option - no since its trail 60 days
... View more
03-24-2022
12:07 AM
@BigDataAvengers, Has any of the replies helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
03-18-2022
04:19 AM
Hi All , After some research googling/stackoverflow/blogs/medium i got a way to try to run the Docker image on Mac m1 Ram - i have set 10 gb macbook air M1 Official Cloudera Installation directory https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html https://hub.docker.com/r/hortonworks/sandbox-hdp/tags before running the step by step installation i have tweaked the script in order to fit for Mac M1 step 0 First validate if Rosetta2 is configured correctly. You can test with alpine - very small linux docker container. docker run --rm -ti --platform linux/amd64 alpine:latest uname -a The output should be something like this: Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
Digest: sha256:d6d0a0eb4d40ef96f2310ead734848b9c819bb97c9d846385c4aca1767186cd4
Status: Downloaded newer image for alpine:latest
Linux 35d1145cff16 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 Linux Step 1 i have updated the docker-deploy-hdp30.sh in a editor and modified following line: docker run --rm -ti --platform linux/amd64 alpine:latest uname -a Step 2 navigated to assets/generate-proxy-deploy-script.sh in an editor & scrolled to the bottom. There's docker run in line 204, added a platform switch after it like this docker run --name sandbox-proxy --network=cda \\
--platform linux/amd64 \\
-v $absPath/assets/nginx.conf:/etc/nginx/nginx.conf \\ Step 3 After doing the changes triggered the command from Terminal sh docker-deploy-hdp30.sh Step 4 i see during step 3 execution the log has triggered an error , below the detailed info sh docker-deploy-hdp30.sh
+ registry=hortonworks
+ name=sandbox-hdp
+ version=3.0.1
+ proxyName=sandbox-proxy
+ proxyVersion=1.0
+ flavor=hdp
+ echo hdp
+ mkdir -p sandbox/proxy/conf.d
+ mkdir -p sandbox/proxy/conf.stream.d
+ docker pull hortonworks/sandbox-hdp:3.0.1
3.0.1: Pulling from hortonworks/sandbox-hdp
70799bbf2226: Pull complete
40963917cdad: Pull complete
3fe9adbb8d7e: Pull complete
ee3ec4e8cb3d: Pull complete
7ea5917732c0: Pull complete
2d951411620c: Pull complete
f4c5e354e7fd: Pull complete
22ffa6ef360f: Pull complete
2060aa0f3751: Pull complete
ca01ba34744d: Pull complete
83326dded077: Pull complete
eb3d71b90b73: Pull complete
bdd1cab41c81: Pull complete
500cc770c4bd: Pull complete
0cb1decd5474: Pull complete
b9591f4b6855: Pull complete
f28e56086127: Pull complete
e7de4e7d0bca: Pull complete
ec77967d2166: Pull complete
4fdcae170114: Pull complete
6347f5df8ffc: Pull complete
6a6ecc232709: Pull complete
ea845898ff50: Pull complete
02135573b1bf: Pull complete
cb0176867cd8: Pull complete
3c08321268fd: Pull complete
82e82a97c465: Pull complete
8aaaa48ed101: Pull complete
74b321ac2ac5: Pull complete
569da02c0a66: Pull complete
af40820407ef: Pull complete
Digest: sha256:7b767af7b42030fb1dd0f672b801199241e6bef1258e3ce57361edb779d95921
Status: Downloaded newer image for hortonworks/sandbox-hdp:3.0.1
docker.io/hortonworks/sandbox-hdp:3.0.1
+ docker pull hortonworks/sandbox-proxy:1.0
1.0: Pulling from hortonworks/sandbox-proxy
951bdea65c93: Pull complete
4b9047c5fbbb: Pull complete
773156407aae: Pull complete
d8524176841d: Pull complete
Digest: sha256:42e4cfbcbb76af07e5d8f47a183a0d4105e65a1e7ef39fe37ab746e8b2523e9e
Status: Downloaded newer image for hortonworks/sandbox-proxy:1.0
docker.io/hortonworks/sandbox-proxy:1.0
+ '[' hdp == hdf ']'
+ '[' hdp == hdp ']'
+ hostname=sandbox-hdp.hortonworks.com
++ docker images
++ grep hortonworks/sandbox-hdp
++ awk '{print $2}'
+ version=3.0.1
+ docker network create cda
bec5a9e5019dc309591b013444b1e07154c62f3c504d8fcfcae080acfd89cf78
+ docker run --platform linux/amd64 --privileged --name sandbox-hdp -h sandbox-hdp.hortonworks.com --network=cda --network-alias=sandbox-hdp.hortonworks.com -d hortonworks/sandbox-hdp:3.0.1
9b6776f780589eabfed07022627332f964e33444309a564b8df4a8207b27e512
+ echo ' Remove existing postgres run files. Please wait'
Remove existing postgres run files. Please wait
+ sleep 2
+ docker exec -t sandbox-hdp sh -c 'rm -rf /var/run/postgresql/*; systemctl restart postgresql-9.6.service;'
Failed to get D-Bus connection: No such file or directory
+ sed s/sandbox-hdp-security/sandbox-hdp/g assets/generate-proxy-deploy-script.sh
+ mv -f assets/generate-proxy-deploy-script.sh.new assets/generate-proxy-deploy-script.sh
+ chmod +x assets/generate-proxy-deploy-script.sh
+ assets/generate-proxy-deploy-script.sh
+ uname
+ grep MINGW
+ chmod +x sandbox/proxy/proxy-deploy.sh
+ sandbox/proxy/proxy-deploy.sh
07a314bfbc25482a05f622af9f5aabe6642332652696270c6673d740a12f0c7a Failed to get D-Bus connection: No such file or directory how do i validate the status of all the applications and more over i am unable to open the localhost:1080 1) how to check the status of all the services ? 2) whether my installation is incomplete ? 3) do i have to enable something ? can some one help ? # Additional Reference https://javamana.com/2021/02/20210206191356735v.html https://stackoverflow.com/questions/71489739/deploying-hortonworks-sandbox-on-docker-on-mac-m1-installation-error-arm64/71521909#71521909
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
03-16-2022
12:58 PM
@steven-matison Thanks for the response i have setup CDP is working fine ,since the image which i have installed is 6 year old image and it is missing KUDU/Kafka/Nifi and so...on i thought of using the HDP sandbox since it is having the latest necessary applications compared to the once i have tested the image https://hub.docker.com/r/cloudera/quickstart/ which is a 6 years old image and i have installed successfully ,however during the installation i did noticed encountered a warning but the installation went fine and i have tested some examples working fine as expected in MAC M1 , So my confusion over here is how come Horton works sandbox images is not working in MAC m1 in docker ,what i also noticed is the script which i mentioned is some thing different when compared agains the 6 year old images which i have tested. anyways things are getting complicated ,to keep it short i will stick to the 6 year old images ,sicne it is open source (free) , i don't want to touch anything which is limited trail/ asking me to provide card details (cloud(aws/gcp/cloudera cloud version).....
... View more
01-03-2022
03:32 AM
@bdworld2 Take a look of this doc which can help about the Maven use. https://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo.html
... View more