Member since
12-10-2015
76
Posts
30
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2330 | 03-10-2021 08:35 AM | |
1622 | 07-25-2019 06:34 AM | |
3762 | 04-20-2016 10:03 AM | |
2845 | 04-11-2016 03:07 PM |
03-10-2021
08:35 AM
Ok, I solved it using the cdp cli. The problem was that from web ui of Cloudera Management Console it is not possible to insert the identity for ranger, while from cli it is possible. Below are the scripts for creating the data lake environment: cdp environments create-azure-environment \
--environment-name <ENV_NAME> \
--credential-name <CREDENTIAL_NAME> \
--region "AZURE_REGIONE_NAME" \
--security-access cidr=0.0.0.0/0 \
--no-enable-tunnel \
--public-key "ssh-rsa ..." \
--log-storage storageLocationBase=abfs://logs@<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net,managedIdentity=/subscriptions/xxx/resourcegroups/<RG_NAME>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<ENV_NAME>-LoggerIdentity \
--use-public-ip \
--existing-network-params networkId=<ENV_NAME>-Vnet,resourceGroupName=<ENV_NAME>,subnetIds=CDP \
--free-ipa instanceCountByGroup=1
cdp environments set-id-broker-mappings \
--environment-name <ENV_NAME> \
--data-access-role /subscriptions/xxx/resourceGroups/<RG_NAME>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<ENV_NAME>-DataAccessIdentity \
--ranger-audit-role /subscriptions/xxx/resourceGroups/<RG_NAME>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<ENV_NAME>-RangerIdentity \
--set-empty-mappings
cdp datalake create-azure-datalake \
--datalake-name <ENV_NAME> \
--environment-name <ENV_NAME> \
--cloud-provider-configuration managedIdentity=/subscriptions/xxx/resourcegroups/<RG_NAME>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<ENV_NAME>-AssumerIdentity,storageLocation=abfs://data@<STORAGE_ACCOUNT_NAME>.dfs.core.windows.net \
--scale LIGHT_DUTY \
--runtime 7.2.7 Here instead the dockerfile for those wishing to have the cdp-cli in cointainer: FROM python
RUN apt update \
&& apt upgrade -y \
&& apt install -y \
groff \
less
RUN git clone https://github.com/cloudera/cdpcli.git \
&& cd cdpcli \
&& pip install .
... View more
07-25-2019
06:29 AM
Hi @jessica moore, see this https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md
... View more
04-24-2018
12:08 PM
The input file is in the text/html format, and the output file must be in csv format, because it will feed the database.
... View more
09-06-2018
07:15 PM
I got the same issue! At the end, I deactivated and activated back SSL, so it generated new certificates for Nifi Cluster.
... View more
11-29-2016
10:07 PM
The maximum number of files in HDFS depends on the amount of memory
available for the NameNode. Each file object and each block object
takes about 150 bytes of the memory. For example, if you have 10 million files
and each file has 1 one block each, then you would need about 3GB of
memory for the NameNode. If I had 10 million files, each using a block, then we would be using:
10 million + 10 million = 20 million * 150 = 3,000,000,000 bytes = 3 GB MEMORY. Keep in mind the NameNode will need memory for other processes. So to support 10 million files then your NameNode will need much more than 3GB of memory.
... View more
04-26-2016
10:23 AM
1 Kudo
Ok, in advaced zookeeper.log4j set: log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log ${zookeeper.log.dir} is missing in defaul configuration.
... View more
04-21-2016
02:29 PM
3 Kudos
You need use sqoop for direct import on hive form sql. First download sql jdbc
https://www.microsoft.com/en-us/download/details.aspx?id=11774 place jar on sqoop master server:
/usr/hdp/current/sqoop-server/lib use this command for import: import --connect jdbc:sqlserver://[SQL_SERVER_NAME]:[SQL_PORT]/[DB-NAME] --username "[SQL_USERNAME]" --password "[PASSWORD]" --query '[INSERT QUERY HERE] WHERE $CONDITIONS' -m 1 --hive-import --hive-database [DB_HIVE] --create-hive-table --hive-table [HIVE_TABLE]
if use --query you must have WHERE $CONDITIONS on query.
... View more
08-16-2016
06:44 AM
@Davide Isoardi Good Afternoon, sir.
May i ask you something ? Is http://www.ecubecenter.it/Hadoop-Big-Data-Twitter-Map/ using this https://github.com/disoardi/tweetsdemo_with_ambari_views. It has been a while that i lost my mind to work my undergraduate project. Maybe, i will start again from this. Thanks before
... View more
04-15-2016
07:13 PM
@Davide Isoardi - You are welcome! 🙂
... View more
- « Previous
-
- 1
- 2
- Next »