This article will show you how to interact with Atlas APIs in CDP-public to create tags and associate tags with entities (in preparation for use with Ranger's tag based policies)   In Cloudera CDP-public offering, Apache Atlas is a part of SDX DataLake cluster that is created when you create your first Environment: Introduction to Data Lakes Pre-requisites A. First, you will need to find the Atlas endpoint using the Cloudera CDP management console: Accessing Data Lake services Sample Atlas endpoint: https://pse-722-cdp-xxxxx.cloudera.site/pse-722-cdp-dl/cdp-proxy-api/atlas/api/atlas/   B. Next, you will need to set your user's workload password Setting the workload password Now you can use the following sample bash code to interact with Atlas APIs from a CentOS instance outside CDP:   From Atlas endpoint, you can extract the first 2 params below. You will also need to set your username and password: export datalake_name='pse-722-cdp-dl' export lake_ip='pse-722-cdp-xxxxx.cloudera.site' export user='abajwa' export password='nicepassword' export atlas_curl="curl -k -u ${user}:${password}" export atlas_url="https://${lake_ip}:443/${datalake_name}/cdp-proxy-api/atlas/api/atlas" After forming the above variables, you can use them to run some basic GET and POST commands to import tags and glossary into Atlas.     #test API by fetching Atlas typedefs ${atlas_curl} ${atlas_url}/v2/types/typedefs #download sample Glossary wget https://github.com/abajwa-hw/masterclass/blob/master/ranger-atlas/HortoniaMunichSetup/data/export-glossary.zip #import sample Glossary into Atlas curl -v -k -X POST -u ${user}:${password} -H "Accept: application/json" -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@export-glossary.zip ${atlas_url}/import #import sample tags wget https://github.com/abajwa-hw/masterclass/raw/master/ranger-atlas/HortoniaMunichSetup/data/classifications.json #import sample tags into Atlas curl -v -k -X POST -u ${user}:${password} -H "Accept: application/json" -H "Content-Type: application/json" ${atlas_url}/v2/types/typedefs -d @classifications.json     At this point, you should be able to see the newly imported tags and glossary entities in your Atlas UI. Next, you can search for any Hive entity (this should get automatically created in Atlas when the Hive table is created) and associate it with a tag.     #find airlines_new_orc.airports entity in Atlas ${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_table?attr:qualifiedName=airlines_new_orc.airports@cm #fetch guid for airlines_new_orc.airports guid=$(${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_table?attr:qualifiedName=airlines_new_orc.airports@cm | jq '.entity.guid' | tr -d '"') #use guid to associate a tag REFERENCE_DATA to airlines_new_orc.airports entity ${atlas_curl} ${atlas_url}/entities/${guid}/traits \ -X POST -H 'Content-Type: application/json' \ --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"REFERENCE_DATA","values":{}}' #confirm now entity shows REFERENCE_DATA tag (also will be visible via UI) ${atlas_curl} ${atlas_url}/v2/entity/uniqueAttribute/type/hive_table?attr:qualifiedName=airlines_new_orc.airports@cm | grep REFERENCE_DATA     Now that you have entities tagged with a tag, you can use Ranger to create a "tag-based policy". Tag-based Services and Policies Other sample code to associate tags Atlas: How to automate associating tags/classifications to HDFS/Hive/HBase/Kafka entities using REST APIs   ... View more

Security/Governance/GDPR Demo on CDP-Private Cloud Base 7.x Summary This article explains how to quickly set up Cloudera Security/Governance/GDPR (Worldwide Bank) demo using Cloudera Data Platform - Private Cloud Base (formerly known as CDP-Data Center). It can be deployed either on AWS using AMI or on your own setup via provided script What's included Single node CDP 7.1.7 including: Cloudera Manager (60-day trial license included) - for managing the services Kerberos - for authentication (via local MIT KDC) Ranger - for authorization (via both resource/tag-based policies for access and masking) Atlas - for governance (classification/lineage/search) Zeppelin - for running/visualizing Hive queries Impala/Hive 3 - for Sql access and ACID capabilities Spark/HiveWarehouseConnector - for running secure SparkSQL queries Worldwide Bank artifacts Demo hive tables Demo tags/attributes and lineage in Atlas Demo Zeppelin notebooks to walk through a demo scenario Ranger policies across HDFS, Hive/Impala, Hbase, Kafka, SparkSQL to showcase: Tag-based policies across HDP components Row-level filtering in Hive columns Dynamic tag-based masking in Hive/Impala columns Hive UDF execution authorization Atlas capabilities like  Classifications (tags) and attributes Tag propagation Data lineage Business glossary: categories and terms GDPR Scenarios around consent and data erasure via Hive ACID Hive ACID / MERGE labs Option 1: Steps to deploy on your own setup Launch a vanilla Centos 7 VM (8 cores / 64GB RAM / 100GB storage) and perform the documented network prereqs. Then set up a single node CDP cluster using this GitHub (instead of "base" CM template choose the "wwbank_krb.json" template) as follows: yum install -y git #setup KDC curl -sSL https://gist.github.com/abajwa-hw/bca3d23fe146c3ebd59a9b5fd19480a3/raw | sudo -E sh git clone https://github.com/fabiog1901/SingleNodeCDPCluster.git cd SingleNodeCDPCluster ./setup_krb.sh gcp templates/wwbank_krb.json #Setup worldwide bank demo using script curl -sSL https://raw.githubusercontent.com/abajwa-hw/masterclass/master/ranger-atlas/setup-dc-703.sh | sudo -E bash Once the script completes, restart Zeppelin once (via CM) for it to pick up the demo notebooks Option 2: Steps to launch prebuilt AMI on AWS Login in to the AWS EC2 console using your credentials Select the AMI from ‘N. California’ region by clicking one of the links below: CDP 7.1.7 here CDP 7.1.4 here CDP 7.1.3 here  CDP 7.1.1 here  CDP 7.0.3 here  Now choose instance type: select m4.4xlarge and click Next: Note: If you choose a smaller instance type from the above recommendation, not all services may come up. In Configure Instance Details, ensure Auto-assign Public IP is enabled and click Next: In Add storage, use at least 100 GB and click Next: In Add Tags, add tags needed to prevent instances from being terminated. Click Next: In Configure Security Group, create a new security group and select All traffic and open all ports to only your IP. The below image displays my IP address:  In Review Instance Launch, review your settings and click Launch: Create and download a new key pair (or choose an existing one). Then click Launch instances: Click the shown link under Your instances are now launching: This opens the EC2 dashboard that shows the details of your launched instance: Make note of your instance’s Public IP (which will be used to access your cluster). If the Public IP is blank, wait for a couple of minutes for this to be populated. After five to ten minutes, open the below URL in your browser to access Cloudera Manager (CM) console: http://<PUBLIC IP>:7180. Login as admin/admin: At this point, CM may still be in the process of starting all the services. You can tell by the presence of the blue operation notification near the bottom left of the page. If so, just wait until it is done: (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP>​ Tail the startup log: tail -f /var/log/cdp_startup.log Once you see “cluster is ready!”, you can proceed.  Once the blue operation notification disappears and all the services show a green checkmark, the cluster is fully up. If any services fail to start, use the hamburger icon next to SingleNodeCluster > Start button to start. Accessing cluster resources CDP urls Access CM at :7180 as admin/admin Access Ranger at :6080. Ranger login is admin/BadPass#1 Access Atlas at :31000. Atlas login is admin/BadPass#1 Access Zeppelin at :8885. Zeppelin users logins are: joe_analyst = BadPass#1 ivanna_eu_hr = BadPass#1 etl_user = BadPass#1 Demo walkthrough Run queries as joe_analyst Open Zeppelin and login as joe_analyst. Find his notebook by searching for "worldwide" using the text field under the Notebook section. Select the notebook called Worldwide Bank - Joe Analyst: On the first launch of the notebook, you will be prompted to choose interpreters. You can keep the defaults. Ensure to click the Save button: Run through the notebook. This notebook shows the following:  MRN/password masked via tag policy. The following shows the Ranger policy that enables this: In the Dynamic Column Level Masking, address, nationalID, credit card numbers are masked using Hive column policies specified in Ranger. Notice that birthday and age columns are masked using the custom mask: It also shows a prohibition policy where zipcode, insuranceID, and blood type cannot be combined in a query: It shows tag-based policies.  Attempts to access an object tagged with EXPIRES_ON accessed after the expiry date will be denied. As we will show later, the fed_tax column of tax_2015 table is tagged in Atlas as EXPIRED_ON with an expiry date of 2016. Hence, it should not be allowed to be queried: Also attempts to access objects tagged with PII will be denied as per policy. Only HR is allowed. As we will show later, the SSN column of tax_2015 table is tagged as PII in Atlas: Attempts to access cost_savings.claim_savings table as an analyst will fail because there is a policy that a minimum of 60% data quality score is required for analysts. As we will see, this table is tagged in Atlas as having a score of 51%: The same queries can also be run via SparkSQL via spark-shell (as described above). The following is the sample query for joe_analyst: hive.execute("SELECT surname, streetaddress, country, age, password, nationalid, ccnumber, mrn, birthday FROM worldwidebank.us_customers").show(10) hive.execute("select zipcode, insuranceid, bloodtype from worldwidebank.ww_customers").show(10) hive.execute("select * from cost_savings.claim_savings").show(10) Confirm using Ranger audits that the queries ran as joe_analyst. Also, notice that column names, masking types, IPs, and policy IDs were captured. Also notice tags (such as DATA_QUALITY or PII) are captured along with their attributes. Also, notice that these audits were captured for operations across Hive, Hbase, Kafka, and HDFS: Run queries as ivanna_eu_hr Once services are up, open Ranger UI and also log in to Zeppelin as ivanna_eu_hr.  Find her notebook by searching for hortonia using the text field under the Notebook section. Select the notebook called Worldwide Bank - Ivana EU HR: On the first launch of the notebook, you may be prompted to choose interpreters. You can keep the defaults, ensure you click Save button: Run through the notebook cells using Play button at the top right of each cell (or Shift-Enter): This notebook highlights the following: Row-level filtering: As Ivana can only see data for European customers who have given consent (even though she is querying ww_customers table which contains both US and EU customers). The following is the Ranger Hive policy that enables this feature: It also shows that since Ivana is part of the HR group, there are no policies that limit her access. Hence, so she can see raw passwords, nationalIDs, credit card numbers, MRN #, birthdays, etc. The last cells show that tag-based policies. Once you successfully run the notebook, you can open the Ranger Audits to show the policies and that the queries ran as her and that row filtering occurred (notice ROW_FILTER access type): Run queries as etl_user Similarly, you can log in to Zeppelin as etl_user and run his notebook as well This notebook shows how an admin would handle GDPR scenarios like the following using Hive ACID capabilities: When a customer withdraws consent (so they no longer appear in searches) When a customer requests their data to be erased Run Hive/Impala queries from Hue Alternatively, you can log in to Hue as joe_analyst and select Query > Editor > Hive, and click Saved queries to run Joe's sample queries via Hive:      You can also switch the editor to Impala to run Joe's sample queries via Impala to show tag-based access policy working for Impala:  In CDP 7.1.1, Impala also supports column-based masking: Alternatively, you can log in to Hue as ivanna_eu_hr and click Saved queries to run Ivanna's sample queries via Hive: Run SparkSQL queries via Hive Warehouse Connector (HWC) To run secure SparkSQL queries (using Hive Warehouse Connector): Connect to instance via SSH using your keypair Authenticate as the user you want to run queries as via keytabs: kinit -kt /etc/security/keytabs/joe_analyst.keytab joe_analyst/$(hostname -f)@CLOUDERA.COM Start SparkSql using HiveWarehouseConnector: spark-shell --jars /opt/cloudera/parcels/CDH/jars/hive-warehouse-connector-assembly*.jar --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://$(hostname -f):10000/default;" --conf "spark.sql.hive.hiveserver2.jdbc.url.principal=hive/$(hostname -f)@CLOUDERA.COM" --conf spark.security.credentials.hiveserver2.enabled=false Import HWC classes and start the session: import com.hortonworks.hwc.HiveWarehouseSession import com.hortonworks.hwc.HiveWarehouseSession._ val hive = HiveWarehouseSession.session(spark).build() Run queries using hive.execute(). Example: hive.execute("select * from cost_savings.claim_savings").show(10) The following is a sample script to automate above for joe_analyst here: /tmp/masterclass/ranger-atlas/HortoniaMunichSetup/run_spark_sql.sh     Troubleshooting Zeppelin In case you encounter Thrift Exception like the following, it's likely the session was expired: Just scroll to the top and click the Gears icon (near top right) to display the interpreters and restart the JDBC one: Atlas walkthrough Log in to Atlas and show the Hive columns tagged as EXPIRES_ON: To see the table name, you can select Table in the Column dropdown: Now, notice the table name is also displayed: Select the fed_tax column and open the Classifications tab to view the attributes of the tag (expiry_date) and value: To save this search, click the Save As button near the bottom left. Provide a Name and click Create to save: Similarly, you can query for Hive tables tagged with DATA_QUALITY: Click on claim_savings to see that the quality score associated with this table is less than 60%: Click back, and select the claims_view table instead. Click the Lineage tab. This shows that this table was derived from the claims_saving table: Click on the Classifications tab and notice that because the table claims_view table was derived from (claims_savings) and had a DATA_QUALITY tag, the tag was automatically propagated to claims_view table itself (i.e. no one had to manually tag it): Use Atlas to query for hive_tables and pick provider_summary to show lineage and impact: You can use the Audits tab to see audits on this table: You can use the Schema tab to inspect the table schema: Navigate to the Classification tab to see how you can easily see all entities tagged with a certain classification (across Hive, Hbase, Kafka, HDFS etc): Navigate to the Glossary tab to see how you can define Glossary categories and terms, as well as search for any entities associated with those terms: Navigate to Atlas home page and notice the option to create a new entity: The following is a sample out of the box entity types that you can create: Selecting an entity type (e.g. hdfs_path) displays the required and optional fields that you need to manually create the new entity:  Hive ACID/Merge walkthrough In Zeppelin, there are two Hive-related notebooks provided to demonstrate Hive ACID and MERGE capabilities. Log in to Zeppelin as etl_user to be able to run these: The notebooks contain tutorials that walk through some of the theory and concepts before going through some basic examples: Appendix: The following are some older AMI links (for HDP releases): For HDP 3.1.4: click here For HDP 3.1 with Knox SSO: click here For HDP 2.6.5 with Knox SSO: click here For HDP 2.6.5: click here ... View more

Summary: The release of HDF 3.3 brings about a significant number of improvements in HDF. This article shows how you can use ambari-bootstrap project to easily generate a blueprint and deploy either HDF only clusters or combined HDP/HDF clusters in 5 easy steps. To quickly setup a single node setup, prebuilt AMIs are available for AWS as well as a script that automates these steps, so you can deploy the cluster in a few commands. Steps for each of the below option are described in this article: A. Single-node prebuilt AMIs on AWS B. Single-node fresh installs C. Multi-node fresh installs A. Single-node prebuilt AMI on AWS: Steps to launch the AMI 1. Launch Amazon AWS console page in your browser by clicking here and sign in with your credentials. Once signed in, you can close this browser tab. 2. Select the AMI from ‘N. California’ region by clicking one of the below options To spin up HDP 3.1/HDF 3.3, click here To spin up HDF 3.3 only cluster, click here Now choose instance type: select ‘m4.2xlarge’ and click Next Note: if you choose a smaller instance type from the above recommendation, not all services may come up 3. Configure Instance Details: leave the defaults and click ‘Next’ 4. Add storage: keep at least the default of 800 GB and click ‘Next’ 5. Optionally, add a name or any other tags you like. Then click ‘Next’ 6. Configure security group: create a new security group and select ‘All traffic’ to open all ports. For production usage, a more restrictive security group policy is strongly encouraged. As an instance only allow traffic from your company’s IP range. Then click ‘Review and Launch’ 7. Review your settings and click Launch 8. Create and download a new key pair (or choose an existing one). Then click ‘Launch instances’ 9. Click the shown link under ‘Your instances are now launching’ 10. This opens the EC2 dashboard that shows the details of your launched instance 11. Make note of your instance’s ‘Public IP’ (which will be used to access your cluster). If it is blank, wait 1-2 minutes for this to be populated. 12. After 5-10 minutes, open the below URL in your browser to access Ambari’s console: http://<PUBLIC IP>:8080. Login as user:admin and pass:StrongPassword (see previous step) 13. At this point, Ambari may still be in the process of starting all the services. You can tell by the presence of the blue ‘op’ notification near the top left of the page. If so, just wait until it is done. (Optional) You can also monitor the startup using the log as below: Open SSH session into the VM using your key and the public IP e.g. from OSX: ssh -i ~/.ssh/mykey.pem centos@<publicIP> Tail the startup log: tail -f /var/log/hdp_startup.log Once you see “cluster is ready!” you can proceed 14. Once the blue ‘op’ notification disappears and all the services show a green check mark, the cluster is fully up. B. Single-node install: Launch a fresh CentOS/RHEL 7 instance with 4+cpu and 16GB+ RAM and run below. Do not try to install HDF on a env where Ambari or HDP are already installed (e.g. HDP sandbox or HDP cluster) To deploy HDF 3.3 only cluster, run below export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/b5565d7e7f9beffd8dd57a970dc54266/raw | sudo -E sh To deploy HDF 3.3/HDP3.1 combined cluster, run below export host_count=1 curl -sSL https://gist.github.com/abajwa-hw/d7cd1c0232c1af46ee2c465e4871ddc6/raw | sudo -E sh Once launched, the script will install Ambari and use it to deploy HDF cluster Note: this script can also be used to install multi-node clusters after step #1 below is complete (i.e. after the agents on non-AmabriServer nodes are installed and registered). Just change the value of the host_count variable C. Multi-node HDF 3.3 install: 0. Launch your RHEL/CentOS 7 instances where you wish to install HDF. In this example, we will use 4 m4.xlarge instances. Select an instance where ambari-server should run (e.g. node1) 1. After choosing a host where you would like Ambari-server to run, first let's prepare the other hosts. Run below on all hosts where Ambari-server will not be running (e.g. node2-4). This will run pre-requisite steps, install Ambari-agents and point them to Ambari-server host: export ambari_server=<FQDN of host where ambari-server will be installed>;#replace this export install_ambari_server=false export ambari_version=2.7.3.0 curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 2. Run remaining steps on host where Ambari-server is to be installed (e.g. node1). The below commands run pre-reqs and install Ambari-server export db_password="StrongPassword" # MySQL password export nifi_password="StrongPassword" # NiFi password must be at least ten chars export hdf_ambari_mpack_url="http://public-repo-1.hortonworks.com/HDF/amazonlinux2/3.x/updates/3.3.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.3.0.0-165.tar.gz" export ambari_version=2.7.3.0 #install bootstrap yum install -y git python-argparse cd /tmp git clone https://github.com/seanorama/ambari-bootstrap.git #Runs pre-reqs and install ambari-server export install_ambari_server=true curl -sSL https://raw.githubusercontent.com/seanorama/ambari-bootstrap/master/ambari-bootstrap.sh | sudo -E sh ; 3. On the same node, install MySQL and create databases and users for Schema Registry and SAM sudo yum localinstall -y https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm sudo yum install -y epel-release mysql-connector-java* mysql-community-server # MySQL Setup sudo systemctl enable mysqld.service sudo systemctl start mysqld.service #extract system generated Mysql password oldpass=$( grep 'temporary.*root@localhost' /var/log/mysqld.log | tail -n 1| sed 's/.*root@localhost: //') #create sql file that # 1. reset Mysql password to temp value and create druid/superset/registry/streamline schemas and users # 2. sets passwords for druid/superset/registry/streamline users to ${db_password} cat << EOF > mysql-setup.sql ALTER USER 'root'@'localhost' IDENTIFIED BY 'Secur1ty!';uninstall plugin validate_password;CREATE DATABASE registry DEFAULT CHARACTER SET utf8; CREATE DATABASE streamline DEFAULT CHARACTER SET utf8;CREATE USER 'registry'@'%' IDENTIFIED BY '${db_password}'; CREATE USER 'streamline'@'%' IDENTIFIED BY '${db_password}';GRANT ALL PRIVILEGES ON registry.* TO 'registry'@'%' WITH GRANT OPTION ; GRANT ALL PRIVILEGES ON streamline.* TO 'streamline'@'%' WITH GRANT OPTION ;commit; EOF #execute sqlfile mysql -h localhost -u root -p"$oldpass" --connect-expired-password < mysql-setup.sql #change Mysql password to StrongPassword mysqladmin -u root -p'Secur1ty!' password StrongPassword #test password and confirm dbs created mysql -u root -pStrongPassword -e 'show databases;' 4. On the same node, install Mysql connector jar and then HDF mpack. Then restart Ambari so it recognizes HDF stack: sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar sudo ambari-server install-mpack --mpack=${hdf_ambari_mpack_url} --verbose sudo ambari-server restart At this point, if you wanted you could use Ambari install wizard to install HDF you can do that as well. Just open http://<Ambari host IP>:8080 and login and follow the steps in the doc. Otherwise, to proceed with deploying via blueprints follow the remaining steps. 4. On the same node, provide minimum configurations required for install by creating configuration-custom.json. You can add to this to customize any component's property that is exposed by Ambari cd /tmp/ambari-bootstrap/deploy cat << EOF > configuration-custom.json { "configurations": { "ams-grafana-env": { "metrics_grafana_password": "${ambari_password}" }, "kafka-broker": { "offsets.topic.replication.factor": "1" }, "streamline-common": { "jar.storage.type": "local", "streamline.storage.type": "mysql", "streamline.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/streamline", "registry.url" : "http://localhost:7788/api/v1", "streamline.dashboard.url" : "http://localhost:9089", "streamline.storage.connector.password": "${db_password}" }, "registry-common": { "jar.storage.type": "local", "registry.storage.connector.connectURI": "jdbc:mysql://$(hostname -f):3306/registry", "registry.storage.type": "mysql", "registry.storage.connector.password": "${db_password}" }, "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "${nifi_password}" }, "nifi-registry-properties": { "nifi.registry.db.password": "${nifi_password}" }, "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "${nifi_password}" } } } EOF 5. Then run below as root to generate a recommended blueprint and deploy the cluster install. Make sure to set host_count to the total number of hosts in your cluster (including Ambari server) sudo su cd /tmp/ambari-bootstrap/deploy/ export host_count=<Number of total nodes> export ambari_stack_name=HDF export ambari_stack_version=3.3 export cluster_name="HDF" export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY AMBARI_METRICS KNOX" ./deploy-recommended-cluster.bash You can now login into Ambari at http://<Ambari host IP>:8080 and sit back and watch your HDF cluster get installed! Notes: a) This will only install Nifi on a single node of the cluster by default b) Nifi Certificate Authority (CA) component will be installed by default. This means that if you wanted to, you could enable SSL to be enabled for Nifi out of the box by including a "nifi-ambari-ssl-config" section in the above configuration-custom.json: "nifi-ambari-ssl-config":{ "nifi.toolkit.tls.token":"hadoop", "nifi.node.ssl.isenabled":"true", "nifi.security.needClientAuth":"true", "nifi.toolkit.dn.suffix":", OU=HORTONWORKS", "nifi.initial.admin.identity":"CN=nifiadmin, OU=HORTONWORKS", "content":"<property name='Node Identity 1'>CN=node-1.fqdn, OU=HORTONWORKS</property><property name='Node Identity 2'>CN=node-2.fqdn, OU=HORTONWORKS</property><property name='Node Identity 3'>node-3.fqdn, OU=HORTONWORKS</property>" }, Make sure to replace node-x.fqdn with the FQDN of each node running Nifi c) As part of the install, you can also have an existing Nifi flow deployed by Ambari. First, read in a flow.xml file from existing Nifi system (you can find this in flow.xml.gz). For example, run below to read the flow for the Twitter demo into an env var twitter_flow=$(curl -L https://gist.githubusercontent.com/abajwa-hw/3a3e2b2d9fb239043a38d204c94e609f/raw) Then include a "nifi-ambari-ssl-config" section in the above configuration-custom.json when you run the tee command - to have ambari-bootstrap include the whole flow xml into the generated blueprint: "nifi-flow-env":{ "properties_attributes":{}, "properties":{"content":"${twitter_flow}"} } d) In case you would like to review the generated blueprint before it gets deployed, just set the below variable as well: export deploy=false .... The blueprint will be created under /tmp/ambari-bootstrap*/deploy/tempdir*/blueprint.json Sample blueprints Sample generated blueprint for 4 node HDF 3.3 only cluster is provided for reference here: { "Blueprints": { "stack_name": "HDF", "stack_version": "3.3" }, "host_groups": [ { "name": "host-group-1", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIFI_CA" }, { "name": "STREAMLINE_SERVER" } ] }, { "name": "host-group-4", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "METRICS_COLLECTOR" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "STREAMLINE_SERVER" } ] }, { "name": "host-group-2", "components": [ { "name": "NIFI_MASTER" }, { "name": "DRPC_SERVER" }, { "name": "METRICS_GRAFANA" }, { "name": "KAFKA_BROKER" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "STREAMLINE_SERVER" }, { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "NIMBUS" }, { "name": "ZOOKEEPER_CLIENT" }, { "name": "KNOX_GATEWAY" }, { "name": "NIFI_REGISTRY_MASTER" }, { "name": "REGISTRY_SERVER" }, { "name": "STORM_UI_SERVER" } ] }, { "name": "host-group-3", "components": [ { "name": "METRICS_MONITOR" }, { "name": "SUPERVISOR" }, { "name": "ZOOKEEPER_SERVER" }, { "name": "STREAMLINE_SERVER" } ] } ], "configurations": [ { "nifi-ambari-config": { "nifi.security.encrypt.configuration.password": "StrongPassword" } }, { "nifi-registry-ambari-config": { "nifi.registry.security.encrypt.configuration.password": "StrongPassword" } }, { "ams-hbase-env": { "hbase_master_heapsize": "512", "hbase_regionserver_heapsize": "768", "hbase_master_xmn_size": "192" } }, { "nifi-logsearch-conf": {} }, { "storm-site": { "metrics.reporter.register": "org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter", "topology.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink\", \"parallelism.hint\": 1, \"whitelist\": [\"kafkaOffset\\..+/\", \"__complete-latency\", \"__process-latency\", \"__execute-latency\", \"__receive\\.population$\", \"__sendqueue\\.population$\", \"__execute-count\", \"__emit-count\", \"__ack-count\", \"__fail-count\", \"memory/heap\\.usedBytes$\", \"memory/nonHeap\\.usedBytes$\", \"GC/.+\\.count$\", \"GC/.+\\.timeMs$\"]}]", "storm.local.dir": "/hadoop/storm", "storm.cluster.metrics.consumer.register": "[{\"class\": \"org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter\"}]" } }, { "registry-common": { "registry.storage.connector.connectURI": "jdbc:mysql://ip-xxx-xx-xx-xx9.us-west-1.compute.internal:3306/registry", "registry.storage.type": "mysql", "jar.storage.type": "local", "registry.storage.connector.password": "StrongPassword" } }, { "registry-env": {} }, { "registry-logsearch-conf": {} }, { "streamline-common": { "streamline.storage.type": "mysql", "streamline.storage.connector.connectURI": "jdbc:mysql://ip-xxx-xx-xx-xx9.us-west-1.compute.internal:3306/streamline", "streamline.dashboard.url": "http://localhost:9089", "registry.url": "http://localhost:7788/api/v1", "jar.storage.type": "local", "streamline.storage.connector.password": "StrongPassword" } }, { "nifi-registry-properties": { "nifi.registry.db.password": "StrongPassword" } }, { "ams-hbase-site": { "hbase.regionserver.global.memstore.upperLimit": "0.35", "hbase.regionserver.global.memstore.lowerLimit": "0.3", "hbase.tmp.dir": "/var/lib/ambari-metrics-collector/hbase-tmp", "hbase.hregion.memstore.flush.size": "134217728", "hfile.block.cache.size": "0.3", "hbase.rootdir": "file:///var/lib/ambari-metrics-collector/hbase", "hbase.cluster.distributed": "false", "phoenix.coprocessor.maxMetaDataCacheSize": "20480000", "hbase.zookeeper.property.clientPort": "61181" } }, { "storm-env": {} }, { "streamline-env": {} }, { "ams-site": { "timeline.metrics.service.webapp.address": "localhost:6188", "timeline.metrics.cluster.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.downsampler.event.metric.patterns": "topology\.%", "timeline.metrics.host.aggregate.splitpoints": "kafka.network.RequestMetrics.ResponseQueueTimeMs.request.OffsetFetch.98percentile", "timeline.metrics.service.handler.thread.count": "20", "timeline.metrics.service.watcher.disabled": "false", "timeline.metrics.host.aggregator.ttl": "86400" } }, { "kafka-broker": { "log.dirs": "/kafka-logs", "offsets.topic.replication.factor": "1" } }, { "ams-grafana-env": { "metrics_grafana_password": "StrongPassword" } }, { "streamline-logsearch-conf": {} }, { "zoo.cfg": { "dataDir": "/hadoop/zookeeper" } }, { "ams-env": { "metrics_collector_heapsize": "512" } } ] }<br> Sample cluster.json for this 4 node cluster: { "blueprint": "recommended", "default_password": "hadoop", "host_groups": [ { "hosts": [ { "fqdn": "ip-XX-XX-XX-XXX.us-west-1.compute.internal" } ], "name": "host-group-1" }, { "hosts": [ { "fqdn": "ip-XX-XX-XX-XXX.us-west-1.compute.internal" } ], "name": "host-group-3" }, { "hosts": [ { "fqdn": "ip-xxx-xxx-xxx-xxx.us-west-1.compute.internal" } ], "name": "host-group-4" }, { "hosts": [ { "fqdn": "ip-xx-xx-xx-xxx.us-west-1.compute.internal" } ], "name": "host-group-2" } ] } ... View more