Deploying HDF Across Firewalls

[#1]

HDF Version: 2.1.2

Test Environment:

Single AWS VPC with multiple subnets, EC2 Security Groups with port rules controlling access between subnets, AWS LogFlow to track accepted/rejected traffic, 1 EC2 instance for Management, 3 EC2 instances for worker nodes, Laptop on public IP for remote access.

Deployment / Testing Method:

1. Installed baseline of Ambari, Ambari Infra, Ambari Metrics, NiFi Certificate Authority, Zookeeper Cluster, NiFi Cluster, Clients
2. Added Ranger
3. Enabled SSL across all services
4. Tested all commonly used interfaces, checked for rejected traffic

Not Tested:

• Using External LDAP/AD services
• Using external Certificate Authority
• Connecting to Auxiliary services

Deployment Topology

Zone	M(anagement)	D(ata)	R(emote)	P(ublic)
Members	Management Node	Worker Nodes 1,2,3	Other NiFi Cluster	Users
Services	Ambari Infra (Infra Solr), Ranger, Metrics Collector, Grafana, Metrics Monitor, Nifi Certificate Authority, Clients (Infra Solr, ZooKeeper)	NiFi, Zookeeper, Metrics Monitor, Clients (Zookeper)	NiFi	Browser, ssh client

Firewall Rules:

Source Zone	Dest Zone	Port	Notes
M	D	22	ssh, if used for deployment
M	D	8670	Ambari Agent
M	D	3000, 61300, 61310, 61330, 61320, 61388, 61288, 61181, 2181, 60200, 6188	Ambari Metrics Service
M	D	2181, 61181	ZooKeeper
D	M	8080 (http), 8443 (https)	Ambari Interface
D	M	8440, 8441	Ambari Agents
D	M	6182, 6080, 8886	Ranger Services
D	M	3000, 61300, 61310, 61330, 61320, 61388, 61288, 61181, 2181, 60200, 6188	Ambari Metrics Service
P	M	8080 (http), 8443 (https)	Ambari
P	M	6080	Ranger
P	M	3000	Grafana
P	M	8886	Solr Admin
P	M	22	ssh
P	D	22	ssh (optional)
P	D	9090 (http), 9091(https)	NiFi Interface
D	R	9090 (http), 9091(https)	NiFi Interface & Data Transfer

Additional Port Considerations:

• Ports for all Zones to connect to LDAP/AD if used
• Ports for all Zones to send Logging and Alerts (smtp etc.) to other systems
• Ports for NiFi to connect to target systems, e.g. HDFS, Hive, Kafka, etc.
• You will require access to your CA to generate and move certificates; it is probably not necessary to open a Port for direct connection