Yes, it's a known CSV with a header line. I wonder if there's a trick to use the column names and expression language to avoid manual re-typing. ... View more

There are 2 areas of focus in your question: Getting data into HDP. Here, as @David Streever mentioned, NiFi can be a great fit (and maintain data lineage of all streams coming into your data lake). Think of any place you might have considered Flume - it will be a good candidate. Orchestrating processing and feeds in HDP. What you described in the original question were all the right concerns, but you should really be looking at Falcon [1] to have a higher-level visibility and controls into the workflow than Oozie. Falcon will use and generate Oozie workflows under the hood, but will expose a nice DSL and UI for higher-level constructs. [1] http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/ch_hdp_data_governance_overview.html ... View more

FWIW, PermGen space has been removed from Java 8, this last param will generate a warning. ... View more

Had a chance to hash out details with Joe further, posting here for a record. The input/output port terminology can get confusing quickly, based on what side of the communications you're looking at A client adds a Remote Process Group (RPG) to a local canvas/instance. The dialog prompts for a remote NiFi instance (UI port). A remote NiFi instance specifies the nifi.remote.input.socket.port property to designate which port to use for incoming site-to-site communications. A RPG's input ports are then visible in our local NiFi instance as possible connections to send data to (granted we have permissions). A RPG's output ports are optional, and can be used to pull data from remotely, or implement a request/response like pattern with a remote instance. The inbound site-to-site port multiplexes all RPG inbound port communications. In case of a client talking to a remote NiFi cluster the following applies: The port to be specified in the RPG UI is the NiFi Cluster Manager (NCM) address. Technically, in a cluster, talking to a node UI directly is illegal and won't work. A local client must be able to reach every node in a cluster for site-to-site communications. The actual port is specified by each node via nifi.remote.input.socket.port. Some more details here https://nifi.apache.org/docs.html ... View more

Thanks Joe, I'll follow up with you offline ... View more