@Matt Clarke Per your suggestion, I turned on debugging in the nifi-app.log file and captured about 10 seconds of a run. Files are still not being ingested and I'm starting to ret really frustrated Log file attachednifi-applog.zip ... View more

@Matt Clarke Quick update: I tried changing the owner:group to nifi:nifi based on the ps -ef | grep nifi Still not seeing any queued FlowFiles nor am I seeing anything in hdfs ... View more

@Matt Clarke Your observations are correct, the processors seem to be configured fine, but the GetFile doesn't retrieve the CSV. I ran the ps, but not sure where to find the user that owns nifi (see screenshot) I've attached the latest log file. I don't see any WARN or ERROR logs...nifi-applog.zip ... View more

NiFi UI view- ... View more

@Gaurav Sharma I think I figured it out. When you asked me to check zoo.cfg, I noticed there were a block of ports ZK communicates over server.1=hdp.c.my-project-1519895027175.internal:2888:3888 server.2=slave1.c.my-project-1519895027175.internal:2888:3888 server.3=slave2.c.my-project-1519895027175.internal:2888:3888 In GCP, you have to configure the firewall manually (or that's what I'm doing at least). Once I added that range of ports to the firewall, I restarted the ZK servers and most of my services work now! I say most, because Spark2 History Server and Zeppelin still aren't working. But I'll open another thread for those. ... View more

@Gaurav Sharma It is a new cluster. Four total nodes [mike_w_wong@hdp ~]$ sudo echo 'stat'| nc hdp.c.my-project-1519895027175.internal 2181 This ZooKeeper instance is not currently serving requests [mike_w_wong@slave1 ~]$ sudo echo 'stat'| nc slave1.c.my-project-1519895027175.internal 2181 This ZooKeeper instance is not currently serving requests [mike_w_wong@slave2 ~]$ sudo echo 'stat'| nc slave2.c.my-project-1519895027175.internal 2181 This ZooKeeper instance is not currently serving requests Checking the zoo.cfg now ... View more

@Geoffrey Shelton Okot They do match From Ambari- From the VMs- [mike_w_wong@hdp ~]$ hostname -f hdp.c.my-project-1519895027175.internal [mike_w_wong@slave1 ~]$ hostname -f slave1.c.my-project-1519895027175.internal [mike_w_wong@slave2 ~]$ hostname -f slave2.c.my-project-1519895027175.internal ... View more

Also, looking at the log file after restarting ZK- 2018-05-21 19:44:50,301 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /35.231.170.209:42398 2018-05-21 19:44:50,301 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-05-21 19:44:50,301 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /35.231.170.209:42398 (no session established for client) 2018-05-21 19:44:53,030 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /35.231.170.209:42408 2018-05-21 19:44:53,031 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-05-21 19:44:53,031 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /35.231.170.209:42408 (no session established for client) 2018-05-21 19:44:54,354 - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@383] - Cannot open channel to 3 at election address slave2.c.my-project-1519895027175.internal/35.231.220.224:3888 java.net.SocketTimeoutException: connect timed out Again, Ambari still shows ZK to be running. ... View more

One thing I noticed, in the zookeeper log folder, the name of the .out file is NOT the full FQDN, just the machine name: hdp machine- -rw-r--r--. 1 zookeeper hadoop 3993 May 21 19:40 zookeeper.log -rw-r--r--. 1 zookeeper hadoop 10485997 May 21 17:47 zookeeper.log.1 -rw-r--r--. 1 zookeeper hadoop 3993 May 21 19:40 zookeeper-zookeeper-server-hdp.out slave1 -rw-r--r--. 1 zookeeper hadoop 0 May 21 19:38 zookeeper.log -rw-r--r--. 1 zookeeper hadoop 10486239 May 20 06:15 zookeeper.log.1 -rw-r--r--. 1 zookeeper hadoop 0 May 21 19:38 zookeeper-zookeeper-server-slave1.out ... View more