Member since
02-25-2016
32
Posts
16
Kudos Received
0
Solutions
10-09-2017
05:24 PM
1 Kudo
Sadly no, @Tilmann Piffl. We ended up with one NiFi outside the HDP cluster network and one inside the cluster network. Then we had the two talk to each other over Site-to-Site and the internal one could write to HDFS directly with PutHDFS.
... View more
05-31-2017
09:02 PM
Great find, thanks!
... View more
05-31-2017
07:30 PM
I should have added - this is HDF 2.1.2 so NiFi 1.1.0
... View more
05-31-2017
07:28 PM
@Matt Clarke thanks, but I am running ntp on all machines, my /etc/localtime all point to the same place (in my case /usr/share/zoneinfo/America/Toronto), and, crucially, the date command on each system returns the same thing, which reports back as EDT, eg: Wed May 31 13:21:29 EDT 2017 I don't think the problem lies with my servers...
... View more
05-31-2017
04:09 PM
I'm in the America/New_York timezone and my servers run ntp, and they show the right time. The header of NiFi running on those servers is off by one hour, eg system time says 11:00 and NiFi says 10:00. This question is very similar to https://community.hortonworks.com/questions/76724/change-timezone-in-nifi.html but I've tried that solution and no joy. This feels like a daylight savings problem, as the gui says 10:00:00 EST, and really that should be EDT since at time of writing it is summer. I'm using Oracle Java 1.8.0_71. Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi
04-11-2017
06:15 PM
1 Kudo
I'm trying to run a Spark streaming app that reads from a NiFi output port (over HTTPS). Spark is running on YARN (HDP 2.4.2) and the nodes have Java 7 installed (and 8 is not an option in the short term). When I build my jar that depends on NiFiReceiver I used the latest version (1.1.2 as of writing) but submitting that jar to spark running on Java 7 results in "Unsupported major.minor version 52.0". That suggests the jars have been built on Java 8. Before I go trying to build these jars myself, do they exist in a 7-compatible version anywhere?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Spark
02-24-2017
02:48 PM
Thanks for writing this up @Matt Clarke - very helpful. Do you have a rule of thumb for a maximum heap size? Is there a limit where garbage collection will surely cause more problems than any gains from further increases in heap?
... View more
02-23-2017
08:22 PM
I have a "client" NiFi instance that has multiple tenants, and each of those tenants wish to use site-to-site communication to a NiFi cluster over SSL. The cluster can assign privileges to the incoming client connection based on the client cert (eg a cluster user matching 'CN=client.local, OU=NIFI'), but I don't see a way of differentiating between the client tenants on the cluster side. Ideally I'd say UserA from the client can only access InputPortA on the cluster but not InputPortB, even though UserB from the client side can access InputPortB. It seems like all client tenant identities get squashed down to the CN of the client cert. Is there a way to keep one identity distinct throughout the chain? Would multiple SANs (Subject Alternative Name) in the client NiFi cert help here? Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2017
08:05 PM
1 Kudo
Apologies for asking two questions at once, but they're very closely related. I have a NiFi cluster configured with kerberos and SSL that I would like to use with multiple tenants with strict policies in order to keep different groups from reading and/or altering each others' data. Each of the tenants need to write to HDFS and/or Kafka (which are in turn both kerberized).
HDFS The PutHDFS processor allows me to specify a kerberos principal and keytab, but that implies the linux user running the NiFi process has to have read access to the keytab, which means any NiFi tenant could specify any keytab as long as they knew the path. Is there a way to keep those identities distinct and secure? Kafka The PutKafka processors only allow referencing the Kerberos Service Name from a jaas file that NiFi needs to know about at startup. That suggests to me that all tenants in NiFi would have to connect to Kafka as the same principal. Is there a way to do this that allows each tenant to have their own identity when talking to Kafka? Thanks in advance for any tips!
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Kafka
-
Apache NiFi
02-22-2017
02:32 PM
Thanks for confirming my suspicions @kkawamura - looks like I'll have to find another way.
... View more