As part of a script I'm writing, I'm wanting to get the HDFS location of a list of Hive schemas that's passed in via a text file. The best way I can think to do this is to run a beeline command in a loop that performs a `describe schema` command and extracts the HDFS location of each schema from this output. However, this will require me to authenticate every time I run this command which is inefficient. Is there a better way to programtically get the HDFS locations of a list of Hive schemas? ... View more

Depending on the user's configuration on their computer, when trying to login to NiFi they either come in as <username> or <username>@<realm>. If the user comes in as <username>@<realm>, for NiFi I have to add them as this user in Ranger instead of their normal username, and then give permissions on <username>@<realm>. This makes permissions difficult to manage. In NiFi's configuration, I have nifi.security.identity.mapping.pattern.kerb set to `EMAILADDRESS=(.*?), CN=([^@]+)` and nifi.security.identity.mapping.value.kerb set to `$2`. I thought this should match up to the `@` sign so only the username would be passed to NiFi for authentication but this doesn't seem to be the case. What's going on here? ... View more

I am currently deleting a large amount of small files and am needing to monitor the number of blocks pending deletion so I do not put too much of a load on the cluster. I know this number can be viewed in the NameNode UI, but I would like to run a command instead which I can then use in combination with `watch` so I can get continuous updates. Does such a command exist? ... View more

I am wanting to delete a process group from NiFi by using a curl command to the API. I tried using the following command. curl -k -X DELETE -H "Authorization: Bearer $token" $cluster_url/nifi-api/process-groups/$process_group_uuid But I get the error that the revision must be specified. When I delete a process group using the UI, I can see using Chrome dev tools that both a revision and client ID are specified. How can I get these values so I can use curl to delete the process group? ... View more

I am writing a script that will dynamically create a process group with a given name based on user input. I then am wanting to deploy a template inside of this process group immediately after it gets created. Since NiFi generates the UUID for the new process group after the POST to the API, I am stumped on how to get the UUID of the newly created process group. How can I accomplish this so I can then deploy a template inside of it? I tried generating a UUID myself for the process group in bash by using `uuidgen` so I could easily track this in a variable, however the POST fails for some reason when I send this along with the name. Here is the curl that is failing when I provide a uuid I generated. curl -k -X POST -v -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"revision":{"clientID":"'$uuid'","version":0},"component":{"name":"'$CLUSTER_NAME' '$FLOW_NAME'"}}' https://myhost.com:9091/nifi-api/process-groups/$EXISTING_PROCESS_GROUP/process-groups/ If I remove the `clientID` part of the JSON, the POST works and NiFi generates a UUID for me, which results in my original problem. ... View more

I am developing a flow where I need the hostname of where a bulletin is coming from. Currently, a large portion of my flow is getting this by me first manually determining what hostname is associated with which bulletinNodeId, and then using the RouteOnAttribute processor to map this to the actual hostname. This isn't going to be scalable if I need to first manually determine which bulletinNodeId is associated with which hostname. Is there a better way to go about getting this hostname? ... View more

I am constantly seeing this error message in the nifi-app.log file. 2018-04-16 11:00:36,383 ERROR [nifi.async.batch_nifi.async.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file. java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:271) at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157) at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:524) at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154) at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880) at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:828) at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758) at java.lang.Thread.run(Thread.java:745) Since this is a bug related to the version of NiFi I am on, I do not need to see it as it happens very frequently and is cluttering the log file. How can I exclude this so it is not logged? Is there some configuration I can do in the logback.xml file? ... View more