About josh_nicholson

josh_nicholson · ‎10-10-2018

I have a custom UDF that I have created on several databases in Hive by using the following command through beeline. add JAR hdfs:///my/jar/location/myjar.jar; create function my_schema.myFunction as 'com.class.path.hive.MyFunction' using JAR 'hdfs:////my/jar/location/myjar.jar'; However, if a user then tries to use this UDF on any of the schemas I have created it on, it sometimes does not work and they receive a 'Invalid Function' error. However, it does work on some schemas, and then automatically adds the JAR to the classpath. After it works, they can then go back to the schema that it previously did not work on and use it without any issues. Why is this JAR only being added to the classpath sometimes?

josh_nicholson · ‎10-08-2018

Thanks @Aditya Sirna, I think this will get me what I need. What I'm ultimately trying to get is the HDFS location so I can use it in the script I'm writing. Is performing a describe table and then grepping the output the best way to do this?

josh_nicholson · ‎10-05-2018

As part of a script I'm writing, I'm wanting to get the HDFS location of a list of Hive schemas that's passed in via a text file. The best way I can think to do this is to run a beeline command in a loop that performs a `describe schema` command and extracts the HDFS location of each schema from this output. However, this will require me to authenticate every time I run this command which is inefficient. Is there a better way to programtically get the HDFS locations of a list of Hive schemas?

josh_nicholson · ‎09-17-2018

Thanks for the explanation @Shu. Do you know if there's any accepted way or best practice for monitoring the cluster status, or is this going to be the best way?

josh_nicholson · ‎09-14-2018

I am wanting to know what the best way would be to monitor the status of a NiFi cluster. Currently I get Ambari alerts if the actual NiFi service stops, but I do not receive any alert if a NiFi node disconnects from the NiFi cluster. I was thinking of having something run within NiFi to get the total amount of nodes connected to the cluster and use this to send alerts. Is there some better way of doing this or generally accepted way?

josh_nicholson · ‎09-04-2018

@Matt Clarke I actually have it set to `USE_DN`, but have nifi.security.user.login.identity.provider set to `kerberos-provider`. LDAP provider is commented out in that file.

josh_nicholson · ‎09-04-2018

Depending on the user's configuration on their computer, when trying to login to NiFi they either come in as <username> or <username>@<realm>. If the user comes in as <username>@<realm>, for NiFi I have to add them as this user in Ranger instead of their normal username, and then give permissions on <username>@<realm>. This makes permissions difficult to manage. In NiFi's configuration, I have nifi.security.identity.mapping.pattern.kerb set to `EMAILADDRESS=(.*?), CN=([^@]+)` and nifi.security.identity.mapping.value.kerb set to `$2`. I thought this should match up to the `@` sign so only the username would be passed to NiFi for authentication but this doesn't seem to be the case. What's going on here?

josh_nicholson · ‎08-29-2018

That worked, thanks.

josh_nicholson · ‎08-29-2018

I didn't think of querying the JMX, this works fantastically, thanks! Actually @Jonathan Sneep I am not able to curl to the JMX as I need authorization. This cluster is kerberized and using SSL. I tried searching around on how to pass credentials but couldn't find anything. Any ideas?

josh_nicholson · ‎08-29-2018

I am currently deleting a large amount of small files and am needing to monitor the number of blocks pending deletion so I do not put too much of a load on the cluster. I know this number can be viewed in the NameNode UI, but I would like to run a command instead which I can then use in combination with `watch` so I can get continuous updates. Does such a command exist?

josh_nicholson · ‎08-28-2018

@snukavarapu Thanks for the article, this worked great for me. Is this something you keep continuously updated? If so, what's your strategy for keeping the table updated?

josh_nicholson · ‎08-23-2018

This worked, thanks!

josh_nicholson · ‎08-21-2018

I am wanting to delete a process group from NiFi by using a curl command to the API. I tried using the following command. curl -k -X DELETE -H "Authorization: Bearer $token" $cluster_url/nifi-api/process-groups/$process_group_uuid But I get the error that the revision must be specified. When I delete a process group using the UI, I can see using Chrome dev tools that both a revision and client ID are specified. How can I get these values so I can use curl to delete the process group?

josh_nicholson · ‎07-31-2018

@Pierre Villard This works for me if I leave out the `clientID` part as NiFi will generate it's own, but I have a use case where I would like to provide my own UUID. Is this possible? I tried using `uuidgen` to get a UUID in bash but the POST then fails.

josh_nicholson · ‎07-31-2018

I am writing a script that will dynamically create a process group with a given name based on user input. I then am wanting to deploy a template inside of this process group immediately after it gets created. Since NiFi generates the UUID for the new process group after the POST to the API, I am stumped on how to get the UUID of the newly created process group. How can I accomplish this so I can then deploy a template inside of it? I tried generating a UUID myself for the process group in bash by using `uuidgen` so I could easily track this in a variable, however the POST fails for some reason when I send this along with the name. Here is the curl that is failing when I provide a uuid I generated. curl -k -X POST -v -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"revision":{"clientID":"'$uuid'","version":0},"component":{"name":"'$CLUSTER_NAME' '$FLOW_NAME'"}}' https://myhost.com:9091/nifi-api/process-groups/$EXISTING_PROCESS_GROUP/process-groups/ If I remove the `clientID` part of the JSON, the POST works and NiFi generates a UUID for me, which results in my original problem.

josh_nicholson · ‎07-23-2018

@sbabu Getting the token this way does not work for me as there is a '%' in the password I am using, which causes curl to throw an error. How can I get around this? The script I am developing is going to be used by different people so I cannot know ahead of time to escape any special characters.

josh_nicholson · ‎06-25-2018

@Aakash Singh Did you find out how to do this on a kerberized cluster?

josh_nicholson · ‎06-25-2018

@Smart Solutions Were you able to get an answer for this?

josh_nicholson · ‎06-06-2018

Is there any way to currently get user-level metrics for YARN jobs? I was hoping to get total amount of jobs submitted by the user and total amount of job failures. I am also wanting to get user-level metrics for Hive queries for the same thing, total amount of queries and failures. I was able to find information on getting user-level HDFS metrics but am struggling with these two other services.

josh_nicholson · ‎05-09-2018

I am wanting to monitor NiFi's JVM heap usage, in a similar way to Ambari does this, by using the MemoryMonitoring reporting task. However, since NiFi allows you to create reporting tasks for each JVM memory pool, how would I go about replicating Ambari's heap usage monitoring? Or is there another best practice for monitoring memory usage in NiFi?

josh_nicholson · ‎05-07-2018

@Matt Clarke Gotcha. One more question, since it appears the monitorActivity just sends a status, and not the flowfile, is there anyway I can get the attributes of the last flowfile? I'm wanting to include the hostname and original bulletin message in the OK status email, but using the "inactive" relationship seems to make this not possible as the attributes from the original flowfile aren't carried over.

josh_nicholson · ‎05-07-2018

Thanks for the suggestion @Abdelkrim Hadjidj. I'm going to use this to send the initial alert and then use @Matt Clarke's suggestion to send an OK alert when the issue has been resolved.

josh_nicholson · ‎05-07-2018

@Matt Burgess Thanks for opening the JIRA Matt. As a workaround in the meantime, I discovered I can use ${s2s.host} to get the host name, I just then need to use an UpdateAttribute processor to add this as an actual property to the flowfile.

josh_nicholson · ‎05-04-2018

@Matt Clarke Thanks for the suggestion, this should give me what I want. One thing that I thought of as I was implementing this however, is currently I have six different disks I am monitoring with six different DiskUsage reporting tasks, as I have different NiFi repositories split over these disks. To keep things clean, these are all being picked up by one SiteToSiteBulletinReportingTask and being submitted to the same process group for alerting. If say disk1 went over the threshold, then disk2 went over while disk1 was still over the threshold, I would never receive an email alert about disk2. The only way around this I can think of is having a separate MonitorActivity for each disk, but then this requires hardcoding which I would rather avoid as this makes it difficult to deploy this monitoring between environments with different disk configurations. Can you think of any way around this?

josh_nicholson · ‎05-04-2018

Currently I have a flow in place that will monitor disk usage by using the MonitorDiskUsageReportingTask in conjunction with the SiteToSiteBulletinReportingTask . At the end of the flow, I have the flowfile, which is essentially each bulletin, being put to email along with a message describing the alert. My issue is, since I have this reporting task set to run every minute to ensure we get the most up to date alerts regarding disk usage, I will receive an email every minute once the disk usage goes over the threshold I set. Is there a way I can configure it so only one email is sent once this goes over the threshold? And then maybe have an "OK" status email go out after the disk usage has decreased beneath the threshold?

josh_nicholson · ‎05-04-2018

@Matt Burgess This would be better as it would be easier to map the hostname but still not a perfect solution. Also, I didn't see this as a field in the bulletin's JSON. I'm currently running HDF 3.0.1 (NiFi 1.2) so I'm assuming it's not in my version?

josh_nicholson · ‎05-03-2018

I am developing a flow where I need the hostname of where a bulletin is coming from. Currently, a large portion of my flow is getting this by me first manually determining what hostname is associated with which bulletinNodeId, and then using the RouteOnAttribute processor to map this to the actual hostname. This isn't going to be scalable if I need to first manually determine which bulletinNodeId is associated with which hostname. Is there a better way to go about getting this hostname?

josh_nicholson · ‎04-16-2018

HDF 3.1.1 is not compatible with HDP 2.6.1. I was told by Hortonworks that HDP 3.0 will be compatible and I must wait until its release.

josh_nicholson · ‎04-16-2018

Hi Matt, I was able to implement these steps and it did exactly what I needed. Thanks!

josh_nicholson · ‎04-16-2018

I am constantly seeing this error message in the nifi-app.log file. 2018-04-16 11:00:36,383 ERROR [nifi.async.batch_nifi.async.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file. java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:271) at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43) at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157) at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:524) at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154) at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880) at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:828) at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758) at java.lang.Thread.run(Thread.java:745) Since this is a bug related to the version of NiFi I am on, I do not need to see it as it happens very frequently and is cluttering the log file. How can I exclude this so it is not logged? Is there some configuration I can do in the logback.xml file?

Online	Offline
Last Visited	‎10-18-2018 07:47 PM

Date Registered	‎11-30-2017 03:54 PM
Last Visited	‎10-18-2018 07:47 PM
Total Messages Posted	44
Total Kudos Received	6

Re: Ambari not reflecting NiFi version after upgra...

Re: Are there currently any tools available for pe...

How to make JAR files in HDFS available for custom...

Re: How can I run multiple beeline commands in a s...

How can I run multiple beeline commands in a scrip...

Re: How to monitor NiFi cluster status?

How to monitor NiFi cluster status?

Re: Mapping kerberos pattern for NiFi

Mapping kerberos pattern for NiFi

Re: Is there a command to find the number of block...

Re: Is there a command to find the number of block...

Is there a command to find the number of blocks pe...

Re: Identify where most of the small file are loca...

Re: How can I delete a process group using the API...

How can I delete a process group using the API?

Re: Create process groups using NiFi 1.x REST API?

How to get the UUID of a process group after creat...

Re: Accessing REST API of Kerberized NiFI Cluster ...

Re: Using Phoenix sqlline utility to browse Ambari...

Re: Need to understand impact for setting tez.task...

How to get user level metrics for Hive or YARN?

Best practice for monitoring NiFi JVM usage?

Re: Best practice for email monitoring for NiFi?

Re: Best practice for email monitoring for NiFi?

Re: How to get hostname of a bulletin rather than ...

Re: Best practice for email monitoring for NiFi?

Best practice for email monitoring for NiFi?

Re: How to get hostname of a bulletin rather than ...

How to get hostname of a bulletin rather than the ...

Re: Ambari not reflecting NiFi version after upgra...

Re: How can I filter the NiFi log to exclude a cer...

How can I filter the NiFi log to exclude a certain...