Re: Has Anyone Used Dr. Elephant with HDP 2.4?

hsowell · ‎09-16-2016

Great article!

hsowell · ‎09-11-2016

Thanks @Bryan Bende, I think this is a great solution. I'm currently using two attributes to help create directory structures dynamically in HDFS: /tmp/data_staging/${SourceAttribute}/${data.date} I was planning to get around the lack of multiple correlation attribute support by doing a route on attribute processor to different merge content processors then back to a single HDFS processor using the original attributes. With your suggestion I can do a much cleaner and more flexible solution. After merging the two attributes with an UpdateAttribute processor, I'll send the data to a MergeContent processor where I'll bin the files on the new combined attribute, and then to my putHDFS processor. The merged attribute will persist (e.g., dataSource1_20160911), which I can then do something like the following to continue dynamically creating directories: /tmp/data_staging/${convergedSourceDateAttribute:substringBefore('_')}/${convergedSourceDateAttribute:substringAfter('_')} Does that seem reasonable?

hsowell · ‎09-11-2016

I'm using the merge content processor and have successfully been using the "Correlation Attribute Name" to bin together like files using a single flowfile attribute with expression language. I would like to start using two attributes to bin and merge files. Is this possible and any help on proper syntax for this would be appreciated.

hsowell · ‎08-10-2016

The Apache Nifi community recently released the beta version of Apache Nifi 1.0.0. This version comes with significant updates, which include a UI refresh, transition to zero master clustering, added multi-tenant authorization, and templates that are now deterministically ordered allowing for version controlled templates! The beta also boasts nine new processors, bringing the total to 165. The full list of release notes can be found here. Below I will do a very basic walkthrough of some of the UI changes from 0.7.0 to the 1.0.0 beta: Original Apache Nifi 0.7.0 Flow New Apache Nifi 1.0.0 Beta Outside of the more modernized look and feel, there are some key UI Changes: (1) Apache Nifi 1.0.0 beta now includes a status bar showing statistics of the overall flow, including bytes in and out, number of started and stopped processors, processors in error, last refresh, etc. (2) The beta also has new collapsable navigation and operation panes (3) There's a new drop down menu where you can access information like flow summary, provenance, and the bulletin board (e.g., error messages). (4) The search field is much more prominent now and allows users to search through complex flows to quickly find and jump-to processor and other elements on the flow

hsowell · ‎06-27-2016

@Victor Xu It looks like you're feeding a list as an argument into this UDF when it's expecting an INT, causing the mismatch. Can you provide any more information on what you're trying to accomplish?

hsowell · ‎06-27-2016

Hi @henryon wen, I would follow these instructions for setting up email alerts in Ambari via SMTP: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Ambari_Users_Guide/content/_configuring_notifications.html And follow Amazon's SES instructions for sending email through their SMTP interface: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html I haven't tried this yet, but it seems straightforward. Please let me know if it works.

hsowell · ‎06-13-2016

@Timothy Spann I've seen a great demo of this done by @Olivier Renault. Olivier - can you confirm?

hsowell · ‎06-13-2016

@Christopher Frankland here are some resources I would recommend checking out: https://community.hortonworks.com/articles/136/how-to-search-for-text-in-an-image.html By @Saptak Sen Another fun one using Apache Nifi by @Jeremy Dyer https://community.hortonworks.com/articles/28380/nifi-ocr-using-apache-nifi-to-read-childrens-books.html And here's a great tutorial with its use: http://hortonworks.com/hadoop-tutorial/indexing-and-searching-text-within-images-with-apache-solr/ Here's also a blog post that is older, so please check commands, but it has some important lessons when it comes to accuracy, the quality and resolution of the PDF will greatly affect your results. http://kiirani.com/2013/03/22/tesseract-pdf.html Posting this as I think it's interesting as well to examine other effects on the source data that might affect accuracy: http://www.assistivetechnology.vcu.edu/wp-content/uploads/sites/1864/2013/09/pxc3882784.pdf

hsowell · ‎06-12-2016

@Milind Rao have you tried using the yarn logs -applicationId <> -am ALL option?

hsowell · ‎06-12-2016

@Micky Woo the article you're pointing to is demonstrating the tag based policy ranger technical preview. While it's not GA in the Apache Ranger .5x line, it is targeted as a development theme for .6x: https://cwiki.apache.org/confluence/display/RANGER/Tag+Based+Policies

hsowell · ‎06-12-2016

This error happens when you're trying to truncate an external table. Truncate needs to target a native/managed table or an exception will be thrown. Here's a really great reference: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

hsowell · ‎06-12-2016

@Marek Martofel yes you can. Here's a full list of the commands, including commands for starting and stopping individual services. https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Shell You can also use Ambari's API to accomplishe the same task: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=41812517

hsowell · ‎06-12-2016

@Ammar Rizvi Here's a really good example on Stackoverflow that provides both an xpath and regex example similar to what you're trying to accomplish: http://stackoverflow.com/questions/30052325/reading-xml-using-pig

hsowell · ‎06-12-2016

@d mahesh Can you provide some more details on what you're seeing? Are there any errors, etc.

hsowell · ‎06-12-2016

@Manoj Gursahani Here are some resources on energy use cases. They don't all apply to solar energy specifically, but you can see how Hadoop could be using similarly: Webinar - http://hortonworks.com/webinar/how-iot-delivering-a-smarter-energy-grid/ http://hortonworks.com/webinar/using-big-data-enable-smarter-energy/ http://hortonworks.com/blog/making-energy-smarter-in-the-united-kingdom/ http://hortonworks.com/blog/apache-hadoop-the-energy-softgrid-and-my-imaginary-tesla/ Here's an interesting one from Stanford, although its views on streaming capabilities is certainly dated: https://energyclub.stanford.edu/big-data-and-the-smart-grid-is-hadoop-the-answer/

hsowell · ‎06-10-2016

@David Henson, @David Van Tries I was working with another person encountering this exact issue and we went back and forth looking for a solution. They were finally able to resolve it when they switched from Internet Explorer to Firefox. I don't know for sure if the browser was the root issue, but seeing that this hasn't been resolved, it is worth a try.

hsowell · ‎05-06-2016

Have you tried something like the solution posted here: http://stackoverflow.com/questions/36958125/ansible-roles-packages-ansible-galaxy-error-on-instalation-mac-osx

hsowell · ‎02-03-2016

I'm not exactly sure what is causing the error you are coming across, but it may be easier to complete the install from a locally hosted repo. Broad strokes, you'll create a temporary HTTP server, download the the repos, place them on the HTTP server, configure the yum clients on all the nodes, and proceed with your install. Here's a link to our documentation that covers this process. Hopefully it will get you past this point. http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_using_a_local_repository.html Use the "No Internet Access" instructions.

hsowell · ‎01-31-2016

I've experienced the same issue on three different installs, Centos 6 & 6.5, Java 8, HDP 2.3.2 & 2.3.4, sandbox and cluster. On the clusters I was installing NiFi on a separate node.

hsowell · ‎12-18-2015

@Guilherme Braccialli If you've already analyzed the columns you can issue a describe table command to get column stats: "As of Hive 0.10.0, the optional parameter FOR COLUMNS computes column statistics for all columns in the specified table (and for all partitions if the table is partitioned). See Column Statistics in Hive for details. To display these statistics, use DESCRIBE FORMATTED [db_name.]table_name column_name [PARTITION (partition_spec)]." https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables

hsowell · ‎12-16-2015

Sqoop is looking for the column precision for `col3`, and because there isn't a parenthesis and precision following 'varchar', it's throwing the error. The Jira you highlighted shows the same problem. Good news, they've written the patch for this and it should be coming in the next version of Sqoop.

hsowell · ‎12-10-2015

Can you share the code you're using to kick this off? I think the hadoop-auth jar is missing in your application class path.

hsowell · ‎12-10-2015

@Mike Owo I would highly recommend checking out @Ali Bajwa's Ambari Nifi Service, which allows you to manage and monitor Nifi from Ambari. https://github.com/abajwa-hw/ambari-nifi-service

hsowell · ‎11-25-2015

Awesome stuff, Ali. Thank you! I've installed this on the sandbox without any issues and I was preparing to get it running on a small demo cluster. Have you attempted to install the service on a cluster and were there any quirks we should look out for during that install (other than the ambari vs ambari-server commands)? If not I'll try it out and let you know what, if any, additional steps were needed to get it running.

Online	Offline
Last Visited	‎08-23-2018 01:47 PM

Date Registered	‎09-28-2015 09:08 PM
Last Visited	‎08-23-2018 01:47 PM
Total Messages Posted	24
Total Kudos Received	19

Re: Has Anyone Used Dr. Elephant with HDP 2.4?

Re: Can anybody suggest a reliable and accurate wa...

Re: apache ranger 0.5.3 supporting tags

Re: Big Data for Solar Energy

Re: Reposync (CentOS 7) is failing

Re: Creating a Kibana dashboard of Twitter data pu...

Re: Using the MergeContent processor in Nifi, can ...

Using the MergeContent processor in Nifi, can I us...

Nifi 1.0.0 Beta UI Introduction

Re: SemanticException caused by 'Argument type mis...

Re: how to setting ambari to sending alert email ...

Re: Has Anyone Used Dr. Elephant with HDP 2.4?

Re: Can anybody suggest a reliable and accurate wa...

Re: Combined Application Logs

Re: apache ranger 0.5.3 supporting tags

Re: hive overwrtite table

Re: ambari-shell - can I start/stop only one servi...

Re: Piggybank not parsing XML

Re: Metron UI not displaying the Storm Events How...

Re: Big Data for Solar Energy

Re: HIVE Query exection fails to create tables... ...

Re: what version of setuptools required for Metron...

Re: Reposync (CentOS 7) is failing

Re: HDF (nifi-1.1.1.0-12) doesn't start using amba...

Re: Viewing Hive Column or Table level Statistics

Re: Cannot create a constant column in HCatalog

Re: Exception in thread "main" java.lang.NoSuchFie...

Re: does anyone know why nifi not included in hdp ...

Re: Ambari service to deploy/manage Hortonworks Io...