About henrysowell

henrysowell · ‎09-16-2016

Great article!

henrysowell · ‎09-11-2016

Thanks @Bryan Bende, I think this is a great solution. I'm currently using two attributes to help create directory structures dynamically in HDFS: /tmp/data_staging/${SourceAttribute}/${data.date} I was planning to get around the lack of multiple correlation attribute support by doing a route on attribute processor to different merge content processors then back to a single HDFS processor using the original attributes. With your suggestion I can do a much cleaner and more flexible solution. After merging the two attributes with an UpdateAttribute processor, I'll send the data to a MergeContent processor where I'll bin the files on the new combined attribute, and then to my putHDFS processor. The merged attribute will persist (e.g., dataSource1_20160911), which I can then do something like the following to continue dynamically creating directories: /tmp/data_staging/${convergedSourceDateAttribute:substringBefore('_')}/${convergedSourceDateAttribute:substringAfter('_')} Does that seem reasonable?

henrysowell · ‎09-11-2016

I'm using the merge content processor and have successfully been using the "Correlation Attribute Name" to bin together like files using a single flowfile attribute with expression language. I would like to start using two attributes to bin and merge files. Is this possible and any help on proper syntax for this would be appreciated.

henrysowell · ‎08-10-2016

The Apache Nifi community recently released the beta version of Apache Nifi 1.0.0. This version comes with significant updates, which include a UI refresh, transition to zero master clustering, added multi-tenant authorization, and templates that are now deterministically ordered allowing for version controlled templates! The beta also boasts nine new processors, bringing the total to 165. The full list of release notes can be found here. Below I will do a very basic walkthrough of some of the UI changes from 0.7.0 to the 1.0.0 beta: Original Apache Nifi 0.7.0 Flow New Apache Nifi 1.0.0 Beta Outside of the more modernized look and feel, there are some key UI Changes: (1) Apache Nifi 1.0.0 beta now includes a status bar showing statistics of the overall flow, including bytes in and out, number of started and stopped processors, processors in error, last refresh, etc. (2) The beta also has new collapsable navigation and operation panes (3) There's a new drop down menu where you can access information like flow summary, provenance, and the bulletin board (e.g., error messages). (4) The search field is much more prominent now and allows users to search through complex flows to quickly find and jump-to processor and other elements on the flow

henrysowell · ‎06-27-2016

Hi @henryon wen, I would follow these instructions for setting up email alerts in Ambari via SMTP: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Ambari_Users_Guide/content/_configuring_notifications.html And follow Amazon's SES instructions for sending email through their SMTP interface: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-email-smtp.html I haven't tried this yet, but it seems straightforward. Please let me know if it works.

henrysowell · ‎06-13-2016

@Timothy Spann I've seen a great demo of this done by @Olivier Renault. Olivier - can you confirm?

henrysowell · ‎06-13-2016

@Christopher Frankland here are some resources I would recommend checking out: https://community.hortonworks.com/articles/136/how-to-search-for-text-in-an-image.html By @Saptak Sen Another fun one using Apache Nifi by @Jeremy Dyer https://community.hortonworks.com/articles/28380/nifi-ocr-using-apache-nifi-to-read-childrens-books.html And here's a great tutorial with its use: http://hortonworks.com/hadoop-tutorial/indexing-and-searching-text-within-images-with-apache-solr/ Here's also a blog post that is older, so please check commands, but it has some important lessons when it comes to accuracy, the quality and resolution of the PDF will greatly affect your results. http://kiirani.com/2013/03/22/tesseract-pdf.html Posting this as I think it's interesting as well to examine other effects on the source data that might affect accuracy: http://www.assistivetechnology.vcu.edu/wp-content/uploads/sites/1864/2013/09/pxc3882784.pdf

henrysowell · ‎06-12-2016

@Micky Woo the article you're pointing to is demonstrating the tag based policy ranger technical preview. While it's not GA in the Apache Ranger .5x line, it is targeted as a development theme for .6x: https://cwiki.apache.org/confluence/display/RANGER/Tag+Based+Policies

henrysowell · ‎06-12-2016

This error happens when you're trying to truncate an external table. Truncate needs to target a native/managed table or an exception will be thrown. Here's a really great reference: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL

henrysowell · ‎06-12-2016

@Manoj Gursahani Here are some resources on energy use cases. They don't all apply to solar energy specifically, but you can see how Hadoop could be using similarly: Webinar - http://hortonworks.com/webinar/how-iot-delivering-a-smarter-energy-grid/ http://hortonworks.com/webinar/using-big-data-enable-smarter-energy/ http://hortonworks.com/blog/making-energy-smarter-in-the-united-kingdom/ http://hortonworks.com/blog/apache-hadoop-the-energy-softgrid-and-my-imaginary-tesla/ Here's an interesting one from Stanford, although its views on streaming capabilities is certainly dated: https://energyclub.stanford.edu/big-data-and-the-smart-grid-is-hadoop-the-answer/

henrysowell · ‎05-06-2016

Have you tried something like the solution posted here: http://stackoverflow.com/questions/36958125/ansible-roles-packages-ansible-galaxy-error-on-instalation-mac-osx

henrysowell · ‎02-03-2016

I'm not exactly sure what is causing the error you are coming across, but it may be easier to complete the install from a locally hosted repo. Broad strokes, you'll create a temporary HTTP server, download the the repos, place them on the HTTP server, configure the yum clients on all the nodes, and proceed with your install. Here's a link to our documentation that covers this process. Hopefully it will get you past this point. http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_using_a_local_repository.html Use the "No Internet Access" instructions.

henrysowell · ‎01-31-2016

I've experienced the same issue on three different installs, Centos 6 & 6.5, Java 8, HDP 2.3.2 & 2.3.4, sandbox and cluster. On the clusters I was installing NiFi on a separate node.

henrysowell · ‎12-18-2015

@Guilherme Braccialli If you've already analyzed the columns you can issue a describe table command to get column stats: "As of Hive 0.10.0, the optional parameter FOR COLUMNS computes column statistics for all columns in the specified table (and for all partitions if the table is partitioned). See Column Statistics in Hive for details. To display these statistics, use DESCRIBE FORMATTED [db_name.]table_name column_name [PARTITION (partition_spec)]." https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables

henrysowell · ‎12-16-2015

Sqoop is looking for the column precision for `col3`, and because there isn't a parenthesis and precision following 'varchar', it's throwing the error. The Jira you highlighted shows the same problem. Good news, they've written the patch for this and it should be coming in the next version of Sqoop.

henrysowell · ‎12-10-2015

@Mike Owo I would highly recommend checking out @Ali Bajwa's Ambari Nifi Service, which allows you to manage and monitor Nifi from Ambari. https://github.com/abajwa-hw/ambari-nifi-service

henrysowell · ‎11-25-2015

Awesome stuff, Ali. Thank you! I've installed this on the sandbox without any issues and I was preparing to get it running on a small demo cluster. Have you attempted to install the service on a cluster and were there any quirks we should look out for during that install (other than the ambari vs ambari-server commands)? If not I'll try it out and let you know what, if any, additional steps were needed to get it running.

Online	Offline
Last Visited	‎08-23-2018 01:47 PM

Date Registered	‎09-28-2015 09:08 PM
Last Visited	‎08-23-2018 01:47 PM
Total Messages Posted	24
Total Kudos Received	19

Re: Has Anyone Used Dr. Elephant with HDP 2.4?

Re: Can anybody suggest a reliable and accurate wa...

Re: apache ranger 0.5.3 supporting tags

Re: Big Data for Solar Energy

Re: Reposync (CentOS 7) is failing

Re: Creating a Kibana dashboard of Twitter data pu...

Re: Using the MergeContent processor in Nifi, can ...

Using the MergeContent processor in Nifi, can I us...

Nifi 1.0.0 Beta UI Introduction

Re: how to setting ambari to sending alert email ...

Re: Has Anyone Used Dr. Elephant with HDP 2.4?

Re: Can anybody suggest a reliable and accurate wa...

Re: apache ranger 0.5.3 supporting tags

Re: hive overwrtite table

Re: Big Data for Solar Energy

Re: what version of setuptools required for Metron...

Re: Reposync (CentOS 7) is failing

Re: HDF (nifi-1.1.1.0-12) doesn't start using amba...

Re: Viewing Hive Column or Table level Statistics

Re: Cannot create a constant column in HCatalog

Re: does anyone know why nifi not included in hdp ...

Re: Ambari service to deploy/manage Hortonworks Io...