About joe_harvyy

joe_harvyy · ‎03-24-2018

Hello, I thought that NiFi 1.5 can report lineage to Atlas but I couldn't find how to it. I've checked the reporting tasks but couldn't find Atlas reporting task. Does anyone succeeded in using this new feature? Thanks Joe

joe_harvyy · ‎10-01-2017

Hello, There are several tools in HDP that doesn't use Yarn (Storm, HBase, etc). If I have OS, HBase, Storm and other tools taking resources on my cluster, how does Yarn know how much resources he owns for its application? What are the best practices for multi-tenancy and isolation in this case? How can I isolate IO with Yarn? is this something coming in future versions? Thanks

joe_harvyy · ‎06-13-2017

Hi @rkovacs Thank you for your reply. I understand that I can not upgrade a Cloudbreak cluster since Ambari upgrade is not supported. What do you mean by HDP upgrade is supported? Thanks for the clarification

joe_harvyy · ‎06-12-2017

Hi, How can I upgrade a cluster deployed by Cloudbreak? If I upgrade through Ambari, what's the impact on Cloudbreak relation? (API, version, Autoscaling, etc) Thanks

joe_harvyy · ‎03-12-2017

Hi I have two HDF cluster, one secured (with SSL enabled) and the other without security. Can I have the cluster 2 (unsecure) send data to cluster 1 (secured) with S2S ? if yes, what's the required configuration? Thanks

joe_harvyy · ‎01-29-2017

Hi @Matt Burgess Thanks for your detailed answer. Your first suggestion looks interesting. I'll give it a try. I still have a question on ConvertJsonToSQL if you can help https://community.hortonworks.com/questions/80362/jdbc-connection-pool-for-convertjsontosql.html

joe_harvyy · ‎01-29-2017

Hi, What's the role of the JDBC Connection Pool property of the ConvertJsonToSQL processor ? How to use this processor with Hive ? I noticed that HiveConnectionPool was possible choice for this property in previous NiFi version but it's no more the case ? Thanks

joe_harvyy · ‎01-27-2017

@Binu Mathew Thanks for your reply. No I don't need data in JSON, I just need to ingest directly into Hive. How do you ingest data in Hive in your suggestion ? PutHiveQL waits for an SQL statement

joe_harvyy · ‎01-27-2017

Hi @mqureshi I don't agree with your statement. If it's the case, why there's processors like getFile, getHDFS, QueyTable etc ?

joe_harvyy · ‎01-27-2017

I want to use NiFi to ingest CSV file (getFile) to Hive (PutHiveQL). Since HiveQL need the SQL statement, how can I generate it ? A solution would be getFile -> InferAvroSchema -> ConvertCSVToAvro -> ConvertAvroToJSon -> ConvertJSONtoSQL -> PutHiveQL. This looks so complex and resource consuming. Any suggestions ?

Online	Offline
Last Visited	‎03-24-2018 10:28 PM

Date Registered	‎01-02-2017 03:29 PM
Last Visited	‎03-24-2018 10:28 PM
Total Messages Posted	18
Total Kudos Received	3

Reporting NiFi lineage to Atlas

Yarn behaviour with external tools

Re: Upgrade a cluster deployed by Cloudbreak

Upgrade a cluster deployed by Cloudbreak

NiFi S2S between secure and unsecure clusters

Re: From CSV to Hive via NiFi

JDBC Connection Pool for ConvertJsonToSQL

Re: From CSV to Hive via NiFi

Re: From CSV to Hive via NiFi

From CSV to Hive via NiFi