Reporting NiFi lineage to Atlas

joe_harvyy · ‎03-24-2018

Hello, I thought that NiFi 1.5 can report lineage to Atlas but I couldn't find how to it. I've checked the reporting tasks but couldn't find Atlas reporting task. Does anyone succeeded in using this new feature? Thanks Joe

joe_harvyy · ‎11-07-2017

Hello, Is there any recommendation on number of concurrent users with Hive LLAP and Druid ? is it acceptable to have 100 or 1000 users with these solutions? If no, is their any other solution to achieve this scale? Thanks Joe

joe_harvyy · ‎10-01-2017

Hello, There are several tools in HDP that doesn't use Yarn (Storm, HBase, etc). If I have OS, HBase, Storm and other tools taking resources on my cluster, how does Yarn know how much resources he owns for its application? What are the best practices for multi-tenancy and isolation in this case? How can I isolate IO with Yarn? is this something coming in future versions? Thanks

joe_harvyy · ‎06-13-2017

Hi @rkovacs Thank you for your reply. I understand that I can not upgrade a Cloudbreak cluster since Ambari upgrade is not supported. What do you mean by HDP upgrade is supported? Thanks for the clarification

joe_harvyy · ‎06-12-2017

Hi, How can I upgrade a cluster deployed by Cloudbreak? If I upgrade through Ambari, what's the impact on Cloudbreak relation? (API, version, Autoscaling, etc) Thanks

joe_harvyy · ‎03-13-2017

My questions are for HDF not HDP

joe_harvyy · ‎03-12-2017

Hi, From my understanding, enabling SSL in NiFi and connecting to LDAP is enough for security. What's the advantage of HDF Kerberisation ? is it only for Kafka and Storm ? Can I have a non-Kerberised HDF cluster talking with a Kerberised HDF cluster Thanks

joe_harvyy · ‎03-12-2017

Hi I have two HDF cluster, one secured (with SSL enabled) and the other without security. Can I have the cluster 2 (unsecure) send data to cluster 1 (secured) with S2S ? if yes, what's the required configuration? Thanks

joe_harvyy · ‎01-29-2017

Hi @Matt Burgess Thanks for your detailed answer. Your first suggestion looks interesting. I'll give it a try. I still have a question on ConvertJsonToSQL if you can help https://community.hortonworks.com/questions/80362/jdbc-connection-pool-for-convertjsontosql.html

joe_harvyy · ‎01-29-2017

Hi, What's the role of the JDBC Connection Pool property of the ConvertJsonToSQL processor ? How to use this processor with Hive ? I noticed that HiveConnectionPool was possible choice for this property in previous NiFi version but it's no more the case ? Thanks

joe_harvyy · ‎01-27-2017

@Binu Mathew Thanks for your reply. No I don't need data in JSON, I just need to ingest directly into Hive. How do you ingest data in Hive in your suggestion ? PutHiveQL waits for an SQL statement

joe_harvyy · ‎01-27-2017

Hi @mqureshi I don't agree with your statement. If it's the case, why there's processors like getFile, getHDFS, QueyTable etc ?

joe_harvyy · ‎01-27-2017

I want to use NiFi to ingest CSV file (getFile) to Hive (PutHiveQL). Since HiveQL need the SQL statement, how can I generate it ? A solution would be getFile -> InferAvroSchema -> ConvertCSVToAvro -> ConvertAvroToJSon -> ConvertJSONtoSQL -> PutHiveQL. This looks so complex and resource consuming. Any suggestions ?

joe_harvyy · ‎01-27-2017

Thanks @Eugene Koifman Can you point to an updated complete and updated documentation/book on Hive features ? (ACID, LLAP, etc)

joe_harvyy · ‎01-27-2017

Hi, I created a simple Hive table (everything default) and were surprised that I could insert data into it doing INSERT INTO TABLE etc. The Hive manual claims that insert is available only for tables that support ACID. However, I didn't set anything to enable ACID support for my table. Also, data is not in ORC. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingvaluesintotablesfromSQL Can you please explain this behavior ? Thanks

joe_harvyy · ‎01-17-2017

Hi, I am able to backup all my hive tables by backuping the HDFS folder /apps/hive/warehouse and the MySQL database (mysql dump and restore). How can I backup some Hive tables only? for HDFS files it's easy. But how can I backup/restore the metadata from MySQL for only those tables ? Cheers! Joe

Online	Offline
Last Visited	‎03-24-2018 10:28 PM

Date Registered	‎01-02-2017 03:29 PM
Last Visited	‎03-24-2018 10:28 PM
Total Messages Posted	18
Total Kudos Received	3

Reporting NiFi lineage to Atlas

Number of concurrent users in LLAP/Druid

Yarn behaviour with external tools

Re: Upgrade a cluster deployed by Cloudbreak

Upgrade a cluster deployed by Cloudbreak

Re: Kerberised HDF cluster

Kerberised HDF cluster

NiFi S2S between secure and unsecure clusters

Re: From CSV to Hive via NiFi

JDBC Connection Pool for ConvertJsonToSQL

Re: From CSV to Hive via NiFi

Re: From CSV to Hive via NiFi

From CSV to Hive via NiFi

Re: Hive insert and ACID

Hive insert and ACID

Backup specific Hive table