About Debu77

Debu77 · ‎08-20-2019

Hi, Greetings and wish you are doing good ! I am using Hadoop 3.1 version. I am connected Atlas to our Enterprise Kafka Cluster. My objective is to visualise the metadata of the Kafka Topics within Atlas User Interface. With the above objective in mind, I would like to ask the following questions: 1. What kind of data lineage facility for Kafka topics is available in Atlas ? For example, I have some metadata listed in a Kafka topic and can I visualise the same in Atlas 2. Just like Hive tables, can we have a similar data lineage across Kafka Topics? 3. What kind of Data Governance use cases can be catered through Kafka in Atlas ? 4. Does Atlas uses Kafka to communicate with Rangers ? Will appreciate for your feedback as per your earliest convenience please. Thanks, Debu

Debu77 · ‎06-28-2019

Hi Everyone, Greetings ! I want to integrate Atlas to a Kafka Cluster , which is outside the Hortonworks Data Platform. Would you please be able to help let me know whether it's possible or is it mandatory that the Kafka cluster needs to exist within the HDP itself Thanks, Debu

Debu77 · ‎04-08-2017

Hi, I am using HDP 2.5. I want to perform a delete operation from a Hive table. I have created a table with the following command: hive> create table test (x int, y string) clustered by (x) into 2 buckets stored as ORC tblproperties ("transactional" = "true"); OK Time taken: 0.148 seconds Further, I have set the following Hive properties: SET hive.support.concurrency=true; SET hive.enforce.bucketing=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET hive.compactor.initiator.on=true; SET hive.compactor.worker.threads=1; set hive.optimize.sort.dynamic.partition=false; Then on, I am doing a test insert into the hive table leveraging the following command: INSERT INTO TEST VALUES (1,a); Unfortunately my Hive CLI shell is getting hung and I have to issue Ctrl + C command to get out of the shell. May I know why is this happening please? Is it due to the fact that my Hive schema / database contains a mixture of ACID and non-ACID tables ? Any suggestion to resolve this problem will be very helpful. Thanks in advance !

Debu77 · ‎04-08-2017

Hi, I am using HDP 2.5. I want to perform a delete operation from a Hive table. I have created a table with the following command: hive> create table test (x int, y string) clustered by (x) into 2 buckets stored as ORC tblproperties ("transactional" = "true"); OK Time taken: 0.148 seconds Further, I have set the following Hive properties: SET hive.support.concurrency=true; SET hive.enforce.bucketing=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET hive.compactor.initiator.on=true; SET hive.compactor.worker.threads=1; set hive.optimize.sort.dynamic.partition=false; Then on, I am doing a test insert into the hive table leveraging the following command: INSERT INTO TEST VALUES (1,a); Unfortunately my Hive CLI shell is getting hung and I have to issue Ctrl + C command to get out of the shell. May I know why is this happening please? Is it due to the fact that my Hive schema / database contains a mixture of ACID and non-ACID tables ? Any suggestion to resolve this problem will be very helpful. Thanks in advance !

Online	Offline
Last Visited	‎09-27-2019 06:50 AM

Date Registered	‎06-02-2019 02:20 AM
Last Visited	‎09-27-2019 06:50 AM
Total Messages Posted	4

Query regarding Data Lineage for Kafka Topics in A...

Can we integrate an external Kafka cluster with At...

Hortonworks 2.5 - Hive ACID Insert is getting hung

Re: How to activate ACID transactions in Hive with...