Member since
06-02-2019
4
Posts
0
Kudos Received
0
Solutions
08-20-2019
11:14 AM
Hi, Greetings and wish you are doing good ! I am using Hadoop 3.1 version. I am connected Atlas to our Enterprise Kafka Cluster. My objective is to visualise the metadata of the Kafka Topics within Atlas User Interface. With the above objective in mind, I would like to ask the following questions: 1. What kind of data lineage facility for Kafka topics is available in Atlas ? For example, I have some metadata listed in a Kafka topic and can I visualise the same in Atlas 2. Just like Hive tables, can we have a similar data lineage across Kafka Topics? 3. What kind of Data Governance use cases can be catered through Kafka in Atlas ? 4. Does Atlas uses Kafka to communicate with Rangers ? Will appreciate for your feedback as per your earliest convenience please. Thanks, Debu
... View more
Labels:
06-28-2019
12:08 PM
Hi Everyone, Greetings ! I want to integrate Atlas to a Kafka Cluster , which is outside the Hortonworks Data Platform. Would you please be able to help let me know whether it's possible or is it mandatory that the Kafka cluster needs to exist within the HDP itself Thanks, Debu
... View more
Labels:
04-08-2017
06:27 PM
Hi, I am using HDP
2.5. I want to perform a delete operation from a Hive table. I have created a
table with the following command: hive> create table test (x int, y string) clustered by (x) into 2 buckets
stored as ORC tblproperties ("transactional" = "true"); OK Time taken: 0.148 seconds Further, I have set the following Hive properties: SET hive.support.concurrency=true; SET hive.enforce.bucketing=true; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET hive.compactor.initiator.on=true; SET hive.compactor.worker.threads=1; set hive.optimize.sort.dynamic.partition=false; Then on, I am doing a test insert into the hive table leveraging the
following command: INSERT INTO TEST VALUES (1,a); Unfortunately my Hive CLI shell is getting hung and I have to issue Ctrl + C
command to get out of the shell. May I know why is this happening please? Is it
due to the fact that my Hive schema / database contains a mixture of ACID and
non-ACID tables ? Any suggestion to resolve this problem will be very helpful. Thanks in advance !
... View more
- Tags:
- Data Processing
- Hive
Labels:
04-08-2017
06:17 PM
Hi, I am using HDP 2.5. I want to perform a delete operation from a Hive table. I have created a table with the following command: hive> create table test (x int, y string) clustered by (x) into 2
buckets stored as ORC tblproperties ("transactional" = "true");
OK
Time taken: 0.148 seconds Further, I have set the following Hive properties: SET hive.support.concurrency=true;
SET hive.enforce.bucketing=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
SET hive.compactor.initiator.on=true;
SET hive.compactor.worker.threads=1;
set hive.optimize.sort.dynamic.partition=false; Then on, I am doing a test insert into the hive table leveraging the following command: INSERT INTO TEST VALUES (1,a); Unfortunately my Hive CLI shell is getting hung and I have to issue Ctrl + C command to get out of the shell. May I know why is this happening please? Is it due to the fact that my Hive schema / database contains a mixture of ACID and non-ACID tables ? Any suggestion to resolve this problem will be very helpful. Thanks in advance !
... View more