Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Falcon - Hive External Table eviction with HCat

Highlighted

Falcon - Hive External Table eviction with HCat

Contributor

HDP 2.3, Falcon, HCatalog, Hive Metastore, Oozie as Scheduler

-----------------------------------------------------------------------------------------

I am trying to do External Hive Table Eviction using a FEED entity which depends on the following Cluster entity. In my understanding, the falcon runs a hcat api with table uri with type storage mentioned in the feed xml.

The feed successfully remove the External Hive Table Partition from Hive Metadata but the HDFS resource path for the external hive table partition is not getting removed. Is this the expected behavior (Dropping Table Partition will not drop external resource path using hcat api) or I am doing something wrong? I can see messages in application logs dropping table partitions but no error at all.

Cluster.xml

---------------------------------------------------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<cluster xmlns="uri:falcon:cluster:0.1" name="PRI" description="DC" colo="houston">

<interfaces>

<interface type="readonly" endpoint="hftp://<active-namenode>:50070" version="2.7.0"/>

<interface type="write" endpoint="hdfs://<Namenode-Servicename>:8020" version="2.7.0"/>

<interface type="execute" endpoint="<Active-ResourceManager>:8050" version="2.7.0"/>

<interface type="workflow" endpoint="http://<oozie-server>:11000/oozie/" version="4.2.0"/>

<interface type="messaging" endpoint="tcp://<falcon-server-activemq>:61616?daemon=true" version="5.1.6"/>

<interface type="registry" endpoint="thrift://<hive-metastore-server>:9083" version="1.2.1"/>

</interfaces>

<locations>

<location name="staging" path="/data/falcon/staging"/>

<location name="temp" path="/data/falcon/temp"/>

<location name="working" path="/data/falcon/work"/>

</locations>

<ACL owner="falcon" group="users" permission="0755"/>

</cluster>

FEED.xml

-------------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<feed xmlns="uri:falcon:feed:0.1" name="sc">

<availabilityFlag>_SUCCESS</availabilityFlag>

<frequency>days(1)</frequency>

<timezone>UTC</timezone>

<late-arrival cut-off="hours(2)"/>

<clusters>

<cluster name="PRI" type="source">

<validity start="2016-04-03T10:46Z" end="2099-04-04T10:46Z"/>

<retention limit="months(72)" action="delete"/>

</cluster>

</clusters>

<table uri="catalog:default:test-table#test_partition=${YEAR}-${MONTH}-${DAY}"/>

<ACL owner="falcon" group="users" permission="0755"/>

<schema location="hcat" provider="hcat"/>

<properties> <property name="timeout" value="hours(1)"/> </properties>

</feed>

1 REPLY 1

Re: Falcon - Hive External Table eviction with HCat

Super Guru

@Sushil Saxena when you drop a partition on a hive external table through a hive client (ie beeline, the behavior is what you are experiencing. I expect the exact same behavior if done via falcon. You can have the data removed if it is a hive internal table or create a similar feed on retention for the underlying hdfs file directory.