Reply
Explorer
Posts: 6
Registered: ‎07-20-2017

PreRegistering Metadata in Cloudera Navigator is not working

[ Edited ]

Hi,

 

We have a requirement to tag some information in Cloudera Navigator to certain hive tables using an automated process once the table is created in HIve.

 

We are trying to achieve that using the Navigator Rest API. We are trying to use the POST method to Pre-register some information, expecting that once Navigator extracts this table information, the metadata that we pre-registered will be linked to the table in Navigator.

 

The above process is mentioned in the cloudera documentation. Please see the link below and search for preregister.

 

Editing HDFS and Hive Metadata Using the Navigator Metadata API

 

However it is not working as expected. 

 

For example:

1. I create a hive managed table named "test_table" under a db called "pos_test"

2. Now, the navigator metadata extraction process will take some pre-defined time to extract this table meta information to navigator db.

3. So, I am trying to pre-register the metadata for this table with the tags I need.

4.  I use the following POST URL to pre-register, passing the original_name and the parent_path

 

curl http://localhost:7187/api/v10/entities/ -u uname:pass -X POST -H\
"Content-Type: application/json" -d '{
"originalName": "test_table",
"sourceId": "4",
"parentPath":"/pos_test",
"name":"test_table",
"description":"My description",
"tags": ["cz_autotagged"]
}'

 

 

5. I get a success response back, with the following identity "2382707" for the Hive table

 

6. Now I wait for the Cloudera Navigator extraction process to extract the table information and make it avaible in Navigator search.

 

7. The table is now available in Search. But I see that a different identity called "2382710" has been assigned to it, and the pre-registered metadata is not linked to this table.

 

Is the pre-registering process that I followed correct?

Why is the pre-registered metadata not linked when the table is extracted?

How does the linking process work( based on what keys?)

 

Any help regarding this will be much appreciated.

 

Thanks

renjoe

 

 

Explorer
Posts: 6
Registered: ‎07-20-2017

Re: PreRegistering Metadata in Cloudera Navigator is not working

Hi,

We are in a deadline situation. Any help would be much appreciated. I am new to this community. How do we get the attention of the cloudera moderators for posts?

Regards
Renju
PSK
New Contributor
Posts: 10
Registered: ‎12-13-2016

Re: PreRegistering Metadata in Cloudera Navigator is not working

I will be trying this soon. 

 

This is from my understanding - I ll leave it to the experts to correct me if this is incorrect.

 

According to this link that shows the steps for metadata preregistration, you need to get the ID using curl and then use that ID to pre-register metadata.

 

e.g. to pre-register a Hive table that is yet to be sourced into Navigator, below query to get the ID and then use that ID with the query you posted.

 

http://localhost:7187/api/v10/entities/?query=((type:SOURCE)AND(sourceType:Hive))%27

Let me know if this works out for you.

Explorer
Posts: 6
Registered: ‎07-20-2017

Re: PreRegistering Metadata in Cloudera Navigator is not working

Hi PSK,

 

Thanks for your reply.

 

That is a misunderstanding. The whole point behind metadata pre-registation is that we would not be knowing the id of the entity that is going to be created, but can pre-register some metadata for a "to-be" created table/file.

 

The api call that you are referring to below is for getting the id of the source type entity (see type:SOURCE), and not the table entity. 

http://localhost:7187/api/v10/entities/?query=((type:SOURCE)AND(sourceType:Hive))%27

I have already got the id of the source entity, and have mentioned it in my request. Please see below. The "sourceId": "4" entry in the request below is retrieved using the above query. This is the source id for all the hive tables.

 

curl http://localhost:7187/api/v10/entities/ -u uname:pass -X POST -H\
"Content-Type: application/json" -d '{
"originalName": "test_table",
"sourceId": "4",
"parentPath":"/pos_test",
"name":"test_table",
"description":"My description",
"tags": ["cz_autotagged"]
}'

 

Hope I have made it clearer.

 

Regards

renjoe

 

Highlighted
PSK
New Contributor
Posts: 10
Registered: ‎12-13-2016

Re: PreRegistering Metadata in Cloudera Navigator is not working

What do you get when you query (thru API) for the entity ID - 2382707?

Announcements
New solutions