question Re: Entity ID generation in Navigator in Support Questions

Entity ID generation in Navigator

geochap — Mon, 21 Nov 2016 13:22:38 GMT

The entity indentity field returned via Navigator API calls appears to be a MD5 hash of key entity fields. E.g. from looking at the navigator SDK source an HDFS entity's identity should be the hash of the sourceId and the fileSystemPath (separated by ##):

https://github.com/cloudera/navigator-sdk/blob/master/model/src/main/java/com/cloudera/nav/sdk/model/HdfsIdGenerator.java#L30

But if I try to reproduce a sample entity's identity value using that approach I can't get the same identity value that actually comes back in API for that entity.

Any ideas why the identities might differ? Is a salt value used in the production version when hashing, or does the separator differ in some cases?

I'm testing on CDH 5.8.0.

Re: Entity ID generation in Navigator

geochap — Mon, 21 Nov 2016 22:39:54 GMT

The issue is that the guava MD5 hasher hashes a string as a series of utf-16 chars if not passed an encoding:

http://stackoverflow.com/questions/21287714/hashing-issue-between-guava-versions

I was testing using a different MD5 implementation which treated the string as utf-8 encoded. Once I changed my code to treat the same way guava does, the IDs matched.

Re: Entity ID generation in Navigator

mark — Fri, 02 Dec 2016 23:52:48 GMT

Hi there,

Although this might work right now, we're using a different algorithm to calculate the entity ID starting in C5.10, which ships next month. We'll automatically translate from the old entity ID to the new one transparently, but you're best off using our API to determine the entity ID -- specifically:

api/v10/entities/?query=<entity_query>

I hope this helps. Regards, Mark.

Re: Entity ID generation in Navigator

geochap — Sat, 03 Dec 2016 00:48:21 GMT

Thanks for the info.

I don't have a real need to generate ids this way at the moment -- mostly I was just curious how it was done. I'd seen the parent link in navigator from an HDFS entity to its parent and figured you must be generating that functionally from other available info since the parent ID isn't part of the entity payload (later I realized the page could probably just be using a relation query to get it). Also, I was wondering whether I could create custom entities without using the java plugin.

I've been using the api as you suggest and for the most part I've found what I've needed between the docs, the sdk code, and a fair amount of experimentation.