Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

S3 Data discovery in Cloudera navigator

S3 Data discovery in Cloudera navigator

Contributor

Hello,

 

I am trying to create an environment to have metadata and lineage information in Cloudera Navigator. For this I am trying to add the connection of S3 and thinking of running the backup sync from HDFS to S3. Does this collects the metadata lineage information in Cloudera Navigator.

 

Is there any other way to collect metadata in Cloudera Navigator.

 

Thanks & Regards,

Rajesh

9 REPLIES 9
Highlighted

Re: S3 Data discovery in Cloudera navigator

Cloudera Employee

Rajesh, 

 

Have you reviewed this document?

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/navigator_s3_metadata_extraction.htm...

 

What version of CDH and Director are you using?

 

Thanks
Seth

Re: S3 Data discovery in Cloudera navigator

Contributor

Hi,

 

I am trying to collect S3 objects metadata in ClouderaNavigator and did the following configuration after which I was not able to see any sourceType for S3 in Cloudera Navigator. Could you please help on this

Cloudera Version: CDH 6.1.0, Parcels

Added S3 account as an External Account in Cloudera Manager and enabled metadata

 

S3Account.PNG

After that I see the S3 Extraction was enabled in Cloudera Navigator Administration Advanced tab

Clouderanavigator.PNG

After that I was not able to get any metadata or sourcetype added for S3 in Cloudera Navigator. Did i miss something here?

 

Thanks & Regards,

Rajesh

Re: S3 Data discovery in Cloudera navigator

Cloudera Employee

Rajesh, 

Directions for use with CDH6x [1][2] will guide you on what will be extracted to S3.

 

 

LINKS:

[1] https://www.cloudera.com/documentation/enterprise/6/6.2/topics/navigator_s3_metadata_extraction.html 

[2] https://www.cloudera.com/documentation/enterprise/6/6.2/topics/navigator_s3.html

 

Thanks
Seth

Re: S3 Data discovery in Cloudera navigator

Contributor

Hi Seth,

 

Thank you for the link. I followed the document and created default configuration and also provided Amazon S3 and SQS access to the user I used. Still I was not able to see any metadata or source type for S3 in Cloudera navigator. Could you please help on this? Also, please let me know how I can get the live support to resolve this issue quickly.

 

Thanks & Regards,

Rajesh

Re: S3 Data discovery in Cloudera navigator

Contributor

Hello @seth ,

 

Can you please help on this issue?

 

Thanks & Regards,

Rajesh

Re: S3 Data discovery in Cloudera navigator

Contributor

Hello,

Can you please help on this issue.

Thanks & Regards,

Rajesh

Re: S3 Data discovery in Cloudera navigator

Contributor

Hello,

I have tried to do custom configuration and done setup for SQS and SNS and I see that the messages are part of the Queue but not received in Clouderanavigator. Could you please help on this.

SQSConfiguration.PNGSQSMessages

 

Thanks & Regards,

Rajesh

Re: S3 Data discovery in Cloudera navigator

Cloudera Employee

Rajesh, Live support would be apart of a support entitlments and you can open a case here [1]

 

At this point we would need to check your configuration via a support bundle from this cluster . Is that possible?

 

[1] https://my.cloudera.com/support.html

Re: S3 Data discovery in Cloudera navigator

Contributor

Hi Seth,

 

Thanks for the response. I have generated the bundle data file and not sure that it can contain any credentials. Let me confirm from my side. Can you please confirm that it will get credentials information too?

 

 

Thanks & Regards,

Rajesh