About CK71

Bill Havanki · ‎08-29-2019

Hi CK, Here are some tips to help answer your question. First, use cloud storage services, like S3 on AWS or ADLS on Azure, for keeping data like Navigator lineage information. Those services provide availability and reliability automatically. Hadoop and other services can be configured to use cloud storage in various ways instead of local block (hard drive) storage. Sometimes it's not efficient or high-performing enough to exclusively use cloud storage. For example, a typical data analysis job may have several stages where data is read, processed, and then written back, and those round trips to storage services can be slower and cost more money than local drive access. So, think about adjusting how data is managed, so that intermediate data resides in local block storage, but final results are sent to cloud storage for safekeeping. Once all of the important data is safe in cloud storage, it becomes less important to keep cluster instances running. You can even destroy entire clusters, including master / manager nodes, knowing that the data is safe in cloud storage. At this point, you will want to use automation tools, like Altus Director or Cloudbreak, so that you can easily spin up new clusters that are configured to pull their initial data from cloud storage. Then, you only run clusters when you need them. If that isn't feasible, you can still do something like what you suggest, with clusters that have some permanent nodes and some transient ones. If so, ensure that those transient nodes do not keep important state that isn't safe elsewhere. For example, YARN node managers are stateless, so scaling nodes only housing those ("worker" nodes) is an easy goal to achieve. By constrast, HDFS datanodes store file data, so those aren't as easy to scale down - you can, though, as long as they are decommissioned properly using Cloudera Manager or Ambari, which the cloud automation tools handle for you.

CK71 · ‎07-10-2019

Hi All Why use Atlas as it is not a proper Business Data Catalog which can by used by business users (only by data stewards probably) and enterprise still need a Business Data Catalog (such as Waterline, Alation, Collibra, etc)? We can just get one of these tools directly which can then sync its own meta-data and tags and synch back to Atlas/Ranger to protect the data access. Do we really need to keep Atlas around if there is such a Business Data Catalogue in place? And how Ranger will manage the Data Access Control in such case? Thanks CK

CK71 · ‎07-10-2019

hi All, I would like to capture end-to-end metadata from the source (Kafka producer) until its final consumption. Assume, the producer submits the feed, then NIFI picks up this feed from Kafka topic, does transformation and saves it into Hive. To get this end to end lineage, is the following correct: 1) On-Board the new feed by manually defining new types on Navigator for the Source as Kafka_Source_1 (which is the Kafka Producer) 2) Assuming, Navigator has the hook to create Kafka topics automatically as Types in the Navigator. For Eg: Kafka_Topic_1, etc. 3) As the producer keeps sending the messages to Kafka, NIFI will read such messages, perform transformation, extract the technical and business meta-data, and then create the related instances with Business Glossary, Operational Meta-data (Entities) in Navigator by calling REST API (custom implementation on NIFI Data pipeline) including creation of Hive endpoint as Type. ( I wish, NIFI would better manage the metadata internally and then export to Atlas in batches.) Is there an easier way doing this? 4) Thus, from Navigator, I will see the lineage as: Kafka_Source_1 --> Kafka_Topic_1 --> Message_Transformation_112121 --> Hive_Path_1 If the above flow is OK, 5) Assuming, NIFI is calling the REST API to create the instances for each bulk read from Kafka, the duplicate definitions created will be consolidated within Navigator to give one lineage line from this Source to Destination. 6) If the same feed has its schema (or nay of the business, technical, operational meta-data) changed, how this change will be captured by Navigator? It will create a new lineage or a new version of it. So how the dynamic meta-data is managed? Thanks in advance CK

CK71 · ‎07-09-2019

hi All, I would like to capture end-to-end metadata from the source (Kafka producer) until its final consumption. Assume, the producer submits the feed, then NIFI picks up this feed from Kafka topic, does transformation and saves it into Hive. To get this end to end lineage, is the following correct: 1) On-Board the new feed by manually defining new types on Atlas for the Source as Kafka_Source_1 (which is the Kafka Producer) 2) Assuming, Atlas has the hook to create Kafka topics automatically as Types in the Atlas. For Eg: Kafka_Topic_1, etc. 3) As the producer keeps sending the messages to Kafka, NIFI will read such messages, perform transformation, extract the technical and business meta-data, and then create the related instances with Business Glossary, Operational Meta-data (Entities) in Atlas by calling REST API (custom implementation on NIFI Data pipeline) including creation of Hive endpoint as Type. ( I wish, NIFI would better manage the metadata internally and then export to Atlas in batches.) Is there an easier way doing this? 4) Thus, from Atlas, I will see the lineage as: Kafka_Source_1 --> Kafka_Topic_1 --> Message_Transformation_112121 --> Hive_Path_1 If the above flow is OK, 5) Assuming, NIFI is calling the REST API to create the instances for each bulk read from Kafka, the duplicate definitions created will be consolidated within Atlas to give one lineage line from this Source to Destination. 6) If the same feed has its schema (or nay of the business, technical, operational meta-data) changed, how this change will be captured by Atlas? It will create a new lineage or a new version of it. So how the dynamic meta-data is managed? Thanks in advance CK

Harsh J · ‎07-09-2019

Yes that is correct, and the motivations/steps-to-use are reflected here too: https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_s3guard.html Note: On your point of 'load data from S3 into HDFS', it is better stated as simply 'read data from S3', where HDFS gets used as a transient storage (where/when required). There does not need to be a 'download X GiB data from S3 to HDFS first, only then begin jobs' step, as distributed jobs can read off of S3 via s3a:// URLs in the same way they do from HDFS hdfs://.

CK71 · ‎07-01-2019

Hi Bill Thanks a lot for the long explanation. So two options exists (please correct me If I am wrong): 1) Use persistent master nodes and expand for extra temp workloads on demand 2) Alternatively, Have components (Hive, Navigator, etc) save their metadata into S3 or RDS so that all cluster can be teared off and then created from scracth repeately. (That is a better option for cost savings) Many Thanks, Cengiz

CK71 · ‎05-20-2019

Thanks for this. I think, we can summarize this as follows: * If only External Hive Table is used to process S3 data, the technical issues regarding consistency, scalable meta-data handling would be resolved. * If External & Internal Hive Tables are used in combination to process S3 data, the technical issues regarding consistency, scalable meta-data handling and data locality would be resolved. * If Spark alone is used on top of S3, the technical issues regarding consistency with (in memory processing), scalable meta-data handling would be resolved. As Spark will perform transient storage in memory and only read the initial data from S3 and write back the result.

CK71 · ‎05-11-2019

Hi All I am evaluating DS WorkBench, so appreciate your views on how Data Science Workbench supports the following: - Model Parallelism: Support of Horovod out of the box? - Platform Agnostic: Any platform specific implementation. How easy to migrate to other platforms? - Language Supported: Python, R - Framework Supported: Scikit Learn, XGBoost, Tensor, Keras, MXNet, etc - Collaborative Environment: Can share experiments, models with teams. - Integration with MLFlow: To creating platform agnostic model formats: Edge, Tensor Serving, SparkML, Pickle, etc. - CI/CD Pipelines support - AB Testing - Monitoring Regards CK

CK71 · ‎05-07-2019

Hi All I want to capture Business (KPI's) and technical (number of records) meta-data into Atlas for each file being ingested.During ETL, I will capture Operational Meta-data (KPI's and successfully processed record) into Atlas. Is it possible to create a Micro Strategy report showing discrepancy of what was ingested and what is the final discrepancy by comparing business/technical metadata from feeds to custom operational metadata (KPI's) that belongs to final entities created as part of that lineage? I do not think REST API will be a good interface to do that. Should Atlas be used for this type for Reconciliation process or suggesting some other approach? Regards CK

CK71 · ‎10-14-2018

We have pipelines to read hundreds of batch files over the NIFI to read, validate each file (its formats specific to each file) and then ingest into HDFS. But configuring each file, creating its own validation rules over NIFI GUI is pain. Can we use any open source tool or any other method to make this configuration easier? Should we use schema approach instead where the source supplies a schema with the file and schema Id in the file? NIFI will open up the file and validate it against its schema in a generic way? Is there any piece of code doing that? Thanks..

Online	Offline
Last Visited	‎07-27-2019 10:07 AM

Date Registered	‎05-11-2019 01:11 AM
Last Visited	‎07-27-2019 10:07 AM
Total Messages Posted	13

Re: Temporary Public Cloud Clusters

How to integrate Atlas with Business Data Catalogu...

Navigator - Metadata Management Approach

Atlas metadata Management Approach

Re: S3Guard Suggested to help fix Consistency

Re: Best Practice for Temporary Hadoop Cluster in ...

Re: S3 loading into HDFS

Data Science Workbench - clarifications

Atlas & Reconciliations

How to configure hundreds data pipelines easily