Hopefully the OP is sorted out by now, but for anyone else who comes across this: one of the best step-by-step instructions on Atlas relationships I've found is this comment by Vinayak Marriaya on the Atlas Jira; re-producing here, in case that link dies:   Following are the json's to create relationships 1. Creating EntityDef json 2. Creating Entity json 3. Creating RelationshipDef json 4. Creating Relationship instance json Step-1 Creating EntityDef POST - http://localhost:21000/api/atlas/v2/types/typedefs   { "entityDefs": [ { "category": "ENTITY", "name": "type_a" }, { "category": "ENTITY", "name": "type_b" } ] }   Step-2 Creating Entity POST - http://localhost:21000/api/atlas/v2/entity/bulk   { "entities": [ { "typeName": "type_a", "guid": "-72ddefefS2UY" }, { "typeName": "type_b", "guid": "-JL1HddDOfdf" } ] }   Step-3 Creating RelationshipDef POST - http://localhost:21000/api/atlas/v2/types/typedefs   { "relationshipDefs": [ { "propagateTags": "ONE_TO_TWO", "description": "description ASSOCIATION between type5 and type6", "relationshipCategory": "ASSOCIATION", "typeVersion": "1.0", "attributeDefs": [ { "name": "LinkInformation", "typeName": "string", "isOptional": true, "cardinality": "SINGLE", "valuesMinCount": 0, "valuesMaxCount": 1, "isUnique": false, "isIndexable": false, "includeInNotification": false, "searchWeight": -1 } ], "endDef2": { "name": "type_b_rel_attribute", "isLegacyAttribute": false, "isContainer": false, "cardinality": "SINGLE", "type": "type_b", "description": "description with name: type_2_rel_attribute" }, "endDef1": { "name": "type_a_rel_attribute", "isLegacyAttribute": false, "isContainer": false, "cardinality": "SET", "type": "type_a", "description": "description with name: type_1_rel_attribute" }, "guid": "-2fsdfjhsewl04", "name": "association_type_a_and_type_b" } ] }   Step-4 Creating Relationship instance POST- http://localhost:21000/api/atlas/v2/relationship   { "typeName": "association_type_a_and_type_b", "end1": { "typeName": "type_a", "guid": "b4dae5e8-a606-4e41-9ce3-8f35245f389e" (guid of type_a entity) }, "propagateTags": "NONE", "provenanceType": 0, "propagatedClassifications": [], "end2": { "typeName": "type_b", "guid": "23c2f3c1-dd74-4190-a6d1-b012c44cbb6d" (guid of type_b entity) }, "blockedPropagatedClassifications": [], "guid": "-bjbfdfddfeffef", "attributes": { "LinkInformation": "TestingInformation" } }     Following is the output of the above API call. Output of type_a entity GET - http://localhost:21000/api/atlas/v2/entity/guid/\{guid of type_a entity}   { "referredEntities": {}, "entity": { "typeName": "type_a", "guid": "b4dae5e8-a606-4e41-9ce3-8f35245f389e", "isIncomplete": false, "status": "ACTIVE", "createdBy": "admin", "updatedBy": "admin", "createTime": 1632121151626, "updateTime": 1632121151626, "version": 0, "relationshipAttributes": { "type_a_rel_attribute": [ { "guid": "23c2f3c1-dd74-4190-a6d1-b012c44cbb6d", "typeName": "type_b", "entityStatus": "ACTIVE", "relationshipType": "association_type_a_and_type_b", "relationshipGuid": "ec64783a-58d7-4265-87d6-c1535ce2d9b7", "relationshipStatus": "ACTIVE", "relationshipAttributes": { "typeName": "association_type_a_and_type_b", "attributes": { "LinkInformation": "TestingInformation" } } } ] }, "labels": [] } }   Output of type_b entity GET - http://localhost:21000/api/atlas/v2/entity/guid/\{guid of type_b entity} { "referredEntities": {}, "entity": { "typeName": "type_b", "guid": "23c2f3c1-dd74-4190-a6d1-b012c44cbb6d", "isIncomplete": false, "status": "ACTIVE", "createdBy": "admin", "updatedBy": "admin", "createTime": 1632121151626, "updateTime": 1632121151626, "version": 0, "relationshipAttributes": { "type_b_rel_attribute": { "guid": "b4dae5e8-a606-4e41-9ce3-8f35245f389e", "typeName": "type_a", "entityStatus": "ACTIVE", "relationshipType": "association_type_a_and_type_b", "relationshipGuid": "ec64783a-58d7-4265-87d6-c1535ce2d9b7", "relationshipStatus": "ACTIVE", "relationshipAttributes": { "typeName": "association_type_a_and_type_b", "attributes": { "LinkInformation": "TestingInformation" } } } }, "labels": [] } } As you can see, LinkInformation is specified under "attributes" and when you search for the entity using GUID as you mentioned in jira, you will be able to see the value of "LinkInformation".   ... View more

Hi Christ,   Hope you are doing good! I see that you would like to configure the Atlas UI in different languages. On testing, though I didn't do any changes from the Service configs but we can achieve this via the Web-browser Settings itself. a) Like for Chrome browser: Go to Settings >> Search for 'language' >> click on language tab and add your preferred language on top of the list.  b) OR you can just Right-click on the Atlas UI webpage and click on 'Translate to' and select the required, that would change just your Atlas UI session language quick (if that suits your requirement): Ref: https://support.google.com/chrome/answer/173424?co=GENIE.Platform%3DDesktop&hl=en    Regards, Shantanu Singh ... View more

Hello,   I'm not done with this issue, for information our HDP use a KDC hosted on an Active Directory wich is used for authenticatation. For security matters we wants to change HDP account passwords on a regular basis : - When I click on "Kerberos/regenerate keytabs" everything is ok for 90% of the accounts (password is changed and keytab regenerated), but as stated in my first post for 10% of the accounts nothing is done, so I have to remove unmodified keytabs and click on "Only regenerate keytabs for missing hosts and components", the keytabs are regenerated but passwords for those accounts are not modified in the AD.    I've tried for the 10% accounts : - to change password in AD, generate keytabs on the AD and push the keytabs on HDP boxes => keytabs are not recognized by HDP - to change password in AD, generate keytabs on the HDP boxes and push the keytabs on HDP boxes => keytabs are not recognized by HDP (pre-authentication failed error message)    So my question is : how can I easily change all the HDP account passwords without breaking Kerberos authentication ?    The only working method I've found is to fully disable kerberos on the HDP cluster (to remove all accounts in the AD) and activate again kerberos on HDP cluster (which creates accounts in the AD with new passwords).   any help on this matter would be greatly appreciated.  regards, ... View more

Hi CV, I see that you have setup Standalone Atlas on CDH edgenode. Though I haven't tested this setup, ideally Apache Atlas should have the pre-defined system types (DataSet). Let me know if you tried creating custom Entities via Atlas API to check if it's getting reflected in the Atlas UI or with the atlas hive import script if it popultaes the meta[1] <atlas package>/hook-bin/import-hive.sh Also, are you observing any exceptions on Atlas Metadata Application logs? You can also attach the atlas configs here to compare validate with CDP/HDP cluster configs if something missing. [1] https://atlas.apache.org/1.2.0/Hook-Hive.html  ... View more

Hello there, The below property would be required if you would like to set RPC encryption[1]: hadoop.rpc.protection = privacy authentication : authentication only (default); integrity : integrity check in addition to authentication; privacy : data encryption in addition to integrity   RPC encryption [2]:The most common way for a client to interact with a Hadoop cluster is through RPC.  A client connects to a NameNode (NN) over RPC protocol to read or write a file. RPC connections in Hadoop use Java’s Simple Authentication & Security Layer (SASL) which supports encryption. When hadoop.rpc.protection property is set to 'privacy' the data over RPC is encrypted with symmetric keys.    Ref: [1] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/configuring-wire-encryption/content/enabling_rpc_encryption.html  Kindly check the below Additional references for Wire Encryption and RPC Encryption blog post with detailed explanation[3]: Ref: [2] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/configuring-wire-encryption/content/wire_encryption.html  Ref: [3] https://blog.cloudera.com/wire-encryption-hadoop/  Ref: [4] Apache Jir Ref: Hadoop in Secure Mode: https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/SecureMode.html    Hope this helps! Let me know if you have any queries. ... View more

Hello there, I understand your use-case to save up some HDFS space. Though I haven't tested zipping possibilities for hdfs level files[2]. Alternately, you may consider reviewing HDFS Erasure Coding[1] if that suits your requirement:  ErasureCoding in HDFS significantly reduces storage overhead while achieving similar or better fault tolerance through the use of parity cells (similar to RAID5). Prior to the introduction of EC, HDFS used 3x replication for fault tolerance exclusively, meaning that a 1GB file would use 3 GB of raw disk space. With EC, the same level of fault tolerance can be achieved using only 1.5 GB of raw disk space. Please refer the below article[1] for more insights on EC: Ref[1]: https://blog.cloudera.com/hdfs-erasure-coding-in-production/ [2] https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-choosing-configuring-data-compression.html ... View more