About ssainath

ssainath · ‎02-11-2019

@Priyank Rupera Please take a look at https://github.com/apache/atlas/blob/master/addons/models/1000-Hadoop/1030-hive_model.json on how Hive is modeled in Atlas and based on that you can model your own type . hive_column is added as an attribute to hive_table type : https://github.com/apache/atlas/blob/master/addons/models/1000-Hadoop/1030-hive_model.json#L254-#L266 where hive_column is a type and entities can be created of hive_column type.

ssainath · ‎01-07-2019

@Owez Mujawar Could you run kafka console consumer on topic ATLAS_HOOK and ATLAS_ENTITIES when you create a table and check if the messages are flowing to the topic ?

ssainath · ‎01-02-2019

@Satya Nittala , Your requirement is not really clear to me . In the Question description you mentioned about Atlas HBase table (sounded like a Atlas restore & backup). In the further comments , I see you want Hive Backup. Can you please let us know the exact issue and requirement ?

ssainath · ‎12-19-2018

@Satya Nittala Do you require Hive backup and restore ?

ssainath · ‎12-14-2018

@Owez Mujawar What is the offsets.topic.replication.factor in kafka properties ? Is it greater than the number of kafka brokers ? Can you check kafka logs for error messages like the following ? ERROR [KafkaApi-1001] Number of alive brokers 'x' does not meet the required replication factor 'x' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

ssainath · ‎12-13-2018

@Owez Mujawar Looks like Hive Hook messages are not read by Atlas. Can you check if Hive hook is enabled correctly and can you check the following permissions ? hive's permission to publish to ATLAS_HOOK topic atlas's permission to read from ATLAS_HOOK topic and publish to ATLAS_ENTITIES topic.

ssainath · ‎12-12-2018

@Owez Mujawar It is not possible to see lineage of tables created before Atlas Hive hook is enabled. Once hive hook is enabled , and then when new tables,lineage are created , you will be able to see the lineage. Let's consider the scenario: 1. Atlas is not installed on the cluster and no hive hook is enabled . Tables , views , (table1,view1)) etc are created in Hive. 2. Install Atlas on the cluster . Run import-hive.sh to let the script contact Hive metastore and populate Atlas with existing Hive data. This script creates all tables , but lineage is not created (except for the external table). 3. Now , Enable hive hook .Create tables and views now . Since Hive hook is enabled , the lineage should be created in Atlas without running any import script. Since hook is enabled , we don't need to run the import-hive.sh script anymore.

ssainath · ‎12-11-2018

@Owez Mujawar Once you enable Hive hook , you should see the table, lineage , all DDL changes of Hive in Atlas. Please refer http://atlas.apache.org/Hook-Hive.html to enable hive hook. import-hive.sh script is a one time script (no harm running multiple times though), which is used when on-boarding Atlas for the first time to sync metadata of Hive with Atlas. Import-hive.sh script basically contacts Hive metastore and populates Hive entities and sends it to Atlas. The script has no way to know how a CTAS table is created and what is the source of the table , hence lineage cannot be created. But it can know the details of external table. Hook captures live events and sends the information to Atlas.

ssainath · ‎12-10-2018

@Owez Mujawar Import-hive.sh doesn't capture the lineage for internal tables. When Hive hook is enabled , and when you execute the queries, you can see the lineage in Atlas.

ssainath · ‎12-06-2018

@Anpan K Yes , Atlas stores information in HBase. For fast information retrieval , the data need to be indexed and retrieved when queried. Atlas uses Janus Graph. Janus graph has 2 types of indexes : composite and mixed indexes. Composite indexes are supported with primary storage backend (HBase in this case), mixed graph indexes require indexing backend for full text search , numeric range search etc., Here , Solr is used by Atlas as indexing backend. You may read about Janus Graph for more detailed information.

ssainath · ‎12-06-2018

@Satya Nittala What is your requirement ? There is a migration utility , using which you can use to export all data from Atlas and import into another.

ssainath · ‎11-22-2018

@David Mason Is the Atlas Hive Hook enabled ? To capture live Hive metadata events , Hook should be enabled. To check if Hive hook for Atlas is enabled , go to Hive -> check if Enable Atlas Hook is checked and org.apache.atlas.hive.hook.HiveHook is present in hive.exec.post.hooks

ssainath · ‎11-22-2018

@Anji Raju UI shows only few types. You may add a property atlas.ui.editable.entity.types=* in atlas-application properties to list down all the types or atlas.ui.editable.entity.types=hive_table,hive_column to list down only hive_table and hive_column types.

ssainath · ‎11-20-2018

@Anji Raju >> Can we any way load the data manually into Atlas without hooking up with HIVE/SQOOP or so components?? Would you like to create the hive entities manually by yourself in Atlas without Hive sending it ? You can use Atlas REST APIs to do that . But would you mind sharing the requirement behind this ? >> If i have to install HIVE on the same instance, how should I hook it up?? Please refer to http://atlas.apache.org/Hook-Hive.html.

ssainath · ‎11-09-2018

1.Basic search by using POST is sophisticated way compared to GET , as POST allows you to provide more params which GET cannot. 2. I go with Sarath's answer . "excludeDeletedEntities": true will exclude the DELETED entities. 3. 10k is the default max limit of results which can be changed to n using : atlas.search.maxlimit=n

ssainath · ‎10-23-2018

@Satya Nittala Sorry to not have asked this in the previous comment. Could you also please send the type definition of following types? 1.mssql_table4 2.mssql_db1 I will try to create the types/entity in my instance and reproduce the issue you are facing. What is the version of Atlas or HDP ?

ssainath · ‎10-23-2018

@Satya Nittala As error message says , "{"error":"Null value not allowed for multiplicty {lower=1, upper=1, isUnique=false} " , the attribute is created with upper = 1 , lower = 1 , meaning , maximum and minimum number of values the attribute can have is 1. Hence, when no value is provided , Atlas throws exception. Can you please share the type definition ( JSON posted to create the type) and the entity defintion (JSON posted to create the entity) ?

ssainath · ‎10-23-2018

@Satya Nittala Can you send the result of GET REST API call ? Also, while you are on the page , do you see any exception in console tab of browser ?

ssainath · ‎09-27-2018

@Takefumi Oide The tag associated to the deleted entity can't be disassociated and hence can't be deleted.

ssainath · ‎08-23-2018

@manikandan ayyasamy HBase hook support for Atlas is introduced in HDP 3.0 . Please upgrade HDP 2.6 to 3.0 . Also note that Atlas 0.8.0 is installed in HDP 2.6.

ssainath · ‎08-23-2018

@Zakir Hossain This is because ZHXMRT user doesn't have authorization to read the zakirtestdb entity. If you log in as user who has read privileges (example : admin) , you should be able to click.

ssainath · ‎08-10-2018

Eva Xiao , Messages are processed in the order they are received by Atlas. If message#1 has any error , only then message#2 is processed and action#3 specified by you can happen.

ssainath · ‎08-09-2018

Yes , it is important. Consider there are 2 events from Hive : 1. Rename an Hive table ( example : employee to employee_personal) 2. Add a column to the renamed Hive table. ( add address field to employee_personal) When Atlas Hive hook is configured , messages are sent for the above 2 events. Say , If message #2 is received first by Atlas first , employee_personal is not yet known to Atlas. Hence Atlas creates employee_personal hive_table entity with address field column + other columns. then , when message #1 is received , Atlas renames existing employee hive_table entity to employee_personal . Now , there are 2 employee_personal entities in Atlas, whereas in Hive , there is only 1 employee_personal table Hence , order is *very* important for Atlas being a Governance and Metadata management framework!

ssainath · ‎05-06-2018

Added the exception stack trace seen while attempting to snapshot of hbase table . hbase(main):009:0> snapshot 'ns_beckf:hbase_table_avnlm','ss3' ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=ss3 table=ns_beckf:hbase_table_avnlm type=FLUSH } had an error. Procedure ss3 { waiting=[] done=[ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894] } at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:380) at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1124) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:82) at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:306) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:369) ... 6 more Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:349) at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:173) at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:193) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:189) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:53) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Take a snapshot of specified table. Examples: hbase> snapshot 'sourceTable', 'snapshotName' hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH => true} Why is this error thrown?

ssainath · ‎04-22-2018

@Amir Mamo I guess you edited the atlas application properties file and restarted Atlas via Ambari. The property shall be added via Ambari. Add the property to Custom Atlas Application properties in Atlas Configs and start Atlas via Ambari (as suggested by stack advisor).

ssainath · ‎03-29-2018

@Nicolas Schneider Please refer to http://atlas.apache.org/Import-Export-API.html

ssainath · ‎03-09-2018

What is the difference between getCompactionCompressionType and getCompressionType? In https://hbase.apache.org/apidocs/src-html/org/apache/hadoop/hbase/client/ColumnFamilyDescriptor.html , the description look same for both APIs. /** 094 * @return Compression type setting. 095 */ 096 Compression.Algorithm getCompactionCompressionType(); 097 /** 098 * @return Compression type setting. 099 */ 100 Compression.Algorithm getCompressionType();

ssainath · ‎03-09-2018

@Alisha Vaz POST the attached create-tag.txt to following REST API to create a tag named PII http://localhost:21000/api/atlas/v2/types/typedefs?type=classification After tag creation , get the GUID of the entity in Atlas which has to be associated to the tag. If you want to associate to an hive_table entity by name employee which is in default database , fetch the GUID of the entity using DSL search : typename = hive_table , query = where qualifiedName="default.employee@cl1" select __guid Fetch the GUID and post the associate-tag.txt to the following REST API: http://localhost:21000/api/atlas/v2/entity/bulk/classification In the associate-tag.txt file , in "entityGuids" json array ,replace the existing GUID with the GUID fetched from the DSL search query. Please refer to http://atlas.apache.org/api/v2/resource_TypesREST.html for more information on creating,updating,deleting tags and http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_addClassification_POST for associating tags to entities.

ssainath · ‎03-07-2018

@Satya Nittala 1. Please use correct port . By default , Atlas in non-SSL environment is configured to use 21000. 2. curl requires "@" for providing files. Example : -d @user/testplace.json 3. To update type , "PUT" (not POST. To create types use POST, to update types use "PUT") the JSON to http://atlashost:21000/api/atlas/v2/types/typedefs 4. As already mentioned , classification/tag best suits your requirement. Its highly recommended to use tags instead of updating types.

ssainath · ‎03-06-2018

@Satya Nittala yes , POST the JSON body attached in the file to http://localhost:21000/api/atlas/v2/types/typedefs?type=classification In the tag definition , name is the name of the tag , and attributeDefs is an JSON array of attribute definitions. I have added expiry_date attribute of type date in the example. Once the tag is created , the tag can be associated to the hive_table entity by POSTing the attached tag-association.txt to http://localhost:21000/api/atlas/v2/entity/bulk/classification in tag-association.txt , "name" is the name of tag . attribute values can be provided in "attributes". entityGuids is the list of all the GUIDs of entity the tag should be associated to. In this array , you can provide the hive_table GUID.

Online	Offline
Last Visited	‎04-08-2020 12:04 AM

Date Registered	‎01-31-2016 04:53 PM
Last Visited	‎04-08-2020 12:04 AM
Total Messages Posted	96
Total Kudos Received	91

Re: How to create Table as an Entity and column of...

Re: Atlas V1 Entity is not clickable. Can not see ...

Re: Is order really important for hook notifiicati...

Re: how to transfer atlas tags between HDP cluster...

Re: REST API for assigning tag to an entity

Re: How to create Table as an Entity and column of...

Re: Lineage is not visible for Hive Table in Atlas

Re: How to connect & get data from Atlas HBase Tab...

Re: How to connect & get data from Atlas HBase Tab...

Re: Lineage is not visible for Hive Table in Atlas

Re: Lineage is not visible for Hive Table in Atlas

Re: Lineage is not visible for Hive Table in Atlas

Re: Lineage is not visible for Hive Table in Atlas

Re: Lineage is not visible for Hive Table in Atlas

Re: solr and atlas

Re: How to connect & get data from Atlas HBase Tab...

Re: Atlas Metadata not being created when objects ...

Re: Manually load data into Apache Atlas

Re: Manually load data into Apache Atlas

Re: How to get the active hive tables count in Apa...

Re: Unable to load the data in Apache atals UI

Re: I am getting "{"error":"Null value not allowed...

Re: Unable to load the data in Apache atals UI

Re: How do I delete an Atlas tag which are associa...

Re: how to fetch Hbase column name in ATLAS?

Re: Atlas V1 Entity is not clickable. Can not see ...

Re: Is order really important for hook notifiicati...

Re: Is order really important for hook notifiicati...

Hbase Snapshot Error : Unable to complete flush af...

Re: How to add custom types to the "create new ent...

Re: how to transfer atlas tags between HDP cluster...

Difference between CompactionCompression and compr...

Re: REST API for assigning tag to an entity

Re: Can i able to add new column in hive_table in ...

Re: Can i able to add new column in hive_table in ...