Member since
01-31-2016
96
Posts
91
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
599 | 02-11-2019 01:04 PM | |
390 | 08-23-2018 06:22 AM | |
468 | 08-09-2018 11:29 AM | |
467 | 03-29-2018 04:55 PM | |
594 | 03-09-2018 04:09 PM |
02-11-2019
01:04 PM
1 Kudo
@Priyank Rupera Please take a look at https://github.com/apache/atlas/blob/master/addons/models/1000-Hadoop/1030-hive_model.json on how Hive is modeled in Atlas and based on that you can model your own type . hive_column is added as an attribute to hive_table type : https://github.com/apache/atlas/blob/master/addons/models/1000-Hadoop/1030-hive_model.json#L254-#L266 where hive_column is a type and entities can be created of hive_column type.
... View more
01-07-2019
05:31 AM
@Owez Mujawar Could you run kafka console consumer on topic ATLAS_HOOK and ATLAS_ENTITIES when you create a table and check if the messages are flowing to the topic ?
... View more
01-02-2019
08:48 AM
@Satya Nittala , Your requirement is not really clear to me . In the Question description you mentioned about Atlas HBase table (sounded like a Atlas restore & backup). In the further comments , I see you want Hive Backup. Can you please let us know the exact issue and requirement ?
... View more
12-19-2018
12:00 PM
@Satya Nittala Do you require Hive backup and restore ?
... View more
12-14-2018
05:25 AM
@Owez Mujawar What is the offsets.topic.replication.factor in kafka properties ? Is it greater than the number of kafka brokers ? Can you check kafka logs for error messages like the following ? ERROR [KafkaApi-1001] Number of alive brokers 'x' does not meet the required replication factor 'x' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
... View more
12-13-2018
06:20 AM
@Owez Mujawar Looks like Hive Hook messages are not read by Atlas. Can you check if Hive hook is enabled correctly and can you check the following permissions ? hive's permission to publish to ATLAS_HOOK topic atlas's permission to read from ATLAS_HOOK topic and publish to ATLAS_ENTITIES topic.
... View more
12-12-2018
09:01 AM
@Owez Mujawar It is not possible to see lineage of tables created before Atlas Hive hook is enabled. Once hive hook is enabled , and then when new tables,lineage are created , you will be able to see the lineage. Let's consider the scenario: 1. Atlas is not installed on the cluster and no hive hook is enabled . Tables , views , (table1,view1)) etc are created in Hive. 2. Install Atlas on the cluster . Run import-hive.sh to let the script contact Hive metastore and populate Atlas with existing Hive data. This script creates all tables , but lineage is not created (except for the external table). 3. Now , Enable hive hook .Create tables and views now . Since Hive hook is enabled , the lineage should be created in Atlas without running any import script. Since hook is enabled , we don't need to run the import-hive.sh script anymore.
... View more
12-11-2018
04:55 AM
@Owez Mujawar Once you enable Hive hook , you should see the table, lineage , all DDL changes of Hive in Atlas. Please refer http://atlas.apache.org/Hook-Hive.html to enable hive hook. import-hive.sh script is a one time script (no harm running multiple times though), which is used when on-boarding Atlas for the first time to sync metadata of Hive with Atlas. Import-hive.sh script basically contacts Hive metastore and populates Hive entities and sends it to Atlas. The script has no way to know how a CTAS table is created and what is the source of the table , hence lineage cannot be created. But it can know the details of external table. Hook captures live events and sends the information to Atlas.
... View more
12-10-2018
04:47 AM
@Owez Mujawar Import-hive.sh doesn't capture the lineage for internal tables. When Hive hook is enabled , and when you execute the queries, you can see the lineage in Atlas.
... View more
12-06-2018
01:19 PM
@Anpan K Yes , Atlas stores information in HBase. For fast information retrieval , the data need to be indexed and retrieved when queried. Atlas uses Janus Graph. Janus graph has 2 types of indexes : composite and mixed indexes. Composite indexes are supported with primary storage backend (HBase in this case), mixed graph indexes require indexing backend for full text search , numeric range search etc., Here , Solr is used by Atlas as indexing backend. You may read about Janus Graph for more detailed information.
... View more
12-06-2018
12:49 PM
@Satya Nittala What is your requirement ? There is a migration utility , using which you can use to export all data from Atlas and import into another.
... View more
11-22-2018
11:49 AM
@David Mason Is the Atlas Hive Hook enabled ? To capture live Hive metadata events , Hook should be enabled. To check if Hive hook for Atlas is enabled , go to Hive -> check if Enable Atlas Hook is checked and org.apache.atlas.hive.hook.HiveHook is present in hive.exec.post.hooks
... View more
11-22-2018
06:23 AM
@Anji Raju UI shows only few types. You may add a property atlas.ui.editable.entity.types=* in atlas-application properties to list down all the types or atlas.ui.editable.entity.types=hive_table,hive_column to list down only hive_table and hive_column types.
... View more
11-20-2018
06:07 AM
@Anji Raju >> Can we any way load the data manually into Atlas without hooking up with HIVE/SQOOP or so components?? Would you like to create the hive entities manually by yourself in Atlas without Hive sending it ? You can use Atlas REST APIs to do that . But would you mind sharing the requirement behind this ? >> If i have to install HIVE on the same instance, how should I hook it up?? Please refer to http://atlas.apache.org/Hook-Hive.html.
... View more
11-09-2018
02:49 PM
1.Basic search by using POST is sophisticated way compared to GET , as POST allows you to provide more params which GET cannot. 2. I go with Sarath's answer . "excludeDeletedEntities": true will exclude the DELETED entities. 3. 10k is the default max limit of results which can be changed to n using : atlas.search.maxlimit=n
... View more
10-23-2018
12:35 PM
@Satya Nittala Sorry to not have asked this in the previous comment. Could you also please send the type definition of following types? 1.mssql_table4 2.mssql_db1 I will try to create the types/entity in my instance and reproduce the issue you are facing. What is the version of Atlas or HDP ?
... View more
10-23-2018
09:15 AM
@Satya Nittala As error message says , "{"error":"Null value not allowed for multiplicty {lower=1, upper=1, isUnique=false} " , the attribute is created with upper = 1 , lower = 1 , meaning , maximum and minimum number of values the attribute can have is 1. Hence, when no value is provided , Atlas throws exception. Can you please share the type definition ( JSON posted to create the type) and the entity defintion (JSON posted to create the entity) ?
... View more
10-23-2018
08:46 AM
@Satya Nittala Can you send the result of GET REST API call ? Also, while you are on the page , do you see any exception in console tab of browser ?
... View more
09-27-2018
06:33 PM
2 Kudos
@Takefumi Oide The tag associated to the deleted entity can't be disassociated and hence can't be deleted.
... View more
08-23-2018
06:28 AM
@manikandan ayyasamy HBase hook support for Atlas is introduced in HDP 3.0 . Please upgrade HDP 2.6 to 3.0 . Also note that Atlas 0.8.0 is installed in HDP 2.6.
... View more
08-23-2018
06:22 AM
@Zakir Hossain This is because ZHXMRT user doesn't have authorization to read the zakirtestdb entity. If you log in as user who has read privileges (example : admin) , you should be able to click.
... View more
08-10-2018
05:53 AM
Eva Xiao , Messages are processed in the order they are received by Atlas. If message#1 has any error , only then message#2 is processed and action#3 specified by you can happen.
... View more
08-09-2018
11:29 AM
2 Kudos
Yes , it is important. Consider there are 2 events from Hive : 1. Rename an Hive table ( example : employee to employee_personal) 2. Add a column to the renamed Hive table. ( add address field to employee_personal) When Atlas Hive hook is configured , messages are sent for the above 2 events. Say , If message #2 is received first by Atlas first , employee_personal is not yet known to Atlas. Hence Atlas creates employee_personal hive_table entity with address field column + other columns. then , when message #1 is received , Atlas renames existing employee hive_table entity to employee_personal . Now , there are 2 employee_personal entities in Atlas, whereas in Hive , there is only 1 employee_personal table Hence , order is *very* important for Atlas being a Governance and Metadata management framework!
... View more
05-06-2018
08:36 PM
Added the exception stack trace seen while attempting to snapshot of hbase table . hbase(main):009:0> snapshot 'ns_beckf:hbase_table_avnlm','ss3'
ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=ss3 table=ns_beckf:hbase_table_avnlm type=FLUSH } had an error. Procedure ss3 { waiting=[] done=[ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894] }
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:380)
at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1124)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts
at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:82)
at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:306)
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:369)
... 6 more
Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts
at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:349)
at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:173)
at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:193)
at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:189)
at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:53)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Take a snapshot of specified table. Examples:
hbase> snapshot 'sourceTable', 'snapshotName'
hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH => true} Why is this error thrown?
... View more
Labels:
04-22-2018
01:49 PM
@Amir Mamo I guess you edited the atlas application properties file and restarted Atlas via Ambari. The property shall be added via Ambari. Add the property to Custom Atlas Application properties in Atlas Configs and start Atlas via Ambari (as suggested by stack advisor).
... View more
03-29-2018
04:55 PM
@Nicolas Schneider Please refer to http://atlas.apache.org/Import-Export-API.html
... View more
03-09-2018
08:39 PM
What is the difference between getCompactionCompressionType and getCompressionType? In https://hbase.apache.org/apidocs/src-html/org/apache/hadoop/hbase/client/ColumnFamilyDescriptor.html , the description look same for both APIs. /**
094 * @return Compression type setting.
095 */
096 Compression.Algorithm getCompactionCompressionType();
097 /**
098 * @return Compression type setting.
099 */
100 Compression.Algorithm getCompressionType();
... View more
Labels:
03-09-2018
04:09 PM
@Alisha Vaz POST the attached create-tag.txt to following REST API to create a tag named PII http://localhost:21000/api/atlas/v2/types/typedefs?type=classification After tag creation , get the GUID of the entity in Atlas which has to be associated to the tag. If you want to associate to an hive_table entity by name employee which is in default database , fetch the GUID of the entity using DSL search : typename = hive_table , query = where qualifiedName="default.employee@cl1" select __guid Fetch the GUID and post the associate-tag.txt to the following REST API: http://localhost:21000/api/atlas/v2/entity/bulk/classification In the associate-tag.txt file , in "entityGuids" json array ,replace the existing GUID with the GUID fetched from the DSL search query. Please refer to http://atlas.apache.org/api/v2/resource_TypesREST.html for more information on creating,updating,deleting tags and http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_addClassification_POST for associating tags to entities.
... View more
03-07-2018
08:35 AM
@Satya Nittala 1. Please use correct port . By default , Atlas in non-SSL environment is configured to use 21000. 2. curl requires "@" for providing files. Example : -d @user/testplace.json 3. To update type , "PUT" (not POST. To create types use POST, to update types use "PUT") the JSON to http://atlashost:21000/api/atlas/v2/types/typedefs 4. As already mentioned , classification/tag best suits your requirement. Its highly recommended to use tags instead of updating types.
... View more
03-06-2018
12:50 PM
@Satya Nittala yes , POST the JSON body attached in the file to http://localhost:21000/api/atlas/v2/types/typedefs?type=classification In the tag definition , name is the name of the tag , and attributeDefs is an JSON array of attribute definitions. I have added expiry_date attribute of type date in the example. Once the tag is created , the tag can be associated to the hive_table entity by POSTing the attached tag-association.txt to http://localhost:21000/api/atlas/v2/entity/bulk/classification in tag-association.txt , "name" is the name of tag . attribute values can be provided in "attributes". entityGuids is the list of all the GUIDs of entity the tag should be associated to. In this array , you can provide the hive_table GUID.
... View more