Re: How to create Table as an Entity and column of...

bthodla · ‎08-19-2020

I have the same issue and in my case it is a Kerberos-enabled cluster. I modified the beeline command as follows: ${HIVE_HOME}/bin/beeline -u "jdbc:hive2://<server_name>:10015/default ssl=true;principal=<principal_name>/<hostname>@<REALM>;auth=kerberos; '' ''" "$@" I get two errors: - Unknown HS2 problem when communicating with Thrift server. - Error: Could not open client transport with JDBC Uri: jdbc:hive2://<node_name>:10015/default: Invalid status 21 (state=08S01,code=0) I would appreciate any help.

irish · ‎05-07-2020

Does this mean that I solr CLI is not working or has to something with solr.

thomaskreutzer · ‎12-21-2019

You need to find the parent that has the association and delete the references. Example: I have a hive table with a column that is SSN and it had two tags associated to it. Even after the association was removed it still contains the history. When you delete you get an error message. Given type {TagName} has references. In this case I know I have a hive_column that had this reference and I need to find the proper GUID in order to delete this reference and get this test setup out of my UI. First I do a search for these items above in the Atlas UI. Check the option for Show historical entities in the event yours has been deleted. The information is displayed in the UI and this is the one I am after. Next I will use the developer tools in chrome and generally I clear out any of the history to help reduce any confusion as to what I am looking for. Next click on ssn or your entity that was associated to a classification. The url is in the panel for the get request, and the GUID in the URL is what you are after. Delete the classification from the GUID. curl -iv -u tkreutzer \ -X DELETE http://yourhost:21000/api/atlas/v2/entity/guid/ec8a34c7-db67-41b8-a14c-32a19d2166bf/classification/SSNHR Now you can delete the Classification from the UI, barring there are no other associations. If so, rinse and repeat for each hive column. I will probably try to figure out a way to do this via REST with Python later in a way that finds all associated GUID's for us. Hope this help... Cheers

ssakalabhaktula · ‎12-09-2019

i found the link with bit of google search ... https://atlas.apache.org/0.8.4/Import-Export-API.html hope that is the one you referring.

ssainath · ‎01-07-2019

@Owez Mujawar Could you run kafka console consumer on topic ATLAS_HOOK and ATLAS_ENTITIES when you create a table and check if the messages are flowing to the topic ?

zakir_hossain · ‎08-23-2018

Thanks, it was authorization problem.

ssainath · ‎05-06-2018

Added the exception stack trace seen while attempting to snapshot of hbase table . hbase(main):009:0> snapshot 'ns_beckf:hbase_table_avnlm','ss3' ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=ss3 table=ns_beckf:hbase_table_avnlm type=FLUSH } had an error. Procedure ss3 { waiting=[] done=[ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894] } at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:380) at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1124) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via ctr-e138-1518143905142-288394-01-000002.hwx.site,16020,1525628682894:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:82) at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:306) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:369) ... 6 more Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: java.io.IOException: Unable to complete flush after 3 attempts at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:349) at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:173) at org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:193) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:189) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:53) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Take a snapshot of specified table. Examples: hbase> snapshot 'sourceTable', 'snapshotName' hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH => true} Why is this error thrown?

tyu · ‎03-11-2018

Compactor calls getCompactionCompressionType() and uses it during compaction: this.compactionCompression = (this.store.getColumnFamilyDescriptor() == null) ? Compression.Algorithm.NONE : this.store.getColumnFamilyDescriptor().getCompactionCompressionType(); While getCompressionType() is used by the flusher - see code in DefaultStoreFlusher#flushSnapshot(): writer = store.createWriterInTmp(cellsCount, store.getColumnFamilyDescriptor().getCompressionType(), false, true, snapshot.isTagsPresent(), false);

ssainath · ‎03-09-2018

@Alisha Vaz POST the attached create-tag.txt to following REST API to create a tag named PII http://localhost:21000/api/atlas/v2/types/typedefs?type=classification After tag creation , get the GUID of the entity in Atlas which has to be associated to the tag. If you want to associate to an hive_table entity by name employee which is in default database , fetch the GUID of the entity using DSL search : typename = hive_table , query = where qualifiedName="default.employee@cl1" select __guid Fetch the GUID and post the associate-tag.txt to the following REST API: http://localhost:21000/api/atlas/v2/entity/bulk/classification In the associate-tag.txt file , in "entityGuids" json array ,replace the existing GUID with the GUID fetched from the DSL search query. Please refer to http://atlas.apache.org/api/v2/resource_TypesREST.html for more information on creating,updating,deleting tags and http://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_addClassification_POST for associating tags to entities.

nittala999 · ‎03-01-2018

Thank You @Sharmadha Sainath , It is working Fine 🙂

amirmam · ‎04-22-2018

you have guessed right. works now, thanks!

saba_baig · ‎07-11-2017

Thanks @Sharmadha Sainath

smartdatabundle · ‎06-26-2017

Thank you su much @Sharmadha Sainath! Your comment is very useful !!! I tested these commands : $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list MY_HOSTNAME:6667 --topic ATLAS_HOOK Then in Kafka Procuder console: {"version": {"version": "1.0.0"}, "message": {"entities": [{"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id": {"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id": "-1467290565135246000", "version": 0, "typeName": "hdfs_path", "state": "ACTIVE"}, "typeName": "hdfs_path", "values": {"qualifiedName": "TestKafka", "owner": "admin", "description": "Test kafka", "path":"/user/data/testkafka.csv", posixPermissions": null, "createTime": "1970-01-01T00:00:00.000Z", "isSymlink": false, "extendedAttributes": null, "numberOfReplicas": 0, "name": "DataKFK"}, "traitNames": [], "traits": {} }], "type": "ENTITY_CREATE", "user": "admin"} } So, Atlas notifies into ATLAS_ENTITIES topic this new entity. I could see it on Atlas UI ! Have a good day !

amestry · ‎06-16-2017

These are good questions. I hope I am able to do justice to them with my answers. To elaborate little more on what @Sarath Subramanian said. Kafka is used to do the work of relaying the notifications from Hive to Atlas. Hive publishes to a topic and Atlas subscribes to that and thus receives the notifications. There has been some discussion on using Atlas for MySQL and Oracle. I have not seen any implementation yet. This is possible, provided these 2 products have notification mechanisms. From what I know, these have database change triggers that be used to call a REST API or push some message onto a queue or publish to Kafka. For Oracle, this is what i found. Hope this helps.

ssainath · ‎01-10-2017

Can you check if Ranger policies required for Atlas are in place with proper permissions ? Example : in your Ranger , verify under cl1_hbase (cl1 is cluster name) if policies are created for the hbase tables atlas_titan and ATLAS_ENTITY_AUDIT_EVENTS. Attached screenshots for a working cluster setup.

bhagan · ‎01-06-2017

You can read messages to the console from a particular offset using the Simple Consumer CLI: https://cwiki.apache.org/confluence/display/KAFKA/System+Tools Search for Simple Consumer.

mjohnson · ‎10-26-2016

The article: Modify Atlas Entity properties using REST API commands contains a full description for how to update both the comment and description entity properties for Atlas managed hive_table types.

ssainath · ‎09-20-2016

Adding this dependency resolved the issue . <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.7.3</version> </dependency> Thanks!

Online	Offline
Last Visited	‎04-08-2020 12:04 AM

Date Registered	‎01-31-2016 04:53 PM
Last Visited	‎04-08-2020 12:04 AM
Total Messages Posted	96
Total Kudos Received	92

Re: How to create Table as an Entity and column of...

Re: Atlas V1 Entity is not clickable. Can not see ...

Re: Is order really important for hook notifiicati...

Re: how to transfer atlas tags between HDP cluster...

Re: REST API for assigning tag to an entity

Re: Hive JDBC : Could not open client transport wi...

Re: Atlas UI is not working

Re: How do I delete an Atlas tag which are associa...

Re: how to transfer atlas tags between HDP cluster...

Re: Lineage is not visible for Hive Table in Atlas

Re: Atlas V1 Entity is not clickable. Can not see ...

Hbase Snapshot Error : Unable to complete flush af...

Re: Difference between CompactionCompression and c...

Re: REST API for assigning tag to an entity

Re: How description field will get populate in Apa...

Re: How to add custom types to the "create new ent...

Re: Select clause is not working in Atlas DSL

Re: Kafka Producer on Apache Atlas via command lin...

Re: How Atlas is notified when there is a change i...

Re: Atlas metadata UI not working after enable Ran...

Re: Read messages from kafka topic

Re: How to add OR modify comment for an existing h...

Re: I am trying to create HDFS directory using ja...