Member since
07-30-2019
93
Posts
96
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1298 | 08-12-2016 02:57 PM | |
| 1798 | 05-02-2016 09:14 PM |
09-24-2016
02:34 PM
It seems that my VM caused several headaches. I launched a fresh VM and Hive View / Atlas are working great.
... View more
09-23-2016
07:40 PM
The behavior looks to indicate a VM gone bad. I am not seeing these issues after downloading a newer version of the sandbox.
... View more
09-23-2016
04:05 PM
Running the import-hive.sh script found in the hook-bin/ directory of Atlas begins a database object import just fine but then fails with this error : 2016-09-23 15:18:27,918 WARN - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:62)
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - HDFS data model is already registered!
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Hive data model is already registered!
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing hive metadata
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered with id c09b0a90-f9f6-4b1e-85ed-dd8aa617d44e. Updating it.
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing objects from databaseName : default
[main] WARN org.apache.atlas.AtlasClient - Handled exception in calling api api/atlas/entities
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 14 more
[main] WARN org.apache.atlas.AtlasClient - Exception's cause: class java.net.SocketTimeoutException
Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 14 more Failed to import Hive Data Model!!! Possibly related to this I see that Hive View is also not working properly; however, cli access to hive is still working.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
09-23-2016
03:29 PM
1 Kudo
Tried to load Hive View and I get the following error: Service 'hiveserver' check failed:
java.sql.SQLException: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook)
java.sql.SQLException: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook)
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:348)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
at org.apache.ambari.view.hive2.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:49)
at org.apache.ambari.view.hive2.actor.StatementExecutor.runStatement(StatementExecutor.java:87)
at org.apache.ambari.view.hive2.actor.StatementExecutor.handleMessage(StatementExecutor.java:70)
at org.apache.ambari.view.hive2.actor.HiveActor.onReceive(HiveActor.java:38)
at akka.actor.UntypedActor$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I'm using hdp 2.5 sandbox. Any ideas?
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
09-23-2016
03:23 PM
@Ayub Pathan I was able to get the script run but now I get the following error 2016-09-23 15:18:27,918 WARN - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:62)
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - HDFS data model is already registered!
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Hive data model is already registered!
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing hive metadata
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered with id c09b0a90-f9f6-4b1e-85ed-dd8aa617d44e. Updating it.
[main] INFO org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing objects from databaseName : default
[main] WARN org.apache.atlas.AtlasClient - Handled exception in calling api api/atlas/entities
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 14 more
[main] WARN org.apache.atlas.AtlasClient - Exception's cause: class java.net.SocketTimeoutException
Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 14 more
Failed to import Hive Data Model!!!
... View more
09-22-2016
06:42 PM
I imported data into Hive through the Ambari View. I don't see the table in Atlas. Any idea how to get Atlas to pick up my new tables and begin to track lineage?
... View more
Labels:
09-21-2016
04:29 PM
5 Kudos
Once you have Atlas up and running( see this for getting started. ) you will want to create your first tag and use it to tag data to begin exploring what Atlas can do. Lets start by creating a tag that will be called "PII" which we will later use it to tag Personally Identifiable data. Log into Atlas and select the + icon on the homepage. Enter "PII" for the tag name and click create. That's it! Now we have a tag to use. Now click on the search tab select "dsl" and from the drop down select "hive_table" hit enter and select the "customer" table. You should see the summary details for the customer table in hive. Select the schema tab. Here you see all columns available in the customer data table. Let's mark the "account_num" field as "PII". Next to "account_num" click the + icon and select "PII" from the drop down. Now the column has been tagged and both searchable from Atlas as well as configured to be administered through Hive for auditing and permissions. Click the "Tags" tab in Atlas and search for "PII" to see your field show up.
... View more
Labels:
09-21-2016
02:09 PM
1 Kudo
There are several dependencies for Atlas. I wrote it up in an article for the next people who run into snags. https://community.hortonworks.com/articles/57315/how-to-get-atlas-up-and-running-in-hdp-25-sandbox.html#comment-57678
... View more
09-21-2016
02:00 PM
Are you on the 2.5 sandbox? If so when you search in atlas are you in the "search" tab? If you search hive in the "tags" tab it will not find anything.
... View more
09-19-2016
11:14 PM
5 Kudos
If you haven't already done so first download the Sandbox from the Hortonworks Website and import it to Virtualbox/VMware. Once you start the Virtual Machine you are directed to the landing page for your new sandbox. http://localhost:8888/ Here you have 2 Paths to choose from. If you are completely new to HDP I encourage you to browse the "New to HDP" section. Click on Advanced HDP to get to the links for the components we will be using in this tutorial. If you look at the Atlas link you see the Note: Off by default. Enable by logging in as raj_ops into Ambari You will need to start in Ambari as user ... raj_ops Go ahead and click the link for Ambari and login. If you see a page that looks like this you are doing great! Atlas has dependencies on other Apache projects. We need to make sure HDFS, Hbase, Kafka and Ambari Infra are all started. Click on each service and then under the service actions drop down select start. After verifying all components are up and running go ahead and follow the same process for Atlas. You can now use the Quick links to navigate to the Atlas Home Page : Atlas Login using the credentials provided on the Quick Links page. You can now search for artifacts and begin exploring Atlas. Try running a search for "hive" and selecting the "customer" table result. You get all the facts about that data source like the Schema and lineage seen below. Hope this was helpful. If there are any Atlas features you would like to see articles for let me know and I will use that feedback to drive future post topics.
... View more