Using Hive Hook: File does not exist: atlas-client-0.7-incubating-SNAPSHOT.jar

[#1]

After configure the Hive Hook according http://atlas.apache.org/Bridge-Hive.html, I could create table in the Hive CLI. Then I could find this table in Atlas Web UI and it showed that no lineage data was found.

So I search this question in this community, I find this tutorial of @Ryan Cicak:

https://community.hortonworks.com/articles/36121/using-apache-atlas-to-view-data-lineage.html

According this tutorial, I run the following command in Hive CLI:

create table brancha(full_name string, ssn string, location string);

It would be executed successfully.

But when I tried to run this:

insert into brancha(full_name,ssn,location) values ('ryan', '111-222-333', 'chicago'); 

It reported Error which is like this:

hive> insert into brancha(full_name,ssn,location) values ('ryan', '111-222-333', 'chicago'); 
Query ID = hadoop_20160627200051_392a732d-cf49-4d13-a10d-cea68fb32217
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://localhost:9000/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

It means that the JAR file:atlas-client-0.7-incubating-SNAPSHOT.jar doesn't exist.

I also searched other solutions, and I run command of CTAS style:

CREATE TABLE table1 AS SELECT * FROM table2;

It reported the same error mentioned above.

~

In order solve this problem, I post my setting of <atlas-package>/conf/atlas-application.properties if it can be helpful:

#########  Graph Database Configs  #########
# Graph Storage
#atlas.graph.storage.backend=berkeleyje
#atlas.graph.storage.directory=${sys:atlas.home}/data/berkley

#Hbase as stoarge backend
atlas.graph.storage.backend=hbase
#For standalone mode , specify localhost
#for distributed mode, specify zookeeper quorum here - For more information refer http://s3.thinkaurelius.com/docs/titan/current/hbase.html#_remote_server_mode_2
atlas.graph.storage.hostname=localhost
atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000

#Solr
#atlas.graph.index.search.backend=solr

# Solr cloud mode properties
#atlas.graph.index.search.solr.mode=cloud
#atlas.graph.index.search.solr.zookeeper-url=localhost:2181

#Solr http mode properties
#atlas.graph.index.search.solr.mode=http
#atlas.graph.index.search.solr.http-urls=http://localhost:8983/solr

# Graph Search Index
#ElasticSearch
atlas.graph.index.search.backend=elasticsearch
atlas.graph.index.search.directory=${sys:atlas.home}/data/es
atlas.graph.index.search.elasticsearch.client-only=false
atlas.graph.index.search.elasticsearch.local-mode=true
atlas.graph.index.search.elasticsearch.create.sleep=2000


#########  Notification Configs  #########
atlas.notification.embedded=true
atlas.kafka.data=${sys:atlas.home}/data/kafka
atlas.kafka.zookeeper.connect=localhost:9026
atlas.kafka.bootstrap.servers=localhost:9027
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.auto.offset.reset=smallest
atlas.kafka.hook.group.id=atlas


#########  Hive Lineage Configs  #########
# This models reflects the base super types for Data and Process
#atlas.lineage.hive.table.type.name=DataSet
#atlas.lineage.hive.process.type.name=Process
#atlas.lineage.hive.process.inputs.name=inputs
#atlas.lineage.hive.process.outputs.name=outputs

## Schema
atlas.lineage.hive.table.schema.query.hive_table=hive_table where name='%s'\, columns
atlas.lineage.hive.table.schema.query.Table=Table where name='%s'\, columns

## Server port configuration
#atlas.server.http.port=21000
#atlas.server.https.port=21443

#########  Security Properties  #########

# SSL config
atlas.enableTLS=false

#truststore.file=/path/to/truststore.jks
#cert.stores.credential.provider.path=jceks://file/path/to/credentialstore.jceks

#following only required for 2-way SSL
#keystore.file=/path/to/keystore.jks

# Authentication config

# enabled:  true or false
atlas.http.authentication.enabled=false
# type:  simple or kerberos
atlas.http.authentication.type=simple

#########  Server Properties  #########
atlas.rest.address=http://localhost:21000
# If enabled and set to true, this will run setup steps when the server starts
#atlas.server.run.setup.on.start=false

#########  Entity Audit Configs  #########
atlas.audit.hbase.tablename=ATLAS_ENTITY_AUDIT_EVENTS
atlas.audit.zookeeper.session.timeout.ms=1000
atlas.audit.hbase.zookeeper.quorum=localhost:2181

#########  High Availability Configuration ########
atlas.server.ha.enabled=false
#### Enabled the configs below as per need if HA is enabled #####
#atlas.server.ids=id1
#atlas.server.address.id1=localhost:21000
#atlas.server.ha.zookeeper.connect=localhost:2181
#atlas.server.ha.zookeeper.retry.sleeptime.ms=1000
#atlas.server.ha.zookeeper.num.retries=3
#atlas.server.ha.zookeeper.session.timeout.ms=20000
## if ACLs need to be set on the created nodes, uncomment these lines and set the values ##
#atlas.server.ha.zookeeper.acl=<scheme>:<id>
#atlas.server.ha.zookeeper.auth=<scheme>:<authinfo>


#### atlas.login.method {FILE,LDAP,AD} ####
atlas.login.method=FILE

### File path of users-credentials
atlas.login.credentials.file=${sys:atlas.home}/conf/users-credentials.properties

Most of settings are default, I never change them.

And I never start the Hiveserver2 and the service of metastore of hive. All the commands are executed in the Hive CLI(Command-Line Interface).

[#2]

Hi @Ethan Hsieh

Can you try running the commands in beeline or Ambari View? I wouldn't recommend running in Hive CLI - let me know if Beeline or Ambari View output the same error.

[#3]

Hi, @Ryan Cicak

I try it in the beeline, and the error is same. it seems that the atlas-client-*.jar is missing.

Where could I find the client.jar ?

Here is the report:

hadoop@eite:~$ beeline
Beeline version 1.2.1 by Apache Hive
beeline> !connect jdbc:hive2://localhost:10000/default
Connecting to jdbc:hive2://localhost:10000/default
Enter username for jdbc:hive2://localhost:10000/default: hadoop
Enter password for jdbc:hive2://localhost:10000/default: ******
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/default> insert into brancha(full_name,ssn,location) values ('ryan', '111-222-333', 'chicago'); 
INFO  : Number of reduce tasks is set to 0 since there's no reduce operator
WARN  : Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
INFO  : Cleaning up the staging area file:/usr/local/hadoop/tmp/mapred/staging/hadoop1956486297/.staging/job_local1956486297_0002
ERROR : Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: hdfs://localhost:9000/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar)'
java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)
    at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
    at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
    at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1)

[#4]

Hi, @Ryan Cicak

I can find the file atlas-client-0.7-incubating-SNAPSHOT.jar in this path:

/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar

But why it search this file with HDFS prefix: hdfs://localhost:9000

hdfs://localhost:9000/usr/local/data-governance/apache-atlas-0.7-incubating-SNAPSHOT/hook/hive/atlas-client-0.7-incubating-SNAPSHOT.jar)

How to change the path of JAR files from a HDFS path to the path of local file system?

[#5 · Accepted solution]

@Ethan Hsieh

Atlas client should be available to hive if the HIVE_AUX_JARS_PATH is set to proper location (for local machine installation, it should be set to 'HIVE_AUX_JARS_PATH=<atlas package>/hook/hive').

Can you please check if this is set properly? Please execute below command and find out if it is included in the running hive process just like below. If it is not, please take appropriate action.

lsof -p 24932 | grep jar | grep -i atlas


java    24932 hive  mem    REG             254,16   379806  3148643 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-typesystem-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16  5137159  3148644 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/kafka_2.10-0.10.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16  3203471  3148637 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-reflect-2.10.4.jar
java    24932 hive  mem    REG             254,16    64723  3148640 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-client-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16   745492  3148638 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/kafka-clients-0.10.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16   597252  3148630 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-core_2.10-3.2.11.jar
java    24932 hive  mem    REG             254,16    57339  3148632 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/hive-bridge-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16    81388  3148633 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/guice-multibindings-4.0.jar
java    24932 hive  mem    REG             254,16    41144  3148636 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-common-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16    84053  3148635 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-ast_2.10-3.2.11.jar
java    24932 hive  mem    REG             254,16  7126372  3148645 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-library-2.10.4.jar
java    24932 hive  mem    REG             254,16 14445780  3148634 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-compiler-2.10.4.jar
java    24932 hive  mem    REG             254,16   853905  3148642 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scalap-2.10.4.jar
java    24932 hive  mem    REG             254,16    26484  3148639 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/hdfs-model-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16    76504  3148641 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-notification-0.7.0.2.5.0.0-831.jar
java    24932 hive  mem    REG             254,16    69460  3148631 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-native_2.10-3.2.11.jar
java    24932 hive    4r   REG             254,16    69460  3148631 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-native_2.10-3.2.11.jar
java    24932 hive    5r   REG             254,16    76504  3148641 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-notification-0.7.0.2.5.0.0-831.jar
java    24932 hive    6r   REG             254,16    26484  3148639 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/hdfs-model-0.7.0.2.5.0.0-831.jar
java    24932 hive    7r   REG             254,16   853905  3148642 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scalap-2.10.4.jar
java    24932 hive    8r   REG             254,16 14445780  3148634 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-compiler-2.10.4.jar
java    24932 hive    9r   REG             254,16  7126372  3148645 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-library-2.10.4.jar
java    24932 hive   10r   REG             254,16    84053  3148635 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-ast_2.10-3.2.11.jar
java    24932 hive   11r   REG             254,16    41144  3148636 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-common-0.7.0.2.5.0.0-831.jar
java    24932 hive   12r   REG             254,16    81388  3148633 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/guice-multibindings-4.0.jar
java    24932 hive   13r   REG             254,16    57339  3148632 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/hive-bridge-0.7.0.2.5.0.0-831.jar
java    24932 hive   14r   REG             254,16   597252  3148630 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/json4s-core_2.10-3.2.11.jar
java    24932 hive   15r   REG             254,16   745492  3148638 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/kafka-clients-0.10.0.2.5.0.0-831.jar
java    24932 hive   16r   REG             254,16    64723  3148640 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-client-0.7.0.2.5.0.0-831.jar
java    24932 hive   17r   REG             254,16  3203471  3148637 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/scala-reflect-2.10.4.jar
java    24932 hive   18r   REG             254,16  5137159  3148644 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/kafka_2.10-0.10.0.2.5.0.0-831.jar
java    24932 hive   19r   REG             254,16   379806  3148643 /grid/0/hdp/2.5.0.0-831/atlas/hook/hive/atlas-typesystem-0.7.0.2.5.0.0-831.jar