Hello guys, Currently,my requirement is to process data of 3 hive tables and store it's result into another hive table.Therefore,I have created 3 hive uri input feed ,1 hive uri output feed and 1 process which is accepting this 3 feed entity as input and genereting output in output feed.Process entity giving error as: FAILED: RuntimeException MetaException(message:java.lang.ClassNotFoundException Classorg.openx.data.jsonserde.JsonSerDe not found) Actually i am getting this error in oozie when scheduled process entity gets failed I understood that error is so simple and it's caused because of serDe jar is missing somewhere within hive/lib or oozir/share/lib. I tried the following solutions: 1) add a jar to hive lib folder or add it to oozie lib. Jar: json-serde-1.3.8-SNAPSHOT-jar- with-dependencies.jar;. But still getting same error.if i add the jar through hive cli using ADD JAR command then query run smoothly there. My xml are: 1) observationInputFeed.xml <feed xmlns='uri:falcon:feed:0.1' name='observationInputFeed' description='This is observation table'> <tags>table=observation</tags> <frequency>hours(1)</frequency> <timezone>UTC</timezone> <clusters> <cluster name='hiveCluster' type='source'> <validity start='2016-05-31T09:00Z' end='2016-06-06T09:00Z'/> <retention limit='days(1)' action='delete'/> <table uri='catalog:falconexample:observation1#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> </cluster> </clusters> <table uri='catalog:falconexample:observation1#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> <ACL owner='ambari-qa' group='users' permission='0755'/> <schema location='hcat' provider='hcat'/> ______________________________________________ 2) patientInputFeed.xml <feed xmlns='uri:falcon:feed:0.1' name='patientInputFeed' description='This is Patient table'> <tags>table=patient</tags> <frequency>hours(1)</frequency> <timezone>UTC</timezone> <clusters> <cluster name='hiveCluster' type='source'> <validity start='2016-05-31T09:00Z' end='2016-06-06T09:00Z'/> <retention limit='days(1)' action='delete'/> <table uri='catalog:falconexample:patient1#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> </cluster> </clusters> <table uri='catalog:falconexample:patient1#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> <ACL owner='ambari-qa' group='users' permission='0755'/> <schema location='hcat' provider='hcat'/> </feed> ____________________________________________ 3) patientprocessedOutputFeed.xml <feed xmlns='uri:falcon:feed:0.1' name='patientprocessedOutputFeed' description='This is patientprocessed table'> <tags>table=observationInputFeed</tags> <frequency>hours(1)</frequency> <timezone>UTC</timezone> <clusters> <cluster name='hiveCluster' type='source'> <validity start='2016-05-31T09:00Z' end='2016-06-06T09:00Z'/> <retention limit='days(1)' action='delete'/> <table uri='catalog:falconexample:Patient_proce#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> </cluster> </clusters> <table uri='catalog:falconexample:Patient_proce#ds=${YEAR}-${MONTH}-${DAY}-${HOUR}'/> <ACL owner='ambari-qa' group='users' permission='0755'/> <schema location='hcat' provider='hcat'/> </feed> _____________________________________________ 4) And 4th one is also very similar to above feed xml except table name. ______________________________________________ Please help my guys, i dont understand why oozie throwing ClassNotFoundException message:java.lang.ClassNotFoundException even though jar is present at proper location. ... View more

I have ran two or more examples of hive data pipeline line using Apache Falcon by creating Hive table URI(input feed/output feed) Feed.Here are the problem statements, statements: 1) Inserting data from one hive table to another table. 2)loading data from HDFS to hive table. Above both data pipeline are running perfectly but now my requirement is somewhat like, Requirement:- I have 3 external hive tables(without partition),written 1 SELECT query on top of that and wants to load data into another hive table using Falcon.I know, I will have to use INSERT OVERWRITE INTO TABLE table2...... SELECT col1,col2 from table1 query but a question is Does Falcon will allow me to create TABLE URI FEED on without partition table? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Tables: 1) patient 2)Observation 3)DiagnosticReport ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- SELECT query: select p.id,p.gender, p.Age, p.birthdate, o.component[1].valuequantity.value, o.component[1].valuequantity.unit from (select *, floor(datediff(to_date(from_unixtime(unix_timestamp())), to_date(birthdate)) / 365.25) as Age FROM patient) p inner join DiagnosticReport d on p.id = substr(d.subject.reference,9) inner join Observation o on p.id = substr(o.subject.reference,9) where p.Age>17 and p.Age<86 and o.component[1].valuequantity.value <140; Thanks in advance, Please help me. ... View more

Hello guys, My atlas-ranger machine is up and all services are running smothly over the node. A question is, atlas just displaying lineage for hive,if we fire a query through beeline.I meant,it's providing lineage/metadata for hive tables only when we perform hive operation by connecting to hiveserver2 using beeline(i.e, only for jdbc connection). Why it's not capturing hive metadata,when we use hive cli? What I should do to resolve this problem? ... View more

Hello Everyone, My Data pipeline is running perfectly in Apache Falcon. Data Pipeline Problem Statement: I am loading data from HDFS location to Hive table(Partitioned table). Question: I didn't seen lineage for this whole process.I mean,Atlas doesn't showed me any type of icon in lineage diagram. Why Apache Atlas do not showing me lineage for falcon? Is it supported in Atlas(Version 0.5)? Thanks in Advance. ... View more