Let me try with checkpoint. Thanks for your reply. @graghu  ... View more

Yes , I am also facing same issue . Any update on it or workaround? ... View more

@EricL    Thanks for your reply. I am getting below error in hiveserver log.   10:20:02 START Executing current statement for: 'SCIP_Test_Env' [Hive] 10:20:03 FAILED [SELECT - 0 rows, 0.574 secs] [Code: 0, SQL State: 08S01] org.apache.hive.org.apache.thrift.transport.TTransportException: HTTP Response code: 500     ... View more

Accidently , I marked this answer as resolved .  @rajkumar_singh  I am getting below output after executing "hdfs groups <username>" command.   <username>@<kerberose principle > : domain users dev_sudo   As i am not much aware of cluster configuration  So , Could you please help me to understand the output of this command. ... View more

I am also not able to understand, whether compaction is enabled on my table or not?   Does anyone know how to filter the output of show compactions query. ... View more

Hi all,   Above solution is failing at one scenario, Scenario: if multiple flow files processed at a time and  landed in the nifi queue which is used after update query ( i.e. puthiveql which increment processed_file_cnt by one for every flow file ) processor ,then there might be chances of triggering the next flow multiple times and that is wrong.   Because we do select processed_file_cnt first and then doing the comparison for processed_file_cnt with input_file_cnt.  ... View more

Hi ,   My assumptions was wrong , putsql processor does execute update query per flowfile  ... View more

Hi , Thanks for your replay.   Yes ,we can use wait and notify processor. But how would i ensure that previous has been completed or failed. What is that criteria or condition?   Let me explain you current nifi flow,we have 3 layers and in/out for each layer always requires 4 flowfiles (number of flowfiles depend on input parameters). How can i integrate wait/notify pattern by keeping this in mind?   Do you have any nifi example where it's already implemented? Could you please share nifi template xml with me.       ... View more

Hi @DennisJaheruddi  Thank you for writing to me. There are some situation or places in NiFi flow ,where i require to execute SQL INSERT for each flowfilea and at some places need to execute SQL INSERT only once irrespective on number of flowfiles. Could you please point me in right direction for the same? ... View more

Hi @DennisJaheruddi  Thanks for you reply. Don't you think, if we use wait and Notify processor then pipeline will get proceeded ahead for next execution after the expiration of Expiration Duration attributes value of wait processor. The situation may occur like the execution is not completed for previous layer but NiFi flow get triggered due expiration of above attribute. It seems wrong behavior to me. ... View more

Thank you Rguruvannagari. This solution really worked for me. ... View more

Hi Ayub, Here is the link for above same question, https://community.hortonworks.com/questions/75818/issue-regarding-apache-atlas-rest-api-to-create-hi.html ... View more

Hi Ayub, We have two table i.e. table4 and table5 having columns Id, Name, Age. we have inserted entity metadata, lineage metadata of both table in Atlas and able to see the schema and lineage graph in Atlas. After that I have deleted the entity metadata of table5 and reinserted entity metadata of table5. Next I have inserted the same lineage metadata (earlier lineage JSON metadata) of both table in atlas; however not able to see the lineage graph of two table. getting below response message from atlas server. {"requestId":"qtp662559856-30620 - 26dbb640-9629-4c29-b209-32331e52962e","entities":{}} Please find here the below lineage JSON metadata and let me know the mistake I have done. lineagejson.txt so after deleting table hive table entity and reinserting same metadata(i.e. hive table entity JSON data) we are unable to see the lineage in atlas ,so what could be the reason behind this? what actually mistake I am making in lineage json second time because of which I am not getting lineage? Do I need to change the value of process id or process name in JSON ? ... View more

Hi Ayub, The above issue is solved. Actually there was mistake in JSON. Whenever we have multiple columns in a table we must have to provide different random GUID long number (not same for all the column) even though it's negative number only in such a case Apache atlas will able to different column names otherwise,will get same name for all columns in Apache Atlas UI. To make this work I have just provided different random ID for each columns as follows: Please find attached the correct JSON file : ... View more

Thank you Ayub, it's working fine for me, I have one query here, I have one table table5 has columns id, name and age, After inserting the table5 metadata into Atlas I am getting the repeated column I.e. name only. however not getting metadata for Id and age column. Please find here the below table5 JSON & let me know if any mistake is there in JSON. Please find attached Atlas image I am getting output like shown in image. atlas-snapshot.png [ { "traits":{ }, "traitNames":[ ], "values":{ "ownerType":2, "owner":"root", "qualifiedName":"default@Sandbox", "clusterName":"Sandbox", "name":"default", "description":"emr hive database", "location":"hdfs:\/\/sandbox.hortonworks.com:8020\/apps\/hive\/\/warehouse", "parameters":{ } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_db", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_db", "id":"-11893021824425525", "state":"ACTIVE", "version":0 } }, { "traits":{ }, "traitNames":[ ], "values":{ "owner":"root", "temporary":false, "lastAccessTime":"2017-01-03T11:02:53.000Z", "qualifiedName":"default.table5@Sandbox", "columns":[ { "traits":{ }, "traitNames":[ ], "values":{ "owner":"root", "qualifiedName":"default.table5.name@Sandbox", "name":"name", "type":"string", "table":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_table", "id":"-11893021824425524", "state":"ACTIVE", "version":0 } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_column", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_column", "id":"-11893021824425522", "state":"ACTIVE", "version":0 } }, { "traits":{ }, "traitNames":[ ], "values":{ "owner":"root", "qualifiedName":"default.table5.id@Sandbox", "name":"id", "type":"int", "table":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_table", "id":"-11893021824425524", "state":"ACTIVE", "version":0 } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_column", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_column", "id":"-11893021824425522", "state":"ACTIVE", "version":0 } }, { "traits":{ }, "traitNames":[ ], "values":{ "owner":"root", "qualifiedName":"default.table5.age@Sandbox", "name":"age", "type":"int", "table":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_table", "id":"-11893021824425524", "state":"ACTIVE", "version":0 } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_column", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_column", "id":"-11893021824425522", "state":"ACTIVE", "version":0 } } ], "tableType":"MANAGED_TABLE", "sd":{ "traits":{ }, "traitNames":[ ], "values":{ "qualifiedName":"default.table5@Sandbox_storage", "storedAsSubDirectories":false, "location":"hdfs:\/\/sandbox.hortonworks.com:8020\/apps\/hive\/warehouse\/table5", "compressed":false, "inputFormat":"org.apache.hadoop.mapred.TextInputFormat", "outputFormat":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", "parameters":{ }, "serdeInfo":{ "values":{ "serializationLib":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", "parameters":{ "serialization.format":"1" } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct", "typeName":"hive_serde" }, "table":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_table", "id":"-11893021824425524", "state":"ACTIVE", "version":0 }, "numBuckets":-1 }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_storagedesc", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_storagedesc", "id":"-11893021824425523", "state":"ACTIVE", "version":0 } }, "createTime":"2017-01-03T11:02:53.000Z", "name":"table5", "partitionKeys":[ ], "parameters":{ "totalSize":"0", "rawDataSize":"0", "numRows":"0", "COLUMN_STATS_ACCURATE":"{\"BASIC_STATS\":\"true\"}", "numFiles":"0", "transient_lastDdlTime":"1482917693" }, "db":{ "traits":{ }, "traitNames":[ ], "values":{ "ownerType":2, "owner":"root", "qualifiedName":"default@Sandbox", "clusterName":"Sandbox", "name":"default", "description":"emr hive database", "location":"hdfs:\/\/sandbox.hortonworks.com:8020\/apps\/hive\/\/warehouse", "parameters":{ } }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_db", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_db", "id":"-11893021824425525", "state":"ACTIVE", "version":0 } }, "retention":0 }, "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "typeName":"hive_table", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "typeName":"hive_table", "id":"-11893021824425524", "state":"ACTIVE", "version":0 } } ] ... View more

Thanks to see you gain Ayub, Could you please post what changes you have made in above json? did you change guid somewhere to link to dataset or something else? ... View more

Hi Ayub, I am able to set lineage between table1 and table2 successfully but now my requirement like, Consider,I already have created hive table using hive query, it's metadata is also present in altas and I want to link or create lineage between this already created table and the one which i will going to create using REST API,to do this what changes I need to make in json file which we are using to create hive_process? which one is that property, you have set in json file because of it we can link table1 and table2? ... View more

Hi Ayub, As we have created two dataset entities and set the lineage between them also. Consider I have already created hive table(i.e .patient_raw_info) and it's metadata is also present in atlas and now I want to create lineage between already exist dataset(i.e. patient_raw_info) and the one which I will going to create by using your REST API (i.e. patient_validated_dataset) so my question is How can I create hive_process between already exist dataset and the other one? what changes I need to make in json file which we are using to create hive_process (i.e. lineage) ? I can create third table(i.e. hive_entity) by using same json file that is fine but what about json data for lineage? How can I link them from, patient_raw_info--->patient_validated_dataset ... View more