Hi All, I want to create the custom hashmap accumulator in spark for one of my use case. I have already referred and implemented Accumulator as per code is given on below links but not found an end-to-end example for the same. I am trying to do it in java. https://www.brosinski.com/post/extending-spark-accumulators/ Problem : I want to know, How Can I increment value for each key in the same accumulator within foreach() transformation? ... View more

Hi all, We are performing foreach operation on rdd , within it parsing each line of rdd to validate the schema of input. Along with this functionality i want to keep on update the custom java object to maintain the metadata of validation. Remember that ,we are expecting to make visible same custom object to other executors. Thanks in advance. ... View more

Hi All, We have a JSON file as input to the spark program(which describe schema definition, constraints which we want to check on each column though spark) and we want to perform some data quality checks(Not NULL, UNIQUE) and schema validations too. JSON File: { "id":"1", "name":"employee", "source":"local", "file_type":"text", "sub_file_type":"csv", "delimeter":",", "path":"/user/all/dqdata/data/emp.txt", "columns":[ {"column_name":"empid","datatype":"integer","constraints":["not null","unique"],"values_permitted":["1","2"]}, {"column_name":"empname","datatype":"string","constraints":["not null","unique"],"values_permitted":["1","2"]}, {"column_name":"salary","datatype":"double","constraints":["not null","unique"],"values_permitted":["1","2"]}, {"column_name":"doj","datatype":"date","constraints":["not null","unique"],"values_permitted":["1","2"]}, {"column_name":"location","string":"number","constraints":["not null","unique"],"values_permitted":["1","2"]} ] } Sample CSV input : empId,empname,salar,dob,location 1,a,10000,11-03-2019,pune 2,b,10020,14-03-2019,pune 3,a,10010,15-03-2019,pune a,1,10010,15-03-2019,pune Keep in mind that, 1) intentionally I have put the invalid data for empId and name field(check last record). 2) The number of column in json file are not fixed? Question: How can I ensure that an input data file contains all the records as per the given datatype(in JSON) file or not? We have tried below things: 1) If we try to load the data from the CSV file using a data frame by applying external schema, then the spark program immediately throws some cast exception(NumberFormatException, etc) and it abnormally terminates the program. But I want to continue the execution flow and log the specific error as "Datatype mismatch error for column empID". Above scenario works only when we call some RDD action on data frame which I felt a weried way to validate schema. Please guide me, How we can achieve it in spark? Thanks in advance. ... View more

HI, Our team working on Hadoop framework/technologies with Hive, Ambari, Ranger and planning to create a dashboard, which provides following information : Execution time of a Hive query. Size of data generated by the created table. Frequency of using each hive schema, table and each column of respective table. User/ Application name or ID firing any query. Resources usage of each application /User. For now, we are trying to use HiveMetastore and Ranger audit logs to access the above mentioned information. Is there any other better way to fetch the information above ? Kindly let me know, if I need to provide any more information. ... View more

Hi everyone, I don't have end-to-end example or script flow for my use case but there is an idea in my mind regarding "Automatic tag Detection" likewise "waterlinedata" tool does for us. I am just looking for any pre-available hortonworks tool/library which will analyze my data and will come out with results(i.e. by from learning data the tool/library will suggest me the Tag). for example, if have employee dataset and there are two columns in it SSN and "date_of_birth". so library or tool will learn this employee dataset and will suggest me that, these column should be tagged unser PII(Personal Identificable Infomration). so is it possible in hortonworks or in any other tools/library? I think we can achieve same thing using python scikit-learn library but can we do it using hortonworks in-built algorithm? Thanks in advance. ... View more

Hi All, I want to create hbase_table entity using Apache Atlas REST API.For this I am using below JSON structure and using below command to POST that JSON to Atlas server.Please keep in mind that I am using HDP 2.5 Horton works sandbox. Could you please help me to solve it? Command: curl -v -H 'Accept: application/json' -H 'Content-Type: application/json; charset=UTF-8' -u username:password -d @hbase_table.json http://<host>:<port>/api/atlas/entities but getting a error as : Error: {"error":"Unknown datatype: hbase_column","stackTrace":"org.apache.atlas.typesystem.exception.TypeNotFoundException: Unknown datatype: hbase_column\n\tat org.apache.atlas.typesystem.types.TypeSystem.getDataType.............} Please find JSON structure and let me know Is there any mistake in JSON: [ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995057000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" }, "typeName":"hbase_column", "values":{ "qualifiedName":"default.webtable.anchor.cssnsi@cluster1", "type":"string", "owner":"crawler", "name":"cssnsi" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995105000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" }, "typeName":"hbase_column", "values":{ "qualifiedName":"default.webtable.anchor.mylookca@cluster1", "type":"string", "owner":"crawler", "name":"mylookca" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995127000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" }, "typeName":"hbase_column", "values":{ "qualifiedName":"default.webtable.contents.html@cluster1", "type":"string", "owner":"crawler", "name":"html" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995148000", "version":0, "typeName":"hbase_column_family", "state":"ACTIVE" }, "typeName":"hbase_column_family", "values":{ "name":"anchor", "inMemory":true, "description":"The anchor column family that stores all links", "versions":3, "compression":"zip", "blockSize":128, "qualifiedName":"default.webtable.anchor@cluster1", "columns":[ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995057000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995105000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" } ], "owner":"crawler" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995232000", "version":0, "typeName":"hbase_column_family", "state":"ACTIVE" }, "typeName":"hbase_column_family", "values":{ "name":"contents", "inMemory":false, "description":"The contents column family that stores the crawled content", "versions":1, "compression":"lzo", "blockSize":1024, "qualifiedName":"default.webtable.contents@cluster1", "columns":[ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995127000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" } ], "owner":"crawler" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995263000", "version":0, "typeName":"hbase_table", "state":"ACTIVE" }, "typeName":"hbase_table", "values":{ "columnFamilies":[ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995148000", "version":0, "typeName":"hbase_column_family", "state":"ACTIVE" }, "typeName":"hbase_column_family", "values":{ "name":"anchor", "inMemory":true, "description":"The anchor column family that stores all links", "versions":3, "compression":"zip", "blockSize":128, "qualifiedName":"default.webtable.anchor@cluster1", "columns":[ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995057000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995105000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" } ], "owner":"crawler" }, "traitNames":[ 　 ], "traits":{ 　 } }, { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995232000", "version":0, "typeName":"hbase_column_family", "state":"ACTIVE" }, "typeName":"hbase_column_family", "values":{ "name":"contents", "inMemory":false, "description":"The contents column family that stores the crawled content", "versions":1, "compression":"lzo", "blockSize":1024, "qualifiedName":"default.webtable.contents@cluster1", "columns":[ { "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"1466771970995127000", "version":0, "typeName":"hbase_column", "state":"ACTIVE" } ], "owner":"crawler" }, "traitNames":[ 　 ], "traits":{ 　 } } ], "name":"webtable", "description":"Table that stores crawled information", "qualifiedName":"default.webtable@cluster1", "isEnabled":true, "namespace":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"71cbc76ed1bf4e92865a13624cac7a2a", "version":0, "typeName":"hbase_namespace", "state":"ACTIVE" } }, "traitNames":[ 　 ], "traits":{ 　 } } ] ... View more

Hi All, We have two table i.e. table4 and table5 having columns Id, Name, Age. we have inserted entity metadata, lineage metadata of both table in Atlas and able to see the schema and lineage graph in Atlas. After that I have deleted the entity metadata of table5 and reinserted entity metadata of table5. Next I have inserted the same lineage metadata (earlier lineage JSON metadata) of both table in atlas; however not able to see the lineage graph of two table. getting below response message from atlas server. {"requestId":"qtp662559856-30620 - 26dbb640-9629-4c29-b209-32331e52962e","entities":{}} Please find here the below lineage JSON metadata and let me know the mistake I have done. lineagejson.txt so after deleting table hive table entity and reinserting same metadata(i.e. hive table entity JSON data) we are unable to see the lineage in atlas ,so what could be the reason behind this? what actually mistake I am making in lineage json second time because of which I am not getting lineage? Do I need to change the value of process id or process name in JSON ? ... View more