Member since
11-30-2016
85
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3222 | 03-15-2017 07:33 AM |
08-09-2017
01:05 PM
Hi, Can we merge 2 different JSON coming from 2 different flow files using any processor of NiFi. For example : Input 1 : {
"Table": "myTable",
"Database": "Database",
"Column": "Column1"
} Input 2 : {
"Table": "myTable",
"Database": "Database",
"Column": "Column2"
} Now the challenge is how we can merge above 2 JSON based on matching condition of Table and Database and merge content of both JSON in a one JSON dictionary. Output : {
"Table": "myTable",
"Database": "Database",
"Column": [
"Column2",
"Column1"
]
} Thank you in advance, Subash
... View more
Labels:
- Labels:
-
Apache NiFi
07-11-2017
10:01 AM
I am using HDP 2.6.1
... View more
07-11-2017
09:58 AM
@Sarath Subramanian, Yes i can see the "__Process.outputs" for the CTAS column. Attaching output below : {"requestId":"pool-2-thread-4 - 4718810e-6302-48cb-9b75-d6d665242731","query":"g.V(\"__guid\",\"90b6074b-532e-4562-8b37-42a8636926ac\").inE.toList()","queryType":"gremlin","results":[{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkg7r-2pk0jc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856568","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pkw1e-2pk0pc-opp1-2pk0gw","label":"__hive_column.table","outVertex":"163856784","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pl8oi-2pkdcg-osut-2pk0gw","label":"__hive_storagedesc.table","outVertex":"163873168","__timestamp":"1499685457722"},{"__modifiedBy":"admin","__state":"ACTIVE","__createdBy":"admin","inVertex":"163856480","__modificationTimestamp":"1499686956258","id":"2pmiia-2pkpzk-1a80l-2pk0gw","label":"__Process.outputs","outVertex":"163889552","__timestamp":"1499685462156"}],"count":4}
Let me know where i am going wrong.
... View more
07-10-2017
12:51 PM
Hey @Geoffrey Shelton Okot, Yes Hive Hook enabled, I can see the lineage of hive_table. Only the hive_column_lineage is not showing up.
... View more
07-10-2017
11:28 AM
I am trying to see the column level lineage in apache atlas 0.8. To get the lineage i have created a table by using CTAS statement over a subset of columns of a different table. As per this link, Ideally i should get the lineage.but in my Atlas, Lineage is not populating. Thank you, Subash
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache HBase
-
Apache Hive
06-29-2017
09:04 AM
I have tried maybe 100 of times with some tweaks but every time i am facing the same error. u'Unable to deserialize json \\"{\\\\\\"structTypes\\\\\\": [], \\\\\\"traitTypes\\\\\\": [], \\\\\\"classTypes\\\\\\": [{\\\\\\"hierarchicalMetaTypeName\\\\\\": \\\\\\"org.apache.atlas.typesystem.types.ClassType\\\\\\", \\\\\\"superTypes\\\\\\": [], \\\\\\"attributeDefinitions\\\\\\": [{\\\\\\"name\\\\\\": \\\\\\"multiplicityRequired\\\\\\", \\\\\\"reverseAttributeName\\\\\\": \\\\\\"\\\\\\", \\\\\\"multiplicity\\\\\\": \\\\\\"required\\\\\\", \\\\\\"dataTypeName\\\\\\": \\\\\\"string\\\\\\", \\\\\\"isUnique\\\\\\": \\\\\\"true\\\\\\", \\\\\\"isIndexable\\\\\\": \\\\\\"true\\\\\\", \\\\\\"isComposite\\\\\\": \\\\\\"true\\\\\\"}], \\\\\\"typeDescription\\\\\\": \\\\\\"\\\\\\", \\\\\\"typeName\\\\\\": \\\\\\"test\\\\\\"}], \\\\\\"enumTypes\\\\\\": []}\\"'} I am following this post, here i guess post request is successful. Let me know where i am doing wrong. https://community.hortonworks.com/articles/124/atlas-api-tips-create-trait-type-example.html curl Command : curl -u atlas_user:password -H 'Content-Type: application/json; charset=UTF-8' -X POST -d '{"enumTypes": [],"structTypes": [],"traitTypes": [{"superTypes":[],"hierarchicalMetaTypeName":"org.apache.atlas.typesystem.types.TraitType","typeName": "EXPIRES_ON", "attributeDefinitions":[{ "name": "expiry_date", "dataTypeName": "date", "multiplicity": "required","isComposite": false,"isUnique": false,"isIndexable": true,"reverseAttributeName": null}]}],"classTypes": []}' http://hostname:21000/api/atlas/types
I am attaching my python code as well for reference. import requests
import json
hostname = 'hostname:21000'
headers = {'Content-Type': 'application/json; charset=UTF-8'}
atlas_user_name = 'username'
atlas_password = 'password'
create_type ={
"enumTypes": [],
"structTypes": [],
"traitTypes": [
{
"superTypes":[],
"hierarchicalMetaTypeName":
"org.apache.atlas.typesystem.types.TraitType",
"typeName": "EXPIRES_ON",
"attributeDefinitions": [
{
"name": "expiry_date",
"dataTypeName": "date",
"multiplicity": "required",
"isComposite": "false",
"isUnique": "false",
"isIndexable": "true",
"reverseAttributeName": ""
}
]
}
],
"classTypes": []
}
type_create_post = requests.post(hostname+ '/api/atlas/types' , headers=headers,data = json.dumps(create_type),auth =(atlas_user_name,atlas_password))
print type_create_post.json()
entity_create_post = requests.post(hostname+ '/api/atlas/types' , headers=headers,data = json.dumps(create_entity),auth =(atlas_user_name,atlas_password))
print entity_create_post.json()
... View more
Labels:
- Labels:
-
Apache Atlas
06-28-2017
01:28 PM
Hey Eyad, Ty. I will try to implement the same.
... View more
06-28-2017
11:24 AM
Hey @Smart Data, Ty can you please let me know how to create type and entity for Nifi Data flow
... View more
06-28-2017
07:29 AM
To bring HDF lineage into Atlas. I would like to create a new entity on HDF, Can we achieve it in Atlas 0.7. If we can how to create a new entity on an existing HDF to bring the lineage. thank you, Subash
... View more
Labels:
- Labels:
-
Apache Atlas
06-23-2017
07:26 AM
We have configured password-less ssh onto our edge node. From one of my NiFi instance, I am able to ssh into the edge node by using this simple command ssh username@edge-node-hostname
When i try this similar command using execute process. I am running into error. error : "host key verification failed" error : "Pseudo-Terminal will not be allocated because stdin is not a terminal" error : No Such file or directory. these are the arguments, which i have passed in the executeProcess, Let me know where I am doing wrong. Any documentation related to this with example will also help. Command : sh command arguments : ssh username@edge-node-hostname
... View more
Labels:
- Labels:
-
Apache NiFi