Apache Atlas: Indexing Relationship data into Elasticsearch

suri07 — Thu, 09 Feb 2023 19:49:55 GMT

In have Atlas setup with Cassandra and ES. In ES entity data gets indexed in janusgraph_vertex_index. I want a way to add relationship data as well, so that I can support queries having filters on both Vertex and Edge attributes. How can I enable this use case using Atlas?

Re: Apache Atlas: Indexing Relationship data into Elasticsearch

vats — Tue, 05 Aug 2025 20:41:25 GMT

Hello @suri07

To support search filtering across both vertex and edge attributes in Apache Atlas without breaking compatibility:

Denormalize the edge attribute into the vertex during ingestion.

Add needed relationship info (like producer process type, owner, etc.) as attributes on the entity itself.
Use /v2/entity/bulk to ingest this during lineage creation.

Bonus: Automating This

If you're using a metadata ingestion pipeline (e.g., from Airflow, Glue, custom ETL):

Fetch related entities' properties.
Inject them as attributes during entity creation.
Keep a mapping logic (e.g., process’s jobType → dataset’s producing_job_type).

question Apache Atlas: Indexing Relationship data into Elasticsearch in Support Questions

Apache Atlas: Indexing Relationship data into Elasticsearch

Re: Apache Atlas: Indexing Relationship data into Elasticsearch