Skip to main content
suri07
suri07
Bronze ⭐️⭐️⭐️
February 9, 2023
Question

Apache Atlas: Indexing Relationship data into Elasticsearch

In have Atlas setup with Cassandra and ES. In ES entity data gets indexed in janusgraph_vertex_index. I want a way to add relationship data as well, so that I can support queries having filters on both Vertex and Edge attributes. How can I enable this use case using Atlas?

1 reply

vats
Gold ⭐️
vats
Gold ⭐️
August 5, 2025

Hello @suri07

To support search filtering across both vertex and edge attributes in Apache Atlas without breaking compatibility:

Denormalize the edge attribute into the vertex during ingestion.

  • Add needed relationship info (like producer process type, owner, etc.) as attributes on the entity itself.

  • Use /v2/entity/bulk to ingest this during lineage creation.

Bonus: Automating This

If you're using a metadata ingestion pipeline (e.g., from Airflow, Glue, custom ETL):

  • Fetch related entities' properties.

  • Inject them as attributes during entity creation.

  • Keep a mapping logic (e.g., process’s jobType → dataset’s producing_job_type).

Translate
Terms & ConditionsAccessibility statement