I’m working with two Elasticsearch indexes -> accounts and contacts. These indexes are related in such a way that each account can be associated to contacts. However, not all contact data is denormalized into the accounts index.
Now I need to query accounts and filter the results based on some fields present only in the associated contacts index. This essentially requires a join-like behavior across two indexes.
Constraints:
Both indexes have multiple shards, so I can’t use lookup-joins provided by elasticsearch, as it currently supports single shard capability.
I also cannot use data enrichment either because the data in both indexes changes frequently, and keeping enrich policies updated in near real-time is not practical.
A full denormalization isn’t ideal either due to data duplication and sync complexity.