In one of my past projects, we were using Amazon DynamoDB as our primary database. The system was designed to deliver high availability and low-latency access for transactional data. Our platform had multiple pages that listed items from DynamoDB, and for a while, things worked smoothly. We were fetching all the data at once and implementing client-side pagination to handle the display.

However, as the data grew in volume, we started experiencing performance bottlenecks. Our APIs began timing out due to the increasing payload sizes. Fetching large datasets in a single call was no longer feasible, and this started affecting user experience. Given our constraints in terms of time and resources, migrating to another primary database was not a viable solution.

The Workaround: Using OpenSearch as a Secondary Store

To overcome this limitation, we decided to integrate Amazon OpenSearch as a secondary database. We chose specific tables that needed fast search and retrieval, and began syncing them to OpenSearch. This hybrid approach significantly improved our response time and search capability, especially for those UI pages that displayed lists or required filtering.

The Sync Challenge

While this solution improved read performance, it introduced a new challenge — how to keep the data in sync between DynamoDB and OpenSearch in real-time. If the data in OpenSearch was stale or inconsistent, it could mislead users and defeat the purpose of fast access. That’s when we discovered the power of DynamoDB Streams.

What are DynamoDB Streams?

It’s a time-ordered log of item-level changes in a table. With streams enabled, any modification triggers an event that Lambda can process to:

• Push new or updated records to OpenSearch
• Remove deleted documents from the index
• Handle versioning or deduplication if needed

This made it possible for us to create a seamless sync mechanism: whenever a data attribute changed in DynamoDB, a Lambda function was triggered to update the corresponding document in OpenSearch.

Stream Configuration Options

You can control what data gets passed to Lambda:

• Keys Only: Just the primary key attributes
• New Image: The full item after modification
• Old Image: The full item before modification
• New and Old Images: Both versions

For syncing to OpenSearch, New Image or New and Old Images is usually the way to go. We used both versions to compare and sync only the changed data fields

Architecture Overview

Here’s a high-level view of how the pieces connect:

[DynamoDB Table] ─► [DynamoDB Stream] ─► [AWS Lambda] ─► [OpenSearch Index]

Components:

• DynamoDB Table: Your primary transactional store
• Stream: Captures every change
• Lambda: Processes the changes
• OpenSearch: Indexes data for fast search

Why It Matters

Instead of migrating or over-engineering a search solution, this setup gives you:

• Real-time sync between DynamoDB and OpenSearch
• Search-optimized reads without impacting write performance
• Low latency for UIs displaying large or filtered datasets

Why This Works Well

DynamoDB is an OLTP (Online Transaction Processing) database, optimized for transactional workloads. It is not designed for analytical queries or full-text search, which is where OpenSearch excels.

By using DynamoDB Streams:

• We avoided data staleness in the secondary store.
• Maintained sequential consistency, as the Lambda was invoked in the order of operations.
• Configured retries on failure to ensure that syncs were eventually successful, thereby making the system fault-tolerant.

This combination of services allowed us to scale effectively without compromising on performance or data integrity.

Final Thoughts

This pattern can be used when the system requires:

• Search/filtering on dashboards or tables
• Real-time indexing of high write-volume data
• A sync solution that “just works”

DynamoDB + Streams + OpenSearch: a modern data pipeline that scales with your app.

DynamoDB Streams supports a wide range of use cases where real-time data processing can be valuable. In our case, integrating DynamoDB Streams with AWS Lambda to sync data to OpenSearch turned out to be a game-changer. It helped us maintain a fast and responsive UI while continuing to use DynamoDB as our primary data source. For any system facing similar scalability issues, this is a highly effective pattern that’s worth considering. It’s plug-and-play with AWS services and lets you focus on features — not infrastructure.