Scaling Real‑Time Discovery: Inside Layers’ PlanetScale Migration

Layers is the only Search and Merchandising platform built exclusively for Shopify Plus. Our next-gen platform serves enterprise-level Shopify Plus users with large catalogs, at a fraction of the cost of competing tools. Layers currently powers hundreds of millions of dynamic requests daily (and growing quickly!).

The platform is designed, built, and backed by a team specializing in ultra-high-throughput applications for finance and aerospace companies. We used the same principles for latency, reliability, and security when building Layers. We believe latency and reliability should be foundational to product development and your first competitive advantages.

Downtime and high costs with RDS

Over the years, we have used everything from self-managed Postgres on AWS EC2 to DigitalOcean Managed DBs and, most recently, AWS RDS for the past two years.

Our read queries can be pretty intensive and are disk-bound. Paired with generally useless data from RDS Performance Insights that only highlighted the underlying disk issues we knew about, we were forced to constantly explore database offerings that promised to be faster or offered granular insights into queries. For the past year, we have resorted to increasing disk size on RDS to increase IOPS to mitigate these performance issues. This drove our database costs up significantly.

The final straw was nearly 45 minutes of downtime during a major version upgrade on RDS that required no changes for us, was promised as zero-downtime with Blue/Green deployments, and had been coordinated with AWS Support over 4–5 weeks. Luckily, this downtime occurred overnight during a very low traffic period; however, it still required us to pay out directly to impacted customers per our SLA.

We decided it was time to address all these problems, so we started looking for a new solution.

Why PlanetScale

My team and I have followed PlanetScale for over a year and have heard constant positive remarks from colleagues, partners, and customers. We have often discussed migrating to MySQL to take advantage of PlanetScale’s offering.

In a past life, I specialized in Pagespeed Performance and spent years neck deep in Chrome performance profiles, shaving a few ms off of JS functions. 3ms of blocking time from an event that fires hundreds or thousands of times a second quickly becomes a significant problem.

What caught my attention more than the positive remarks and impressive Metal benchmarks were the social posts and blog articles from PlanetScale breaking down Vitess performance, queries, etc.; they resonated with me and felt familiar, as I have followed and constantly read articles from many pagespeed experts.

While many other DB providers constantly released articles about using extensions, why they were better, etc., reading the PlanetScale articles felt different. I learned something compared to the posts that ultimately advertised the other platforms.

It felt like PlanetScale was focused on building an exceptional offering, and the writing focused on educating others based on the findings from that process.

Migrating from RDS to PlanetScale

Out of trauma from the failed major version upgrade a few weeks before deciding to move, we performed this migration overnight at our lowest traffic period, 12AM — 2AM EST.

As a platform that integrates with a third party for all underlying data, we have infrastructure and processes to recover from outages and “replay” write events to our database. This made the migration process relatively simple.

Initially, we planned to use AWS DMS, but after a few hours of setup and an emergency call with AWS Support, we discovered some incompatibilities with our setup. Minor downtime would be required to set up AWS DMS. As we have the processes to “replay” writes, we decided to use the dump and restore method.

We started by putting our queues and customer-facing dashboard into maintenance mode, effectively pausing non-essential writes to the database. A dump was created on EC2 server and then imported into PlanetScale. After hand-checking the data and running our test suite, we disabled maintenance mode, let the backlog of jobs process, refreshed indices, warmed critical tables, and then hot-swapped the environment variables on ECS/Fargate to begin reading from PlanetScale.

This process took roughly 90 minutes end-to-end without downtime and did not impact a single end-user.

Immediate performance improvements

Immediately upon hot-swapping the environment variables, our P95 query time dropped significantly across the board.

The graph after the hot swap is shown above. The best part? We improperly configured the application for read replicas and spent over a week querying the primary. You can see further in the graph that P95 drops even further after implementing read replicas.

The most drastic improvement was with our pgvector workload. We saw a 91% decrease in latency. It isn’t easy to illustrate this in an aggregate chart as we generate SQL for each query at the time of request using a proprietary query builder that inlines some parameters rather than using bindings, generates CTEs, and flattens queries.

The query above is from July 21st. It reads from nine tables, performs 17 concurrent vector searches, and aggregates the results before reranking them.

Below is a comparable query from August 17th.

Since moving to PlanetScale, over 43 days of cumulative latency have been saved daily.

Fast and helpful support responses

In the past two years, I have spent over fourteen hours on calls with AWS RDS Support. Half of that time was spent talking to someone who had next to no experience with Postgres or was quoting Aurora docs when we did not use Aurora.

Coming into the PlanetScale migration, I anticipated having to pay for a massive support package to receive guidance on architecture. The reality was nothing like that.

We were immediately given direct access to intelligent support staff and the engineers who built the Postgres offering. Many questions were asked to determine the best architecture or ensure support for extensions. For every single question, there was a response that was not only fast but thoughtful and informational. The team at PlanetScale was eager to dive in and understand our use cases to ensure we hit the ground running.

Since the migration, the support has remained incredible. We continue to receive rapid, thoughtful responses to basic questions, issues, and theoretical scenarios we are exploring. I cannot say enough about the support throughout this process. It has truly been a pleasure working with everyone.

Impact on end-users and product development

I wish I had decided to migrate to PlanetScale sooner. It has had a significant impact on the platform and product development. Not only are responses served to end-users in over half the time, but the latency savings have enabled us to further implement real-time AI/ML technology in areas that previously were pushed out due to the latency addition.

PlanetScale has single-handedly increased our competitive advantage and given us the tools to tackle items that have been left in the backlog for years. The Insights and Anomaly detection have allowed us to quickly identify and solve bottlenecks previously implemented to mitigate issues with former providers. We even identified bugs in the order of operations for some processes that have existed since the platform’s conception.

We all sleep better at night knowing that IOPS is no longer an issue and queries will not fail under traffic spikes simply because read/writes are being throttled. In a year, we will look back at what has been accomplished, and I am confident that much of it would not have been possible without PlanetScale.