Azure AI Search, Microsoft’s top serverless option for the retrieval part of RAG, has unique sizing, scaling, and pricing logic. While it conceals many complexities of server based solutions, it demands specific knowledge of its configurations.

There are two special tiers that behave a bit differently. The free tier should only be used for demos and educational purposes. Its resources are shared with other services, making the performance somewhat unpredictable, and the limits are very tight. Even in a proof of concept, you want to gain insights into the query throughput and other performance metrics on your specific data, which you can’t reliably do in the Free tier. Hence, even when you do a proof of concept, you should at least use the Basic tier.

The Standard S3HD tier supports up to 3,000 indices, compared to a maximum of 200 in other tiers. It is designed for multi-tenant applications, a distinct scenario from typical use cases. For the rest of this text, we will assume that you need at most 200 indices. Multi-tenant applications and the Standard S3HD tier will not be covered

Sizing Azure AI Search for a proof of concept is usually easier than in a production environment. Two things help us in a typical proof of concept situation:

1. The number of users and queries is usually low and not growing quickly.
2. You don’t need to adhere to SLAs.

Hence, we usually don’t need multiple replicas. Instead, we focus on the total storage needed. This number is impossible to estimate with any precision. Microsoft’s documentation says that the only way to know the size of an index is to build it. But as there are vast differences between the total storage available from one service tier to the next, you will probably have an idea which one you need. If in doubt, choose the higher tier. Migrating from one tier to another means rebuilding the search service completely. You don’t want to do that in a typical proof of concept situation. Another viable option is to choose a tier that may or may not suffice to accommodate all of your data, and live with the risk of not being able to ingest all documents. Finally, the number of indices you need may also influence your choice of a service tier. Once you’ve chosen your tier and figured out your true storage needs, you know the right number of partitions and therefore also the monthly cost you have to expect. 

There are two important things that you can only find out via testing:

1. The size of an index.
2. The query performance on a different service tier, with a different number of replicas or changes in indices. Increasing the number of replicas helps query performance, but the speedup is not proportional to the number of replicas and may differ for different queries.

For a production setting, this means that all sizing and scaling decisions need a lot of testing. You can’t prepare by laying out a scaling strategy that will tell you how to choose your parameters in a given situation. Instead, you must design your architecture to facilitate frequent, realistic, and efficient testing. Testing takes time: Not only a migration to a different tier takes time and effort with its complete rebuild. Even changing the number of replicas or partitions in the configuration may take several hours to be completed, due to the potentially large data volumes to be moved.

To be ready for testing, you need to prepare your architecture and your devops-processes:

• Architecture: Plan for a traffic split / traffic copy that routes somes of your traffic to a separate Azure AI search service when you perform a test.
• Devops: Fully automate the build process of your testing service, including, but not limited to, the use of infrastructure as code.