Azure AI Service in Containers: Ultimate Flexibility Anywhere

Azure AI Containers

In my earlier blog posts about Azure AI language services, I explored how non-cloud solutions can be used in specific scenarios, particularly when offline capabilities are essential. Microsoft addresses this growing demand by enabling Azure AI services to be hosted in containers. The benefits? Greater control over data for privacy-sensitive industries, improved processing speeds by bringing the workload closer to your environment, and freedom from Azure’s API limits, such as maximum calls per minute.

In this post, we’ll dive into why you might consider deploying Azure AI services in containers and how to set them up on Azure or on-premises. Spoiler alert: The containerised version of Azure AI services runs on Linux!

Why Use Azure AI Cognitive Services in Containers?

Azure AI containers offer several advantages:

1. High Throughput & Low Latency

• Containers remove transaction limits, making them perfect for bulk processing, like digitising documents with OCR or analysing historical data.
• Hosting containers close to your infrastructure reduces latency and improves speed.

2. Control Over Data

• Azure ensures customer data is encrypted and isolated by region, but API calls may be temporarily stored for up to 48 hours.
• Some industries, such as healthcare or finance, have strict regulations requiring all data to remain on-premises.

3. Offline Mode or Reduced Bandwidth Usage

• In scenarios with limited or unstable internet access—like ships at sea or remote regions—containers offer critical offline functionality.
• Containers are also suited for high-critical applications such as emergency services requiring real-time speech-to-text transcription.

4. Customised Control

• Gain full control over updates and versioning. Containers offer version-controlled, isolated environments, giving flexibility over when and how updates are applied.
• Containers can be created and removed dynamically, even in the free version.

Are you concerned about AI security and responsible AI practices? I highly recommend exploring this insightful blog post by Jakub from Azure AI Content Safety Studio.

Requirements for Setting Up Azure AI Language Containers

To deploy Azure AI services in containers, you’ll need:

1. An Azure AI services subscription (free or standard tier).
2. A container platform, such as Docker, Kubernetes, Azure Kubernetes Service (AKS), or Azure Container Instances (ACI).
3. Linux hosting capabilities for the container runtime.
4. Network connectivity to Azure’s billing endpoint unless offline mode is being used.

Below is an overview of a sample setup of Azure AI services with containers in Azure:

Azure AI Container Image Repository

Azure AI containers are hosted in Microsoft’s container registry at mcr.microsoft.com/azure-cognitive-services.

For example, to pull the latest Azure AI Language container for language detection:

docker pull mcr.microsoft.com/azure-cognitive-services/textanalytics/language:latest

Note: The image size for the latest container version is around 10.88 GB.

Billing and Connectivity

Azure AI containers report billing data every 10–15 minutes. Unless in offline mode, this is the only required connection to Azure, ensuring no additional data leaves the container.

Microsoft requires this connection to verify usage and ensure billing accuracy. If you’re operating in offline mode, usage is tracked differently through an annual commitment model (see below).

On-Premises Local Deployment

For a simple on-premises setup on a local machine, I used Docker:

1. Pull the container image:

docker pull mcr.microsoft.com/azure-cognitive-services/textanalytics/language:latest

2. Run the container with your billing endpoint and API key:

docker run -it -p 5000:5000 --memory 6g --cpus 2 mcr.microsoft.com/azure-cognitive-services/textanalytics/language Eula=accept Billing=https://<yourendpoint>.cognitiveservices.azure.com ApiKey=<yourAPIKey>

Now, you’ve got Azure AI Language running locally!

Azure Setup deployment

Setting up containers on Azure is just as straightforward but with additional options for logging and monitoring. For instance:

• Integrate Log Analytics to track container usage and performance.
• Use Application Insights by adding the InstrumentationKey parameter to your setup for detailed application monitoring.

Steps on how deploy Azure Ai Services in containers on Azure:

1. Create a new container Instance, most default options can be used
2. Provide the image repository on mcr.microsoft.com/azure-cognitive-services/textanalytics/language:latest

3. Provide open port connections to 5000 TCP:

4. Add the required parameters Eula, Billing, ApiKey and optionally your InstrumentationKey to application insights:

Using Azure AI Containers in Offline Mode

Offline mode is a specialised configuration for environments with no internet connectivity or strict data regulations. Examples include:

• Devices or servers without permanent internet access.
• Remote locations with occasional connectivity.
• Organisations who are prohibited from sending data to the cloud.

Azure offers annual commitment tiers for offline containers instead of pay-as-you-go pricing. Offline mode requires Microsoft approval and is available only under specific conditions. A clear business case must be submitted via Microsoft Partner channels.

Once approved, you can download and activate offline containers using a license key. Be aware that not all features are available in offline mode.

More details: Azure Cognitive Services Disconnected Containers

Additional Features in Azure AI Containers

Azure AI containers provide some endpoints not available in cloud-hosted services. These endpoints, exposed via Swagger, are useful for monitoring and troubleshooting:

• GET /ready: Checks if the container is ready to accept queries.
• GET /status: Verifies the validity of the API key without triggering a query.

These endpoints are excellent for setting up liveness, readiness, and startup probes in container orchestration platforms like Kubernetes.

For offline containers, additional endpoints allow access to usage logs:

• GET /records/usage-logs
• GET /records-usage-logs/{month}/{year}

Challenges and Considerations

Deploying Azure AI services in containers shifts the operational model from PaaS to IaaS, requiring you to manage:

• Infrastructure: Hosting, scaling, updates, and load balancing.
• Costs: While Azure AI service pricing remains, you’ll incur additional hosting and maintenance expenses.

Common issues you may encounter:

Invalid API Key: Ensure the key is correct and matches your service region.

Container is not in a valid state. Subscription validation failed with status ‘Invalid’. Api Key is invalid. You must provide a valid API key to run the container. Find your API key and service region under Keys and Endpoint at your Azure Cognitive Services resource blade in the Azure Portal. For more information please visit https://aka.ms/cscontainers-faq.

Incorrect Billing Endpoint: Verify your container has the required network access to Azure’s billing endpoint.

Submitting metering to ‘https://xxx.cognitiveservices.azure.com/’. Failed to reach billing endpoint: ‘The SSL connection could not be established, see inner exception.’. Trying 9 more times.
fail: Microsoft.CloudAI.Containers.Http.CloudClient[0]
      Failed to reach billing endpoint: ‘The SSL connection could not be established, see inner exception.’. Trying 9 more times.

For diagnostics purposes, Microsoft provides a container-specific diagnostics image:

docker run --rm mcr.microsoft.com/azure-cognitive-services/diagnostic Eula=accept Billing={ENDPOINT_URI} ApiKey={API_KEY}

This is what to expect when connected to log analytics:

Conclusion

Azure AI containers open up possibilities that go beyond the standard Azure cloud setup. They’re perfect for businesses handling privacy-sensitive data, high-throughput workloads, or needing offline capabilities. With containers, you can run Azure’s AI services wherever it makes sense—on-premises, in a private cloud, or even in remote locations with unreliable internet.

The real value lies in flexibility. You can mix and match cloud and on-prem workloads, customise deployments, and control updates to suit your specific needs. It’s a chance to solve challenges like latency, compliance, or API limitations that the standard cloud setup might not handle well.

Of course, this shift comes with added responsibility. Hosting your own infrastructure means managing scaling, monitoring, and updates, but for the right use cases, it’s worth the effort. Containers let you shape Azure AI to fit your operations, not the other way around. If you’re ready to take AI deployment into your own hands, this is the way to go.