Accelerate AI with JSON Mode in Azure OpenAI & Semantic Kernel

AI Integration solutions on Azure often require consistent JSON outputs, especially when dealing with structured data. However, obtaining reliable responses in JSON can be tricky. In this post, I will explain why I shifted from the standard Azure.AI.OpenAI NuGet package to Semantic Kernel. I will also explore how Semantic Kernel helps generate guaranteed JSON outputs—often referred to as JSON mode OpenAI JSON mode—making your AI solutions more robust.

Why I Switched from the Azure.AI.OpenAI NuGet Package

The Azure.AI.OpenAI NuGet package is straightforward. It is an official Microsoft offering, so integration with Azure OpenAI JSON endpoints is direct. In typical scenarios, you can send prompts and receive responses without any issues.

However, if you need to OpenAI return JSON reliably, you may run into challenges ensuring that your Large Language Model (LLM) consistently adheres to your desired JSON structure. In my last project, I required OpenAI JSON output to align with specific schemas. This kind of typed JSON response is important to be consistent or the application would otherwise crash.

When I discovered that JSON mode Azure OpenAI was not fully supported in the NuGet package, I began looking for an alternative. Semantic Kernel stood out because of its superior handling of structured prompts, including the ability to return JSON in line with an OpenAI JSON schema.

The Challenge of Azure OpenAI JSON Responses

LLMs are excellent at generating human-like text, but they are not always perfect when returning structured data. Even if your system prompt asks explicitly for JSON, the LLM might not follow the schema you specify.

For example, you might ask for OpenAI structure outputs API JSON and receive incomplete or malformed JSON. This problem is so common that I ended up removing unwanted markdown sections ( “`json) or re-prompting the model whenever JSON parsing failed.

Although these workarounds sometimes function, they lead to “smelly” code. It is hardly an ideal solution.

Exploring Azure OpenAI JSON Support

Interestingly, the Azure OpenAI REST API does offer extra parameters to guarantee JSON outputs. Through the Chat Completions endpoint, you can specify a response format such as:

• Text (default): Plain text responses.
• JsonObject: A guaranteed JSON response.
• JsonSchema: A typed JSON response (an OpenAI api json schema) with specific fields.

However, this feature is not yet fully exposed in the Azure.AI.OpenAI NuGet package. Hence, you may have to rely on REST calls to use OpenAI structure API JSON schemas.

Note: Some models do not support JSON schema formats. Always check model compatibility before using these features.

Introducing Semantic Kernel

Semantic Kernel is a lightweight, open-source development kit for building AI agents in C#. It supports multiple AI providers, including Azure OpenAI. With Semantic Kernel, you can integrate advanced AI features into your applications more smoothly.

The library emphasises modularity, offering support for telemetry, hooks, and filters to tailor AI workflows. Many find Semantic Kernel to be more future-proof, especially when it comes to handling Azure OpenAI JSON mode and generating OpenAI structure outputs api JSON schemas for typed responses.

Official documentation: Semantic Kernel Overview

Easier Integration for Typed JSON Results

Semantic Kernel is great at producing OpenAI JSON output that matches a desired schema. You no longer need to define the entire JSON schema manually. Instead, you can pass a C# class, and Semantic Kernel takes care of generating the JSON schema OpenAI on the fly.

Here is a quick example (source: Semantic Kernel Blog Post):

var kernelBuilder = Kernel.CreateBuilder();

var config = new AzureOpenAIConfig()
{
   Endpoint = "https://xxxx.openai.azure.com/",
   APIKey = "yourapikeyhere",
   Deployment = "gpt-4o-mini",
   Auth = AzureOpenAIConfig.AuthTypes.APIKey
};

Kernel kernel = kernelBuilder.Build();


var executionSettings = new OpenAIPromptExecutionSettings
{
    ResponseFormat = typeof(Book), // Specify response format class
};

var result = await kernel.InvokePromptAsync("What is the most famous book?", new(executionSettings));


public class Book
{
    public string Title { get; set; }
    public int ReleaseDate { get; set; }
    public string Author { get; set; }
}

In this example, the OpenAI structure outputs API JSON schemas are automatically handled, ensuring consistent JSON responses that match your C# type. This removes the burden of having to parse or validate JSON manually.

Conclusion

If you are looking to produce guaranteed JSON responses—complete with schema validation—Semantic Kernel is likely your best option today. Despite initial disappointment that the NuGet package does not expose these features yet, I am still pleased with how Semantic Kernel bridges the gap.

Happy prompting, and best of luck with your Azure OpenAI JSON integrations!