Optimizing Email Triage Using Azure AI Language Studio and Power Platform

Understanding the Problem: Email Triage

Insurance companies often deal with a shared mailbox where clients can send any type of inquiry or claim information. Even after converting emails into a case management system, these cases or tickets need to be assigned to the right team for resolution. Manually managing this shared inbox is time-consuming and monotonous. Companies with high email volumes struggle to keep up, especially in the mornings when a bulk load of new emails arrives, creating stress and backlogs for the processing team.

In this blog post, we’ll explore how to automate email triage for an insurance company’s claims mailbox using Azure AI services to assign emails to the correct team. Our goal is to automate the extraction of policy numbers using Named Entity Recognition (NER) and employ Text Classification to determine the appropriate department for each email. This process will enhance email classification within a case management system managed in Microsoft Dynamics Customer Engagement (CE), with the Power Platform integrated. By combining the Power Platform with Azure Language Studio (formerly Cognitive Services), we aim to streamline this challenge. This example is also applicable to industries beyond insurance.

Common examples of email processing automation include:

• Reading, classifying, and routing emails to the appropriate individual or team.
• Extracting data from the body of emails or their attachments.

By leveraging Azure AI Language Studio and Power Platform, we can significantly improve efficiency and accuracy in handling high volumes of emails, reducing manual workload and ensuring prompt responses to client inquiries.

Solution Architecture with Azure Language Studio

In this use case, emails sent to claims@company.com are received in Dynamics CE with Dataverse. Once an email is registered, a Dataverse trigger starts a flow with Power Automate. Here’s how the implementation works:

• Step 1-2: Receive and register emails in Dynamics CE using Dataverse, triggering Power Automate.
• Step 3: Submit the email subject and body to Azure AI Language Studio. Use Named Entity Recognition to extract the policy number if it’s present.
• Step 4: If a policy number is found, contact Dataverse to convert the policy number to the corresponding department. This approach ensures an exact match to the department.
• Step 5: If no policy number is found, submit the email subject and body to Azure AI Language Studio for text classification to detect the department.
• Step 6: Assign the email case to the correct department.

Azure AI Language (formerly Azure Cognitive Services) is a cloud-based service providing Natural Language Processing (NLP) features for understanding and analyzing text. This approach avoids complex Machine Learning algorithms, opting for a faster time to market.

The goal is to create a scalable solution with no custom development, relying solely on low-code solutions, and providing an easy-to-use interface for end-users to create new language models and improve the AI engine. While this example uses Power Automate, Logic Apps can also be used as they support the Language Studio connector.

Extracting Policy Numbers with Named Entity Recognition (NER)

Named entity recognition (NER) — sometimes referred to as entity chunking, extraction is the process of identifying and categorizing key information, or entities, in text, such as names, places, organizations, and quantities. In our case, we use NER to detect policy numbers from emails. By training the language service model, we can accurately highlight policy numbers within email content, as shown in the screenshot below.

Accurate data labeling is crucial for the NER model’s success. Generally, more labeled data improves model performance, provided the labels are accurate. While there’s no fixed number of labels needed for optimal performance, Microsoft recommend labeling at least 50 documents per class to ensure good results. The key to an effective model is reducing ambiguity in your labeling schema and maintaining high data quality.

Focusing on policy numbers allows us to direct emails to the appropriate department efficiently, eliminating guesswork and speeding up the email triage process.

Department Detection with Text Classification

When an email doesn’t include a matching policy number, we need to sort it by category. Emails related to life insurance are classified under “life,” health insurance emails under “health,” and all other types go under “non-life.”

We’ll train the language service model using single-label text classification to detect these categories. The screenshot below demonstrates this process, showing how the email (even though it includes a policy number) is categorized into one of the three groups.

Power automate flow

Microsoft power platform provides a Cognitive Services adapter to connect with the Text Analytics service in Azure Language service. This connector is available for both Power Automate and Logic Apps, making it easy to integrate and automate text analysis tasks within your workflows.

Named entity recognition

Single Label Classification

Monitoring and Optimization

Model performance metrics

In Azure AI Language Studio, you can easily check how well your model predicts and how accurate it is. Microsoft provides recommendations for actions you can take to improve the model’s accuracy. This helps ensure your email triage process for insurance claims is as efficient and reliable as possible.

Monitor performance in production

To keep track of machine learning model performance in production, Azure Machine Learning offers robust monitoring tools. These tools provide detailed signals and alerts to help you spot potential issues early. This allows you to assess risks and identify any blind spots.

You can set up both basic and advanced monitoring for models deployed on Azure Machine Learning online or batch endpoints, and even for external deployments. Additionally, you can customize monitoring signals and metrics, interpret the results, and integrate with Azure Event Grid for smooth operations.

Confidence score

Azure AI Language Studio provides a confidence score along with its results. This score indicates how confident the AI model is in its output. It’s crucial to understand that AI doesn’t operate on absolute certainties but rather on trained models that make educated guesses. The confidence score helps gauge the reliability of these guesses.

Using the confidence score can be very helpful in deciding whether to trust the AI’s result or to perform a manual review. For instance, if the confidence score is below 0.5, it might be wise to have a human review the result to ensure accuracy.

"documents": [
    {
        "id": "565bbcb7-e2cd-4aec-be47-f19a5ec26c8d",
        "class": [
            {
                "category": "Non-Life",
                "confidenceScore": 0.39
            }
        ],
        "warnings": []
    }
]

Throttling limits

Be aware of the cognitive services connector throttling limits with the connector, as mentioned on the Microsoft documentation there is a limit present on how many calls are allowed per connection:

Adaptive Learning

The goal here is to continuously improve AI models based on feedback and new data.

To implement an Adaptive Learning strategy for Azure AI Language Studio, you can establish a continuous feedback loop that enables your AI models to improve over time. Start by collecting feedback on the AI’s performance from end-users and system logs, particularly focusing on misclassifications, edge cases, and evolving email content patterns.

Integrate this feedback into a retraining pipeline using Azure Machine Learning, which regularly updates your models based on new labeled data and corrected predictions.

Automate this process by scheduling regular model evaluations and retraining sessions, ensuring that the models adapt to new types of inquiries and changes in language usage.

Further optimizations

There’s even more we can do to improve this email triage system:

Customer Data: Utilize customer email data to narrow down the relevant department types based on active and past contracts. This refinement can significantly boost the system’s accuracy.

Sentiment Analysis: Assess the tone of incoming emails to prioritize them based on urgency or customer sentiment. Implement this by using Azure Cognitive Services to detect sentiment and creating rules in Power Automate to promptly escalate negative or urgent emails to human agents.