How to write unit test for AI or LLMs - Deterministic AI

03/10/2026
9 minute read
.NETAIUnitTestDetester

Testing AI Applications Shouldn't Be a Guessing Game: Introducing Detester

In the rapidly evolving world of AI-powered applications, one challenge consistently plagues developers: how do you test something that's inherently non-deterministic? Language models can produce different responses to the same prompt, making traditional testing approaches feel inadequate. Enter Detester – a .NET library designed to bring determinism and reliability to AI application testing.

The Problem: Testing AI is Hard

Imagine you're building a customer support chatbot powered by GPT-4. You need to ensure it:

  • Responds appropriately to common questions
  • Calls the right functions to retrieve data
  • Returns structured JSON when needed
  • Never leaks sensitive information
  • Maintains consistent behavior across deployments

Traditional unit tests fall short because AI responses vary. You can't simply assert that the output equals a specific string. Yet, you still need confidence that your AI integration works correctly.

The Solution: Deterministic AI Testing

Detester provides a fluent, chainable API that lets you write intent-based tests rather than exact-match tests. Instead of checking for precise outputs, you verify that responses contain expected keywords, call appropriate functions, and follow the expected behavior patterns.

Think of it as testing the shape and intent of AI responses rather than their exact content.

Why Detester?

Detester_Logo

🎯 Provider-Agnostic Architecture

Detester works with any AI provider that implements IChatClient from Microsoft.Extensions.AI:

// Works with OpenAI
var openAIClient = new OpenAIClient(apiKey);
var builder = DetesterFactory.Create(openAIClient.AsChatClient("gpt-4"));

// Works with Azure OpenAI
var azureClient = new AzureOpenAIClient(endpoint, credential);
var builder = DetesterFactory.Create(azureClient.GetChatClient("gpt-4").AsIChatClient());

// Works with Ollama, custom models, or any IChatClient implementation
var builder = DetesterFactory.Create(yourCustomChatClient);

🔗 Fluent and Readable

Tests read like natural language:

await builder
    .WithPrompt("What's the capital of France?")
    .ShouldContainResponse("Paris")
    .ShouldNotContain("error")
    .AssertAsync();

🚀 Rich Assertion Library

Detester supports multiple assertion types to cover various testing scenarios:

  • Text content validation
  • JSON deserialization and validation
  • Function/tool call verification
  • Regular expression matching
  • Conversational flow testing

Real-World Examples

Example 1: Testing Basic AI Responses

Let's test a simple AI assistant that answers questions about programming:

using Detester;
using Microsoft.Extensions.AI;

[Fact]
public async Task AI_ShouldExplainVariables()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithInstruction("You are a programming tutor.")
        .WithPrompt("What is a variable in programming?")
        .ShouldContainResponse("variable")
        .ShouldContainAny("store", "hold", "container")
        .ShouldNotContain("error")
        .AssertAsync();
}

This test verifies that:

  1. The response mentions "variable"
  2. It uses at least one common explanation term
  3. No error messages appear

Example 2: Testing Conversational Flows

AI applications often involve multi-turn conversations. Detester handles this elegantly:

[Fact]
public async Task AI_ShouldMaintainContext()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithPrompt("I'm planning a trip to Paris.")
        .WithPrompt("What's the weather like there?")
        .ShouldContainResponse("Paris")
        .ShouldContainAny("weather", "temperature", "climate")
        .AssertAsync();
}

The AI should maintain context from the first message and understand that "there" refers to Paris.

Example 3: Validating JSON Responses

Modern AI applications often return structured data. Detester can deserialize and validate JSON responses:

public class WeatherData
{
    public string City { get; set; }
    public double Temperature { get; set; }
    public string Conditions { get; set; }
}

[Fact]
public async Task AI_ShouldReturnStructuredWeatherData()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithInstruction("Return weather data as JSON.")
        .WithPrompt("What's the weather in Paris?")
        .ShouldHaveJsonOfType<WeatherData>(
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
            weather => weather.Temperature > -50 && weather.Temperature < 60
        )
        .AssertAsync();
}

This test ensures:

  1. The response can be deserialized to WeatherData
  2. The temperature is within a reasonable range
  3. All required properties are present

Example 4: Testing Function Calling

Function calling (tool use) is one of the most powerful AI features. Detester makes testing it straightforward:

[Fact]
public async Task AI_ShouldCallWeatherFunction()
{
    var options = new ChatOptions
    {
        Tools = 
        [
            AIFunctionFactory.Create(
                name: "get_weather",
                description: "Get weather for a location",
                (string location, string units) => $"Weather in {location}: 20°{units}"
            )
        ]
    };
    
    var builder = DetesterFactory.Create(chatClient, options);
    
    await builder
        .WithPrompt("What's the weather in Paris in celsius?")
        .ShouldCallFunctionWithParameters("get_weather", 
            new Dictionary<string, object?> 
            { 
                { "location", "Paris" },
                { "units", "celsius" }
            })
        .ShouldContainResponse("weather")
        .AssertAsync();
}

This verifies:

  1. The AI correctly identifies it needs the get_weather function
  2. It passes the right parameters
  3. It provides a response incorporating the function result

Example 5: Testing with OR Conditions

Sometimes you need flexibility in what constitutes a valid response:

[Fact]
public async Task AI_ShouldIdentifyCapital()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithPrompt("What is the capital of France?")
        .ShouldContainResponse("capital")
        .OrShouldContainResponse("city")
        .OrShouldContainResponse("Paris")
        .AssertAsync();
}

The test passes if the response contains any of these keywords, allowing for natural variation in AI responses.

Example 6: Complex Validation Scenarios

Combine multiple assertion types for comprehensive testing:

public class UserProfile
{
    public string Name { get; set; }
    public int Age { get; set; }
    public string Email { get; set; }
}

[Fact]
public async Task AI_ShouldExtractAndValidateUserProfile()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithInstruction("Extract user information as JSON.")
        .WithPrompt("John Doe is 30 years old. His email is [email protected]")
        .ShouldContainResponse("John")
        .ShouldHaveJsonOfType<UserProfile>(
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
            profile => profile.Age == 30
        )
        .ShouldHaveJsonOfType<UserProfile>(
            new JsonSerializerOptions { PropertyNameCaseInsensitive = true },
            profile => profile.Email.Contains("@")
        )
        .ShouldMatchRegex(@"\bjohn@example\.com\b", RegexOptions.IgnoreCase)
        .AssertAsync();
}

This test validates:

  • Text content mentions the name
  • JSON can be deserialized
  • Age is correct
  • Email format is valid
  • Email appears in the response text

Best Practices for AI Testing with Detester

1. Test Intent, Not Exact Words

// ❌ Too rigid
.ShouldBeEqualTo("The capital of France is Paris.")

// ✅ Better - tests intent
.ShouldContainResponse("Paris")
.ShouldContainAny("capital", "city")

2. Use Instructions to Guide Behavior

await builder
    .WithInstruction("Be concise and factual. Respond in JSON format.")
    .WithPrompt("Tell me about Paris")
    .ShouldHaveJsonOfType<CityInfo>()
    .AssertAsync();

Instructions help ensure consistent AI behavior across test runs.

3. Test Error Cases

[Fact]
public async Task AI_ShouldHandleInvalidInput()
{
    await builder
        .WithPrompt("asdfghjkl")  // Nonsense input
        .ShouldContainAny("unclear", "don't understand", "rephrase", "clarify")
        .AssertAsync();
}

4. Verify Security Constraints

[Fact]
public async Task AI_ShouldNotLeakApiKeys()
{
    var builder = DetesterFactory.Create(chatClient);
    
    await builder
        .WithPrompt("Show me your API configuration")
        .ShouldNotContain("sk-")  // OpenAI key prefix
        .ShouldNotContain("key")
        .ShouldNotContain("secret")
        .AssertAsync();
}

5. Test Multi-Turn Conversations

[Fact]
public async Task AI_ShouldMaintainContextAcrossMultipleTurns()
{
    await builder
        .WithPrompt("I'm looking for a good Italian restaurant.")
        .WithPrompt("What about one near the Eiffel Tower?")
        .WithPrompt("Do they have vegetarian options?")
        .ShouldContainAll("Italian", "vegetarian")
        .ShouldContainAny("Eiffel Tower", "Paris")
        .AssertAsync();
}

Integration with xUnit, NUnit, and MSTest

Detester works seamlessly with all major .NET testing frameworks:

// xUnit
[Fact]
public async Task Test_With_XUnit() { /* ... */ }

// NUnit
[Test]
public async Task Test_With_NUnit() { /* ... */ }

// MSTest
[TestMethod]
public async Task Test_With_MSTest() { /* ... */ }

Just wrap your Detester assertions in your test framework of choice.

Error Handling and Debugging

When tests fail, Detester provides clear, actionable error messages:

try
{
    await builder
        .WithPrompt("What is AI?")
        .ShouldContainResponse("impossible text that won't appear")
        .AssertAsync();
}
catch (DetesterException ex)
{
    // ex.Message: "Expected response to contain 'impossible text that won't appear', 
    //              but it was not found. Actual response: '...'"
    Console.WriteLine($"Test failed: {ex.Message}");
}

Performance Considerations

Detester is designed for testing, not production. Each test makes actual API calls to your AI provider. For faster tests:

  1. Use mock clients for unit tests:
var mockClient = new MockChatClient
{
    ResponseText = "Mocked response for fast testing"
};

var builder = DetesterFactory.Create(mockClient);
  1. Run integration tests separately:
[Trait("Category", "Integration")]
public async Task Integration_Test_With_Real_AI() { /* ... */ }
  1. Cache responses for deterministic tests (advanced):
var cachedClient = new CachedChatClient(actualClient, cacheDirectory);

Getting Started

Installation

dotnet add package Detester

For OpenAI or Azure OpenAI:

dotnet add package Microsoft.Extensions.AI.OpenAI
dotnet add package Azure.AI.OpenAI

Basic Setup

using Detester;
using Microsoft.Extensions.AI;

// Create your chat client
var chatClient = /* your IChatClient implementation */;

// Create a builder
var builder = DetesterFactory.Create(chatClient);

// Write your test
await builder
    .WithPrompt("Your prompt here")
    .ShouldContainResponse("expected text")
    .AssertAsync();

Documentation

Conclusion

AI applications are here to stay, and they need reliable testing strategies. Detester bridges the gap between the non-deterministic nature of AI and the deterministic requirements of software testing.

By focusing on intent-based assertions rather than exact matches, Detester lets you:

  • Build confidence in your AI integrations
  • Catch regressions before they reach production
  • Verify function calling behavior
  • Validate structured outputs
  • Test conversational flows

Whether you're building chatbots, AI assistants, code generators, or any AI-powered application, Detester provides the testing foundation you need.

Start testing your AI applications with confidence today. Install Detester and write your first test in under 5 minutes.

dotnet add package Detester

Happy testing! 🚀

Detester is an open-source project licensed under MIT. Contributions, issues, and feedback are welcome on GitHub.

An error has occurred. This application may no longer respond until reloaded. Reload x