Unit testing: how to validate code that is fed to source generator?

[vcaraulean]

Hello

I'm using Verify.SourceGenerators library to verify/assert the output of the generator. The approach works fine. But I lost some time investigating why some outputs were not generated and it turned out that my inputs were not correct - they wouldn't even compile.

Is there a way to validate that the inputs of the generator are valid and can be compiled before running generator in the context of a unit test?

I'm running the generator as follows:

    private static GeneratorDriverRunResult RunGenerators(params string[] sources)
    {
        var compilation = CSharpCompilation.Create(
            nameof(CodeGeneratorTests),
            sources.Select(text => CSharpSyntaxTree.ParseText(text)),
            [
                // To support 'System.Attribute' inheritance, add reference to 'System.Private.CoreLib'.
                MetadataReference.CreateFromFile(typeof(object).Assembly.Location)
            ]);

        return CSharpGeneratorDriver
            .Create(new MyCodeGenerator())
            .RunGenerators(compilation)
            .GetRunResult();
    }

I'm just starting my journey with source generators, help is appreciated! If it's covered or was discussed, will appreciate a link or reference...

Thanks!

[333fred]

You'll have to ask on the Verify repo. This is a common complaint I hear from users of this testing library, and it's worth raising with them. The official testing SDK verifies that diagnostics are either not present or match a list of expected diagnostics, which prevents this issue.

[vcaraulean]

Thanks for feedback!

The code that I provided is not using any of the Verify-specific code. Are there any settings that I can tweak to get some diagnostics and be able to break/throw if code is not valid? I inspected all of the intermediate outputs in the code above (compilation, driver, results) supplied with invalid inputs - diagnostics are all empty and there are no exceptions...

Particularly, even this code never throws on bad inputs:

        var x = driver.RunGeneratorsAndUpdateCompilation(compilation, out _, out var diagnostics);
        if (!diagnostics.IsEmpty)
            throw new Exception("Code generation failed");

[vcaraulean]

I reviewed as well all generator's cookbooks and other docs, tried to follow recommendations and patterns but nothing helped - all diagnostics are empty...

[333fred]

I'm not really sure what you're looking for, to be honest. If you want to get diagnostics from the compilation, you can get diagnostics from the compilation, but we don't provide helpers to validate those diagnostics in Roslyn itself. AFAIR, Verify.SourceGenerators just provides helpers that validate the generated sources, not diagnostics.

[333fred]

driver.RunGeneratorsAndUpdateCompilation(compilation, out _, out var diagnostics);

These diagnostics are only diagnostics that are raised by generators that are run, ie things produced by output sources that raise diagnostics. If you want to get all diagnostics from the entire compilation, then you need to save the compilation, rather than discarding it, and get all diagnostics from it.

/cc @chsienki, it looks like we don't have documentation on that API, and this is a pitfall I've seen before. We should document this better.

[vcaraulean]

Thanks for the hint on looking at the whole compilation.
I have this snipped that allows me to advance and have some error handling:

        var x = driver.RunGeneratorsAndUpdateCompilation(compilation, out var updatedCompilation, out _);
        if (updatedCompilation.GetDiagnostics().Any(d => d.Severity == DiagnosticSeverity.Error))
            throw new Exception("Compilation failed");

[chsienki]

You'll want to check both sets of diagnostics: the ones that come out via the out param are things that the generators raised whilst generating source. The ones on the compilation are any that occur when the generated source is actually compiled.