Add proposed plan for the COM source generator #60143
Conversation
jkoritzinsky
added
design-discussion
jkoritzinsky
requested review from
lambdageek,
stephentoub,
jkotas,
tannergooding,
AaronRobinsonMSFT and
elinor-fung
| We should also provide a marshaller that uses the generated `ComWrappers` type to easily enable marshalling a COM interface in method calls to another COM interface. | ||
|
|
||
| > Open Question: Do we want to enable using the `MarshalAsAttribute` as well for these types? | ||
| > This would require us to hard-code in support for these marshallers into the COM source generator, but would provide better backward compatibility and easier migration. |
There was a problem hiding this comment.
Migration might not be so bad depending on the users in question. I'd like to start with the premise the defaults, unadorned arguments, be the most common which results in COM interface definitions being void of MarshalAs.
|
|
||
| The COM Source Generator should be designed as a Roslyn source generator that uses C# as the "source of truth". By using C# as the source of truth, we can use Roslyn's rich type system to inspect the various parameter and return types and determine the correct marshalling mechanism. Additionally, we would not have to encode policy for mapping non-.NET types to .NET types, which can be an error prone and opinionated process. | ||
|
|
||
| COM has a platform-agnostic source of truth with IDL files and TLBs (type libraries). We propose that conversions from these files to a C# source of truth that the COM source generator can consume should be implemented by .NET CLI tools. These tools can be manually invoked before a build if the source of truth changes rarely, or they can be included in the build pipeline for a project to produce the C# source of truth at build-time. Alternatively, we can provide some of the hooks for the COM source generator as an API, and another source generator that reads IDL or TLB files from the AdditionalFiles collection could re-use the core of the COM source generator to generate similar code directly from the IDL or TLBs. Since Roslyn source generators cannot have dependencies on other generators, we cannot sequence both an IDL/TLB to C# generator and a C# to C# generator together. |
There was a problem hiding this comment.
Just noting that while COM supports IDL and TLB, it is not the case that all COM starts out from IDL or TLB. There are a few cases of COM even in the Windows SDK (D2D1, for example) which only exist as C++ header files.
There are also cases, like most of DirectX, where its not the type of IDL/TLB that you find in "traditional COM". Instead, it uses what is sometimes referred to as "nano COM" which is a limited subset and which doesn't use CoTaskMemAlloc, CLSID (only using IID), or other functionality. Additionally, the IDL files don't have all the required contrsucts to work with the existing tlbgen.exe or related tooling that exists on Full Framework.
I think we should ensure that any tooling here can properly account for and handle these types of scenarios.
- https://github.com/microsoft/clangsharp currently achieves this non-source generator based support by simply parsing and traversing the C/C++ AST, and while it can't handle everything that real C++ supports; it does support effectively all the COM constructs you'd encounter in nano-COM and several of the constructs from full COM.
There was a problem hiding this comment.
Due to the issues with making IDL/TLB the source of truth (as you mention), the proposed plan is to leave TLB/IDL consumption and production for later work and develop a solution that uses a C# source of truth first. That way, projects like ClangSharp could produce C# that the source generator can consume even when the source of truth isn't IDL or a TLB.
There was a problem hiding this comment.
I would prefer to avoid providing any additional tooling beyond what TlbImp or TlbExp could provide. Parsing C/C++ headers looks to be well covered by the clangsharp tool and I'd argue that tool should continue to do that. In this case the proposed source generator would be able to consume the output of clangsharp and treat that as the source of truth. What would be the benefit of supplanting this tool or am I missing some nuance here?
There was a problem hiding this comment.
What would be the benefit of supplanting this tool or am I missing some nuance here?
That was somewhat my point. This section discusses IDL and TLB files and conversions from them to something that the source generator can handle.
However, they are not always available or usable and there exists other tooling that could already generate the required "stuff". ClangSharp for example already generates the full VTBLs and helper methods required to support COM today: https://source.terrafx.dev/#TerraFX.Interop.Windows/um/d3d12/ID3D12Device8.cs,56ac150c61296fc3
It just doesn't have the "stuff" to connect these raw COM views into the COM Wrappers support.
There was a problem hiding this comment.
@AaronRobinsonMSFT thats exactly what my proposal is.
We as the runtime team would provide the “C# source of truth” generator.
Either our team or someone else would provide the TlbImp/Exp replacement, which could generate C# that our generator reads in.
ClangSharp (or another community project) would provide the “C++ to C#” tool that would generate code that our generator reads in to do the work.
The only work we need to do is the “C# source of truth” generator.
There was a problem hiding this comment.
they are not always available or usable and there exists other tooling that could already generate the required "stuff"
I don't think I am getting the point here. The current TlbImp tools generate things that may not work with this if we use new attributes and to some degree we would like to have a single assembly that would contain both definition and goo - should be possible to not have that but that would ideally be P0 given the MCG replacement goal. Consuming a TLB as the source of truth is the canonical way for COM interop so that is the primary focus. DirectX scenarios are P2 at this point but once something has converted those definitions into C# we could consume that and clangsharp seems to be able to handle that so I am back to being confused I think. Is the point here that tools exist that already do part of the job?
The TlbExp tooling however does need a full replacement so that would be another reason TLB is called out.
| // UserProvided.cs | ||
|
|
||
| [Guid("4b69d271-5c99-4f95-b1eb-381e6e689f1a")] | ||
| interface IMyComInterface |
There was a problem hiding this comment.
Is there an example of what an a more complex inheritance hierarchy might look like?
Will users continue needing to "duplicate" base interface members with the new slot keyword like with the built-in COM support?
How do you indicate this is IUnknown vs no base vs IDispatch or something else?
There was a problem hiding this comment.
I will add some examples for these scenarios in my next iteration.
There was a problem hiding this comment.
I'm not sure what no base means in this case because without it what is being generated? These tooling is specific to IUnknown based interfaces so that is where it would stop. The IDispatch base is interesting and I'd imagine we can reuse some existing attributes – ComInterfaceType has baggage but could be reused in this case, ignoring InterfaceIsIInspectable of course.
There was a problem hiding this comment.
Likewise, what's the expected model for common patterns where managed objects (e.g. interfaces) don't translate cleanly to the native signature?
For example, there are often APIs that take the form of HRESULT M(..., void** ppOutput);
If ppOutput is null, the method returns S_FALSE if the passed in parameters would have allowed the method to succeed (it acts effectively as a validation method without allocating or doing other costly internal operations).
This means the signature is effectively out T output where Unsafe.NullRef<T>() is valid; but that has various difficulties among other things and is "traditionally" handled by out IntPtr output or IntPtr* output where you lose all real safety and support, so you aren't significantly "better off" than if you were simply using the underlying vtbls in the first place (outside of basic lifetime tracking).
There was a problem hiding this comment.
I'm not sure what no base means in this case because without it what is being generated? These tooling is specific to IUnknown based interfaces so that is where it would stop. The IDispatch base is interesting and I'd imagine we can reuse some existing attributes – ComInterfaceType has baggage but could be reused in this case, ignoring InterfaceIsIInspectable of course.
@AaronRobinsonMSFT, there exist COM interfaces in a few scenarios that don't inherit from IUnknown. One example is ID3DInclude, which is simply declared using DECLARE_INTERFACE(iface) and so it follows the COM ABI, but is not itself IUnknown. This interface is expected to be implemented by the consumer and passed into a consuming function provided by the library.
The tooling should likely not break down in the face of such an interface declaration; regardless of whether the user implements it alongside IUnknown on the derived type
There was a problem hiding this comment.
there exist COM interfaces in a few scenarios that don't inherit from IUnknown.
That isn't a COM interface then – by definition. The entire ComWrappers API is about providing IUnknown ABI support and removing that is, in my opinion, a non-starter as it breaks down lifetime and identity semantics fast. In fact exposing the ID3DInclude type seems like a "dead-end" because you can't go anywhere from it.
I guess one could attempt to use virtual inheritance to inherit from both IUnknown and ID3DInclude to "unify" them but I don't know if the order of that is actually defined so how would we enable this in a stable way? What am I missing?
There was a problem hiding this comment.
That isn't a COM interface then – by definition.
Sorry, that was a misstatement by me. I meant "there exist interfaces in a few COM related scenarios".
In particular I'm just trying to call out some scenarios that I think are important to consider, at least from one aspect. I don't think they have to be handled the same way or even at all, as long as they are thought about and a decision is made about them.
I just struggle with finding the right words sometimes 😄
There was a problem hiding this comment.
@tannergooding I think this conversation revolves around two principles – (1) let's make sure we support as many scenarios as possible compared to the build-in and (2) how can we ensure our source gen provides maximum benefit for the .NET community. I agree with these principles and I think the tenor of the initial questions demonstrates a similar thinking. Instead of focusing on concrete examples like ID3DInclude, I'll try to adhere to the above principles.
The scenario above, I am sure there are others, demonstrates that we going to fail to please everyone. Therefore in an effort to bring (1) closer to reality we will need to be permissive in some manner. Principle (2) dictates saying "You can build your own source generator from these N assemblies" or "your special cases will require you to perform unnatural build system hacks" aren't acceptable. So, I propose the following:
- We provide a way for users to define their own manually written/defined wrappers (RCW/CCW) for an interface. These would then be discoverable through some attribute or other gesture by the source generator. This would allow the majority of people to benefit from the simple well-defined COM that is likely the common support scenario (for example, WinForms) and at the same time permit special cases to be addressed while not degrading the overall experience if a small minority of types are an issue.
@jkoritzinsky I'd imagine we could have users attribute types for an RCW (a la, IDIC) and/or add attributes to static methods marked with UnmanagedCallersOnly as to where in a CCW generated vtable they wanted to place the method. There are lots of options here but in the end we can focus on typical COM support circa 2002 but permit anyone to say "I want the tool to generate 99 % of my interfaces, but these 2 I will define myself using all the fancy Marshal helpers written in .NET 7 and the tool will fold them into the process naturally."
|
I fully support Option 3 in the markdown document. As I see it, the developer should be able to simply replace
Not necessarily. Remember that you can use Roslyn to detect if a type in the user code does not have an attribute. Hence, we can create
The
Not (entirely) true! People often forget about the the win32metadata project, which contains a representation of just about every COM type in the Windows SDK, including those that ship without IDL files. If this source generator is to be useful, it must be able to interoperate with CsWin32 as far as consuming Windows.Win32.winmd is concerned. As such, why not ship this generator in the CsWin32 bits? A developer can use either the existing generator, or this COM interop generator, but if they use both, the output of the former will be compatible with the latter. |
I've incorporated some of this feedback into Option 5 for the code-gen.
I've provided a specific example of where the "two ComWrappers" problem would arise.
I've addressed the reasons why we wouldn't use the Win32Metadata format in an update part of the doc. The main issues are that the primary use case for the generator is for non-Win32 COM APIs, which the Win32Metadata project has no plans for supporting. Additionally, CsWin32 has no plans of supporting consuming multiple Win32Metadata files, so even hacking together your own Win32Metadata file wouldn't enable a good experience. |
|
|
||
| The third limitation basically would mean that the COM source generator would have to ignore all of the non-COM APIs in the metadata, which would make it a poor replacement/companion for CsWin32, which can already handle all of the cases that the Win32Metadata project uses. | ||
|
|
||
| We would suggest to the CsWin32 team that they should instead adopt using some of our shared code in our generator infrastructure to push all of the mechanical marshalling logic to be generated by the Interop team source generator tooling, while they generate the public surface area how they see fit. |
There was a problem hiding this comment.
For what its worth, Win32Metadata starts out via ClangSharp processing the C/C++ headers.
The process is basically:
C/C++ Headers -> ClangSharpPInvokeGenerator -> C# Source Files -> Win32MetadataSpecificPostProcessing -> WinMD Files
The WinMD Files then get consumed by other projects, such as CsWin32, RsWin32, CppWin32, etc to produce the "consumable output".
Extending the pipeline to expose all the required metadata and to consume that metadata such that CsWin32 can successfully use some of the shared generator infrastructure logic would probably involve:
- ClangSharp (myself)
- Win32Metadata (Steve Otteson)
- CsWin32 (Andrew Arnott)
- Interop Team
That's perfectly all right. I just wanted to ensure that Win32Metadata will be supported, as it's the only managed definition for many desirable Windows native APIs. |
|
Another important point I just thought of: Since you cannot directly feed one source generator's output into another, we will need to have the ability to place the generation-triggering attributes in a user-code file that does not contain the definition of the type that code should be generated for. Even though private COM APIs are the primary target of this project, please do not forget that this it will also be used with Windows APIs in a great many cases. |
@wjk. This gets slightly off topic, but I maintain https://github.com/terrafx/terrafx.interop.windows, which is powered by https://github.com/microsoft/clangsharp (which I also maintain). The bindings are 1-to-1 raw/unsafe interop bindings and about as close to You end up doing a bit more manual management and unsafe code yourself; but at the benefit of less overhead, things that are correct by default, no missing functionality, additional type safety, and no having to unnecessarily deal with |
|
@tannergooding Please accept my apologies for forgetting about your library. In hindsight that entire paragraph was rather obviously biased towards only my use-cases anyway, so I deleted it. We must remember that the use-cases that we foresee do not necessarily contain every use-case someone will try to use this source generator for. (One of my major criticisms of the Linux movement is the refusal for maintainers to consider points of view, use cases, or design patterns that they did not think of or do not agree with personally.) But let's get back on topic here, shall we? 😄 |
|
/cc: @JeremyKuhne @weltkante FYI |
| } | ||
| ``` | ||
|
|
||
| This attribute could be applied to any interface to generate the marshalling code for either RCW or CCW scenarios. The `ExportMetadata` property would be used by any tool that wants to generate a metadata file, like a TlbExp successor, to determine which interfaces to export. |
There was a problem hiding this comment.
The
ExportMetadataproperty
It might be better to be explicit about what the "metadata" format. Based on some offline conversations it would seem IDL is what is going to be the source generator's artifact. From that, a TLB could be generated if needed.
Also, I just realized that expecting this to be a TLB is going to make the cross-platform scenario weird since the attribute would mean nothing without us implementing CreateTypeLib() for non-Windows... blech.
ghost
locked as resolved and limited conversation to collaborators
The interop team has decided to invest some time into creating a COM source generator. This doc describes some of the goals of the project, various checkpoints where we can stop our work, and opens a discussion for how we want users to identify which types to generate source for.