CONTRIBUTING.md

Contributing to F# Data

This page should provide you with some basic information if you're thinking about contributing to the F# Data package. It gives a brief summary of the library structure, how type providers are written and how the F# Data library handles multi-targeting (to make the providers available for Desktop as well as Portable libraries).

• This page can be edited by sending a pull request to F# Data on GitHub, so if you learn something when playing with F# Data, please record your findings here!

• If you want to discuss a feature (a good idea!), or if you want to look at suggestions how you might contribute, check out the Issue list on GitHub or send an email to the F# Open-Source mailing list.

Solution files

The root directory contains a number of Visual Studio solutions (*.sln) files that group the projects in the main logical groups:

• FSharp.Data.sln contains the main projects that implement the F# Data functionality (such as runtime and design-time type provider libraries).

• FSharp.Data.Tests.sln is a library with tests for F# Data and it also contains the content of this web site (as *.fsx and *.md) files. Look here if you want to edit the documentation!

Projects and multi-targeting

One problem with developing type providers is supporting multiple versions of the .NET platform. Type providers consist of two components:

• Runtime is the part of the type provider that is actually used when the compiled F# code that uses the provider runs. This assembly also has the non type-provider components of FSharp.Data: the CSV, HTML and JSON parsers, and the HTTP utilities.

• Design time is the part that is used when editing F# code that uses type provider in your favourite editor or when compiling code. For example, in the CSV provider, this component does the type inference and generates types (that are mapped to runtime components by the compiler).

To support multiple targets, we need a runtime component for every single target (.NET 4.0, Portable profile 47 and Portable profile 7). However, we only need one design time component, because that is always going to be executed on desktop .NET in Visual Studio or Xamarin Studio.

So, there are 3 versions of runtime components and 1 version of design time components. At the moment, this is done by having separate project file for each component, but they share the same files - the project just defines some symbols that are then used to include/exclude parts that are not available on certain platforms using #if.

If you open FSharp.Data.sln, you'll see the following projects for runtime components:

• FSharp.Data - the desktop .NET 4.0 version
• FSharp.Data.Portable47 - F# portable library version (Profile 47 targeting desktop .NET 4.5, Silverlight 5.0, Windows Phone 8 and Windows 8)
• FSharp.Data.Portable7 - F# portable library version (Profile 7 targeting desktop .NET 4.5 and Windows 8)

The design time components are in the following project:

• FSharp.Data.DesignTime

Type provider structure

Several of the F# Data type providers have similar structure - the CSV, JSON, XML and HTML providers all infer the types from structure of a sample input. In addition, they all have a runtime component (CSV parser, HTML parser, JSON parser, and wrapper for XDocument type in .NET).

So, how is a typical type provider implemented? First of all, there are some shared files - in Common and Library subdirectories of the projects. These contain common runtime components (such as parsers, HTTP helpers, etc.)

Next, there are some common design-time components. These can be found in Providers folder (in the 2 design time projects) and contain the ProvidedTypes helpers from the F# team, StructureInference.fs (which implements type inference for structured data) and a couple of other helpers.

A type provider, such as JSON provider, is then located in a single folder with a number of files, typically like this:

• JsonRuntime.fs - the only runtime component. Contains JSON parser and other objects that are called by code generated by the type provider.

• JsonInference.fs - design-time component that infers the structure using the common API in StructureInference.fs.

• JsonGenerator.fs - implements code that generates provided types, adds properties and methods etc. This uses the information infered by inference and it generates calls to the runtime components.

• JsonProvider.fs - entry point that defines static properties of the type provider, registers the provided types etc.

The WorldBank and Freebase providers are different. They do not need inference, but they still distinguish between runtime and design-time components, so you'll find at least two files (and possibly some additional helpers).

Source code

Debugging

To debug the type generation, the best way is to change FSharp.Data.DesignTime project to a Console application, rename Test.fsx to Test.fs and hit the Run command in the IDE, setting the breakpoints where you need them. This will invoke all the type providers manually without locking the files in Visual Studio / Xamarin Studio. You'll also see in the console output the complete dump of the generated types and expressions. This is also the process used for the signature tests.

Assembly replacer

Generating code in type providers (see e.g. JsonGenerator.fs) is a bit tricky, because the generated code needs to contain references to the appropriate runtime assembly (Desktop or Portable profile). This is particularly tricky When using F# quotations to produce the generated code. If the source code contains <@@ foo.Bar @@>, then the quoation has a direct reference to the type of foo from the current assembly.

This is handled by Assembly replacer (see AssemblyReplacer.fs in Providers) which transforms quotations and replaces references with the right versions. For more information about how this works, see also the discussion on GitHub.

Here is a documentation for the AssemblyReplacer type:

When we split a type provider into a runtime assembly and a design time assembly, we can no longer use quotations directly, because they will reference the wrong types. AssemblyReplacer fixes that by transforming the expressions generated by the quotations to have the right types.

On all expressions that we provide to InvokeCode and GetterCode of ProvidedMethod, ProvidedConstructor, and ProvidedProperty, instead of (fun args -> <@@ doSomethingWith(%%args) @@>), we should use (fun args -> let args = replacer.ToDesignTime args in replacer.ToRuntime <@@ doSomethingWith(%%args) @@>).

When creating the ProvidedXYZ type, we have to always specify the runtime type, and when it invokes the function provided to InvokeCode and GetterCode, we to first transform the argument expressions to the design time types, so we can splice it in the quotation, and then after that we have to convert it back to the runtime type.

A further complication arises because Expr.Var's have reference equality, so when can't just create new Expr.Var's with the same variable name and a different type. When transforming them from runtime to design time we keep them in a dictionary, so that when we convert them back to runtime we can return the exact same instance that was provided to us initially.

Another limitation (not only of this method, but in general with type providers) is that we can never use expressions that use F# functions as parameters or return values, we always have to use delegates instead.

[hide]
open System
open System.Reflection
open Microsoft.FSharp.Quotations

All standard type providers obtain an instance of AssemblyReplacer when constructed and then pass it to the code generator, which can use it to generate appropriate code:

type AssemblyReplacer =

  /// Gets the equivalent runtime type
  abstract member ToRuntime : designTimeType:Type -> Type
  
  /// Gets an equivalent expression with all the types 
  /// replaced with runtime equivalents
  abstract member ToRuntime : designTimeTypeExpr:Expr -> Expr

  /// Gets an equivalent expression with all the types 
  /// replaced with designTime equivalents
  abstract member ToDesignTime: runtimeExpr:Expr -> Expr

Documentation

The documentation for the F# Data library is automatically generated using the F# Formatting library. It turns *.md (Markdown with embedded code snippets) and *.fsx files (F# script file with embedded Markdown documentation) to a nice HTML documentation.

• The code for all the documents can be found in the content directory on GitHub. If you find a bug or add a new feature, make sure you document it!

• Aside from direct documentation for individual types, there is also a tutorials folder (on GitHub) where you can add additional samples and tutorials that show some interesting aspects of F# Data.

• If you want to build the documentation, simply run the build.fsx script (GitHub link) which builds the documentation.

Related articles

If you want to learn more about writing type providers in general, here are some useful resources:

• Writing F# Type Providers with the F# 3.0 Developer Preview - An Introductory Guide and Samples

• F# 3.0 Sample Pack contains a number of examples ranging from quite simple, to very complex.

• Tutorial: Creating a Type Provider (F#)