FsCheck


Quick Start

The fastest way to understand how FsCheck works is by writing some properties - FsCheck's terminology for a parametrized test, or a generative test - and run them using the built-in test runner. Later on, we'll describe how they can be integrated with existing test frameworks like NUnit, xUnit.NET or MsTest.

First install FsCheck, open an fsx file and start with:

#r "FsCheck"

open FsCheck

In C#: To easily experiment, start a new console app to execute the snippets below (the output is written to console by default). Alternatively, in LinqPad, reference FsCheck.dll and FSharp.Core.dll, open namespace FsCheck, change the language to "C# statements" and you should be able to execute most of the snippets as well.

A Simple Example

A simple example of a property - a test with parameters - is written as a normal F# function that returns a bool:

let revRevIsOrig (xs:list<int>) = List.rev(List.rev xs) = xs

This property asserts that the reverse of the reverse of a list is the list itself. To check the property, we load this definition in F# interactive and then invoke

Check.Quick revRevIsOrig
Ok, passed 100 tests.

In C#:

Func<int[],bool> revRevIsOrig = xs => xs.Reverse().Reverse().SequenceEqual( xs );
Prop.ForAll(revRevIsOrig).QuickCheck();

When a property fails, FsCheck displays a counter-example. For example, if we define

let revIsOrig (xs:list<int>) = List.rev xs = xs
Check.Quick revIsOrig
Falsifiable, after 2 tests (2 shrinks) (StdGen (338235241,296278002)):
Original:
[3; 0]
Shrunk:
[1; 0]

In C#:

Prop.ForAll<int[]>(xs => xs.Reverse().SequenceEqual(xs))
    .QuickCheck();

FsCheck also shrinks the counter example: it tries to find the minimal counter example that still fails the property. The counter example is indeed minimal: the list must have at least two different elements for the test to fail. FsCheck displays how many times it found a smaller (in some way) counter example and so proceeded to shrink further.

To learn more on how to write properties, see Properties.

What do I do if a test loops or encounters an error?

In this case we know that the property does not hold, but Check.Quick does not display the counter-example. There is another testing function provided for this situation. Repeat the test using

Check.Verbose
or in C#
VerboseCheck()
which displays each test case before running the test: the last test case displayed is thus the one in which the loop or error arises.

To learn more on how to run FsCheck tests see Running Tests.

FsCheck teaches us a lesson

The property above (the reverse of the reverse of a list is the list itself) is not always correct. Consider a list of floats that contains NaN (not a number). Since NaN <> NaN, the reverse of the reverse of [NaN] is not actually equal to [NaN]. FsCheck has a knack for finding this kind of specification problem. To see this error, ask FsCheck to generate lists of floats:

let revRevIsOrigFloat (xs:list<float>) = List.rev(List.rev xs) = xs
Check.Quick revRevIsOrigFloat
Falsifiable, after 13 tests (11 shrinks) (StdGen (338855678,296278002)):
Original:
[8.166666667; -14.36363636; -1.797693135e+308; 2.0; 7.0; 13.58333333; 9.5; nan;
 -7.2; 1.538461538; 1.797693135e+308; -10.33333333]
Shrunk:
[nan]

That said, the example in C# using floats actually works!

Prop.ForAll<double[]>(xs => xs.Reverse().Reverse().SequenceEqual(xs))
    .QuickCheck();

This is because SequenceEquals uses the default equality comparer under the hood, which uses Double's Equals method, which has a special provision for NaN, as you can see in the reference source. If we pass an EqualityComparer that uses == to SequenceEquals, the C# example also fails.

As this seemingly trivial example shows, FsCheck helps you discover interesting properties of your code - and so ultimately, more bugs!

Using FsCheck with other testing frameworks

Once you have finished your initial exploration of FsCheck, you'll probably want to use it with your existing unit test framework to augment unit tests or just to run the properties more easily. Below, we'll show some integration, but we leave it up to you to choose whichever suits your environment best.

Integration with Expecto

Some testing frameworks like Expecto have an out-of-the-box integration with FsCheck. By using one of the runners that support FsCheck out of the box, you gain access the the frameworks' reporting capabilities and integrations with IDEs and build tooling.

Here's a sample:

open Expecto
open Expecto.ExpectoFsCheck

let config = { FsCheck.Config.Default with MaxTest = 10000 }

let properties =
  testList "FsCheck samples" [
    testProperty "Addition is commutative" <| fun a b ->
      a + b = b + a
      
    testProperty "Reverse of reverse of a list is the original list" <|
      fun (xs:list<int>) -> List.rev (List.rev xs) = xs

    // you can also override the FsCheck config
    testPropertyWithConfig config "Product is distributive over addition" <|
      fun a b c ->
        a * (b + c) = a * b + a * c
  ]

Tests.runTests defaultConfig properties

Integration with xUnit

Another frequently used runner is xUnit.NET. Here is how to write the unit test above so it can be run from xUnit.NET:

open global.Xunit

[<Fact>]
let ``Reverse of reverse of a list is the original list``() =
  let revRevIsOrig (xs:list<int>) = List.rev(List.rev xs) = xs
  Check.QuickThrowOnFailure revRevIsOrig
  
[Fact]
public void RevRevIsOrig(){
    Prop.ForAll<int[]>(xs => xs.Reverse().Reverse().SequenceEqual(xs))
        .QuickCheckThrowOnFailure();
}

For xUnit, the test looks like any normal test, and the QuickThrowOnFailure ensures that if the test fails, an exception with the necessary information is raised so xUnit knows the test failed. The output of the test is the same as above.

Using FsCheck with xUnit.NET using the plugin

xUnit.NET is "blessed" with an FsCheck plugin. To use it, install the FsCheck.Xunit NuGet package. The test above can now be written more tersely as follows:

open FsCheck.Xunit

[<Property>]
let ``Reverse of reverse of a list is the original list ``(xs:list<int>) =
  List.rev(List.rev xs) = xs
  

xUnit now shows the test similarly to a regular test, and is able to run it directly.

To learn more on how to use this integration or integration with other frameworks like NUnit, see Running Tests.

namespace FsCheck
val revRevIsOrig : xs:int list -> bool

Full name: QuickStart.revRevIsOrig
val xs : int list
type 'T list = List<'T>

Full name: Microsoft.FSharp.Collections.list<_>
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
Multiple items
module List

from Microsoft.FSharp.Collections

--------------------
type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
  interface IEnumerable
  interface IEnumerable<'T>
  member GetSlice : startIndex:int option * endIndex:int option -> 'T list
  member Head : 'T
  member IsEmpty : bool
  member Item : index:int -> 'T with get
  member Length : int
  member Tail : 'T list
  static member Cons : head:'T * tail:'T list -> 'T list
  static member Empty : 'T list

Full name: Microsoft.FSharp.Collections.List<_>
val rev : list:'T list -> 'T list

Full name: Microsoft.FSharp.Collections.List.rev
val revIsOrig : xs:int list -> bool

Full name: QuickStart.revIsOrig
val revRevIsOrigFloat : xs:float list -> bool

Full name: QuickStart.revRevIsOrigFloat
val xs : float list
Multiple items
val float : value:'T -> float (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.float

--------------------
type float = System.Double

Full name: Microsoft.FSharp.Core.float

--------------------
type float<'Measure> = float

Full name: Microsoft.FSharp.Core.float<_>
namespace Expecto
val config : FsCheck.Config

Full name: QuickStart.config
type Config =
  {MaxTest: int;
   MaxFail: int;
   Replay: StdGen option;
   Name: string;
   StartSize: int;
   EndSize: int;
   QuietOnSuccess: bool;
   Every: int -> obj list -> string;
   EveryShrink: obj list -> string;
   Arbitrary: Type list;
   ...}
  static member Default : Config
  static member Quick : Config
  static member QuickThrowOnFailure : Config
  static member Verbose : Config
  static member VerboseThrowOnFailure : Config
  static member private throwingRunner : IRunner

Full name: FsCheck.Config
property FsCheck.Config.Default: FsCheck.Config
val properties : Test

Full name: QuickStart.properties
val testList : name:string -> tests:Test list -> Test

Full name: Expecto.Tests.testList
val a : int
val b : int
val c : int
Multiple items
module Tests

from Expecto

--------------------
type TestsAttribute =
  inherit Attribute
  new : unit -> TestsAttribute

Full name: Expecto.TestsAttribute

--------------------
new : unit -> TestsAttribute
val runTests : config:Impl.ExpectoConfig -> tests:Test -> int

Full name: Expecto.Tests.runTests
val defaultConfig : Impl.ExpectoConfig

Full name: Expecto.Tests.defaultConfig
namespace Xunit
val ( Reverse of reverse of a list is the original list ) : unit -> 'a

Full name: QuickStart.( Reverse of reverse of a list is the original list )
val revRevIsOrig : (int list -> bool)
namespace FsCheck.Xunit
val ( Reverse of reverse of a list is the original list ) : xs:int list -> bool

Full name: QuickStart.( Reverse of reverse of a list is the original list )
Fork me on GitHub