FsCheck


Running tests

This section describes the various ways in which you can run FsCheck tests:

  • FsCheck has a built-in test runner that is easy to invoke from F#/C# Interactive, commandline program or any test framework. It writes the result of tests to standard output, and you can configure the FsCheck runner to throw an exception on test failure to signal the failure to whichever test runner you use.

  • FsCheck.Xunit integrates FsCheck with xUnit.NET to allow you to specify properties in a terse way. Tests written this way look like native xUnit.NET tests, except they can take arguments.

  • FsCheck.NUnit integrates FsCheck with NUnit to allow you to specify properties in a terse way. Tests written this way look like native NUnit tests, except they can take arguments.

  • FsCheck allows you to register an IRunner implementation that it calls on each outcome and each individual test it runs. This allows a tighter integration with a specific test framework, and offers more control over printing of results.

Using the built-in test runner

Running a single property from an fsx/csx file or a command line runner

From F# the preferred way of running a property is to use the methods on Check; for C# the preferred way is to use the extension methods on the Property type.

Check.One or <property.Check> runs the tests for a single property. It takes as argument an instance of the Config type or Configuration type respectively, which allows you to configure many aspects of how the FsCheck test runner works, among others the size of the test data and the random seed used (so you can reproduce a run). The Config type is an F# record type so is the preferred way to use from F#; for other languages Configuration is a mutable version of it so is easier to use.

Most of the methods on Check and the extension methods like QuickCheck() are a short-hand for running a test with a particular configuration. For example, Check.Quick is equivalent to Check.One(Config.Quick, <property>); respectively <property>.QuickCheck() is a shorthand for <property>.Check(Configuration.Quick).

Take note of Config(uration).Verbose and Check.Verbose/VerboseCheck. These will print the arguments for each test, and are useful for example if your test loops infinitely on some inputs.

Also take note of Check.QuickThrowOnFailure/QuickCheckThrowOnFailure() and Check.VerboseThrowOnFailure/VerboseThrowOnFailure(). These are intended to be used from unit tests executed through an existing test runner. They'll call the FsCheck runner and configure it such that is throws on failure; in all frameworks we are aware of this signals that the test has failed and so is an easy way to integrate FsCheck tests with unit tests you may already have.

Here is an example of how to run a test with a similar configuration to Quick, but that runs 1000 tests and does not print to the output on success:

1: 
Check.One({ Config.Quick with MaxTest = 1000; QuietOnSuccess=true }, fun _ -> true)
1: 
2: 
3: 
4: 
var configuration = Configuration.Quick;
configuration.MaxNbOfTest = 1000;
configuration.QuietOnSuccess = true;
true.ToProperty().Check(configuration);

Running many properties at once with Check.All

Usually, you'll write more than one property to test. FsCheck allows you to group together properties as static members of a class:

1: 
2: 
3: 
type ListProperties =
  static member ``reverse of reverse is original`` (xs:list<int>) = List.rev(List.rev xs) = xs
  static member ``reverse is original`` (xs:list<int>) = List.rev xs = xs

These can be checked at once using:

1: 
Check.QuickAll<ListProperties>()

FsCheck now also prints the name of each test:

--- Checking ListProperties ---
ListProperties.reverse of reverse is original-Ok, passed 100 tests.
ListProperties.reverse is original-Falsifiable, after 1 test (1 shrink) (StdGen (686334613,296521705)):
Original:
[-1; 0]
Shrunk:
[1; 0]

Since all top level functions of a a module are also compiled as static member of a class with the name of the module, you can also use Check.QuickAll to test all the top level functions in a certain module. However, the type of a module is not directly accessible via F#, so you can use the following trick:

1: 
Check.QuickAll typeof<ListProperties>.DeclaringType

Notice also the counterpart of Check.Verbose: Check.VerboseAll.

Running tests using only modules

Arbitrary instances are given as static members of classes, and properties can be grouped together as static members of classes. Top level let functions are compiled as static member of their enclosing module (which is compiled as a class), and so you can simply define your properties and generators as top level let-bound functions, and then register all generators and and all properties at once using the following trick:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
let myprop (i:int) = i >= 0
let mygen = Arb.Default.Int32() |> Arb.mapFilter (fun i -> Math.Abs i) (fun i -> i >= 0)
let helper = "a string"
let private helper' = true

type Marker = class end
Arb.registerByType (typeof<Marker>.DeclaringType)
Check.QuickAll (typeof<Marker>.DeclaringType)
--- Checking FSI_0128 ---
FSI_0128.myprop-Ok, passed 100 tests.
FSI_0128.get_it-Ok, passed 100 tests.

The Marker type is just any type defined in the module, to be able to get to the module's Type. F# offers no way to get to a module's Type directly.

FsCheck determines the intent of the function based on its return type:

  • Properties: public functions that return unit, bool, Property or function of any arguments to those types or Lazy value of any of those types. So myprop is the only property that is run; helper' also returns bool but is private.
  • Arbitrary instances: return Arbitrary<_>

All other functions are respectfully ignored. If you have top level functions that return types that FsCheck will do something with, but do not want them checked or registered, just make them private. FsCheck will ignore those functions.

Using FsCheck.Xunit

To use the integration install the FsCheck.Xunit nuget package. Then:

1: 
2: 
open FsCheck
open FsCheck.Xunit

You can now attribute tests with PropertyAttribute (a subclass of xUnit.NET's FactAttribute). Unlike xUnit.NET's facts, these methods can take arguments and should return a property. FsCheck will be used to generate and shrink the arguments based on the type and the currently registered generators.

An FsCheck test fails from xUnit.NET's perspective if it finds a counter-example, or if the arguments are exhausted. It passes when FsCheck can execute 100 tests (or whatever the configured number of tests is) succesfully.

The PropertyAttribute allows you to customize how FsCheck will run for that method, similar to how you would use the Config type otherwise. For example:

1: 
2: 
3: 
[<Property>]
let ``abs(v) % k equals abs(v % k)`` v (NonZeroInt k) = 
    (abs v) % k = abs(v % k)

Likely one of the most useful configuration options of PropertyAttribute is the ability to register or override an Arbitrary instance just for that test method. You can also use the PropertiesAttribute (note the plural form) to set custom configuration options per class or per module basis. For example:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
type Positive =
    static member Double() =
        Arb.Default.Float()
        |> Arb.mapFilter abs (fun t -> t >= 0.0)

type Negative =
    static member Double() =
        Arb.Default.Float()
        |> Arb.mapFilter (abs >> ((-) 0.0)) (fun t -> t <= 0.0)

[<Properties( Arbitrary=[| typeof<Negative> |] )>]
module ModuleWithProperties =

    [<Property>]
    let ``should use Arb instances from enclosing module``(underTest:float) =
        underTest <= 0.0

    [<Property( Arbitrary=[| typeof<Positive> |] )>]
    let ``should use Arb instance on method``(underTest:float) =
        underTest >= 0.0

Using PropertiesAttribute and PropertyAttribute you can set any configuration. For example in following module:

  • property 1 would use default config + overriden MaxTest = 10 and EndSize = 10 from Properties attribute
  • property 2 would use default config + overriden EndSize = 10 from Properties attribute and MaxTest = 500 from Property attribute
  • property 3 would use default config + overriden MaxTest = 10 and EndSize = 10 from Properties attribute and Replay = "123,456" from Property attribute
 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
[<Properties(MaxTest = 10, EndSize = 10)>] 
module Properties =

    [<Property>]
    let ``property 1`` input =
        true

    [<Property(MaxTest = 500)>]
    let ``property 2`` input =
        true

    [<Property(Replay = "123,456")>]
    let ``property 3`` input =
        true

Using FsCheck.Xunit with TestDriven.Net

TestDriven.Net is a Visual Studio add-in that enables you to easily run a variety of tests while working with code in the IDE. Out of the box, TestDriven.Net can run tests written with FsCheck.Xunit.

However, the user experience may, by default, be slightly less than optimal. TestDriven.Net outputs the state of the test run in an unobtrusive manner, but if test failures occur, it outputs the result of the failing tests to Visual Studio's Output window. If you have Visual Studio configured in such a way that the Output window only appears if there actually is any output, you may be used to interpret the appearance of the Output window as a test failure.

Since the Output window also appears if anything is written to the console in general, this can produce false positives. If you're accustomed to interpret the appearance of the Output window as a sign of a test failure, it can be a little jarring that FsCheck by default always reports a bit of output on success.

If, for that, or other reasons, you want to disable output on success, you can do so:

1: 
2: 
3: 
[<Property(QuietOnSuccess = true)>]
let ``abs(v) % k equals abs(v % k) `` v (NonZeroInt k) = 
    (abs v) % k = abs(v % k)

Setting QuietOnSuccess = true only suppresses the output in case of success; in the case of test failures, output appears as normal.

Capturing output when using FactAttribute

xUnit 2 doesn't capture messages written to the console but instead provides ITestOutputHelper to capture output. ITestOutputHelper has a single method WriteLine and xUnit will automatically pass it in as a constructor argument. FsCheck.Xunit provides overloads for Property.QuickCheck, Property.QuickCheckThrowOnFailure, Property.VerboseCheck and Property.VerboseCheckThrowOnFailure that you can pass an ITestOutputHelper so that xUnit captures FsCheck messages:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
using System;
using FsCheck;
using FsCheck.Xunit;
using Xunit;
using Xunit.Abstractions;

public class Test
{
    private readonly ITestOutputHelper _TestOutputHelper;
    public Test(ITestOutputHelper testOutputHelper)
    {
        _TestOutputHelper = testOutputHelper;
    }

    [Fact]
    public void Test1()
    {
        Prop
            .ForAll(...)
            .VerboseCheckThrowOnFailure(_TestOutputHelper);
    }
}

Using FsCheck.NUnit

To use the integration with NUnit 3 install the FsCheck.NUnit nuget package. Make sure your project(s) has target .NET 4.5 or greater. Then open FsCheck.NUnit.

You can now attribute tests with PropertyAttribute (a subclass of NUnit's TestAttribute). Unlike NUnit tests, these methods can take arguments and should return a property. FsCheck will be used to generate and shrink the arguments based on the type and the currently registered generators.

An FsCheck test fails from NUnit's perspective if it finds a counter-example, or if the arguments are exhausted. It passes when FsCheck can execute 100 tests (or whatever the configured number of tests is) succesfully.

The PropertyAttribute allows you to customize how FsCheck will run for that method, similar to how you would use the Config type otherwise.

1: 
2: 
3: 
4: 
5: 
open FsCheck.NUnit

[<Property>]
let ``Reverse of reverse of a list is the original list ``(xs:list<int>) =
  List.rev(List.rev xs) = xs

Note: the NUnit integration doesn't have the ability, like FsCheck.Xunit, to override Arbitrary instances on a per class or per module basis. Otherwise, it is very similar.

Implementing IRunner

Example 1: to integrate FsCheck with other unit testing frameworks

The Config type that can be passed to the Check.One or Check.All methods takes an IRunner as argument. This i nterface has the following methods:

  • OnStartFixture is called when FsCheck is testing all the methods on a type, before starting any tests.
  • OnArguments is called after every test, passing the implementation the test number, the arguments and the every function.
  • OnShrink is called at every succesful shrink.
  • OnFinished is called with the name of the test and the outcome of the overall test run. This is used in the example below to call Assert statements from a particular unit testing framework - allowing FsCheck to integrate easily. You can leverage another unit testing framework's ability to setup and tear down tests, have a nice graphical runner etc.
 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
let testRunner =
  { new IRunner with
      member __.OnStartFixture t = ()
      member __.OnArguments (ntest,args, every) = ()
      member __.OnShrink(args, everyShrink) = ()
      member __.OnFinished(name,testResult) = 
          match testResult with 
          | TestResult.True _ -> () //let the test runner know that the test passed
          | _ -> () // test failed, or other problem. Notify test runner. Runner.onFinishedToString name testResult
  }
   
let withxUnitConfig = { Config.Default with Runner = testRunner }

Example 2: to customize printing of generated arguments

By default, FsCheck prints generated arguments using sprintf "%A", or structured formatting. This usually does what you expect, i.e. for primitive types the value, for objects the ToString override and so on. If it does not (A motivating case is testing with COM objects - overriding ToString is not an option and structured formatting does not do anything useful with it), you can use the label combinator to solve this on a per property basis, but a more structured solution can be achieved by implementing IRunner. For example:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
let formatterRunner formatter =
  { new IRunner with
      member x.OnStartFixture t =
          printf "%s" (Runner.onStartFixtureToString t)
      member x.OnArguments (ntest,args, every) =
          printf "%s" (every ntest (args |> List.map formatter))
      member x.OnShrink(args, everyShrink) =
          printf "%s" (everyShrink (args |> List.map formatter))
      member x.OnFinished(name,testResult) = 
          let testResult' = match testResult with 
                              | TestResult.False (testData,origArgs,shrunkArgs,outCome,seed) -> 
                                  TestResult.False (testData,origArgs |> List.map formatter, shrunkArgs |> List.map formatter,outCome,seed)
                              | t -> t
          printf "%s" (Runner.onFinishedToString name testResult') 
  }

Using FsCheck.NUnit with TestDriven.Net

TestDriven.Net is a Visual Studio add-in that enables you to easily run a variety of tests while working with code in the IDE. Out of the box, TestDriven.Net can run tests written with FsCheck.NUnit.

However, the user experience may, by default, be slightly less than optimal. TestDriven.Net outputs the state of the test run in an unobtrusive manner, but if test failures occur, it outputs the result of the failing tests to Visual Studio's Output window. If you have Visual Studio configured in such a way that the Output window only appears if there actually is any output, you may be used to interpret the appearance of the Output window as a test failure.

Since the Output window also appears if anything is written to the console in general, this can produce false positives. If you're accustomed to interpret the appearance of the Output window as a sign of a test failure, it can be a little jarring that FsCheck by default always reports a bit of output on success.

If, for that, or other reasons, you want to disable output on success, you can do so:

1: 
2: 
3: 
[<Property(QuietOnSuccess = true)>]
let revUnit (x:char) = 
    List.rev [x] = [x]

Setting QuietOnSuccess = true only suppresses the output in case of success; in the case of test failures, output appears as normal.

namespace FsCheck
namespace System
type Check =
  static member All : config:Config -> unit
  static member All : config:Config * test:Type -> unit
  static member Method : config:Config * methodInfo:MethodInfo * ?target:obj -> unit
  static member One : config:Config * property:'Testable -> unit
  static member One : name:string * config:Config * property:'Testable -> unit
  static member Quick : property:'Testable -> unit
  static member Quick : name:string * property:'Testable -> unit
  static member QuickAll : unit -> unit
  static member QuickAll : test:Type -> unit
  static member QuickThrowOnFailure : property:'Testable -> unit
  ...
static member Check.One : config:Config * property:'Testable -> unit
static member Check.One : name:string * config:Config * property:'Testable -> unit
type Config =
  {MaxTest: int;
   MaxFail: int;
   Replay: StdGen option;
   Name: string;
   StartSize: int;
   EndSize: int;
   QuietOnSuccess: bool;
   Every: int -> obj list -> string;
   EveryShrink: obj list -> string;
   Arbitrary: Type list;
   ...}
    static member Default : Config
    static member Quick : Config
    static member QuickThrowOnFailure : Config
    static member Verbose : Config
    static member VerboseThrowOnFailure : Config
    static member private throwingRunner : IRunner
property Config.Quick: Config
val xs : int list
type 'T list = List<'T>
Multiple items
val int : value:'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int
Multiple items
module List

from Microsoft.FSharp.Collections

--------------------
type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
    interface IReadOnlyCollection<'T>
    interface IEnumerable
    interface IEnumerable<'T>
    member GetSlice : startIndex:int option * endIndex:int option -> 'T list
    member Head : 'T
    member IsEmpty : bool
    member Item : index:int -> 'T with get
    member Length : int
    member Tail : 'T list
    static member Cons : head:'T * tail:'T list -> 'T list
    ...
val rev : list:'T list -> 'T list
static member Check.QuickAll : unit -> unit
static member Check.QuickAll : test:Type -> unit
type ListProperties =
  static member ( reverse is original ) : xs:int list -> bool
  static member ( reverse of reverse is original ) : xs:int list -> bool
val typeof<'T> : Type
val myprop : i:int -> bool
val i : int
val mygen : Arbitrary<int>
module Arb

from FsCheck
type Default =
  static member Array : unit -> Arbitrary<'a []>
  static member Array2D : unit -> Arbitrary<'a0 [,]>
  static member Arrow : unit -> Arbitrary<('a -> 'b)> (requires equality)
  static member BigInt : unit -> Arbitrary<bigint>
  static member Bool : unit -> Arbitrary<bool>
  static member Byte : unit -> Arbitrary<byte>
  static member Char : unit -> Arbitrary<char>
  static member Culture : unit -> Arbitrary<CultureInfo>
  static member DateTime : unit -> Arbitrary<DateTime>
  static member DateTimeOffset : unit -> Arbitrary<DateTimeOffset>
  ...
static member Arb.Default.Int32 : unit -> Arbitrary<int>
val mapFilter : mapper:('a -> 'a) -> pred:('a -> bool) -> a:Arbitrary<'a> -> Arbitrary<'a>
type Math =
  static val PI : float
  static val E : float
  static member Abs : value:sbyte -> sbyte + 6 overloads
  static member Acos : d:float -> float
  static member Asin : d:float -> float
  static member Atan : d:float -> float
  static member Atan2 : y:float * x:float -> float
  static member BigMul : a:int * b:int -> int64
  static member Ceiling : d:decimal -> decimal + 1 overload
  static member Cos : d:float -> float
  ...
Math.Abs(value: decimal) : decimal
Math.Abs(value: float) : float
Math.Abs(value: float32) : float32
Math.Abs(value: int64) : int64
Math.Abs(value: int) : int
Math.Abs(value: int16) : int16
Math.Abs(value: sbyte) : sbyte
val helper : string
val private helper' : bool
type Marker
val registerByType : t:Type -> TypeClass.TypeClassComparison
namespace FsCheck.Xunit
Multiple items
type Property =
  private | Property of Gen<Rose<Result>>
    static member private GetGen : Property -> Gen<Rose<Result>>

--------------------
type PropertyAttribute =
  inherit FactAttribute
  new : unit -> PropertyAttribute
  member Arbitrary : Type []
  member private Config : PropertyConfig
  member EndSize : int
  member MaxFail : int
  member MaxTest : int
  member QuietOnSuccess : bool
  member Replay : string
  member StartSize : int
  ...

--------------------
new : unit -> PropertyAttribute
val ( abs(v) % k equals abs(v % k) ) : v:int -> NonZeroInt -> bool
val v : int
Multiple items
union case NonZeroInt.NonZeroInt: int -> NonZeroInt

--------------------
type NonZeroInt =
  | NonZeroInt of int
    override ToString : unit -> string
    member Get : int
    static member op_Explicit : NonZeroInt -> int
val k : int
val abs : value:'T -> 'T (requires member Abs)
type Double =
  struct
    member CompareTo : value:obj -> int + 1 overload
    member Equals : obj:obj -> bool + 1 overload
    member GetHashCode : unit -> int
    member GetTypeCode : unit -> TypeCode
    member ToString : unit -> string + 3 overloads
    static val MinValue : float
    static val MaxValue : float
    static val Epsilon : float
    static val NegativeInfinity : float
    static val PositiveInfinity : float
    ...
  end
static member Arb.Default.Float : unit -> Arbitrary<double>
val t : double
Multiple items
type PropertiesAttribute =
  inherit PropertyAttribute
  new : unit -> PropertiesAttribute

--------------------
new : unit -> PropertiesAttribute
Multiple items
type Arbitrary<'a> =
  interface IArbitrary
  new : unit -> Arbitrary<'a>
  abstract member Shrinker : 'a -> seq<'a>
  abstract member Generator : Gen<'a>
  override Shrinker : 'a -> seq<'a>

--------------------
new : unit -> Arbitrary<'a>
type Negative =
  static member Double : unit -> Arbitrary<double>
val ( should use Arb instances from enclosing module ) : underTest:float -> bool
val underTest : float
Multiple items
val float : value:'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float
type Positive =
  static member Double : unit -> Arbitrary<double>
val ( should use Arb instance on method ) : underTest:float -> bool
val ( property 1 ) : input:'a -> bool
val input : 'a
val ( property 2 ) : input:'a -> bool
val ( property 3 ) : input:'a -> bool
val ( abs(v) % k equals abs(v % k) ) : v:int -> NonZeroInt -> bool
val using : resource:'T -> action:('T -> 'U) -> 'U (requires 'T :> System.IDisposable)
namespace FsCheck.NUnit
Multiple items
type Property =
  private | Property of Gen<Rose<Result>>
    static member private GetGen : Property -> Gen<Rose<Result>>

--------------------
type PropertyAttribute =
  inherit TestAttribute
  interface IWrapTestMethod
  interface ISimpleTestBuilder
  new : unit -> PropertyAttribute
  member Arbitrary : Type []
  member EndSize : int
  member MaxFail : int
  member MaxTest : int
  member QuietOnSuccess : bool
  member Replay : string
  ...

--------------------
new : unit -> PropertyAttribute
val ( Reverse of reverse of a list is the original list ) : xs:int list -> bool
val testRunner : IRunner
type IRunner =
  interface
    abstract member OnArguments : int * obj list * (int -> obj list -> string) -> unit
    abstract member OnFinished : string * TestResult -> unit
    abstract member OnShrink : obj list * (obj list -> string) -> unit
    abstract member OnStartFixture : Type -> unit
  end
val t : Type
val __ : IRunner
abstract member IRunner.OnArguments : int * obj list * (int -> obj list -> string) -> unit
val ntest : int
val args : obj list
val every : (int -> obj list -> string)
abstract member IRunner.OnShrink : obj list * (obj list -> string) -> unit
val everyShrink : (obj list -> string)
abstract member IRunner.OnFinished : string * TestResult -> unit
val name : string
val testResult : TestResult
type TestResult =
  | True of TestData * bool
  | False of TestData * obj list * obj list * Outcome * StdGen
  | Exhausted of TestData
union case TestResult.True: TestData * bool -> TestResult
val withxUnitConfig : Config
property Config.Default: Config
module Runner

from FsCheck
val formatterRunner : formatter:(obj -> obj) -> IRunner
val formatter : (obj -> obj)
val x : IRunner
abstract member IRunner.OnStartFixture : Type -> unit
val printf : format:Printf.TextWriterFormat<'T> -> 'T
val onStartFixtureToString : t:Type -> string
val map : mapping:('T -> 'U) -> list:'T list -> 'U list
val testResult' : TestResult
union case TestResult.False: TestData * obj list * obj list * Outcome * Random.StdGen -> TestResult
val testData : TestData
val origArgs : obj list
val shrunkArgs : obj list
val outCome : Outcome
val seed : Random.StdGen
val t : TestResult
val onFinishedToString : name:string -> testResult:TestResult -> string
val revUnit : x:char -> bool
val x : char
Multiple items
val char : value:'T -> char (requires member op_Explicit)

--------------------
type char = Char
Fork me on GitHub