FsCheck


Running tests

This section describes the various ways in which you can run FsCheck tests:

Using the built-in test runner

Running a single property from an fsx/csx file or a command line runner

From F# the preferred way of running a property is to use the methods on Check; for C# the preferred way is to use the extension methods on the Property type.

Check.One or <property.Check> runs the tests for a single property. It takes as argument an instance of the Config type or Configuration type respectively, which allows you to configure many aspects of how the FsCheck test runner works, among others the size of the test data and the random seed used (so you can reproduce a run). The Config type is an F# record type so is the preferred way to use from F#; for other languages Configuration is a mutable version of it so is easier to use.

Most of the methods on Check and the extension methods like QuickCheck() are a short-hand for running a test with a particular configuration. For example, Check.Quick is equivalent to Check.One(Config.Quick, <property>); respectively <property>.QuickCheck() is a shorthand for <property>.Check(Configuration.Quick).

Take note of Config(uration).Verbose and Check.Verbose/VerboseCheck. These will print the arguments for each test, and are useful for example if your test loops infinitely on some inputs.

Also take note of Check.QuickThrowOnFailure/QuickCheckThrowOnFailure() and Check.VerboseThrowOnFailure/VerboseThrowOnFailure(). These are intended to be used from unit tests executed through an existing test runner. They'll call the FsCheck runner and configure it such that is throws on failure; in all frameworks we are aware of this signals that the test has failed and so is an easy way to integrate FsCheck tests with unit tests you may already have.

Here is an example of how to run a test with a similar configuration to Quick, but that runs 1000 tests and does not print to the output on success:

Check.One({ Config.Quick with MaxTest = 1000; QuietOnSuccess=true }, fun _ -> true)
var configuration = Configuration.Quick;
configuration.MaxNbOfTest = 1000;
configuration.QuietOnSuccess = true;
true.ToProperty().Check(configuration);

Running many properties at once with Check.All

Usually, you'll write more than one property to test. FsCheck allows you to group together properties as static members of a class:

type ListProperties =
  static member ``reverse of reverse is original`` (xs:list<int>) = List.rev(List.rev xs) = xs
  static member ``reverse is original`` (xs:list<int>) = List.rev xs = xs

These can be checked at once using:

Check.QuickAll<ListProperties>()

FsCheck now also prints the name of each test:

--- Checking ListProperties ---
ListProperties.reverse of reverse is original-Ok, passed 100 tests.
ListProperties.reverse is original-Falsifiable, after 1 test (0 shrinks) (StdGen (928367725, 297043896)):
Original:
[1; 0]

Since all top level functions of a a module are also compiled as static member of a class with the name of the module, you can also use Check.QuickAll to test all the top level functions in a certain module. However, the type of a module is not directly accessible via F#, so you can use the following trick:

Check.QuickAll typeof<ListProperties>.DeclaringType

Notice also the counterpart of Check.Verbose: Check.VerboseAll.

Running tests using only modules

Arbitrary instances are given as static members of classes, and properties can be grouped together as static members of classes. Top level let functions are compiled as static member of their enclosing module (which is compiled as a class), and so you can simply define your properties and generators as top level let-bound functions, and then register all generators and and all properties at once using the following trick:

let myprop (i:int) = i >= 0
let mygen = Arb.Default.Int32() |> Arb.mapFilter (fun i -> Math.Abs i) (fun i -> i >= 0)
let helper = "a string"
let private helper' = true

type Marker = class end
Arb.registerByType (typeof<Marker>.DeclaringType)
Check.QuickAll (typeof<Marker>.DeclaringType)
--- Checking FSI_0124 ---
FSI_0124.myprop-Ok, passed 100 tests.
FSI_0124.get_it-Ok, passed 100 tests.

The Marker type is just any type defined in the module, to be able to get to the module's Type. F# offers no way to get to a module's Type directly.

FsCheck determines the intent of the function based on its return type:

All other functions are respectfully ignored. If you have top level functions that return types that FsCheck will do something with, but do not want them checked or registered, just make them private. FsCheck will ignore those functions.

Using FsCheck.Xunit

To use the integration install the FsCheck.Xunit nuget package. Then:

open FsCheck
open FsCheck.Xunit

You can now attribute tests with PropertyAttribute (a subclass of xUnit.NET's FactAttribute). Unlike xUnit.NET's facts, these methods can take arguments and should return a property. FsCheck will be used to generate and shrink the arguments based on the type and the currently registered generators.

An FsCheck test fails from xUnit.NET's perspective if it finds a counter-example, or if the arguments are exhausted. It passes when FsCheck can execute 100 tests (or whatever the configured number of tests is) succesfully.

The PropertyAttribute allows you to customize how FsCheck will run for that method, similar to how you would use the Config type otherwise. For example:

[<Property>]
let ``abs(v) % k equals abs(v % k)`` v (NonZeroInt k) = 
    (abs v) % k = abs(v % k)

Likely one of the most useful configuration options of PropertyAttribute is the ability to register or override an Arbitrary instance just for that test method. You can also use the PropertiesAttribute (note the plural form) to set custom configuration options per class, per module, or per assembly. For example:

type Positive =
    static member Double() =
        Arb.Default.Float()
        |> Arb.mapFilter abs (fun t -> t > 0.0)

type Negative =
    static member Double() =
        Arb.Default.Float()
        |> Arb.mapFilter (abs >> ((-) 0.0)) (fun t -> t < 0.0)

type Zero =
    static member Double() =
        0.0
        |> Gen.constant
        |> Arb.fromGen

[<assembly: Properties( Arbitrary = [| typeof<Zero> |] )>] do()

module ModuleWithoutProperties =

    [<Property>]
    let ``should use Arb instances from assembly``(underTest:float) =
        underTest = 0.0

    [<Property( Arbitrary=[| typeof<Positive> |] )>]
    let ``should use Arb instance on method``(underTest:float) =
        underTest > 0.0

[<Properties( Arbitrary=[| typeof<Negative> |] )>]
module ModuleWithProperties =

    [<Property>]
    let ``should use Arb instances from enclosing module``(underTest:float) =
        underTest < 0.0

    [<Property( Arbitrary=[| typeof<Positive> |] )>]
    let ``should use Arb instance on method``(underTest:float) =
        underTest > 0.0

Using PropertiesAttribute and PropertyAttribute you can set any configuration. For example in following module:

[<Properties(MaxTest = 10, EndSize = 10)>] 
module Properties =

    [<Property>]
    let ``property 1`` input =
        true

    [<Property(MaxTest = 500)>]
    let ``property 2`` input =
        true

    [<Property(Replay = "123,456")>]
    let ``property 3`` input =
        true

Using FsCheck.Xunit with TestDriven.Net

TestDriven.Net is a Visual Studio add-in that enables you to easily run a variety of tests while working with code in the IDE. Out of the box, TestDriven.Net can run tests written with FsCheck.Xunit.

However, the user experience may, by default, be slightly less than optimal. TestDriven.Net outputs the state of the test run in an unobtrusive manner, but if test failures occur, it outputs the result of the failing tests to Visual Studio's Output window. If you have Visual Studio configured in such a way that the Output window only appears if there actually is any output, you may be used to interpret the appearance of the Output window as a test failure.

Since the Output window also appears if anything is written to the console in general, this can produce false positives. If you're accustomed to interpret the appearance of the Output window as a sign of a test failure, it can be a little jarring that FsCheck by default always reports a bit of output on success.

If, for that, or other reasons, you want to disable output on success, you can do so:

[<Property(QuietOnSuccess = true)>]
let ``abs(v) % k equals abs(v % k) `` v (NonZeroInt k) = 
    (abs v) % k = abs(v % k)

Setting QuietOnSuccess = true only suppresses the output in case of success; in the case of test failures, output appears as normal.

Capturing output when using FactAttribute

xUnit 2 doesn't capture messages written to the console but instead provides ITestOutputHelper to capture output. ITestOutputHelper has a single method WriteLine and xUnit will automatically pass it in as a constructor argument. FsCheck.Xunit provides overloads for Property.QuickCheck, Property.QuickCheckThrowOnFailure, Property.VerboseCheck and Property.VerboseCheckThrowOnFailure that you can pass an ITestOutputHelper so that xUnit captures FsCheck messages:

using System;
using FsCheck;
using FsCheck.Xunit;
using Xunit;
using Xunit.Abstractions;

public class Test
{
    private readonly ITestOutputHelper _TestOutputHelper;
    public Test(ITestOutputHelper testOutputHelper)
    {
        _TestOutputHelper = testOutputHelper;
    }

    [Fact]
    public void Test1()
    {
        Prop
            .ForAll(...)
            .VerboseCheckThrowOnFailure(_TestOutputHelper);
    }
}

Using FsCheck.NUnit

To use the integration with NUnit 3 install the FsCheck.NUnit nuget package. Make sure your project(s) has target .NET 4.5 or greater. Then open FsCheck.NUnit.

You can now attribute tests with PropertyAttribute (a subclass of NUnit's TestAttribute). Unlike NUnit tests, these methods can take arguments and should return a property. FsCheck will be used to generate and shrink the arguments based on the type and the currently registered generators.

An FsCheck test fails from NUnit's perspective if it finds a counter-example, or if the arguments are exhausted. It passes when FsCheck can execute 100 tests (or whatever the configured number of tests is) succesfully.

The PropertyAttribute allows you to customize how FsCheck will run for that method, similar to how you would use the Config type otherwise.

open FsCheck.NUnit

[<Property>]
let ``Reverse of reverse of a list is the original list ``(xs:list<int>) =
  List.rev(List.rev xs) = xs

Note: the NUnit integration doesn't have the ability, like FsCheck.Xunit, to override Arbitrary instances on a per class or per module basis. Otherwise, it is very similar.

Implementing IRunner

Example 1: to integrate FsCheck with other unit testing frameworks

The Config type that can be passed to the Check.One or Check.All methods takes an IRunner as argument. This i nterface has the following methods:

let testRunner =
  { new IRunner with
      member __.OnStartFixture t = ()
      member __.OnArguments (ntest,args, every) = ()
      member __.OnShrink(args, everyShrink) = ()
      member __.OnFinished(name,testResult) = 
          match testResult with 
          | TestResult.True _ -> () //let the test runner know that the test passed
          | _ -> () // test failed, or other problem. Notify test runner. Runner.onFinishedToString name testResult
  }
   
let withxUnitConfig = { Config.Default with Runner = testRunner }

Example 2: to customize printing of generated arguments

By default, FsCheck prints generated arguments using sprintf "%A", or structured formatting. This usually does what you expect, i.e. for primitive types the value, for objects the ToString override and so on. If it does not (A motivating case is testing with COM objects - overriding ToString is not an option and structured formatting does not do anything useful with it), you can use the label combinator to solve this on a per property basis, but a more structured solution can be achieved by implementing IRunner. For example:

let formatterRunner formatter =
  { new IRunner with
      member x.OnStartFixture t =
          printf "%s" (Runner.onStartFixtureToString t)
      member x.OnArguments (ntest,args, every) =
          printf "%s" (every ntest (args |> List.map formatter))
      member x.OnShrink(args, everyShrink) =
          printf "%s" (everyShrink (args |> List.map formatter))
      member x.OnFinished(name,testResult) = 
          let testResult' = match testResult with 
                              | TestResult.False (testData,origArgs,shrunkArgs,outCome,seed) -> 
                                  TestResult.False (testData,origArgs |> List.map formatter, shrunkArgs |> List.map formatter,outCome,seed)
                              | t -> t
          printf "%s" (Runner.onFinishedToString name testResult') 
  }

Using FsCheck.NUnit with TestDriven.Net

TestDriven.Net is a Visual Studio add-in that enables you to easily run a variety of tests while working with code in the IDE. Out of the box, TestDriven.Net can run tests written with FsCheck.NUnit.

However, the user experience may, by default, be slightly less than optimal. TestDriven.Net outputs the state of the test run in an unobtrusive manner, but if test failures occur, it outputs the result of the failing tests to Visual Studio's Output window. If you have Visual Studio configured in such a way that the Output window only appears if there actually is any output, you may be used to interpret the appearance of the Output window as a test failure.

Since the Output window also appears if anything is written to the console in general, this can produce false positives. If you're accustomed to interpret the appearance of the Output window as a sign of a test failure, it can be a little jarring that FsCheck by default always reports a bit of output on success.

If, for that, or other reasons, you want to disable output on success, you can do so:

[<Property(QuietOnSuccess = true)>]
let revUnit (x:char) = 
    List.rev [x] = [x]

Setting QuietOnSuccess = true only suppresses the output in case of success; in the case of test failures, output appears as normal.

namespace FsCheck
namespace System
type Check =
  static member All : config:Config * test:Type -> unit + 1 overload
  static member Method : config:Config * methodInfo:MethodInfo * ?target:obj -> unit
  static member One : config:Config * property:'Testable -> unit + 1 overload
  static member Quick : property:'Testable -> unit + 1 overload
  static member QuickAll : test:Type -> unit + 1 overload
  static member QuickThrowOnFailure : property:'Testable -> unit
  static member QuickThrowOnFailureAll : test:Type -> unit + 1 overload
  static member Verbose : property:'Testable -> unit + 1 overload
  static member VerboseAll : test:Type -> unit + 1 overload
  static member VerboseThrowOnFailure : property:'Testable -> unit
  ...
static member Check.One : config:Config * property:'Testable -> unit
static member Check.One : name:string * config:Config * property:'Testable -> unit
type Config =
  { MaxTest: int
    MaxFail: int
    Replay: StdGen option
    Name: string
    StartSize: int
    EndSize: int
    QuietOnSuccess: bool
    Every: int -> obj list -> string
    EveryShrink: obj list -> string
    Arbitrary: Type list
    ... }
    static member Default : Config
    static member Quick : Config
    static member QuickThrowOnFailure : Config
    static member Verbose : Config
    static member VerboseThrowOnFailure : Config
    static member private throwingRunner : IRunner
property Config.Quick: Config with get
val xs : int list
type 'T list = List<'T>
Multiple items
val int : value:'T -> int (requires member op_Explicit)

--------------------
[<Struct>]
type int = int32

--------------------
type int<'Measure> =
  int
Multiple items
module List

from Microsoft.FSharp.Collections

--------------------
type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
    interface IReadOnlyList<'T>
    interface IReadOnlyCollection<'T>
    interface IEnumerable
    interface IEnumerable<'T>
    member GetReverseIndex : rank:int * offset:int -> int
    member GetSlice : startIndex:int option * endIndex:int option -> 'T list
    static member Cons : head:'T * tail:'T list -> 'T list
    member Head : 'T
    member IsEmpty : bool
    member Item : index:int -> 'T with get
    ...
val rev : list:'T list -> 'T list
static member Check.QuickAll : unit -> unit
static member Check.QuickAll : test:Type -> unit
type ListProperties =
  static member ( reverse is original ) : xs:int list -> bool
  static member ( reverse of reverse is original ) : xs:int list -> bool
val typeof<'T> : Type
val myprop : i:int -> bool
val i : int
val mygen : Arbitrary<int>
module Arb

from FsCheck
type Default =
  static member Array : unit -> Arbitrary<'a []>
  static member Array2D : unit -> Arbitrary<'a0 [,]>
  static member Arrow : unit -> Arbitrary<('a -> 'b)> (requires equality)
  static member BigInt : unit -> Arbitrary<bigint>
  static member Bool : unit -> Arbitrary<bool>
  static member Byte : unit -> Arbitrary<byte>
  static member Char : unit -> Arbitrary<char>
  static member Complex : unit -> Arbitrary<Complex>
  static member ConsoleKeyInfo : unit -> Arbitrary<ConsoleKeyInfo>
  static member Culture : unit -> Arbitrary<CultureInfo>
  ...
static member Arb.Default.Int32 : unit -> Arbitrary<int>
val mapFilter : mapper:('a -> 'a) -> pred:('a -> bool) -> a:Arbitrary<'a> -> Arbitrary<'a>
type Math =
  static member Abs : value: decimal -> decimal + 6 overloads
  static member Acos : d: float -> float
  static member Acosh : d: float -> float
  static member Asin : d: float -> float
  static member Asinh : d: float -> float
  static member Atan : d: float -> float
  static member Atan2 : y: float * x: float -> float
  static member Atanh : d: float -> float
  static member BigMul : a: int * b: int -> int64 + 2 overloads
  static member BitDecrement : x: float -> float
  ...
Math.Abs(value: float32) : float32
Math.Abs(value: sbyte) : sbyte
Math.Abs(value: int64) : int64
Math.Abs(value: int) : int
Math.Abs(value: int16) : int16
Math.Abs(value: float) : float
Math.Abs(value: decimal) : decimal
val helper : string
val private helper' : bool
type Marker
val registerByType : t:Type -> TypeClass.TypeClassComparison
namespace FsCheck.Xunit
Multiple items
type Property =
  private | Property of Gen<Rose<Result>>
    static member private GetGen : Property -> Gen<Rose<Result>>

--------------------
type PropertyAttribute =
  inherit FactAttribute
  new : unit -> PropertyAttribute
  member Arbitrary : Type []
  member private Config : PropertyConfig
  member EndSize : int
  member MaxFail : int
  member MaxTest : int
  member QuietOnSuccess : bool
  member Replay : string
  member StartSize : int
  ...

--------------------
new : unit -> PropertyAttribute
val ( abs(v) % k equals abs(v % k) ) : v:int -> NonZeroInt -> bool
val v : int
Multiple items
union case NonZeroInt.NonZeroInt: int -> NonZeroInt

--------------------
type NonZeroInt =
  | NonZeroInt of int
    override ToString : unit -> string
    static member op_Explicit : NonZeroInt -> int
    member Get : int
val k : int
val abs : value:'T -> 'T (requires member Abs)
[<Struct>]
type Double =
  member CompareTo : value: float -> int + 1 overload
  member Equals : obj: float -> bool + 1 overload
  member GetHashCode : unit -> int
  member GetTypeCode : unit -> TypeCode
  member System.IConvertible.ToBoolean : provider: IFormatProvider -> bool
  member System.IConvertible.ToByte : provider: IFormatProvider -> byte
  member System.IConvertible.ToChar : provider: IFormatProvider -> char
  member System.IConvertible.ToDateTime : provider: IFormatProvider -> DateTime
  member System.IConvertible.ToDecimal : provider: IFormatProvider -> decimal
  member System.IConvertible.ToDouble : provider: IFormatProvider -> float
  ...
static member Arb.Default.Float : unit -> Arbitrary<float>
val t : float
Multiple items
module Gen

from FsCheck

--------------------
type Gen<'a> =
  private | Gen of (int -> StdGen -> 'a)
    interface IGen
    member private Map : f:('a -> 'b) -> Gen<'b>
    static member ( <!> ) : f:('a1 -> 'a2) * a:Gen<'a1> -> Gen<'a2>
    static member ( <*> ) : f:Gen<('a1 -> 'a2)> * a:Gen<'a1> -> Gen<'a2>
    static member ( >>= ) : m:Gen<'a1> * k:('a1 -> Gen<'a2>) -> Gen<'a2>
val constant : v:'b -> Gen<'b>
val fromGen : gen:Gen<'Value> -> Arbitrary<'Value>
Multiple items
type PropertiesAttribute =
  inherit PropertyAttribute
  new : unit -> PropertiesAttribute

--------------------
new : unit -> PropertiesAttribute
Multiple items
type Arbitrary<'a> =
  interface IArbitrary
  new : unit -> Arbitrary<'a>
  abstract member Shrinker : 'a -> seq<'a> + 1 overload
  abstract member Generator : Gen<'a>

--------------------
new : unit -> Arbitrary<'a>
type Zero =
  static member Double : unit -> Arbitrary<float>
val ( should use Arb instances from assembly ) : underTest:float -> bool
val underTest : float
Multiple items
val float : value:'T -> float (requires member op_Explicit)

--------------------
[<Struct>]
type float = Double

--------------------
type float<'Measure> =
  float
type Positive =
  static member Double : unit -> Arbitrary<float>
val ( should use Arb instance on method ) : underTest:float -> bool
type Negative =
  static member Double : unit -> Arbitrary<float>
val ( should use Arb instances from enclosing module ) : underTest:float -> bool
val ( property 1 ) : input:'a -> bool
val input : 'a
val ( property 2 ) : input:'a -> bool
val ( property 3 ) : input:'a -> bool
val ( abs(v) % k equals abs(v % k) ) : v:int -> NonZeroInt -> bool
val using : resource:'T -> action:('T -> 'U) -> 'U (requires 'T :> System.IDisposable)
namespace FsCheck.NUnit
Multiple items
type Property =
  private | Property of Gen<Rose<Result>>
    static member private GetGen : Property -> Gen<Rose<Result>>

--------------------
type PropertyAttribute =
  inherit TestAttribute
  interface IWrapTestMethod
  interface ISimpleTestBuilder
  new : unit -> PropertyAttribute
  member Arbitrary : Type []
  member EndSize : int
  member MaxFail : int
  member MaxTest : int
  member QuietOnSuccess : bool
  member Replay : string
  ...

--------------------
new : unit -> PropertyAttribute
val ( Reverse of reverse of a list is the original list ) : xs:int list -> bool
val testRunner : IRunner
type IRunner =
  abstract member OnArguments : int * obj list * (int -> obj list -> string) -> unit
  abstract member OnFinished : string * TestResult -> unit
  abstract member OnShrink : obj list * (obj list -> string) -> unit
  abstract member OnStartFixture : Type -> unit
val t : Type
val __ : IRunner
abstract member IRunner.OnArguments : int * obj list * (int -> obj list -> string) -> unit
val ntest : int
val args : obj list
val every : (int -> obj list -> string)
abstract member IRunner.OnShrink : obj list * (obj list -> string) -> unit
val everyShrink : (obj list -> string)
abstract member IRunner.OnFinished : string * TestResult -> unit
val name : string
val testResult : TestResult
type TestResult =
  | True of TestData * bool
  | False of TestData * obj list * obj list * Outcome * StdGen
  | Exhausted of TestData
union case TestResult.True: TestData * bool -> TestResult
val withxUnitConfig : Config
property Config.Default: Config with get
module Runner

from FsCheck
val formatterRunner : formatter:(obj -> obj) -> IRunner
val formatter : (obj -> obj)
val x : IRunner
abstract member IRunner.OnStartFixture : Type -> unit
val printf : format:Printf.TextWriterFormat<'T> -> 'T
val onStartFixtureToString : t:Type -> string
val map : mapping:('T -> 'U) -> list:'T list -> 'U list
val testResult' : TestResult
union case TestResult.False: TestData * obj list * obj list * Outcome * Random.StdGen -> TestResult
val testData : TestData
val origArgs : obj list
val shrunkArgs : obj list
val outCome : Outcome
val seed : Random.StdGen
val t : TestResult
val onFinishedToString : name:string -> testResult:TestResult -> string
val revUnit : x:char -> bool
val x : char
Multiple items
val char : value:'T -> char (requires member op_Explicit)

--------------------
[<Struct>]
type char = Char