FsCheck


Tips and Tricks

Properties of functions

Perhaps surprisingly, FsCheck can generate random functions, Func and Actions. As a result, it can check properties of functions. For example, we can check associativity of function composition as follows:

let associativity (x:int) (f:int->float,g:float->char,h:char->int) = ((f >> g) >> h) x = (f >> (g >> h)) x
Check.Quick associativity
Ok, passed 100 tests.

FsCheck can generate all functions with a target type that it can generate. In addition, the functions are pure and total - the former means that if you give a generated function the same value as input, it will keep returning that same value as output, no matter how many times you call it. The latter means that the function does not throw any exceptions and always terminates.

If a counter-example is found, function values will be displayed as <func>. However, FsCheck can show you the generated function in more detail, if you ask it to generate a Function type, which has an embedded "real" function. FsCheck can even shrink Functions. For example:

let mapRec (Fun f) (l:list<int>) =
  not l.IsEmpty ==>
      lazy (List.map f l = ((*f <|*) List.head l) :: List.map f (List.tail l))
Check.Quick mapRec
Falsifiable, after 1 test (3 shrinks) (5902594408163390424,9119776349419056757)
Last step was invoked with size of 2 and seed of (1404904362023207336,17126916857698407705):
Original:
{ 0->-2; 2->-1 }
[2]
Shrunk:
{ 0->-2; 2->0 }
[0]
with exception:
System.Exception: Expected true, got false.

The type Function<'a,'b> - here deconstructed using the single case active pattern Fun - records a map of all the arguments it was called with, and the result it produced. In your properties, you can extract the actual function by pattern matching as in the example. Function is used to print the function, and also to shrink it.

Use pattern matching instead of forAll to use custom generators

To define a generator that generates a subset of the normal range of values for an existing type, say all the even ints, it makes properties more readable if you define a single-case union case, and register a generator for the new type:

type EvenInt = EvenInt of int with
  static member op_Explicit(EvenInt i) = i

type ArbitraryModifiers =
    static member EvenInt() = 
        ArbMap.defaults
        |> ArbMap.arbitrary<int> 
        |> Arb.filter (fun i -> i % 2 = 0) 
        |> Arb.convert EvenInt int
        
let ``generated even ints should be even`` (EvenInt i) = i % 2 = 0
Check.One(Config.Quick.WithArbitrary([typeof<ArbitraryModifiers>]), ``generated even ints should be even``)
Ok, passed 100 tests.

It's now easy to define custom shrink functions as well.

FsCheck uses this pattern frequently, e.g. NonNegativeInt, PositiveInt, StringWithoutNullChars etc. See the default Arbitrary instances on the Arb.Default type.

Also, for these kinds of generators, the Arb.filter, Arb.convert and Arb.mapFilter functions will come in handy.

An equality comparison that prints the left and right sides of the equality

Properties commonly check for equality. If a test case fails, FsCheck prints the counterexample, but sometimes it is useful to print the left and right side of the comparison, especially if you do some complicated calculations with the generated arguments first. To make this easier, you can define your own labelling equality combinator:

let (.=.) left right = left = right |@ sprintf "%A = %A" left right

let testCompare (i:int) (j:int) = 2*i+1  .=. 2*j-1
Check.Quick testCompare
Falsifiable, after 1 test (3 shrinks) (4944255497987765388,7960236367306300667)
Last step was invoked with size of 2 and seed of (7117107965869244564,1795690253827230511):
Label of failing property: 3 = -1
Original:
-2
1
Shrunk:
1
0
with exception:
System.Exception: Expected true, got false.

Of course, you can do this for any operator or function that you often use.

Some ways to run FsCheck tests

Testing mutable types without using Command or StateMachine

For some relatively simple mutable types you might feel more comfortable just writing straightforward FsCheck properties without using the Command or StateMachine API. This is certainly possible, but for shrinking FsCheck assumes that it can re-execute the same test multiple times without the inputs changing. If you call methods or set properties on a generated object that affect its state, this assumption does not hold and you'll see some weird results.

The simplest way to work around this is not to write a generator for your mutable object at all, but instead write an FsCheck property that takes all the values necessary to construct the object, and then simply construct the object in the beginning of your test. For example, suppose we want to test a mutable list:

let testMutableList =
    Prop.forAll (Arb.fromGen(Gen.choose (1,10))) (fun capacity -> 
        let underTest = new System.Collections.Generic.List<int>(capacity)
        Prop.forAll (ArbMap.defaults |> ArbMap.arbitrary<int[]>) (fun itemsToAdd ->
            underTest.AddRange(itemsToAdd)
            underTest.Count = itemsToAdd.Length))
Prop.ForAll(Gen.Choose(1, 10).ToArbitrary(), ArbMap.Default.ArbFor<int[]>(),(capacity, itemsToAdd) => {
    var underTest = new List<int>(capacity);
    underTest.AddRange(itemsToAdd);
    return underTest.Count == itemsToAdd.Length;
})
.QuickCheck();

This works, as a bonus you get shrinking for free.

If you do want to write a generator for your mutable type, this can be made to work but if you mutate a generated object during a test, either:

Replaying a failed test

When you have a failed test, it's often useful for debugging to be able to replay exactly those inputs. For this reason, FsCheck displays the seed of its pseudo-random number generator when a test fails. Look for the bit of text that looks like: (StdGen (1145655947,296144285)).

To replay this test, which should have the exact same output, use the Replay field on Config:

Check.One(Config.Quick.WithReplay(1145655947UL,296144285UL), fun x -> abs x >= 0)

In C#:

Prop.ForAll((int x) => Math.Abs(x) >= 0)
    .Check(Config.Quick.WithReplay(1145655947UL, 296144285UL));

Checking properties in parallel

FsCheck can evaluate properties in parallel. This feature may be useful to speed-up your cpu-heavy properties and custom arbitraries. Also this is invaluable for running asynchronous propertiess, i.e. when you are doing asynchronous IO inside prop. Don't forget to wrap your property in Task or Async in that case.

To run a property in parallel, use the ParallelRunConfig field on Config:

Check.One(
    Config.Quick.WithParallelRunConfig({ MaxDegreeOfParallelism = System.Environment.ProcessorCount }),
     fun x -> abs x >= 0
)

System.Environment.ProcessorCount is a good default for cpu-bound work. For io-bound work it's usually enough to set ParallelRunConfig to 1.

Check.One(
    Config.Verbose.WithParallelRunConfig({ MaxDegreeOfParallelism = 1 } ),
    fun (x:int) -> 
        async { 
            do! Async.Sleep (abs x)
            return true
        }
)
namespace FsCheck
namespace FsCheck.FSharp
namespace System
val associativity : x:int -> f:(int -> float) * g:(float -> char) * h:(char -> int) -> bool
val x : int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

--------------------
[<Struct>]
type int = int32

--------------------
type int<'Measure> =
  int
val f : (int -> float)
Multiple items
val float : value:'T -> float (requires member op_Explicit)

--------------------
[<Struct>]
type float = Double

--------------------
type float<'Measure> =
  float
val g : (float -> char)
Multiple items
val char : value:'T -> char (requires member op_Explicit)

--------------------
[<Struct>]
type char = Char
val h : (char -> int)
type Check =
  static member All : config:Config * test:Type -> unit + 1 overload
  static member Method : config:Config * methodInfo:MethodInfo * ?target:obj -> unit
  static member One : config:Config * property:'Testable -> unit + 1 overload
  static member Quick : property:'Testable -> unit + 1 overload
  static member QuickAll : test:Type -> unit + 1 overload
  static member QuickThrowOnFailure : property:'Testable -> unit
  static member QuickThrowOnFailureAll : test:Type -> unit + 1 overload
  static member Verbose : property:'Testable -> unit + 1 overload
  static member VerboseAll : test:Type -> unit + 1 overload
  static member VerboseThrowOnFailure : property:'Testable -> unit
  ...
static member Check.Quick : property:'Testable -> unit
static member Check.Quick : name:string * property:'Testable -> unit
val mapRec : Function<int,int> -> l:int list -> Property
active recognizer Fun: Function<'a,'b> -> 'a -> 'b
val f : (int -> int)
val l : int list
type 'T list = List<'T>
val not : value:bool -> bool
property List.IsEmpty: bool with get
Multiple items
module List

from Microsoft.FSharp.Collections

--------------------
type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
    interface IReadOnlyList<'T>
    interface IReadOnlyCollection<'T>
    interface IEnumerable
    interface IEnumerable<'T>
    member GetReverseIndex : rank:int * offset:int -> int
    member GetSlice : startIndex:int option * endIndex:int option -> 'T list
    static member Cons : head:'T * tail:'T list -> 'T list
    member Head : 'T
    member IsEmpty : bool
    member Item : index:int -> 'T with get
    ...
val map : mapping:('T -> 'U) -> list:'T list -> 'U list
val head : list:'T list -> 'T
val tail : list:'T list -> 'T list
Multiple items
union case EvenInt.EvenInt: int -> EvenInt

--------------------
type EvenInt =
  | EvenInt of int
    static member op_Explicit : EvenInt -> int
val i : int
type EvenInt =
  | EvenInt of int
    static member op_Explicit : EvenInt -> int
module ArbMap

from FsCheck.FSharp
val defaults : IArbMap
val arbitrary : arbMap:IArbMap -> Arbitrary<'T>
module Arb

from FsCheck.FSharp
val filter : pred:('T -> bool) -> a:Arbitrary<'T> -> Arbitrary<'T>
val convert : convertTo:('T -> 'U) -> convertFrom:('U -> 'T) -> a:Arbitrary<'T> -> Arbitrary<'U>
val ( generated even ints should be even ) : EvenInt -> bool
static member Check.One : config:Config * property:'Testable -> unit
static member Check.One : name:string * config:Config * property:'Testable -> unit
type Config =
  private | Config of {| ArbMap: IArbMap; EndSize: int; Every: (int -> obj list -> string); EveryShrink: (obj list -> string); MaxRejected: int; MaxTest: int; Name: string; ParallelRunConfig: ParallelRunConfig option; QuietOnSuccess: bool; Replay: Replay option; Runner: IRunner; StartSize: int |}
    member WithArbitrary : arbitrary:seq<#Type> -> Config
    member WithEndSize : endSize:int -> Config
    member WithEvery : every:(int -> obj list -> string) -> Config
    member WithEveryShrink : everyShrink:(obj list -> string) -> Config
    member WithMaxRejected : maxRejected:int -> Config
    member WithMaxTest : maxTest:int -> Config
    member WithName : name:string -> Config
    member WithParallelRunConfig : config:ParallelRunConfig option -> Config
    member WithQuietOnSuccess : quietOnSuccess:bool -> Config
    member WithReplay : replay:Replay option -> Config
    ...
property Config.Quick: Config with get
member Config.WithArbitrary : arbitrary:seq<#Type> -> Config
val typeof<'T> : Type
type ArbitraryModifiers =
  static member EvenInt : unit -> Arbitrary<EvenInt>
val left : 'a (requires equality)
val right : 'a (requires equality)
val sprintf : format:Printf.StringFormat<'T> -> 'T
val testCompare : i:int -> j:int -> Property
val j : int
val testMutableList : Property
module Prop

from FsCheck.FSharp
val forAll : arb:Arbitrary<'Value> -> body:('Value -> 'Testable) -> Property
val fromGen : gen:Gen<'Value> -> Arbitrary<'Value>
Multiple items
module Gen

from FsCheck.FSharp

--------------------
type Gen<'T> =
  private | Gen of (int -> Rnd -> struct ('T * Rnd))
    interface IGen
val choose : l:int * h:int -> Gen<int>
val capacity : int
val underTest : Collections.Generic.List<int>
namespace System.Collections
namespace System.Collections.Generic
Multiple items
type List<'T> =
  interface ICollection<'T>
  interface IEnumerable<'T>
  interface IEnumerable
  interface IList<'T>
  interface IReadOnlyCollection<'T>
  interface IReadOnlyList<'T>
  interface ICollection
  interface IList
  new : unit -> unit + 2 overloads
  member Add : item: 'T -> unit
  ...

--------------------
Collections.Generic.List() : Collections.Generic.List<'T>
Collections.Generic.List(collection: Collections.Generic.IEnumerable<'T>) : Collections.Generic.List<'T>
Collections.Generic.List(capacity: int) : Collections.Generic.List<'T>
val itemsToAdd : int []
Collections.Generic.List.AddRange(collection: Collections.Generic.IEnumerable<int>) : unit
property Collections.Generic.List.Count: int with get
property Array.Length: int with get
member Config.WithReplay : replay:Replay option -> Config
static member ConfigExtensions.WithReplay : config:Config * seed:uint64 * gamma:uint64 -> Config
static member ConfigExtensions.WithReplay : config:Config * seed:uint64 * gamma:uint64 * size:int -> Config
val abs : value:'T -> 'T (requires member Abs)
static member ConfigExtensions.WithParallelRunConfig : config:Config * parallelRunConfig:ParallelRunConfig -> Config
member Config.WithParallelRunConfig : config:ParallelRunConfig option -> Config
type Environment =
  static member Exit : exitCode: int -> unit
  static member ExpandEnvironmentVariables : name: string -> string
  static member FailFast : message: string -> unit + 1 overload
  static member GetCommandLineArgs : unit -> string []
  static member GetEnvironmentVariable : variable: string -> string + 1 overload
  static member GetEnvironmentVariables : unit -> IDictionary + 1 overload
  static member GetFolderPath : folder: SpecialFolder -> string + 1 overload
  static member GetLogicalDrives : unit -> string []
  static member SetEnvironmentVariable : variable: string * value: string -> unit + 1 overload
  static member CommandLine : string
  ...
property Environment.ProcessorCount: int with get
property Config.Verbose: Config with get
val async : AsyncBuilder
Multiple items
type Async =
  static member AsBeginEnd : computation:('Arg -> Async<'T>) -> ('Arg * AsyncCallback * obj -> IAsyncResult) * (IAsyncResult -> 'T) * (IAsyncResult -> unit)
  static member AwaitEvent : event:IEvent<'Del,'T> * ?cancelAction:(unit -> unit) -> Async<'T> (requires delegate and 'Del :> Delegate)
  static member AwaitIAsyncResult : iar:IAsyncResult * ?millisecondsTimeout:int -> Async<bool>
  static member AwaitTask : task:Task<'T> -> Async<'T> + 1 overload
  static member AwaitWaitHandle : waitHandle:WaitHandle * ?millisecondsTimeout:int -> Async<bool>
  static member CancelDefaultToken : unit -> unit
  static member Catch : computation:Async<'T> -> Async<Choice<'T,exn>>
  static member Choice : computations:seq<Async<'T option>> -> Async<'T option>
  static member FromBeginEnd : beginAction:(AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T> + 3 overloads
  static member FromContinuations : callback:(('T -> unit) * (exn -> unit) * (OperationCanceledException -> unit) -> unit) -> Async<'T>
  ...

--------------------
type Async<'T> =
static member Async.Sleep : dueTime:TimeSpan -> Async<unit>
static member Async.Sleep : millisecondsDueTime:int -> Async<unit>