Tips and Tricks

Properties of functions

Perhaps surprisingly, FsCheck can generate random functions, Func and Actions. As a result, it can check properties of functions. For example, we can check associativity of function composition as follows:

let associativity (x:int) (f:int->float,g:float->char,h:char->int) = ((f >> g) >> h) x = (f >> (g >> h)) x
Check.Quick associativity
Ok, passed 100 tests.

FsCheck can generate all functions with a target type that it can generate. In addition, the functions are pure and total - the former means that if you give a generated function the same value as input, it will keep returning that same value as output, no matter how many times you call it. The latter means that the function does not throw any exceptions and always terminates.

If a counter-example is found, function values will be displayed as <func>. However, FsCheck can show you the generated function in more detail, if you ask it to generate a Function type, which has an embedded "real" function. FsCheck can even shrink Functions. For example:

let mapRec (Fun f) (l:list<int>) =
  not l.IsEmpty ==>
      lazy (List.map f l = ((*f <|*) List.head l) :: List.map f (List.tail l))
Check.Quick mapRec
Falsifiable, after 1 test (2 shrinks) (StdGen (442963506,296278002)):
{ 0->1; 1->3 }
{ 0->1; 1->0 }

The type Function<'a,'b> - here deconstructed using the single case active pattern Fun - records a map of all the arguments it was called with, and the result it produced. In your properties, you can extract the actual function by pattern matching as in the example. Function is used to print the function, and also to shrink it.

Use pattern matching instead of forAll to use custom generators

To define a generator that generates a subset of the normal range of values for an existing type, say all the even ints, it makes properties more readable if you define a single-case union case, and register a generator for the new type:

type EvenInt = EvenInt of int with
  static member op_Explicit(EvenInt i) = i

type ArbitraryModifiers =
    static member EvenInt() = 
        |> Arb.filter (fun i -> i % 2 = 0) 
        |> Arb.convert EvenInt int

let ``generated even ints should be even`` (EvenInt i) = i % 2 = 0
Check.Quick ``generated even ints should be even``
Ok, passed 100 tests.

It's now easy to define custom shrink functions as well.

FsCheck uses this pattern frequently, e.g. NonNegativeInt, PositiveInt, StringWithoutNullChars etc. See the default Arbitrary instances on the Arb.Default type.

Also, for these kinds of generators, the Arb.filter, Arb.convert and Arb.mapFilter functions will come in handy.

An equality comparison that prints the left and right sides of the equality

Properties commonly check for equality. If a test case fails, FsCheck prints the counterexample, but sometimes it is useful to print the left and right side of the comparison, especially if you do some complicated calculations with the generated arguments first. To make this easier, you can define your own labelling equality combinator:

let (.=.) left right = left = right |@ sprintf "%A = %A" left right

let testCompare (i:int) (j:int) = 2*i+1  .=. 2*j-1
Check.Quick testCompare
Falsifiable, after 1 test (1 shrink) (StdGen (444704913,296278002)):
Label of failing property: 3 = -1

Of course, you can do this for any operator or function that you often use.

Some ways to run FsCheck tests

  • By adding properties and generators to an fsx file in your project. It's easy to execute, just press ctrl-a and alt-enter, and the results are displayed in F# Interactive. Be careful when referencing dlls that are built in your solution; Versions of F# Interactive earlier than 3.1.2 will lock those for the remainder of the session, and you won't be able to build until you quit the session. One solution is to include the source files instead of the dlls, but that makes the process slower. Useful for smaller projects. Difficult to debug though.
  • By making a separate console application. Easy to debug, and no annoying locks on assemblies. Your best option if you use only FsCheck for testing and your properties span multiple assemblies.
  • By using another unit testing framework. Useful if you have a mixed FsCheck/unit testing approach (some things are easier to check using unit tests, and vice versa), and you like a graphical runner. Depending on what unit testing framework you use, you may get good integration with Visual Studio for free. Also have a look at some of the existing integrations with test runners like Xunit.NET, NUnit, Fuchu.

Testing mutable types without using Command or StateMachine

For some relatively simple mutable types you might feel more comfortable just writing straightforward FsCheck properties without using the Command or StateMachine API. This is certainly possible, but for shrinking FsCheck assumes that it can re-execute the same test multiple times without the inputs changing. If you call methods or set properties on a generated object that affect its state, this assumption does not hold and you'll see some weird results.

The simplest way to work around this is not to write a generator for your mutable object at all, but instead write an FsCheck property that takes all the values necessary to construct the object, and then simply construct the object in the beginning of your test. For example, suppose we want to test a mutable list:

let testMutableList =
    Prop.forAll (Arb.fromGen(Gen.choose (1,10))) (fun capacity -> 
        let underTest = new System.Collections.Generic.List<int>(capacity)
        Prop.forAll Arb.from<int[]> (fun itemsToAdd ->
            underTest.Count = itemsToAdd.Length))
Prop.ForAll(Arb.From(Gen.Choose(1, 10)), Arb.From<int[]>(),(capacity, itemsToAdd) => {
    var underTest = new List<int>(capacity);
    return underTest.Count == itemsToAdd.Length;

This works, as a bonus you get shrinking for free.

If you do want to write a generator for your mutable type, this can be made to work but if you mutate a generated object during a test, either:

  • Disable shrinking, typically by wrapping all types into DontShrink; or
  • Clone or otherwise 'reset' the generated mutable object at the beginning or end of every test.

Replaying a failed test

When you have a failed test, it's often useful for debugging to be able to replay exactly those inputs. For this reason, FsCheck displays the seed of its pseudo-random number generator when a test fails. Look for the bit of text that looks like: (StdGen (1145655947,296144285)).

To replay this test, which should have the exact same output, use the Replay field on Config:

Check.One({ Config.Quick with Replay = Some <| Random.StdGen (1145655947,296144285) }, fun x -> abs x >= 0)

In C#:

Prop.ForAll((int x) => Math.Abs(x) >= 0)
    .Check(new Configuration { Replay = FsCheck.Random.StdGen.NewStdGen(1145655947, 296144285)});
namespace FsCheck
namespace System
val associativity : x:int -> f:(int -> float) * g:(float -> char) * h:(char -> int) -> bool

Full name: TipsAndTricks.associativity
val x : int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

type int = int32

Full name: Microsoft.FSharp.Core.int

type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
val f : (int -> float)
Multiple items
val float : value:'T -> float (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.float

type float = Double

Full name: Microsoft.FSharp.Core.float

type float<'Measure> = float

Full name: Microsoft.FSharp.Core.float<_>
val g : (float -> char)
Multiple items
val char : value:'T -> char (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.char

type char = Char

Full name: Microsoft.FSharp.Core.char
val h : (char -> int)
val mapRec : 'a -> l:int list -> 'b

Full name: TipsAndTricks.mapRec
val _arg1 : 'a
val l : int list
type 'T list = List<'T>

Full name: Microsoft.FSharp.Collections.list<_>
val not : value:bool -> bool

Full name: Microsoft.FSharp.Core.Operators.not
property List.IsEmpty: bool
Multiple items
module List

from Microsoft.FSharp.Collections

type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
  interface IEnumerable
  interface IEnumerable<'T>
  member GetSlice : startIndex:int option * endIndex:int option -> 'T list
  member Head : 'T
  member IsEmpty : bool
  member Item : index:int -> 'T with get
  member Length : int
  member Tail : 'T list
  static member Cons : head:'T * tail:'T list -> 'T list
  static member Empty : 'T list

Full name: Microsoft.FSharp.Collections.List<_>
val map : mapping:('T -> 'U) -> list:'T list -> 'U list

Full name: Microsoft.FSharp.Collections.List.map
val head : list:'T list -> 'T

Full name: Microsoft.FSharp.Collections.List.head
val tail : list:'T list -> 'T list

Full name: Microsoft.FSharp.Collections.List.tail
Multiple items
union case EvenInt.EvenInt: int -> EvenInt

type EvenInt =
  | EvenInt of int
  static member op_Explicit : EvenInt -> int

Full name: TipsAndTricks.EvenInt
static member EvenInt.op_Explicit : EvenInt -> int

Full name: TipsAndTricks.EvenInt.op_Explicit
val i : int
type ArbitraryModifiers =
  static member EvenInt : unit -> 'a

Full name: TipsAndTricks.ArbitraryModifiers
Multiple items
static member ArbitraryModifiers.EvenInt : unit -> 'a

Full name: TipsAndTricks.ArbitraryModifiers.EvenInt

type EvenInt =
  | EvenInt of int
  static member op_Explicit : EvenInt -> int

Full name: TipsAndTricks.EvenInt
val ( generated even ints should be even ) : EvenInt -> bool

Full name: TipsAndTricks.( generated even ints should be even )
val left : 'a (requires equality)
val right : 'a (requires equality)
val sprintf : format:Printf.StringFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf
val j : int
val testMutableList : obj

Full name: TipsAndTricks.testMutableList
namespace System.Collections
namespace System.Collections.Generic
Multiple items
type List<'T> =
  new : unit -> List<'T> + 2 overloads
  member Add : item:'T -> unit
  member AddRange : collection:IEnumerable<'T> -> unit
  member AsReadOnly : unit -> ReadOnlyCollection<'T>
  member BinarySearch : item:'T -> int + 2 overloads
  member Capacity : int with get, set
  member Clear : unit -> unit
  member Contains : item:'T -> bool
  member ConvertAll<'TOutput> : converter:Converter<'T, 'TOutput> -> List<'TOutput>
  member CopyTo : array:'T[] -> unit + 2 overloads
  nested type Enumerator

Full name: System.Collections.Generic.List<_>

Collections.Generic.List() : unit
Collections.Generic.List(capacity: int) : unit
Collections.Generic.List(collection: Collections.Generic.IEnumerable<'T>) : unit
union case Option.Some: Value: 'T -> Option<'T>
Multiple items
type Random =
  new : unit -> Random + 1 overload
  member Next : unit -> int + 2 overloads
  member NextBytes : buffer:byte[] -> unit
  member NextDouble : unit -> float

Full name: System.Random

Random() : unit
Random(Seed: int) : unit
val abs : value:'T -> 'T (requires member Abs)

Full name: Microsoft.FSharp.Core.Operators.abs
Fork me on GitHub