FsCheck


Tips and Tricks

Properties of functions

Perhaps surprisingly, FsCheck can generate random functions, Func and Actions. As a result, it can check properties of functions. For example, we can check associativity of function composition as follows:

let associativity (x:int) (f:int->float,g:float->char,h:char->int) = ((f >> g) >> h) x = (f >> (g >> h)) x
Check.Quick associativity
Ok, passed 100 tests.

FsCheck can generate all functions with a target type that it can generate. In addition, the functions are pure and total - the former means that if you give a generated function the same value as input, it will keep returning that same value as output, no matter how many times you call it. The latter means that the function does not throw any exceptions and always terminates.

If a counter-example is found, function values will be displayed as <func>. However, FsCheck can show you the generated function in more detail, if you ask it to generate a Function type, which has an embedded "real" function. FsCheck can even shrink Functions. For example:

let mapRec (Fun f) (l:list<int>) =
  not l.IsEmpty ==>
      lazy (List.map f l = ((*f <|*) List.head l) :: List.map f (List.tail l))
Check.Quick mapRec
Falsifiable, after 1 test (3 shrinks) (StdGen (347585768,296510006)):
Original:
{ 0->2; 1->0 }
[1; 1; 1]
Shrunk:
{ 0->2; 1->0 }
[0]

The type Function<'a,'b> - here deconstructed using the single case active pattern Fun - records a map of all the arguments it was called with, and the result it produced. In your properties, you can extract the actual function by pattern matching as in the example. Function is used to print the function, and also to shrink it.

Use pattern matching instead of forAll to use custom generators

To define a generator that generates a subset of the normal range of values for an existing type, say all the even ints, it makes properties more readable if you define a single-case union case, and register a generator for the new type:

type EvenInt = EvenInt of int with
  static member op_Explicit(EvenInt i) = i

type ArbitraryModifiers =
    static member EvenInt() = 
        Arb.from<int> 
        |> Arb.filter (fun i -> i % 2 = 0) 
        |> Arb.convert EvenInt int
        
Arb.register<ArbitraryModifiers>()

let ``generated even ints should be even`` (EvenInt i) = i % 2 = 0
Check.Quick ``generated even ints should be even``
Ok, passed 100 tests.

It's now easy to define custom shrink functions as well.

FsCheck uses this pattern frequently, e.g. NonNegativeInt, PositiveInt, StringWithoutNullChars etc. See the default Arbitrary instances on the Arb.Default type.

Also, for these kinds of generators, the Arb.filter, Arb.convert and Arb.mapFilter functions will come in handy.

An equality comparison that prints the left and right sides of the equality

Properties commonly check for equality. If a test case fails, FsCheck prints the counterexample, but sometimes it is useful to print the left and right side of the comparison, especially if you do some complicated calculations with the generated arguments first. To make this easier, you can define your own labelling equality combinator:

let (.=.) left right = left = right |@ sprintf "%A = %A" left right

let testCompare (i:int) (j:int) = 2*i+1  .=. 2*j-1
Check.Quick testCompare
Falsifiable, after 1 test (2 shrinks) (StdGen (349902936,296510006)):
Label of failing property: 3 = -1
Original:
2
1
Shrunk:
1
0

Of course, you can do this for any operator or function that you often use.

Some ways to run FsCheck tests

  • By adding properties and generators to an fsx file in your project. It's easy to execute, just press ctrl-a and alt-enter, and the results are displayed in F# Interactive. Be careful when referencing dlls that are built in your solution; Versions of F# Interactive earlier than 3.1.2 will lock those for the remainder of the session, and you won't be able to build until you quit the session. One solution is to include the source files instead of the dlls, but that makes the process slower. Useful for smaller projects. Difficult to debug though.
  • By making a separate console application. Easy to debug, and no annoying locks on assemblies. Your best option if you use only FsCheck for testing and your properties span multiple assemblies.
  • By using another unit testing framework. Useful if you have a mixed FsCheck/unit testing approach (some things are easier to check using unit tests, and vice versa), and you like a graphical runner. Depending on what unit testing framework you use, you may get good integration with Visual Studio for free. Also have a look at some of the existing integrations with test runners like Xunit.NET, NUnit, Fuchu.

Testing mutable types without using Command or StateMachine

For some relatively simple mutable types you might feel more comfortable just writing straightforward FsCheck properties without using the Command or StateMachine API. This is certainly possible, but for shrinking FsCheck assumes that it can re-execute the same test multiple times without the inputs changing. If you call methods or set properties on a generated object that affect its state, this assumption does not hold and you'll see some weird results.

The simplest way to work around this is not to write a generator for your mutable object at all, but instead write an FsCheck property that takes all the values necessary to construct the object, and then simply construct the object in the beginning of your test. For example, suppose we want to test a mutable list:

let testMutableList =
    Prop.forAll (Arb.fromGen(Gen.choose (1,10))) (fun capacity -> 
        let underTest = new System.Collections.Generic.List<int>(capacity)
        Prop.forAll Arb.from<int[]> (fun itemsToAdd ->
            underTest.AddRange(itemsToAdd)
            underTest.Count = itemsToAdd.Length))
Prop.ForAll(Arb.From(Gen.Choose(1, 10)), Arb.From<int[]>(),(capacity, itemsToAdd) => {
    var underTest = new List<int>(capacity);
    underTest.AddRange(itemsToAdd);
    return underTest.Count == itemsToAdd.Length;
})
.QuickCheck();

This works, as a bonus you get shrinking for free.

If you do want to write a generator for your mutable type, this can be made to work but if you mutate a generated object during a test, either:

  • Disable shrinking, typically by wrapping all types into DontShrink; or
  • Clone or otherwise 'reset' the generated mutable object at the beginning or end of every test.

Replaying a failed test

When you have a failed test, it's often useful for debugging to be able to replay exactly those inputs. For this reason, FsCheck displays the seed of its pseudo-random number generator when a test fails. Look for the bit of text that looks like: (StdGen (1145655947,296144285)).

To replay this test, which should have the exact same output, use the Replay field on Config:

Check.One({ Config.Quick with Replay = Some <| Random.StdGen (1145655947,296144285) }, fun x -> abs x >= 0)

In C#:

Prop.ForAll((int x) => Math.Abs(x) >= 0)
    .Check(new Configuration { Replay = FsCheck.Random.StdGen.NewStdGen(1145655947, 296144285)});
namespace FsCheck
namespace System
val associativity : x:int -> f:(int -> float) * g:(float -> char) * h:(char -> int) -> bool
val x : int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int
val f : (int -> float)
Multiple items
val float : value:'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float
val g : (float -> char)
Multiple items
val char : value:'T -> char (requires member op_Explicit)

--------------------
type char = Char
val h : (char -> int)
type Check =
  static member All : config:Config -> unit
  static member All : config:Config * test:Type -> unit
  static member Method : config:Config * methodInfo:MethodInfo * ?target:obj -> unit
  static member One : config:Config * property:'Testable -> unit
  static member One : name:string * config:Config * property:'Testable -> unit
  static member Quick : property:'Testable -> unit
  static member Quick : name:string * property:'Testable -> unit
  static member QuickAll : unit -> unit
  static member QuickAll : test:Type -> unit
  static member QuickThrowOnFailure : property:'Testable -> unit
  ...
static member Check.Quick : property:'Testable -> unit
static member Check.Quick : name:string * property:'Testable -> unit
val mapRec : Function<int,int> -> l:int list -> Property
active recognizer Fun: Function<'a,'b> -> 'a -> 'b
val f : (int -> int)
val l : int list
type 'T list = List<'T>
val not : value:bool -> bool
property List.IsEmpty: bool
Multiple items
module List

from Microsoft.FSharp.Collections

--------------------
type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
    interface IReadOnlyList<'T>
    interface IReadOnlyCollection<'T>
    interface IEnumerable
    interface IEnumerable<'T>
    member GetSlice : startIndex:int option * endIndex:int option -> 'T list
    member Head : 'T
    member IsEmpty : bool
    member Item : index:int -> 'T with get
    member Length : int
    member Tail : 'T list
    ...
val map : mapping:('T -> 'U) -> list:'T list -> 'U list
val head : list:'T list -> 'T
val tail : list:'T list -> 'T list
Multiple items
union case EvenInt.EvenInt: int -> EvenInt

--------------------
type EvenInt =
  | EvenInt of int
    static member op_Explicit : EvenInt -> int
val i : int
type EvenInt =
  | EvenInt of int
    static member op_Explicit : EvenInt -> int
module Arb

from FsCheck
val from<'Value> : Arbitrary<'Value>
val filter : pred:('a -> bool) -> a:Arbitrary<'a> -> Arbitrary<'a>
val convert : convertTo:('a -> 'b) -> convertFrom:('b -> 'a) -> a:Arbitrary<'a> -> Arbitrary<'b>
val register<'t> : unit -> TypeClass.TypeClassComparison
type ArbitraryModifiers =
  static member EvenInt : unit -> Arbitrary<EvenInt>
val ( generated even ints should be even ) : EvenInt -> bool
val left : 'a (requires equality)
val right : 'a (requires equality)
val sprintf : format:Printf.StringFormat<'T> -> 'T
val testCompare : i:int -> j:int -> Property
val j : int
val testMutableList : Property
module Prop

from FsCheck
val forAll : arb:Arbitrary<'Value> -> body:('Value -> 'Testable) -> Property
val fromGen : gen:Gen<'Value> -> Arbitrary<'Value>
Multiple items
module Gen

from FsCheck

--------------------
type Gen<'a> =
  private | Gen of (int -> StdGen -> 'a)
    interface IGen
    member private Map : f:('a -> 'b) -> Gen<'b>
    static member ( >>= ) : m:Gen<'a1> * k:('a1 -> Gen<'a2>) -> Gen<'a2>
    static member ( <!> ) : f:('a1 -> 'a2) * a:Gen<'a1> -> Gen<'a2>
    static member ( <*> ) : f:Gen<('a1 -> 'a2)> * a:Gen<'a1> -> Gen<'a2>
val choose : l:int * h:int -> Gen<int>
val capacity : int
val underTest : Collections.Generic.List<int>
namespace System.Collections
namespace System.Collections.Generic
Multiple items
type List<'T> =
  new : unit -> List<'T> + 2 overloads
  member Add : item:'T -> unit
  member AddRange : collection:IEnumerable<'T> -> unit
  member AsReadOnly : unit -> ReadOnlyCollection<'T>
  member BinarySearch : item:'T -> int + 2 overloads
  member Capacity : int with get, set
  member Clear : unit -> unit
  member Contains : item:'T -> bool
  member ConvertAll<'TOutput> : converter:Converter<'T, 'TOutput> -> List<'TOutput>
  member CopyTo : array:'T[] -> unit + 2 overloads
  ...
  nested type Enumerator

--------------------
Collections.Generic.List() : Collections.Generic.List<'T>
Collections.Generic.List(capacity: int) : Collections.Generic.List<'T>
Collections.Generic.List(collection: Collections.Generic.IEnumerable<'T>) : Collections.Generic.List<'T>
val itemsToAdd : int []
Collections.Generic.List.AddRange(collection: Collections.Generic.IEnumerable<int>) : unit
property Collections.Generic.List.Count: int
property Array.Length: int
static member Check.One : config:Config * property:'Testable -> unit
static member Check.One : name:string * config:Config * property:'Testable -> unit
type Config =
  {MaxTest: int;
   MaxFail: int;
   Replay: StdGen option;
   Name: string;
   StartSize: int;
   EndSize: int;
   QuietOnSuccess: bool;
   Every: int -> obj list -> string;
   EveryShrink: obj list -> string;
   Arbitrary: Type list;
   ...}
    static member Default : Config
    static member Quick : Config
    static member QuickThrowOnFailure : Config
    static member Verbose : Config
    static member VerboseThrowOnFailure : Config
    static member private throwingRunner : IRunner
property Config.Quick: Config
union case Option.Some: Value: 'T -> Option<'T>
Multiple items
type Random =
  new : unit -> Random + 1 overload
  member Next : unit -> int + 2 overloads
  member NextBytes : buffer:byte[] -> unit
  member NextDouble : unit -> float

--------------------
Random() : Random
Random(Seed: int) : Random
Multiple items
union case Random.StdGen.StdGen: int * int -> Random.StdGen

--------------------
type StdGen = | StdGen of int * int
val abs : value:'T -> 'T (requires member Abs)
Fork me on GitHub