Perhaps surprisingly, FsCheck can generate random functions, Func
and Action
s. As a result, it can check properties of
functions. For example, we can check associativity of function composition as follows:
let associativity (x:int) (f:int->float,g:float->char,h:char->int) = ((f >> g) >> h) x = (f >> (g >> h)) x
Check.Quick associativity
FsCheck can generate all functions with a target type that it can generate. In addition, the functions are pure and total -
the former means that if you give a generated function the same value as input, it will keep returning that same value as output,
no matter how many times you call it. The latter means that the function does not throw any exceptions and always terminates.
If a counter-example is found, function values will be displayed as <func>
. However, FsCheck can show
you the generated function in more detail, if you ask it to generate a Function
type, which has an embedded "real" function.
FsCheck can even shrink Function
s. For example:
let mapRec (Fun f) (l:list<int>) =
not l.IsEmpty ==>
lazy (List.map f l = ((*f <|*) List.head l) :: List.map f (List.tail l))
Check.Quick mapRec
Falsifiable, after 1 test (1 shrink) (StdGen (982337070, 297043896)):
Original:
{ 0->2 }
[0]
Shrunk:
{ 0->1 }
[0]
|
The type Function<'a,'b>
- here deconstructed using the single case active pattern Fun
-
records a map of all the arguments it was called with, and the result it produced.
In your properties, you can extract the actual function by pattern matching as in the example.
Function
is used to print the function, and also to shrink it.
To define a generator that generates a subset of the normal range of values for an existing type,
say all the even ints, it makes properties more readable if you define a single-case union
case, and register a generator for the new type:
type EvenInt = EvenInt of int with
static member op_Explicit(EvenInt i) = i
type ArbitraryModifiers =
static member EvenInt() =
Arb.from<int>
|> Arb.filter (fun i -> i % 2 = 0)
|> Arb.convert EvenInt int
Arb.register<ArbitraryModifiers>()
let ``generated even ints should be even`` (EvenInt i) = i % 2 = 0
Check.Quick ``generated even ints should be even``
It's now easy to define custom shrink functions as well.
FsCheck uses this pattern frequently, e.g. NonNegativeInt
, PositiveInt
, StringWithoutNullChars
etc. See the
default Arbitrary instances on the Arb.Default
type.
Also, for these kinds of generators, the Arb.filter
, Arb.convert
and Arb.mapFilter
functions will come in handy.
Properties commonly check for equality. If a test case fails, FsCheck prints the counterexample, but
sometimes it is useful to print the left and right side of the comparison, especially if you
do some complicated calculations with the generated arguments first. To make this easier, you can
define your own labelling equality combinator:
let (.=.) left right = left = right |@ sprintf "%A = %A" left right
let testCompare (i:int) (j:int) = 2*i+1 .=. 2*j-1
Check.Quick testCompare
Falsifiable, after 1 test (1 shrink) (StdGen (983199806, 297043896)):
Label of failing property: 1 = -1
Original:
1
0
Shrunk:
0
0
|
Of course, you can do this for any operator or function that you often use.
-
By adding properties and generators to an fsx file in your project. It's easy to execute, just press
ctrl-a and alt-enter, and the results are displayed in F# Interactive. Be careful when referencing dlls
that are built in your solution; Versions of F# Interactive earlier than 3.1.2 will lock those for the remainder of the session,
and you won't be able to build until you quit the session. One solution is to include the source files
instead of the dlls, but that makes the process slower. Useful for smaller projects. Difficult to debug though.
-
By making a separate console application. Easy to debug, and no annoying locks on assemblies. Your best option
if you use only FsCheck for testing and your properties span multiple assemblies.
-
By using another unit testing framework. Useful if you have a mixed FsCheck/unit testing approach
(some things are easier to check using unit tests, and vice versa), and you like a graphical runner.
Depending on what unit testing framework you use, you may get good integration with Visual Studio for free. Also have a look
at some of the existing integrations with test runners like Xunit.NET, NUnit, Fuchu.
For some relatively simple mutable types you might feel more comfortable just writing straightforward FsCheck properties without
using the Command
or StateMachine
API. This is certainly possible, but for shrinking FsCheck assumes that it can
re-execute the same test multiple times without the inputs changing. If you call methods or set properties on a generated object
that affect its state, this assumption does not hold and you'll see some weird results.
The simplest way to work around this is not to write a generator for your mutable object at all, but instead write an FsCheck property
that takes all the values necessary to construct the object, and then simply construct the object in the beginning of your test. For example, suppose we want to test
a mutable list:
let testMutableList =
Prop.forAll (Arb.fromGen(Gen.choose (1,10))) (fun capacity ->
let underTest = new System.Collections.Generic.List<int>(capacity)
Prop.forAll Arb.from<int[]> (fun itemsToAdd ->
underTest.AddRange(itemsToAdd)
underTest.Count = itemsToAdd.Length))
Prop.ForAll(Arb.From(Gen.Choose(1, 10)), Arb.From<int[]>(),(capacity, itemsToAdd) => {
var underTest = new List<int>(capacity);
underTest.AddRange(itemsToAdd);
return underTest.Count == itemsToAdd.Length;
})
.QuickCheck();
|
This works, as a bonus you get shrinking for free.
If you do want to write a generator for your mutable type, this can be made to work but if
you mutate a generated object during a test, either:
- Disable shrinking, typically by wrapping all types into
DontShrink
; or
- Clone or otherwise 'reset' the generated mutable object at the beginning or end of every test.
When you have a failed test, it's often useful for debugging to be able to replay exactly those inputs. For this reason, FsCheck displays the
seed of its pseudo-random number generator when a test fails. Look for the bit of text that looks like: (StdGen (1145655947,296144285))
.
To replay this test, which should have the exact same output, use the Replay
field on Config
:
Check.One({ Config.Quick with Replay = Some <| Random.StdGen (1145655947,296144285) }, fun x -> abs x >= 0)
In C#:
Prop.ForAll((int x) => Math.Abs(x) >= 0)
.Check(new Configuration { Replay = FsCheck.Random.StdGen.NewStdGen(1145655947, 296144285)});
|
namespace FsCheck
namespace System
val associativity : x:int -> f:(int -> float) * g:(float -> char) * h:(char -> int) -> bool
val x : int
Multiple items
val int : value:'T -> int (requires member op_Explicit)
--------------------
[<Struct>]
type int = int32
--------------------
type int<'Measure> =
int
val f : (int -> float)
Multiple items
val float : value:'T -> float (requires member op_Explicit)
--------------------
[<Struct>]
type float = Double
--------------------
type float<'Measure> =
float
val g : (float -> char)
Multiple items
val char : value:'T -> char (requires member op_Explicit)
--------------------
[<Struct>]
type char = Char
val h : (char -> int)
type Check =
static member All : config:Config * test:Type -> unit + 1 overload
static member Method : config:Config * methodInfo:MethodInfo * ?target:obj -> unit
static member One : config:Config * property:'Testable -> unit + 1 overload
static member Quick : property:'Testable -> unit + 1 overload
static member QuickAll : test:Type -> unit + 1 overload
static member QuickThrowOnFailure : property:'Testable -> unit
static member QuickThrowOnFailureAll : test:Type -> unit + 1 overload
static member Verbose : property:'Testable -> unit + 1 overload
static member VerboseAll : test:Type -> unit + 1 overload
static member VerboseThrowOnFailure : property:'Testable -> unit
...
static member Check.Quick : property:'Testable -> unit
static member Check.Quick : name:string * property:'Testable -> unit
val mapRec : Function<int,int> -> l:int list -> Property
active recognizer Fun: Function<'a,'b> -> 'a -> 'b
val f : (int -> int)
val l : int list
type 'T list = List<'T>
val not : value:bool -> bool
property List.IsEmpty: bool with get
Multiple items
module List
from Microsoft.FSharp.Collections
--------------------
type List<'T> =
| ( [] )
| ( :: ) of Head: 'T * Tail: 'T list
interface IReadOnlyList<'T>
interface IReadOnlyCollection<'T>
interface IEnumerable
interface IEnumerable<'T>
member GetReverseIndex : rank:int * offset:int -> int
member GetSlice : startIndex:int option * endIndex:int option -> 'T list
static member Cons : head:'T * tail:'T list -> 'T list
member Head : 'T
member IsEmpty : bool
member Item : index:int -> 'T with get
...
val map : mapping:('T -> 'U) -> list:'T list -> 'U list
val head : list:'T list -> 'T
val tail : list:'T list -> 'T list
Multiple items
union case EvenInt.EvenInt: int -> EvenInt
--------------------
type EvenInt =
| EvenInt of int
static member op_Explicit : EvenInt -> int
val i : int
type EvenInt =
| EvenInt of int
static member op_Explicit : EvenInt -> int
module Arb
from FsCheck
val from<'Value> : Arbitrary<'Value>
val filter : pred:('a -> bool) -> a:Arbitrary<'a> -> Arbitrary<'a>
val convert : convertTo:('a -> 'b) -> convertFrom:('b -> 'a) -> a:Arbitrary<'a> -> Arbitrary<'b>
val register<'t> : unit -> TypeClass.TypeClassComparison
type ArbitraryModifiers =
static member EvenInt : unit -> Arbitrary<EvenInt>
val ( generated even ints should be even ) : EvenInt -> bool
val left : 'a (requires equality)
val right : 'a (requires equality)
val sprintf : format:Printf.StringFormat<'T> -> 'T
val testCompare : i:int -> j:int -> Property
val j : int
val testMutableList : Property
module Prop
from FsCheck
val forAll : arb:Arbitrary<'Value> -> body:('Value -> 'Testable) -> Property
val fromGen : gen:Gen<'Value> -> Arbitrary<'Value>
Multiple items
module Gen
from FsCheck
--------------------
type Gen<'a> =
private | Gen of (int -> StdGen -> 'a)
interface IGen
member private Map : f:('a -> 'b) -> Gen<'b>
static member ( <!> ) : f:('a1 -> 'a2) * a:Gen<'a1> -> Gen<'a2>
static member ( <*> ) : f:Gen<('a1 -> 'a2)> * a:Gen<'a1> -> Gen<'a2>
static member ( >>= ) : m:Gen<'a1> * k:('a1 -> Gen<'a2>) -> Gen<'a2>
val choose : l:int * h:int -> Gen<int>
val capacity : int
val underTest : Collections.Generic.List<int>
namespace System.Collections
namespace System.Collections.Generic
Multiple items
type List<'T> =
interface ICollection<'T>
interface IEnumerable<'T>
interface IEnumerable
interface IList<'T>
interface IReadOnlyCollection<'T>
interface IReadOnlyList<'T>
interface ICollection
interface IList
new : unit -> unit + 2 overloads
member Add : item: 'T -> unit
...
--------------------
Collections.Generic.List() : Collections.Generic.List<'T>
Collections.Generic.List(collection: Collections.Generic.IEnumerable<'T>) : Collections.Generic.List<'T>
Collections.Generic.List(capacity: int) : Collections.Generic.List<'T>
val itemsToAdd : int []
Collections.Generic.List.AddRange(collection: Collections.Generic.IEnumerable<int>) : unit
property Collections.Generic.List.Count: int with get
property Array.Length: int with get
static member Check.One : config:Config * property:'Testable -> unit
static member Check.One : name:string * config:Config * property:'Testable -> unit
type Config =
{ MaxTest: int
MaxFail: int
Replay: StdGen option
Name: string
StartSize: int
EndSize: int
QuietOnSuccess: bool
Every: int -> obj list -> string
EveryShrink: obj list -> string
Arbitrary: Type list
... }
static member Default : Config
static member Quick : Config
static member QuickThrowOnFailure : Config
static member Verbose : Config
static member VerboseThrowOnFailure : Config
static member private throwingRunner : IRunner
property Config.Quick: Config with get
union case Option.Some: Value: 'T -> Option<'T>
Multiple items
type Random =
new : unit -> unit + 1 overload
member Next : unit -> int + 2 overloads
member NextBytes : buffer: byte [] -> unit + 1 overload
member NextDouble : unit -> float
member Sample : unit -> float
--------------------
Random() : Random
Random(Seed: int) : Random
Multiple items
union case Random.StdGen.StdGen: int * int -> Random.StdGen
--------------------
type StdGen = | StdGen of int * int
val abs : value:'T -> 'T (requires member Abs)