assumption of advancement

A few month ago I played with some mocking frameworks in .Net. There are already some comparisons available (here, here or here). In this blog post I want to show which frameworks are available and which one fits best for agile development.

You could download the source code from github.com.

Software under Test (SUT)

To demonstrate and evaluate all the different mocking framework I had to create a scenario where it is appropriate to use a mocking framework. So I chose an user entity which I want to validate and in a successful case also to persist in a database.

namespace Mocking
{
	public class User
	{
		public IUserGateway Gateway
		{
			get; set;
		}

		public User()
		{
			this.Gateway = new UserGateway();
		}

		public User(IUserGateway gateway)
		{
			this.Gateway = gateway;
		}

		public bool Persist(IUserValidator validator)
		{
			bool bValid = validator.Validate(this);

			if(bValid) bValid = this.Gateway.Persist(this);

			return bValid;
		}
	}
}

This is the User entity class which implements the active record pattern. As you see, there are three injection patterns realized, the constructor injection and setter injection for the gateway and the parameter injection for the validator.

namespace Mocking
{
	public class UserGateway : IUserGateway
	{
		public virtual bool Persist(User user)
		{
			return true;
		}
	}
}

The UserGateway class is responsible to persist the entity. In my scenario it is just a dummy, because I don’t want to setup a database for this demo.

namespace Mocking
{
	public class UserValidator : IUserValidator
	{
		public bool Validate(User user)
		{
			if(user == null) return false;

			return true;
		}
	}
}

The validator is also more or less just a dummy for this demo.

Both interfaces, IUserGateway and IUserValidator, has the same methods, properties as the classes which implement them.

NMock2

One of the first mocking frameworks in .Net which I was looking at was NMock2. Specially the cheat sheet make it really easy to learn this framework.

namespace MockingTest
{
	[TestClass]
	public class NMock2Test
	{
		[TestMethod]
		public void TestPersist()
		{
			Mockery mocks = new Mockery();

			//Create mocks
			IUserGateway mockGateway = mocks.NewMock<IUserGateway>();
			IUserValidator mockValidator = mocks.NewMock<IUserValidator>();

			//Create user
			User user = new User();

			//Expectations
			using(mocks.Ordered)
			{
				Expect.Once.On(mockValidator).Method("Validate").With(user).
Will(Return.Value(true));
				Expect.Once.On(mockGateway).Method("Persist").With(user).
Will(Return.Value(true));
			}

			//Assign Gateway
			user.Gateway = mockGateway;

			//Test method
			Assert.AreEqual(true, user.Persist(mockValidator));

			mocks.VerifyAllExpectationsHaveBeenMet();
		}
	}
}

You could use the AAA-Syntax (Arrange-Act-Assert). Also more extended features like the ordered calls with the using syntax is very nice.

But, there is are also negative points: NMock2 uses “magic strings”. In your Expectations you have to declare the called method with a strings, which makes your tests very brittle when your refactor your code. For example if you have to rename a method, even the compiler could not help you. An other negative point is, that you can only mock interfaces. When you have to mock a class, you have to change your design by adding an interface just for mocking.

NMock3

Recently I discover, that there is already a follower of NMock2 on codeplex: NMock3.

namespace NMock3Test
{
    [TestClass]
    public class NMock3Test
    {
        [TestMethod]
        public void TestPersist()
        {
            MockFactory factory = new MockFactory();

            //Create mocks
            Mock<IUserGateway> mockGateway = factory.CreateMock<IUserGateway>();
            Mock<IUserValidator> mockValidator = factory.CreateMock<IUserValidator>();

            //Create user
            User user = new User();

            //Expectations
            using(factory.Ordered)
            {
                mockValidator.Expects.One.MethodWith(m => m.Validate(user)).WillReturn(true);
                mockGateway.Expects.One.MethodWith(m => m.Persist(user)).WillReturn(true);
            }

            //Assign gateway
            user.Gateway = mockGateway.MockObject;

            //Test method
            Assert.AreEqual(true, user.Persist(mockValidator.MockObject));

            factory.VerifyAllExpectationsHaveBeenMet();
        }
    }
}

The migration is quite easy: Replace the reference from your test project from the NMock2.dll with a new reference to the NMock3.dll. After that you could use the new approach with the MockFactory and the lambda expressions in your Expectations.

NMock3 introduce type safety and readiness for refactorings combined with the easiness of NMock2. They also keep the tradition of a good cheat sheet.

With NMock3 I couldn’t mock a class (even when I followed the exceptions…"). The creation of stubs should be possible. The project is still a young one and so the documentation isn’t that good as it should be. But NMock3 remain an interesting mocking framework.

Simple.Mocking

When I looked for mocking frameworks I also found this new and not very well known framework Simple.Mocking on codeplex. Its also avoid “magic strings” and you could also use the AAA-Syntax. Simple.Mocking has only one committer, Mikael Waltersson.

namespace MockingTest
{
    [TestClass]
    public class SimpleNetTest
    {
        [TestMethod]
        public void TestPersist()
        {
            //Create mocks
            var expectationScope = new ExpectationScope();
            IUserGateway mockGateway = Mock.Interface<IUserGateway>(expectationScope);
            IUserValidator mockValidator = Mock.Interface<IUserValidator>(expectationScope);

            //Create user
            User user = new User();

            //Expectations
            Expect.Once.MethodCall(() => mockValidator.Validate(user)).Returns(true);
            Expect.Once.MethodCall(() => mockGateway.Persist(user)).Returns(true);

            //Assign gateway
            user.Gateway = mockGateway;

            //Test method
            Assert.AreEqual(true, user.Persist(mockValidator));

            AssertExpectations.IsMetFor(expectationScope);
        }
    }
}

The ExpectationsScope is used to avoid to call AssertExpectations.IsMetFor for each mock. This construct is special and I’m (currently) not a big fan of it.

With Simple.Mocking you could just mocking interfaces and delegates, you couldn’t mock classes.

Rhino.Mocks

Rhino.Mocks is one of the famous .Net mocking frameworks. As Simple.Mocking there is also only one committer to this framework, Oren Eini (alias Ayende Rahien).

namespace MockingTest
{
	[TestClass]
	public class RhinoMockTest
	{
		[TestMethod]
		public void TestPersistWithRecordingReplay()
		{
			MockRepository mocks = new MockRepository();

			IUserGateway gateway = mocks.StrictMock<IUserGateway>();
			IUserValidator validator = mocks.StrictMock<IUserValidator>();

			//Create user
			User user = new User();

			//Expectations
			using(mocks.Ordered())
			{
				Expect.Call(validator.Validate(user)).Return(true);
				Expect.Call(gateway.Persist(user)).Return(true);
			}

			//Stop recording, start replay
			mocks.ReplayAll();

			//Assign gateway
			user.Gateway = gateway;

			//Test method
			Assert.AreEqual(true, user.Persist(validator));

			mocks.VerifyAll();
		}

		[TestMethod]
		public void TestPersistWithArrangeActAssert()
		{
			var gateway = MockRepository.GenerateMock<IUserGateway>();
			var validator = MockRepository.GenerateMock<IUserValidator>();

			//Create user
			User user = new User();

			//Expectations
			validator.Expect(x => x.Validate(user)).Return(true);
			gateway.Expect(x => x.Persist(user)).Return(true);

			//Assign gateway
			user.Gateway = gateway;

			//Test method
			Assert.AreEqual(true, user.Persist(validator));

			//Asserts
			validator.VerifyAllExpectations();
			gateway.VerifyAllExpectations();
		}
	}
}

Rhino.Mocks supports two ways to use the framework: the record-replay and the AAA-syntax. My favorite is clearly the AAA-syntax, because there is much less code needed (reduced accidental complexity). Rhino.Mocks uses also Lambda-Expressions to avoid “magic strings”, so refactoring shouldn’t be a problem.

Moq

Moq is another mocking framework which takes advantage of the .Net 3.0 language features like lambda expressions. So Moq doesn’t use “magic strings” and it enables also the AAA-syntax. There are two ways how to create mocks. You could create mocks directly with the Mock class or you could use the MockFactory class. For both scenarios I created a test.

namespace MockingTest
{
	[TestClass]
	public class MoqTest
	{
		[TestMethod]
		public void TestPersist()
		{
			var gateway = new Mock<IUserGateway>();
			var validator = new Mock<IUserValidator>();

			User user = new User();

			//Expectations
			validator.Setup(x => x.Validate(user)).Returns(true).AtMostOnce();
			gateway.Setup(x => x.Persist(user)).Returns(true).AtMostOnce();

			//Assign gateway
			user.Gateway = gateway.Object;

			//Test method
			Assert.AreEqual(true, user.Persist(validator.Object));

			validator.VerifyAll();
			gateway.VerifyAll();
		}

		[TestMethod]
		public void TestPersistWithFactory()
		{
			MockFactory factory = new MockFactory(MockBehavior.Strict);

			//Class works, but methods have to be virtual -> not nice
			var mockGateway = factory.Create<UserGateway>();
			var mockValidator = factory.Create<IUserValidator>();

			User user = new User();

			//Expectations
			mockValidator.Setup(x => x.Validate(user)).Returns(true).AtMostOnce();
			mockGateway.Setup(x => x.Persist(user)).Returns(true).AtMostOnce();

			//Assign gateway
			user.Gateway = mockGateway.Object;

			//Test method
			Assert.AreEqual(true, user.Persist(mockValidator.Object));

			factory.VerifyAll();
		}
	}
}

I prefer the MockFactory approach, because you don’t need to verify each mock (its reduce the test code which is a good thing). Be sure, that in this scenario you don’t forget the parameter MockBehavior.Strict which makes your mock objects real mocks. Without this parameter they are more like Stubs, which don’t fail and just return a default value.

But there is a little problem: If you want to mock a class, then all the methods have to be virtual, but Moq has no problem to mock an interface.

Typemock Isolator

Unfortunately Typemock Isolator isn’t free, but you could get a trial to try this great mocking framework. As you could see in the TestPersistWithReflectiveMocking Typemock Isolator could mock classes, also non-virtual methods. Great. It also doesn’t use “magic strings”. But wait, you could not mock interfaces? With the new AAA-syntax it is possible and it looks very nice.

namespace MockingTest
{
	[TestClass]
	public class TypeMockTest
	{
	    [TestMethod]
	    public void TestPersistWithNaturalMocking()
	    {
	        MockManager.Init();

	        User user = new User();

	        // Natural mocking
	        using(RecordExpectations recorder = new RecordExpectations())
	        {
	            IUserGateway mockGateway = new UserGateway();
	            IUserValidator mockValidator = new UserValidator();

	            //Expectations
	            recorder.ExpectAndReturn(mockValidator.Validate(user), true);
	            recorder.ExpectAndReturn(mockGateway.Persist(user), true);
	        }

	        //Create instances
	        IUserGateway gateway = new UserGateway();
	        IUserValidator validator = new UserValidator();

	        //Assign gateway
	        user.Gateway = gateway;

	        //Method
	        Assert.AreEqual(true, user.Persist(validator));

	        MockManager.Verify();
	    }

	    [TestMethod]
	    public void TestPersistWithReflectiveMocking()
	    {
	        MockManager.Init();

	        //Create mocks
	        Mock mockGateway = MockManager.Mock<UserGateway>();
	        Mock mockValidator = MockManager.Mock<UserValidator>();

	        //Create user
	        UserGateway gateway = new UserGateway();
	        IUserValidator validator = new UserValidator();
	        User user = new User(gateway);

	        //Expectations
	        mockValidator.ExpectAndReturn("Validate", true).Args(user);
	        mockGateway.ExpectAndReturn("Persist", true).Args(user);

	        //Method
	        Assert.AreEqual(true, user.Persist(validator));

	        MockManager.Verify();
	    }

		[TestMethod]
		[Isolated]
		public void TestPersistWithAAA()
		{
			//Arrange
			//Create mocks
			UserGateway gateway = Isolate.Fake.Instance<UserGateway>();
			IUserValidator validator = Isolate.Fake.Instance<IUserValidator>();
            //Create user
			User user = new User(gateway);
            //Expectations
			Isolate.WhenCalled(() => validator.Validate(user)).WillReturn(true);
			Isolate.WhenCalled(() => gateway.Persist(user)).WillReturn(true);

			//Act
			bool bPersist = user.Persist(validator);

			//Assert
			Assert.AreEqual(true, bPersist);
		}
	}
}

Typemock Isolator also supports the record-replay approach (demonstrated in the TestPersistWithNaturalMocking test method). But as you already know, I prefer the AAA-syntax.

The question about Typemock Isolator is, if it is worth to pay for a mocking framework if there are so much other free opportunities?  Typemock Isolator is very powerful and for some companies it is very important to get support and also guarantee that the product will be continued for example ten years. In such a scenario Typemock Isolator would be definitely an option. In reaction of such scenarios, also other providers of mocking frameworks offer commercial support for their frameworks, for example Rhino.Mocks.

The undocumented – NUnit Mocking

It is not very known that there exists also in NUnit a mocking framework. It is possible to mock interfaces and classes if the inherit from MarshalByRefObject.

namespace MockingTest
{
	[TestClass]
	public class NUnitMockTest
	{
		[TestMethod]
		public void TestPersist()
		{
			//Gateway
			DynamicMock mockGateway = new DynamicMock(typeof(IUserGateway));
			IUserGateway gateway = (IUserGateway) mockGateway.MockInstance;

			//Validator
			DynamicMock mockValidator = new DynamicMock(typeof(IUserValidator));
			IUserValidator validator = (IUserValidator)mockValidator.MockInstance;

			//User
			User user = new User(gateway);

			//Expectations
			mockValidator.ExpectAndReturn("Validate", true, user);
			mockGateway.ExpectAndReturn("Persist", true, user);

			Assert.AreEqual(true, user.Persist(validator));
			mockValidator.Verify();
			mockGateway.Verify();
		}
	}
}

The mocking framework is easy to use and if you already use NUnit, then you don’t need an additional framework.

As I know, it isn’t official and there is no documentation. You have also to use “magic strings” for methods and the limitation for classes also reduce the benefit.

Conclusion

For a modern mocking framework it is essential that its enable refactoring by not using “magic strings”. It is also essential for me that it allows to use the AAA-syntax. A third essential thing is that it allows you to mock classes. You don’t want to create interface just for mocking, for example for your domain model. Avoid accidental complexity is very important, so you shouldn’t add unnecessary code just for mocking. That doesn’t mean that “design for testability” is wrong, “design for testability” means, that you design your code in a way, that you have the possibility to test your code, for example you avoid huge classes (god object anti-pattern) with a lot of private methods or you avoid creation of instances in methods (unable to use dependency injection).

So, which framework fulfill all the requirements? NMock3, which inherit the easiness of NMock2 combined with the elimination of the “magic strings”? Or Typemock Isolator which also eliminate the “magic strings” and could even mock classes?

For me, there isn’t currently a clear winner visible, maybe Typemock makes the race. But there are frameworks which I would no longer use, as NMock2 or NUnit mocking. They don’t fulfill my requirements clearly. That doesn’t mean they aren’t suitable for your scenarios.

Also to be clear: There are other topics that should be considered: mocking events, Stubs, etc. The list of the mocking frameworks isn’t complete, for example Moles is a framework which could be interesting too.

Last week I migrated several projects from NUnit to MSTest. The developers use the Developer version of Microsoft Visual Studio Team System, so they have integrated unit-test support for MSTest. In this post I show you all the problems and work I had to migrate the tests from NUnit to MSTest.

Less assert-methods in MSTest
In NUnit you have comparison assert methods, exception asserts, more type asserts, more utility asserts and so on (you see what I mean). There already exist little projects to add more assert methods for MSTest.
The problem with less assert method is the following:

Assert.Greater(nCount, 5);

This snippet is much more readable and maintainable than the following:

Assert.IsTrue(nCount > 5);

But that’s just my opinion. There are for sure people who like the second snippet more than the first one.

Attribute mappings
When you planning to migrate your NUnit Tests to MSTest you will need a mapping between the attributes of NUnit and MSTest. You’ll find on the internet mostly the following configuration:
[TestFixture] -> [TestClass]
[Test] -> [TestMethod]
[SetUp] -> [TestInitialize]
[TearDown] -> [TestCleanup]
[TestFixtureSetUp] -> [ClassInitialize]
[TestFixtureTearDown] -> [ClassCleanup]
That’s fine, but wait a minute! For two of those mappings you couldn’t just use find and replace. TestFixtureSetUp and TestFixtureTearDown aren’t static methods in NUnit. Methods with the attribute ClassInitialize or ClassCleanup has to be static. The method with the attribute ClassInitialize has even a parameter Testcontext. For those two methods you have to do a little bit more work.

TestFixtureSetup vs. ClassInitialize
As I mention above the difference between the attributes TestFixtureSetup and ClassInitialize (or TestFixtureTearDown and ClassCleanup), there’s still the question which one is conceptionally correct. Or why Microsoft did it different in comparison to NUnit.
I found the interesting answer in the book xUnit Test Patterns of Gerard Meszaros (on p. 384). Microsoft implements the JUnit approach, what means there will be for each test method a new instance of the test class. James Newkirk (one of the authors of NUnit) regret the decision in NUnit to us the same instance for all the test methods.

Ignore-attribute, no comment
The developers used a lot the attribute ignore in NUnit to deactivate unit tests for explorating testing. In NUnit you could add a comment to justify why the test should be ignored.
In MSTest the ignore attribute does also exists, but you couldn’t add a comment.

Resharper & MSTest
Since Resharper 4.5 you could also use this amazing tool with MSTest. Before the version 4.5 it was also possible to use it with MSTest, but you have to install a plug-in to do that.
From my perspective the Resharper unit test runner is much more better than the Visual Studio test runner.

ExpectedException and Visual Studio
In Visual Studio Professional and above you have an integrated unit-test runner (similar to eclipse, netbeans, etc.). There were a lot of unit-tests which expected an exception with the type Exception. That’s works fine in Resharper, on the build server (TFS) but not in the test runner of Visual Studio. Here you’ll receive the following warning:
UTA017: MyUnitTest has invalid ExpectedException attribute. The type must inherit from Exception.
This is a bug of the test runner, as others already described.

Execute several assemblies
One of the projects which I migrated from NUnit to MSTest runs unit tests of different assembles. So it had a nunit-file like that:

<NUnitProject>
    <Settings activeconfig="Web">
    <Config name="Web" appbase="Web" configfile="web.config" binpathtype="Auto">
        <assembly path="bin\Product.NUnit.ComponentA.dll"/>
        <assembly path="bin\Product.NUnit.ComponentB.dll"/>
        <assembly path="bin\Product.NUnit.ComponentC.dll"/>
    </Config>
</NUnitProject>

I didn’t found a way to do that in MSTest.

Conclusion
With the migration the developers gain code coverage and the integration of unit-testing in Visual Studio (for those who hadn’t Resharper). On the other hand they lost readability, because they have to express an assertion with code with a primitive assert method.
If I could give you an advice, it’s the following: Migrate, if you have at least one of the Visual Studio Team System Editions (Developer, Test or Suite). With Visual Studio Professional (integrates a unit-test runner) and without Resharper, it is also an option to migrate to MSTest.

When you create your unit tests for a method in the SUT (software under test) you will ask yourself how to structure the code in the test method.
I saw two kind of syntaxes which help to structure the code in a unit test method (well, actually there are at least three, but the third syntax is just chaos, so this is definitely not the way how to do it).

SEVT-Syntax

[TestFixture]
public class ThingTest
{
    [Test]
    public void TestSomething()
    {
        // Setup

        // Exercise

        // Verify

        // Teardown
    }
}

I found this syntax in the xUnit Test Patterns book by Gerard Meszaros. Obviously this syntax is inspired by classic unit frameworks (like junit or nunit). There is one thing, which could be critical: this syntax could be misused when in the teardown block exist code, which has to run always (in other words, which has to be correctly in teardown-method). But when an assert fails, the teardown block would not be invoked. To prevent this case, you have to do extra effort (try/finally construct).
Gerard Meszaros named this usage of the syntax Four-Phase Test. He also showed, that the last phase (the teardown block) could be done by a teardown-method.

AAA-Syntax

[TestFixture]
public class ThingTest
{
    [Test]
    public void TestSomething()
    {
        // Arrange

        // Act

        // Assert
    }
} 

This syntax I found on the blog of William Wake and ayende. I prefer this syntax, because it sounds good and therefore I don’t forget it. I like also the words which are more precise and there are no competition with the setup- or teardown-methods. It is also clear where to put the asserts.

Conclusion

The most important thing is that you structure your code in your unit-test. It is also very important that you test only one concern per test method. A syntax helps you to create readable, maintainable and specific (one test-concern per test method) tests.

There exists a project at Microsoft which generate unit tests based of source code. The name of this interesting project is Pex.
At the first time when I heard about this project I was skeptical.

• Where’s the value of generated tests?
• Doesn’t it break the “test first” approach?

After I saw this screencast from the PDC 2008, I could find the answers for me:
There is a value: It is often tedious to write all the unit tests with the extreme values which could pass to the method. For this tedious work it’s nice that you could generate all this “technical” unit tests. For this reason I would create an assembly where I would generate all this unit tests and separate them from the other unit tests, which I wrote by hand (and with the “test first” approach, like TDD).
The answer for the second question if it brakes the “test first” approach, is yes. But it isn’t a bad thing by nature. The generated tests are more technical stuff and they don’t serve as examples or specification.

When you use a code coverage tool one of the first question is what is a good code coverage. Recently I listened to different podcasts (stackoverflow, scott hanselman) where they discuss this topic. I wasn’t really surprised that there wasn’t one unique opinion.
One opinion was that 100% is a good number, an other opinion was that for non-failing code it doesn’t make sense to build a test, so 100% isn’t a good number for most of the projects.
The last opinion sounds not that bad, but for which code it doesn’t make sense to build unit tests?
Here is a (not complete) list of code elements for which I wouldn’t build a unit-test:

• Properties (.Net)
• Getters (Java)
• Setters (Java)
• Primitive Constructors (no chaining, no big logic in it)
• Services (which just delegate to the business logic)
• GUI (Views)
• Data Access (do not test a framework)
• Services of another component or another application (do not test a library or another application)
• Any Generated Code (do not test a tool, in this case a code generator)

Is there a code element or type which I forget?