There are a number of things related to the mechanics of testing that need to be worked out, such as what types of tests to run, how to structure test projects, and even how to run the tests effectively.
Types of Testing
There are many types of tests in addition to unit tests, some of which frequently get mistakenly referred to as unit tests. They all have their places and all testing is good, so it is easy to fall into the mind set of thinking it does not matter what type of testing is being done. An explanation of some of the types of testing follows:
- Unit Test – This is a small test which ensures that one small piece of code, usually a single method, performs as expected. It is important to keep the test small and divorced from system resources so it can be run fast. In a medium to large application, there may be hundreds or even thousands of unit tests, all of which need to be run often during the development process.
- Integration Test or Functional Test – This is generally larger than a unit test, but not much. The goal of a functional test is to ensure that several pieces of code work together to meet a specified requirement correctly. These tests will generally be allowed to access system resources to ensure that they provide the expected service and to ensure that failure of the resource is handled correctly.
- Acceptance Test or System Test – This is a large scale test, used to ensure that the overall system provides the expected service in a useable and easy manner. Sometimes these are called Black Box tests, because they test only the user interface and need to know nothing about the internal source code. Generally, some manual tests are required since usability is difficult for programs to determine, but large scale system tests can be automated to ensure that crashes do not occur.
Since WatiN tests the UI portion of an application, many developers put such tests into the realm of functional tests or even system tests, but since the tests are generally structured to test small portions of the interface at a time, many would say that these are unit tests. The truth is probably both are correct, depending on how the tests are written and what is being tested, which is true of any test framework. For instance, it is just as easy to write functional tests with NUnit as it is to write unit tests. It is up to the developer to structure their tests to meet the needs of the task at hand. The key is to ensure that all non-trivial functionality is covered with as many different cases as necessary to catch potential bugs, and do it all in an efficient manner.
Structuring Unit Test Projects
Most examples that you will find on how to use WatiN or another UI test environment do not address the critical question of how to organize your unit tests to be able to modify, add or run specific tests easily. Most developers who have worked with unit tests for a while have a system, but since it is a background item to the topic, it usually is not discussed during demonstrations of how a test itself works.
Method One - It is tempting to create a solution for running WatiN unit tests and then just keep plunking tests into it for whatever web page needs to be tested next. This is the path many take when first experiencing the power of unit testing with WatiN, since the projects containing pages to be tested do not have to be referenced. Eventually, the number of tests will grow to the point where it is difficult to establish which tests need to be run at any one time. The following example shows the beginning of such a project.
It can be seen that this is a Windows Forms application because it contains a FormMain. It also can be switched to being a Console application because the MainModule is written to be a control module for all the tests. There are several utility modules to help with the nuts and bolts of testing (Assert.vb, Logger.vb, TestUtilities.vb) and there is one module actually created to test a web page (EquipmentListing.vb). It is easy to see that after 20 or 30 pages have had tests written, this simple seeming list will begin to get cumbersome.
Method Two - A next step frequently taken is to add folders to the solution to contain tests for particular applications or pages. All related tests can then be segregated in this manner, making their identification easier, if not their processing.
This sample shows that even with twenty more tests added to the solution, the folders have permitted the tests to be ordered in such a way that it is easy to find a particular test. The drawback is that each of the folders equates to an entire application, each of which has one or more pages, so the folders can still get quite full. There is no need to continue to run all tests for all applications when no changes have been made to some of them. For example, if working in Equipment only, why run tests for Booking when no changes have been made there. There are ways to run only part of the tests by modifying the controlling code. This leaves open the possibility of forgetting to reactivate tests that are needed later, however.
Method Three - A preferred method is also the most logical once the time is taken to reflect. Each application to be tested should have its own WatiN test project created which tests only that application. Within the project, folders can be arranged for each page to further segregate the tests. Frequently a page can be fully tested with only one test class, but it is still wise to create the separate folders for consistency. Some people like to place this test project within the same solution as the application to be tested, but many feel that it makes for cleaner builds and installation to have a completely separate solution/project for the tests, which is the method promoted here.
This sample shows that the project is only testing the Equipment application. There are folders included to store tests for each of the web pages included in the application. Now the controlling form or module can freely run all tests each time and it is easy to find or add a new test in the proper location.
What some may point to as the one small drawback of this method is that the overhead modules (Logger.vb and Assert.vb) will have to be included in each project. This is true, but is a minor inconvenience that will be made moot when the advantage of testing with a test framework such as NUnit is explained.
Deciding to Control Testing with a Test Framework or an Application
WatiN will work equally well from either a stand alone application, such as a console or Windows Forms application, or from within one of the popular unit testing frameworks, such as NUnit or MbUnit. Both methods have advantages and examples of each will be given.
- Application – Applications allow you to handle things outside of the tests related to the environment in a manner which can increase the speed of the tests.
- Test Frameworks – Frameworks are excellent at revealing what tests are available to be run, which ones passed or failed and providing a method to run groups of tests. Test frameworks can test both UI and background classes all in one project.
The main differences in the two styles are in how the test code is decorated, how the test code is run and how results are reviewed.
Code Differences
The following two examples perform the same test. The first is for a console application and the second is for the NUnit framework:
Imports WatiN.Core
''' <summary>
''' Test the Scorecard application main page
''' </summary>
''' <remarks>
''' Uses Console testing methods
''' </remarks>
Public Class Scorecard
Public Const CustomerName As String = "ctl00$ContentPlaceHolder$ShipperPoolRequest_CustomerName_TextBox"
Private m_appURL As String = "Scorecard/"
Private m_currentURL As String = m_BaseURL & m_appURL & "default.aspx"
Private m_logger As New Logger
''' <summary>
''' Ensure the page initializes itself to have blank or alphabetically low entries in the drop down lists.
''' Ensure the page does not initially display any information.
''' </summary>
''' <remarks></remarks>
Public Sub TestInit()
Dim ie As IE = New IE(m_currentURL)
Assert.IsTrue(ie.ContainsText("Interchange Scorecard"), "Title not found on page")
Assert.IsFalse(ie.SelectList(Find.ByName("Owner_DropDownList")).Text.Equals("MSK"), "Owner should not start pointing to MSK")
Assert.IsFalse(ie.SelectList(Find.ByName("Operator_DropDownList")).Text.Equals("SLX"), "Operator should not start pointing to SLX")
Assert.IsTrue(ie.ContainsText("No Information found"), "Initial page should not find data.")
ie.Close()
End Sub
End Class
The following code reflects the NUnit version of the same test.
Imports WatiN.Core
Imports NUnit.Framework
''' <summary>
''' Test the Scorecard application main page
''' </summary>
''' <remarks>
''' Uses NUnit testing methods
''' </remarks>
<TestFixture()> _
Public Class Scorecard
Private m_appURL As String = "Scorecard/"
Private m_currentURL As String = m_BaseURL & m_appURL & "default.aspx"
Private m_logger As New Logger
''' <summary>
''' Ensure the page initializes itself to have blank or alphabetically low entries in the drop down lists.
''' Ensure the page does not initially display any information.
''' </summary>
''' <remarks></remarks>
<Test()> _
Public Sub TestInit()
Dim ie As IE = New IE(m_currentURL)
Assert.IsTrue(ie.ContainsText("Interchange Scorecard"), "Title not found on page")
Assert.IsFalse(ie.SelectList(Find.ByName("Owner_DropDownList")).Text.Equals("MSK"), "Owner should not start pointing to MSK")
Assert.IsFalse(ie.SelectList(Find.ByName("Operator_DropDownList")).Text.Equals("SLX"), "Operator should not start pointing to SLX")
Assert.IsTrue(ie.ContainsText("No Information found"), "Initial page should not find data.")
ie.Close()
End Sub
End Class
The code for these two tests is almost identical. The only difference is that NUnit requires some decoration of the code with attributes to identify the tests, while the console application will need a main module somewhere to call the test modules and will have to its own assert methods (not shown here).
How Console Applications and Test Frameworks Run Tests
A console application runs unit tests in pretty much the same way it runs any program. A main module makes calls to other methods, which may make calls to additional methods, until all desired tests are run. In our proposed methodology, the project would contain a class for each web page to be tested. Each of these classes would then have a main method to act as a controller for calling the unit test methods within the class. The main entry module for the project (or the Windows Form) would call the controlling method in each class for each page to be tested. Calls to any tests which should not be run during the current iteration would be commented out.
With console applications, there must also be a mechanism created to save and display the results of each test and a way of determining if tests pass or fail.
A test framework will automatically recognize all of the unit tests in the project based on the <Test()> attribute assigned to each test method. These will be displayed in the runner for the test framework. Results will usually display with the familiar red and green indicators, red for failure and green indicating pass. Test frameworks have assert classes built in to be used in determining if the tests pass or fail.
Another way to run NUnit tests is with the TestDriven.NET add-on to the Visual Studio IDE provided by TestRunner.NET. This utility will allow you to right click on any node in the project tree or any test method and run the tests directly from the IDE. The results are shown at the bottom of the screen in the lower left corner and messages will appear in the Output window. TestDriven.NET also allows you to step through both the test and the code being tested with the debugger to determine if a failure is caused by the code or if the test itself has a bug. It also bundles the popular NCover utility to help with determining what lines of code are being tested.
Contents: Table of Contents Previous Page: Introduction Next Page: How to use WatiN to Test Web Pages - Part I