TestDriven.Me

I hope to spend some time discussing testing with VB.NET. I don't think of myself as an expert, but I have had some experience and certainly some opinions. I will try to share some of those opinions along with some information and hopefully generate some discussion so that we all can improve our skills a bit.

This Blog

Syndication

Search

Navigation

March 2008 - Posts

How to Use WatiN to Test Web Pages - Part 1

As I have mentioned elsewhere, you can drive WatiN tests with either an application or a test framework. Examples of each will be given in this section, even though most developers seem to prefer a test framework. The TestDriven.NET add-in for the Visual Studio IDE makes using a test framework even easier.

Testing with an Application

With either a console or a Windows Form application the developer will have to provide certain utilities that are built into test frameworks such as NUnit for determining the success or failure of tests and also for displaying the test results. The MyAssert and Logger classes have been written to provide these services. Full source code for these classes is provided at the end of this section. You are welcome to use them or modify them as needed if you choose to control your tests with an application.

Create a new Application

To begin, create a new project, selecting the Console Application template. You will need to select a name for the application and a place to create it as shown next:

Clicking the OK button will create a bare project like that shown next:

I like to immediately rename Module1.vb to MainModule.vb so it is clear which the controlling module is.

Add References and Utilities

Right click on the project name (WatiNConsoleApplication) and select AddReference from the menu. Add the WatiNCore.DLL from the Bin folder that you installed WatiN into.

The next step is to copy the source of MyAssert.vb and Logger.vb from wherever you have it saved on your drive into the project folder. Then right click on the project name in Solution Explorer and select Add – Existing Item as shown next:

NOTE: If you have not yet saved copies of the utilities locally, you can just use the Add – Class context menu on your solution and then copy and paste the code into the classes as you create them in this solution.

At this point you should be able to get a clean build (Alt-Shift-B). Not very useful yet, but it helps to build and test often so you know that something you just did is either the cause of any problems you encounter, or it caused an earlier created problem to become visible.

Create Modules to be Tested

Let’s start the testing process by creating a class that will contain math functions. Everyone knows what math functions should do, so they will be easy to create tests for. Normally, you would want to write the unit test prior to creating the class, but since this section is focused on how to use the tools, it will be easier to understand if we have a class to test already completed.

Right click on the project name in your solution and select Add – Class. Create a class called MathFunctions.vb.

In the new class enter the GetPercentage function we examined in the Introduction section:

    Public Function GetPercentage(ByVal whatNumber As Double, _
ByVal whatDivisor As Double) As Decimal
        Dim answer As Decimal
 
        answer = (whatNumber / whatDivisor) * 100
         Return answer
    End Function
 

Create Unit Tests

Create a new class that will contain the unit tests for the math functions. Right click on the project name and select Add – Class. Create a class called MathFunctionsTest.vb.

In the new class enter the following test, which is very similar to the NUnit test we used earlier.

    Public Sub TestGetPercentage()
        Dim answer As Decimal
        Dim numberOne As Double
        Dim numberTwo As Double
        Dim myMathFunctions As New MathFunctions
         numberOne = 2.0
        numberTwo = 4.0
        answer = myMathFunctions.GetPercentage(numberOne, numberTwo)
  MyAssert.IsTrue("TestGetPercentage", answer = 50D, _
"GetPercentage returned " & _
            answer.ToString & " when provided " & _
            numberOne.ToString & " and " & numberTwo.ToString & _
            " when expected 50", False)
    End Sub

Once again Alt-Shift-B should get you a clean build, but the test still will not run, because it has not been called from the main module. Modify Sub Main() of the Main Module as shown next:

    Sub Main()
        Dim myMathFunctionsTest As New MathFunctionsTest
 
        myMathFunctionsTest.TestGetPercentage()
    End Sub

Now the program runs just fine, but you have no idea if the test failed or not (unless you remembered from looking at the code in MyAssert.vb and Logger.vb that an error log called ErrorLog.txt would be written to the C:\AppData\ folder if an assert fails). You could look for that, but hopefully it is not there from which you can infer that there were no errors. The missing file still does not give a warm fuzzy feeling that everything is all right though.

This points out another of the drawbacks of the console app. You have to put in some extra logic just to make your logging efforts more useful and more visible. Once we add a few lines to both TestGetPercentage and Sub Main, they look like the following:

    Public Sub TestGetPercentage()
        Dim answer As Decimal
        Dim numberOne As Double
        Dim numberTwo As Double
        Dim myMathFunctions As New MathFunctions
 
        m_logger.LogResultToTextFile("TestGetPercentage Test Start" & _
            StrDup(32, "-"), True, True)
 
        numberOne = 2.0
        numberTwo = 4.0
        answer = myMathFunctions.GetPercentage(numberOne, numberTwo)
        MyAssert.IsTrue("TestGetPercentage", answer = 50D,
           "GetPercentage returned " & _
            answer.ToString & " when provided " & _
            numberOne.ToString & " and " & numberTwo.ToString & _
            " when expected 50", False)
 
        m_logger.LogResultToTextFile("TestGetPercentage Test End " & _
      StrDup(32, "-"), True, True)
 
    End Sub
 
    Sub Main()
        Dim myMathFunctionsTest As New MathFunctionsTest
 
        'Open test log at start of test
        m_logger.LogResultToTextFile("Test Run Begins ", False, True)
 
        myMathFunctionsTest.TestGetPercentage()
 
        m_logger.LogResultToTextFile("Test Run Ends ", True, True)
 
        m_logger.ShowLogs() 
    End Sub

Now when you run the test, you are rewarded with the following file displayed in NotePad:

================================

Wednesday, February 27, 2008 2:36 PM

Test Run Begins 

================================

Wednesday, February 27, 2008 2:36 PM

TestGetPercentage Tests Start --------------------------------

================================

Wednesday, February 27, 2008 2:36 PM

TestGetPercentage Tests End --------------------------------

================================

Wednesday, February 27, 2008 2:36 PM

Test Run Ends

You can see that the test run began, it called the TestGetPercentage module and everything closed up with no error message given. This is rather satisfactory, but if we refactor the code just a little bit, it will be much more extensible for adding more tests:

In the MathFunctionsTest class, add the following functions at the top of the class prior to the current TestGetPercentage subroutine.

    Public Sub RunTests()
 
        TestSetup()
 
        'Call each test that will be run within the class here
        TestGetPercentage()
 
        TestTeardown()
 
    End Sub
 
    Sub TestSetup()
 
        'Open test log at start of test
        m_logger.LogResultToTextFile("MathFunctionsTest Tests Begin", _
True, True)
 
    End Sub
 
    Sub TestTeardown()
 
        m_logger.LogResultToTextFile("MathFunctionsTest Tests End ", _ 
True, True)
 
    End Sub

In the Main Module modify the Sub Main and add the following methods as shown next:

    Sub Main()
        Dim myMathFunctionsTest As New MathFunctionsTest
 
        TestSetup()
 
        myMathFunctionsTest.RunTests()
 
        TestTeardown()
 
    End Sub
 
    Sub TestSetup()
 
        'Open test log at start of test
        m_logger.LogResultToTextFile("Test Run Begins ", False, True)
 
    End Sub
 
    Sub TestTeardown()
 
        m_logger.LogResultToTextFile("Test Run Ends ", True, True)
 
        m_logger.ShowLogs() 
    End Sub

By adding the TestSetup and TestTeardown methods to each class, you can segregate code that needs to be run for all of the tests in the class. The RunTests method in the MathFunctionsTest and future test classes gives you an entry point to call within the class from Sub Main without having to reference every test in the class. That is what RunTests will do as you add further tests to the class.

Running the application now should provide you with the same display of tests started and ended with no errors. Now for the fun. Let’s create a test that will fail, highlighting one of the limitations of our GetPercentage function.

Add the following subroutine to MathFunctionsTest:

    Public Sub TestGetPercentageBadData()
        Dim answer As Decimal
        Dim numberOne As Double
        Dim numberTwo As Double
        Dim myMathFunctions As New MathFunctions
         m_logger.LogResultToTextFile("TestGetPercentage Test Start" & _ 
StrDup(32, "-"), True, True)
 
        numberOne = 2.0
        numberTwo = 0.0
        answer = myMathFunctions.GetPercentage(numberOne, numberTwo)
        MyAssert.IsTrue("TestGetPercentage", answer = 50D,
"GetPercentage returned " & _
            answer.ToString & " when provided " & _
            numberOne.ToString & " and " & numberTwo.ToString & _
            " when expected 50", False)
 
        m_logger.LogResultToTextFile("TestGetPercentage Tests End" & _
StrDup(32, "-"), True, True)
 
    End Sub

And then modify the TestStart sub in MathFunctionsTest to call this new sub after the original test:

    Public Sub RunTests()
 
        TestSetup()
 
        'Call each test that will be run within the class here
        TestGetPercentage()
        TestGetPercentageBadData()
 
        TestTeardown()
 
    End Sub

This will build just fine, but you will get a run-time error like that shown next when you try to run it.

Our tests have shown that the GetPercentage function is not protected against a divide by zero error. Also, the console application cannot log the error and continue with the tests once it reaches a run-time error. Time to break out the requirements for the project. Should the GetPercentage function trap a divide by zero and other errors that might occur or should it just throw an error when the developer tries something that should not be done.

You will find a good number of developers on each side of this question. One side will argue for Try…Catch block in the GetPercentage function to ensure that it does cause a system crash when it receives bad data. The other side will argue that the crash is good, because it shows that there really is a bug somewhere else that is sending the zero in the first place. Adding the Try…Catch block will mask the real error and it may never be corrected!

Our tests have shown that either the GetPercentage function needs to be refactored to have a Try…Catch block, or the tests themselves need to have a Try…Catch block added around each call to the actual method being tested. We are going to suggest not masking the real error and force it to be caught by a later test, so our change to the TestGetPercentageBadData is shown next:

    Public Sub TestGetPercentageBadData()
        Dim answer As Decimal
        Dim numberOne As Double
        Dim numberTwo As Double
        Dim myMathFunctions As New MathFunctions
 
        m_logger.LogResultToTextFile("TestGetPercentage Test Start" & _
StrDup(32, "-"), True, True)
 
        numberOne = 2.0
        numberTwo = 0.0
        Try
            answer = myMathFunctions.GetPercentage(numberOne, _
numberTwo)
        Catch ex As Exception
            m_logger.LogResultToTextFile("TestGetPercentageBadData" & _
" threw a divide by zero error as expected ", & _
True, False)
        End Try
        MyAssert.IsTrue("TestGetPercentage", answer = 0D, _
"GetPercentage returned " & _
            answer.ToString & " when provided " & _
            numberOne.ToString & " and " & numberTwo.ToString & _
            " when expected 0", False)
 
        m_logger.LogResultToTextFile("TestGetPercentage Tests End " & _
StrDup(32, "-"), True, True)
 
    End Sub

Now running the application provides a nice clear report of no errors.

This test is quite a change from the original NUnit test shown earlier, however. This points out the other drawback of using a console application to drive unit tests; the tests are harder to write and not as clear. In that entire sub there are really only two lines important to the test itself, the call to GetPercentage and the assert. The rest is there to make the test run well and to be nice about showing us if it passed or failed.

As an exercise, the reader is invited to complete the coverage of the unit tests for the GetPercentage function by adding tests in the MathFunctionsTest class for sending one or both parameters as negative numbers and also as Nothing to see how the function will behave.

At this point, it seems safe to recommend using NUnit as the framework to run unit tests.

Source Code for Console Application Utilities

The following source is provided for the utilities required to make tests run correctly and report their results from console applications. They are provided, as is, and you are free to include them in your own applications, or to modify them as you see fit:

Logger.vb

This utility is provided to assist in logging test results to two different files. One file will contain detailed information about the tests, which is frequently desired by Quality Assurance testers to ensure that all required test steps are run and to keep track of information about data and the internal states of each test. The second file is for errors and minimal navigational information about which tests are being run.

''' <summary>

''' Create and maintain logs of what happens during a test run.  

''' Two logs will be created, one containing only errors with test start/stop messages,

''' and the other more detailed with indications of all actions taken.

''' </summary>

''' <remarks></remarks>

Public Class Logger

    Private ErrorLogPath As String = "C:/TestData/"

    Private TestLogName As String = ErrorLogPath & "TestLog.txt"

    Private ErrorLogName As String = ErrorLogPath & "ErrorLog.txt"

    ''' <summary>

    ''' Save the error message to the currently defined text file.

    ''' </summary>

    ''' <param name="errorMessage"></param>

    ''' <param name="appendFile"></param>

    ''' <remarks>ErrorLogName contains the name of the currently defined error log file.

    ''' </remarks>

    Public Sub LogResultToTextFile(ByVal errorMessage As String, ByVal appendFile As Boolean, ByVal isError As Boolean)

        Try

            If Dir(ErrorLogPath).Length = 0 Then

                System.IO.Directory.CreateDirectory(ErrorLogPath)

            End If

            If Dir(TestLogName).Length = 0 Then

                Dim textWriter2 As New System.IO.StreamWriter(TestLogName)

                With textWriter2

                    .Write(" ")

                    .Flush()

                    .Close()

                End With

            End If

            If Dir(ErrorLogName).Length = 0 Then

                Dim textWriter2 As New System.IO.StreamWriter(ErrorLogName)

                With textWriter2

                    .Write(" ")

                    .Flush()

                    .Close()

                End With

            End If

            Dim textWriter As New System.IO.StreamWriter(TestLogName, appendFile)

            With textWriter

                .Write(StrDup(32, "=") & vbCrLf)

                .Write(Now.ToLongDateString & " " & Now.ToShortTimeString & vbCrLf)

                .Write(errorMessage & vbCrLf)

                .Close()

            End With

            If isError Then

                Dim errorWriter As New System.IO.StreamWriter(ErrorLogName, appendFile)

                With errorWriter

                    .Write(StrDup(32, "=") & vbCrLf)

                    .Write(Now.ToLongDateString & " " & Now.ToShortTimeString & vbCrLf)

                    .Write(errorMessage & vbCrLf)

                    .Close()

                End With

            End If

        Catch ex As Exception

            MsgBox("There was a system error writing to one of the logs." & vbCrLf & vbCrLf & ex.Message & vbCrLf & vbCrLf & ex.StackTrace, MsgBoxStyle.OkOnly, "Error writing to Log")

        End Try

    End Sub

    ''' <summary>

    ''' Display the error log using the standard NotePad application 

    '''  that should be found in the path of all Windows computers.

    ''' </summary>

    ''' <remarks>

    ''' Usually called at the end of a test run to display any failed test messages.

    ''' </remarks>

    Public Sub ShowLogs()

        Dim dblWordID As Integer

        dblWordID = Shell("notepad " & ErrorLogName, vbNormalFocus)

        AppActivate(dblWordID)

    End Sub

End Class

MyAssert.vb

Asserts are used to determine if a specific test step has passed or failed. These methods are built into test frameworks, but must be provided by the developer if an application is used to drive tests.

Imports WatiN.Core

Public Class MyAssert

    Public Shared Function AreEqual(ByVal whatData As String, ByVal valueOne As String, ByVal valueTwo As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = True Then

                valueOne = ""

            End If

            If IsNothing(valueTwo) = True Then

                valueTwo = ""

            End If

            If valueOne.ToUpper.Equals(valueTwo.ToUpper) Then

                m_logger.LogResultToTextFile(whatData & " matched.", True, False)

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [" & valueOne & "] did not match [" & valueTwo & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function Contains(ByVal whatData As String, ByVal valueOne As String, ByVal valueTwo As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = True Then

                valueOne = ""

            End If

            If IsNothing(valueTwo) = True Then

                valueTwo = ""

            End If

            If valueOne.ToUpper.IndexOf(valueTwo.ToUpper) > 0 Then

                m_logger.LogResultToTextFile(whatData & " contained value.", True, False)

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [" & valueTwo & "] is not contained in [" & valueOne & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function
 
    Public Shared Function IsBlank(ByVal whatData As String, ByVal valueOne As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True
size=3>        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = True Then

                valueOne = ""

            End If

            If valueOne.Length = 0 Then

                m_logger.LogResultToTextFile(whatData & " correctly found to be blank", True, False)

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": contained data when expected to be blank [" & valueOne & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function IsFalse(ByVal whatData As String, ByVal valueOne As Boolean, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = False AndAlso valueOne = False Then

                m_logger.LogResultToTextFile(whatData & " was false.", True, False)

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & " was not false." & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function IsTrue(ByVal whatData As String, ByVal valueOne As Boolean, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = False AndAlso valueOne = True Then

                m_logger.LogResultToTextFile(whatData & " was true.", True, False)

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & " was not true." & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function NotBlank(ByVal whatData As String, ByVal valueOne As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = False AndAlso valueOne.Length > 0 Then

                m_logger.LogResultToTextFile(whatData & " contains data [" & valueOne & "]", True, False)

                aReturnValue = True

            Else

                If IsNothing(valueOne) Then

                    m_logger.LogResultToTextFile(whatData & ": Value to compare is null." & whatMessage & StrDup(64, "<"), True, True)

                    m_logger.LogResultToTextFile(whatData & ": Increase pause length." & StrDup(64, "<"), True, True)

                Else

                    m_logger.LogResultToTextFile(whatData & ": did not contain data when expected to do so." & whatMessage & StrDup(64, "<"), True, True)

                End If

                aReturnValue = False

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function NotContains(ByVal whatData As String, ByVal valueOne As String, ByVal valueTwo As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = True Then

                valueOne = ""

            End If

            If IsNothing(valueTwo) = True Then

                valueTwo = ""

            End If

            If valueOne.ToUpper.IndexOf(valueTwo.ToUpper) > 0 Then

                m_logger.LogResultToTextFile(whatData & ": [" & valueTwo & "] was contained in error in [" & valueOne & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            Else

                m_logger.LogResultToTextFile(whatData & " properly did not contain value.", True, False)

                aReturnValue = True

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

    Public Shared Function NotEqual(ByVal whatData As String, ByVal valueOne As String, ByVal valueTwo As String, Optional ByVal whatMessage As String = "", Optional ByVal expectException As Boolean = False) As Boolean

        Dim aReturnValue As Boolean = True

        Dim m_logger As New Logger

        Try

            If IsNothing(valueOne) = True Then

                valueOne = ""

            End If

            If IsNothing(valueTwo) = True Then

                valueTwo = ""

            End If

            If valueOne.ToUpper.Equals(valueTwo.ToUpper) Then

                m_logger.LogResultToTextFile(whatData & ": [" & valueOne & "] equal when not expected to [" & valueTwo & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            Else

                m_logger.LogResultToTextFile(whatData & " correctly not equal.", True, False)

                aReturnValue = True

            End If

        Catch ex As Exception

            If expectException = True Then

                aReturnValue = True

            Else

                m_logger.LogResultToTextFile(whatData & ": [threw an exception " & ex.Message & "]" & whatMessage & StrDup(64, "<"), True, True)

                aReturnValue = False

            End If

        End Try

        Return aReturnValue

    End Function

End Class

In my next post I will show how to perform unit testing with WatiN using the NUnit framework.

Contents: Table of Contents Previous Page: Unit Testing Methodology Next Page: How to use WatiN to Test Web Pages - Part II

Posted Monday, March 31, 2008 10:36 AM by ddodgen | with no comments

Filed under: WatiN, Unit Tests, VB.NET, Web Page Testing

Unit Testing Methodology

There are a number of things related to the mechanics of testing that need to be worked out, such as what types of tests to run, how to structure test projects, and even how to run the tests effectively.

Types of Testing

There are many types of tests in addition to unit tests, some of which frequently get mistakenly referred to as unit tests. They all have their places and all testing is good, so it is easy to fall into the mind set of thinking it does not matter what type of testing is being done. An explanation of some of the types of testing follows:

Unit Test – This is a small test which ensures that one small piece of code, usually a single method, performs as expected. It is important to keep the test small and divorced from system resources so it can be run fast. In a medium to large application, there may be hundreds or even thousands of unit tests, all of which need to be run often during the development process.
Integration Test or Functional Test – This is generally larger than a unit test, but not much. The goal of a functional test is to ensure that several pieces of code work together to meet a specified requirement correctly. These tests will generally be allowed to access system resources to ensure that they provide the expected service and to ensure that failure of the resource is handled correctly.
Acceptance Test or System Test – This is a large scale test, used to ensure that the overall system provides the expected service in a useable and easy manner. Sometimes these are called Black Box tests, because they test only the user interface and need to know nothing about the internal source code. Generally, some manual tests are required since usability is difficult for programs to determine, but large scale system tests can be automated to ensure that crashes do not occur.

Since WatiN tests the UI portion of an application, many developers put such tests into the realm of functional tests or even system tests, but since the tests are generally structured to test small portions of the interface at a time, many would say that these are unit tests. The truth is probably both are correct, depending on how the tests are written and what is being tested, which is true of any test framework. For instance, it is just as easy to write functional tests with NUnit as it is to write unit tests. It is up to the developer to structure their tests to meet the needs of the task at hand. The key is to ensure that all non-trivial functionality is covered with as many different cases as necessary to catch potential bugs, and do it all in an efficient manner.

Structuring Unit Test Projects

Most examples that you will find on how to use WatiN or another UI test environment do not address the critical question of how to organize your unit tests to be able to modify, add or run specific tests easily. Most developers who have worked with unit tests for a while have a system, but since it is a background item to the topic, it usually is not discussed during demonstrations of how a test itself works.

Method One - It is tempting to create a solution for running WatiN unit tests and then just keep plunking tests into it for whatever web page needs to be tested next. This is the path many take when first experiencing the power of unit testing with WatiN, since the projects containing pages to be tested do not have to be referenced. Eventually, the number of tests will grow to the point where it is difficult to establish which tests need to be run at any one time. The following example shows the beginning of such a project.

It can be seen that this is a Windows Forms application because it contains a FormMain. It also can be switched to being a Console application because the MainModule is written to be a control module for all the tests. There are several utility modules to help with the nuts and bolts of testing (Assert.vb, Logger.vb, TestUtilities.vb) and there is one module actually created to test a web page (EquipmentListing.vb). It is easy to see that after 20 or 30 pages have had tests written, this simple seeming list will begin to get cumbersome.

Method Two - A next step frequently taken is to add folders to the solution to contain tests for particular applications or pages. All related tests can then be segregated in this manner, making their identification easier, if not their processing.

This sample shows that even with twenty more tests added to the solution, the folders have permitted the tests to be ordered in such a way that it is easy to find a particular test. The drawback is that each of the folders equates to an entire application, each of which has one or more pages, so the folders can still get quite full. There is no need to continue to run all tests for all applications when no changes have been made to some of them. For example, if working in Equipment only, why run tests for Booking when no changes have been made there. There are ways to run only part of the tests by modifying the controlling code. This leaves open the possibility of forgetting to reactivate tests that are needed later, however.

Method Three - A preferred method is also the most logical once the time is taken to reflect. Each application to be tested should have its own WatiN test project created which tests only that application. Within the project, folders can be arranged for each page to further segregate the tests. Frequently a page can be fully tested with only one test class, but it is still wise to create the separate folders for consistency. Some people like to place this test project within the same solution as the application to be tested, but many feel that it makes for cleaner builds and installation to have a completely separate solution/project for the tests, which is the method promoted here.

This sample shows that the project is only testing the Equipment application. There are folders included to store tests for each of the web pages included in the application. Now the controlling form or module can freely run all tests each time and it is easy to find or add a new test in the proper location.

What some may point to as the one small drawback of this method is that the overhead modules (Logger.vb and Assert.vb) will have to be included in each project. This is true, but is a minor inconvenience that will be made moot when the advantage of testing with a test framework such as NUnit is explained.

Deciding to Control Testing with a Test Framework or an Application

WatiN will work equally well from either a stand alone application, such as a console or Windows Forms application, or from within one of the popular unit testing frameworks, such as NUnit or MbUnit. Both methods have advantages and examples of each will be given.

Application – Applications allow you to handle things outside of the tests related to the environment in a manner which can increase the speed of the tests.
Test Frameworks – Frameworks are excellent at revealing what tests are available to be run, which ones passed or failed and providing a method to run groups of tests. Test frameworks can test both UI and background classes all in one project.

The main differences in the two styles are in how the test code is decorated, how the test code is run and how results are reviewed.

Code Differences

The following two examples perform the same test. The first is for a console application and the second is for the NUnit framework:

Imports WatiN.Core

''' <summary>
''' Test the Scorecard application main page
''' </summary>
''' <remarks>
''' Uses Console testing methods
''' </remarks>
Public Class Scorecard
    Public Const CustomerName As String = "ctl00$ContentPlaceHolder$ShipperPoolRequest_CustomerName_TextBox"
    Private m_appURL As String = "Scorecard/"
    Private m_currentURL As String = m_BaseURL & m_appURL & "default.aspx"
    Private m_logger As New Logger

    ''' <summary>
    ''' Ensure the page initializes itself to have blank or alphabetically low entries in the drop down lists.
    ''' Ensure the page does not initially display any information.
    ''' </summary>
    ''' <remarks></remarks>
    Public Sub TestInit()
        Dim ie As IE = New IE(m_currentURL)

        Assert.IsTrue(ie.ContainsText("Interchange Scorecard"), "Title not found on page")
        Assert.IsFalse(ie.SelectList(Find.ByName("Owner_DropDownList")).Text.Equals("MSK"), "Owner should not start pointing to MSK")
        Assert.IsFalse(ie.SelectList(Find.ByName("Operator_DropDownList")).Text.Equals("SLX"), "Operator should not start pointing to SLX")

        Assert.IsTrue(ie.ContainsText("No Information found"), "Initial page should not find data.")

        ie.Close()

    End Sub

End Class

The following code reflects the NUnit version of the same test.

Imports WatiN.Core
Imports NUnit.Framework

''' <summary>
''' Test the Scorecard application main page
''' </summary>
''' <remarks>
''' Uses NUnit testing methods
''' </remarks>
<TestFixture()> _
Public Class Scorecard

    Private m_appURL As String = "Scorecard/"
    Private m_currentURL As String = m_BaseURL & m_appURL &       "default.aspx"
    Private m_logger As New Logger

    ''' <summary>
    ''' Ensure the page initializes itself to have blank or alphabetically low entries in the drop down lists.
    ''' Ensure the page does not initially display any information.
    ''' </summary>
    ''' <remarks></remarks>
    <Test()> _
    Public Sub TestInit()
        Dim ie As IE = New IE(m_currentURL)

        Assert.IsTrue(ie.ContainsText("Interchange Scorecard"), "Title not found on page")
        Assert.IsFalse(ie.SelectList(Find.ByName("Owner_DropDownList")).Text.Equals("MSK"), "Owner should not start pointing to MSK")
        Assert.IsFalse(ie.SelectList(Find.ByName("Operator_DropDownList")).Text.Equals("SLX"), "Operator should not start pointing to SLX")

        Assert.IsTrue(ie.ContainsText("No Information found"), "Initial page should not find data.")

        ie.Close()

    End Sub

End Class

The code for these two tests is almost identical. The only difference is that NUnit requires some decoration of the code with attributes to identify the tests, while the console application will need a main module somewhere to call the test modules and will have to its own assert methods (not shown here).

How Console Applications and Test Frameworks Run Tests

A console application runs unit tests in pretty much the same way it runs any program. A main module makes calls to other methods, which may make calls to additional methods, until all desired tests are run. In our proposed methodology, the project would contain a class for each web page to be tested. Each of these classes would then have a main method to act as a controller for calling the unit test methods within the class. The main entry module for the project (or the Windows Form) would call the controlling method in each class for each page to be tested. Calls to any tests which should not be run during the current iteration would be commented out.

With console applications, there must also be a mechanism created to save and display the results of each test and a way of determining if tests pass or fail.

A test framework will automatically recognize all of the unit tests in the project based on the <Test()> attribute assigned to each test method. These will be displayed in the runner for the test framework. Results will usually display with the familiar red and green indicators, red for failure and green indicating pass. Test frameworks have assert classes built in to be used in determining if the tests pass or fail.

Another way to run NUnit tests is with the TestDriven.NET add-on to the Visual Studio IDE provided by TestRunner.NET. This utility will allow you to right click on any node in the project tree or any test method and run the tests directly from the IDE. The results are shown at the bottom of the screen in the lower left corner and messages will appear in the Output window. TestDriven.NET also allows you to step through both the test and the code being tested with the debugger to determine if a failure is caused by the code or if the test itself has a bug. It also bundles the popular NCover utility to help with determining what lines of code are being tested.

Contents: Table of Contents Previous Page: Introduction Next Page: How to use WatiN to Test Web Pages - Part I

Posted Wednesday, March 26, 2008 10:00 AM by ddodgen | with no comments

Filed under: WatiN, NUnit, TestDriven.NET, Unit Tests, VB.NET, Web Page Testing

Introduction

There are a number of ways to test web pages and the server side or back-end classes that support the UI. Many are expensive, some are free or open source. I have chosen to focus on using WatiN for testing web pages, supported by the WatiN Test Recorder. WatiN is an open source tool which extends VB.NET (and C#) for testing the UI portions of a web site, WatiN is basically a .NET version of the popular WatiR test framework used for testing Ruby, and was created by Jeroen van Menen. WatiN Recorder is another open source tool that will help in writing tests for WatiN.

WatiN tests can be run as console applications, Windows Form applications, or from within one of the test frameworks available. I will show examples using the NUnit test framework with the TestDriven.NET Visual Studio add-in, but there are other test frameworks that work equally as well with WatiN, such as MbUnit. I will let others discuss the tools that I am not focused on, not because they are not as good as my choices, but because focus will help with becoming familiar and actually using the tools.

Why Test?

Every developer who is worthy of the title will test his or her code during the process of writing it. No one writes software and puts in it production without ever executing it. Maybe they write a method, run the application, click things and watch the result, or maybe they have a framework that helps them test individual pieces. Unit tests are just a more formal way to test code that has the advantage that the tests are always there; ready to run again to validate that the code is still performing as expected. In many if not most instances, it would probably take less time to write the unit test for a method than it would take a developer to click through the application to ensure that the new method is performing its function, and the test result from the unit test is far less likely to be misinterpreted that just watching the screen to see what happens.

How many times has a maintenance change in one area of an application caused something to break in another? How many times has an application required a change, but the code was so confusing that there was a real fear of changing anything because the entire house of cards might fall down? Granted, with proper design, these sorts of things do not happen in a well thought out and written system, but how many of those exist in the real world.

Basically, testing is a safety net. It helps you to be reasonably sure that your code will perform as expected, even after numerous maintenance changes.

How to Test

Write small tests for each piece of functionality that is not totally obvious, and even those can use a test if you wish. The most insidious bugs are those where you look at the code and just know that this place cannot be where the problem is, and then later realize that you were looking at one thing but seeing another!

Tests are grouped by the methods and functionality that they are testing and then controlled by either an application or framework. NUnit has been a standard test framework in the Microsoft arena for some time, but others are available. Both applications and frameworks can be configured to run all or part of the tests at any time. Since tests are generally run often during the development process. They should be kept small so they can be run fast and so that they test only one specific piece of code or function per test.

Using an application – Create a Main method which will be the entry point for the application. Then create classes for each page that needs to be tested. These classes should each have a controlling method that will be responsible for calling each of the test methods within the class. The main module can then be responsible for calling the controlling methods of each class. Comment out calls to classes or tests that you do not desire to be run at the current time.
Using a test framework – Create a class for each web page that needs to be tested. The test framework will automatically identify the tests in the class and will have a front end application which will display the classes and tests in a tree structure to allow you to select all or a group of tests to run.

Not only should you test that the method performs as expected when given good parameters, but you need to create tests that check bad input and the edges also. That is where most of the bugs occur.

For example, assume that you have a function that provides the percentage that one number is of another such as that listed below:

    Public Function GetPercentage(ByVal whatNumber As Double, ByVal whatDivisor As Double) As Decimal
        Dim answer As Decimal

        answer = (whatNumber / whatDivisor) * 100

Return answer
End Function

This function looks pretty easy, as it should if written well. If you pass 2 and 4 as the parameters, you get back 50% as you expect, so a unit test should be:

    <Test()> _
    Public Sub TestGetPercentage()
        Dim answer As Decimal
        Dim numberOne As Double
        Dim numberTwo As Double

        numberOne = 2.0
        numberTwo = 4.0
        answer = GetPercentage(numberOne, numberTwo)
        Assert.IsTrue(answer = 50D, "GetPercentage returned " & _
            answer.ToString & " when provided " & _
            numberOne.ToString & " and " & numberTwo.ToString & _
            " when expected 50")

End Sub

This is known as testing the happy path. Yes the function performs well when given the logically expected parameters. It is an important test, but you also need to test the edges and unlikely paths. What if the second number is zero for instance? A divide by zero error will occur. What if either number is negative or nothing? The requirements should have covered these possibilities, but even if they did not, the developer must consider them and have the function behave as expected.

Note that parts of this routine may be new to some developers. The <Test()> attribute tells NUnit that this is a unit test so it can discover the test and run it properly. The Assert statement is a NUnit function that will check to see if the statement provided meets expected criteria (equates to true in this case) and provides an optional error message in the event of test failure.

Contents: Table of Contents Previous Page: Table of Contents Next Page: Unit Testing Methodology

Posted Monday, March 10, 2008 11:48 PM by ddodgen | with no comments

Filed under: WatiN, NUnit, WatiN Test Recorder, TestDriven.NET, Unit Tests, VB.NET, Web Page Testing

Let's Get Started

So, I am new to blogging. I plan to use this space to discuss issues related to testing in the VB.NET environment with emphasis on using NUnit, TestDriven.NET and WatiN. There are certainly many other options for testing, but I'll let someone else cover most of those. I want to focus on tools that are free to use and that will help us all to be better developers if we learn to use them in our daily work.

A little about me...I started programming on the TRS-80, which for those that don't know was one of the first microcomputers. It had no hard drive, no floppy, or CD, just a cassette tape and 16MB of RAM. Back before the Internet, BBS systems were all the rage. I ran an Adventure based BBS on a Commodore64 with two floppy disk drives for several years. Now I work for a large shipping company helping to keep track of containers moving around the world. There have been lots of steps in between. I started with MS Basic, learned procedural programming the hard, self-taught way. Finally got my BS in MIS degree some time later. Liked RAD, never liked waterfall, happy with Agile, and eXtreme looks good, which has finally dragged me kicking and screaming into object oriented programming, which was a big step for me.

Now here I am a big proponent for unit testing in particular and testing in general. I plan to spend some time discussing why you would want to make unit testing a part of your development, but primarily I want to discuss how to make unit testing work for you. Since my time is spent writing and maintaining web sites these days, I will focus on testing not only the back end processes such as business and data layers, but also the web pages themselves using WatiN.

This page will contain links to the other pages related to my extended discussion of unit testing using WatiN, WatiN Test Recorder, NUnit, TestDriven.NET with examples provided in VB.NET.