Web Service Testing

Here's some notes about testing a web service:

Web Service Request has a URL that points to .asmx; the Web Service Request has a Query String Parameter, String Body, and Header

1) open a webtest

2) right click the web test name and slect Add Web service Request; an empty request is added to localhost with an empty String Body

3) open a seperate browser window and browse to the asmx to get the details of the SOAP request for each operation (http://server/example/test.asmx)

4) View properties of request and change the URL from http://localhost/ to point to the asmx

5) Expand the request to select the String Body; enter text/xml under Content Type; in the other browser select the operation to be tested; copy the xml request from "?xml version="1.0" encoding="utf-8"? through "soap:Envelope"; go back to the String Body and click the elipsis beside String Body and paste the xml request under the HTTP body string. Then change placeholder values to actual data and click OK to save. Use arrows to navigate since this dialog does not contain a scrollbar. Enter dates in this format: 2006-03-10T07:42:00Z

6) Go back to the browser to get a copy of the URL (without quotes) http://example.com/app/UpdatePerson (located above the start of the XML request). Go back to the Web Service Request; right click and select Add Header. Enter SOAPAction under Name then paste the URL under the Value http://example.com/app/UpdatePerson.

7) Go back to the browser to get the name of the operation UpdatePerson. Go back to the Web Service Request; right click and select Add URL Query String Parameter. Change the name of the Query String Parameter from Parameter1 to some other value like op and enter the name of the operation under Value (UpdatePerson).

8) Add Validation Rules to check for text in the HTTP Response

This MSDN article explains creating a web service test in more detail

Sara Ford's Tip of the Day Sidebar Gadget

Sara Ford works at Microsoft as a Program Manager and blogs about Visual Studio. There's a VS2005 Tip of the Week and a VS2008 Tip of the Day. She used to be a test dev and has some video interviews on MSDN channel 9.

Rob Caron wrote a sidebar gadget for Visual Studio 2008 Tip of the Day. You can download it here.

This is a great resource for people like me who are new to the development environment.

ASP.NET __VIEWSTATE in recorded web tests - Validation of viewstate MAC failed

I'm not a developer, so I wasn't sure what was happening at first. I got a new machine, but couldn't run recorded web tests on it. There were error messages when I tried to run the recorded web tests on the new machine that were recorded on the old machine. Here's how I figured out what was happening.

The original error message didn't make sense to me until later. It said "Validation of viewstate MAC failed. If the application is hosted by a web farm of cluster, ensure that configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster"

At first I thought something was wrong with machine.config or web.config. I searched for the error message and read this forum post about adding changing web.config to add entries to the pages tag:
validateRequest="false"
enableEventValidation="false"
viewStateEncryptionMode ="Never"

That didn't work, so I found the request where the errors happened and tried deleting the Form Post Parameters that referred to __VIEWSTATE=

That didn't work, so I read about __VIEWSTATE this article explained that Viewstate contains user entered information encrypted in a hidden POST form field so users don't have to re-enter information after moving to another page.

Then I recorded a web test to get the updated encrypted VIEWSTATE=/whGSTdfdbdjdfy and copied the updated value to the other web tests. I also found a post that tells how to make the viewstate dynamic, but I only have Visual Studio installed on one machine now.

Don't forget to Run as Administrator on Vista

For Recorded Web Tests; the recorder bar will show up in IE but the controls are disabled unless you run Visual Studio as Administrator. Right click the Start Menu shortcut to select "Run as Administrator".

There's also check box to always run as administrator under the Compatibility tab of the shortcut. My boss told me about that today.

Here's a blog post about troubleshooting the Web Test Recorder bar:

New machine - webtests

We got new laptops recently. They're super awesome; HP Compaq 8710p; 4 GB RAM; 150 GB drive; Intel Core Duo CPU T7100 1.80 GHz; CD/DVD combo drive with burner

Right now I'm updating the web tests to work on this machine. It seems like some parameters in the web tests contain encoded information about the old machine. For certain things I was able to do Find In Files and Replace text values, since web tests are stored in XML format.

Here's what I changed in web tests on new laptop:

1) Replace in Files: path to text files used in Data Sources
2) Replace in Files: web application URL
3) Delete and Insert the Data Sources - left the existing bindings when deleting the data source
4) Record a web test on the new machine to get the updated VIEWSTATE and replace __VIEWSTATE=/hdnsytYTTfwjkwkmi from the old machine with the new value for the new machine in the Form Post Parameters
5) Delete __VIEWSTATE=/WkdujnUWEUiwkmi from the Form Post Parameters

Add Data Source for Text File with Header Row:

1) Click Add Data Source button
2) Select the Microsoft Jet 4.0 OLE DB Provider under the OLE DB Provider choice list
3) Click Advanced button
4) Enter "Text;" without quotes in the first item under Advanced>Extended Properties
5) Click OK
6) Enter path to input file under the "Server or file name" text box - without file name
7) Click Test Connection
8) Click OK at "Test connection succeeded' message
9) Click OK to close the Add Data Source dialog
10) Select check box next to input file used in the web test and click OK

Custom Extraction Rules - Extract Drop Down List Form Field

I used the Josh Christie's sample code to make a Custom Extraction Rule to check if a drop down list exists based on its name.

I hope it's correct; I removed the references to index and changed
e.Response.HtmlDocument.GetFilteredHtmlTags("input") to
e.Response.HtmlDocument.GetFilteredHtmlTags("select")

modified sample code:

Custom Extraction Rules - Extract Form Fields by Index

Here are the sites describing how to add the Custom Extraction Rule to Extract Form Fields by Index:

Josh Christie's blog entry: http://blogs.msdn.com/joshch/archive/2005/11/17/494164.aspx

MSDN library: http://msdn2.microsoft.com/en-us/library/ms243179(VS.80).aspx

Using the example code from Josh Christie:

1. right click the project and select Add>Class, enter a name, and click OK
2. double click the .cs file to open and replace text with sample code from here
3. build the solution
4. open a web test and the new extraction rule is available

Extraction Rules - Extract Form Field

This type of Extraction Rule is used to verify Form Fields exist on the page. Use an Internet Explorer add-in utility to find the form field names.

Here is the download site for the Internet Explorer Developer Toolbar: https://www.microsoft.com/downloads/details.aspx?familyid=E59C3964-672D-4511-BB3E-2D5E1DB91038&displaylang=en

Steps to find the form field names:

1. View Developer Toolbar
2. View DOM
3. Find>Select Element by Click (under the menu of the window in the lower half of the screen)
4. Click on the control to be tested
5. Find Name, double click in the Value column, and Copy the text under the Value (in the middle section)

Add the form field to an Extraction Rule

1. Add an extraction rule to the web test
2. Select Extract Form Field
3. Enter a descriptive name under Context Parameter Name (user defined name)
4. Paste the Form Field Name copied from the DOM Explorer

Form Post Parameters - requires Name - cannot use ID

I have a script written using the Ruby library Watir. I couldn't upgrade it to VSTS because the form fields cannot be accessed by ID in the VSTS Web Test.

There is a testing exercise on Elisabeth Hendrickson's site: http://www.testobsessed.com/exercises/triangle.html

Here's what the Ruby script does
1) click draw with nothing in the sides and verify the triangle type is Invalid
2) enter the same value in all 3 sides, click draw, and verify the triangle type is Equilateral
3) enter values to generate a Right triangle, clcik draw, and verify triangle type is Right
4) enter values to generate a Scalene triangle, clcik draw, and verify triangle type is Scalene
5) enter values to generate a Isosceles triangle, clcik draw, and verify triangle type is Isosceles
6) enter non numeric values in all 3 sides, clcik draw, and verify triangle type is Invalid

The command in Watir is ie.text_field(:id, "side1").set("eee")

In VSTS the Name of the Form Post Parameter is required to set the value.

Retest bug fixes

I am writing web tests that will retest bug fixes in the next version of a reporting application. There's not a good way to retest some of the bugs I found.

Example: unable to check if a control exists when items missing from the report selections unless the name of the control is known. This test will be added after the controls are added.

Another Example: the reports are opened as a pdf in a new window. I don't know how to verify the contents of a pdf file. I tried using the Fiddler utility and saving the file to disk, but none of those requests were captured.

TimeoutException Internal Error! failed to acquire a ReaderLock

I ran a group of web tests from a Test List and got an error on cancel:

Here's a post from the MSDN forum that says to increase the timeout setting in the testrunconfig under Test Timeouts: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=163138&SiteID=1

Test Data Generation

Here's some sites to help generate test data.

Joe Strazzere has a blog post with strings used to check field lengths: http://www.sqablogs.com/jstrazzere/538/Some+Strings+for+Pasting.html

James Bach wrote a Perl script to generate self describing data strings: http://www.satisfice.com/blog/archives/22

Graham King has a script to generate fake credit card numbers in Java and Python. Includes credit card types: Mastercard, VISA 13 digit, VISA 16 digit, AMEX, Discover, Diners Club / Carte Blanche, enRoute, JCB 15 digit, JCB 16 digit, and Voyager. Credit Card Number Generator: http://www.darkcoding.net/index.php/projects/credit-card-generator/

Benjamin Keen's site allows you to select the fields used in the data file: http://www.generatedata.com/#generator

This site will send you a list of fake names, but you have to e-mail a request. They sent me csv file containing 3000 people. Fake Name Generator: http://www.fakenamegenerator.com/

Validation Rule - prohibited text

The Validation Rule can be set to fail a test when prohibited text is found on the page. I used this to fail when the message "error running report" is found.

1. Add Validation Rule
2. Find Text
3. Enter Text to search for and set Pass If Text Found = False

Here is the message if the forbidden text is found:

"The prohibited text 'There was an error running the report.' appeared in the HTML response. FindText=There was an error running the report., IgnoreCase=False, UseRegularExpression=False, PassIfTextFound=False"

(select the row that fails in the test results and click the Details tab to view the error)

Data Source - Choose Tables dialog not updated

I posted yesterday that new text files are missing from the Choose Tables dialog.

I missed a key step; the text file is not missing until you add a second data source.

I modified my post to Microsoft, sent them a test project, and attached a video of the steps.

Modified Steps to Reproduce:

1. Add a Data Source to a web test and save
2. Create a new text file and Save All
3. Add another Data Source to the web test and attempt to select the new text file.

The new file is not being added to the Choose Tables dialog until you exit and reopen Visual Studio. The same thing will happen when files are deleted.

Data Source - new text file missing from Choose Tables

New text file is missing from the Choose Tables dialog until you exit and reopen Visual Studio.

Here's what I've been doing as a workaround: create the web test, create the text file, close Visual Studio, open solution, and add a new Data Source to the web test

I also tried Save All then reopen the solution, but that didn't work.

I reported this as a bug to Microsoft today: https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=297000

Clearing requests in the web test recorder

Clicking the X in the Web Test Recorder will clear all requests and comments from the test.

I incorrectly thought you could select one request to only delete one line.

Adding comments when recording a web test

You can add the comments as the test is being recorded. I was adding comments after the test was finished and they weren't very good.

Just remember to add the comment first.

1. Click the folder button in the recorder pane
2. Enter the comment describing the next set of events and click OK
3. Click the button or whatever needs done

Virtual Tech Ed and other sites

Here's a couple of sites to check out:

http://virtualteched.com/Pages/OnlineSessions/OnlineSessions.aspx

http://www.runasradio.com/default.aspx

http://www.microsoft.com/whdc/winhec/default.mspx

Copy Paste Requests in a recorded web test

New request is pasted above selected request

Select the top node or Web Test name and paste to add the copied request to the end of the list

Example:

invalid date
_http://server/default.aspx
_http://server/page1.aspx
_http://server/page1.aspx
_http://server/page1.aspx

To add copies of the 3 requests for page1 and modify them for page2:

1. Select the first page1 request, right click, and Copy
2. Select the Invalid Date top node, right click, and Paste
3. Select the second page1 request, right click, and Copy
4. Select the Invalid Date top node, right click, and Paste
5. Select the third page1 request, right click, and Copy
6. Select the Invalid Date top node, right click, and Paste

Now there are copies of the last 3 requests

Change the properties and Form Post Parameters to work with another page

Select a different request and paste to add a copy above the selected request

There should be a way to change the order of the Web Requests. What if you record steps in a certain order and then want to change them later?

Delete Test Results

The test results can take up a lot of disk space if you're not careful. I had a test project where the results directory took up 4GB.

Here's how to delete the test results in VSTS:

1. Open the test project
2. Click Test Runs tab in the lower portion 4 tabs after output window
3. Click the Select drop down list
4. Select last item "Local or Connect to remote"
5. Select 2nd radio button "Select results from local test runs" and browse if needed, but it should default to the results of the currently loaded project, and click OK
6. Select the result from the tree
7. Connect if needed
8. Click the X to delete the selected test run and results

Here's the MSDN reference:
http://msdn2.microsoft.com/en-us/library/ms182502(VS.80).aspx

I also deleted the contents of the results directories through Windows Explorer. The results are stored in a file with a .trx extension under C:\Documents and Settings\user\My Documents\Visual Studio 2005\Projects\TestProject\TestResults (I don't know if this way is correct, but it took less time than trying to delete the result of each run through Visual Studio.)

bind form field to data source - comma in text

Surround commas with double quotes to send to a form field from a text record. This is because VSTS uses the comma as a field delimiter. Even if the first record only contains one field.

,,, = empty since it looks at what's before the comma as field one in that record

",,," = three commas

Data source to a text file

Record a web test that does something - enter 01/01/2007 in a form field; now you want to change which values are entered in the form field without recording a new test for each value.

Add data source to web test
select OLE DB Provider = Microsoft Jet 4.0 OLE DB Provider
Enter the path to the text file (without including the filename) under Server or filename
Click Advanced
Extended Properties at the top enter "Text;" (without quotes) if the file has a header row or "Text;HDR=NO;" (make sure it's HDR not Hdr & no quotes) if there is no header row
click Test Connection and OK
Select one or more text files in the dialog to select the tables

Edit the text file and enter the values you want to test with

Select the input field where you entered the date under Form Post Parameters
Right click and select Properties
Drop down the Choice list next to Value and select your input file

Run the test to send over the dates from the input file

Adding data binding to a recorded web test

First you need to create the table or text field with the values sent to the web test. I made a text file with one value in each row. Make sure there are no extra line feeds at the end of the file. Then add the data source to the web test.

Here's the MSDN article that describes adding the data source:
http://msdn2.microsoft.com/en-us/library/ms182546(VS.80).aspx

There's 2 important things to remember with data binding:

1) Select the data source under the QueryString Parameter or the Form Post Parameter - Properties>Value

2) the setting to iterate through all rows in a table or input file is entered in localtestrun.testrunconfig under Web Test there is a radio button called "Number of run iterations" - select One run per data source row

Here is the MSDN page with a walkthrough on adding data binding to a web test: http://msdn2.microsoft.com/en-us/library/ms243142(VS.80).aspx

Automated Testing Tools

Hopefully this information helps someone. I've been using this blog to keep notes about Web Tests in Visual Studio Team System. In the past I would learn something, then forget important details after moving on to the next project.

Here's the tools I have used to implement automated test scripts: (I'm not an expert in any of these tools)

$$$$:
Microsoft Visual Studio 2005 Team Edition for Software Testers - automated web tests, manual tests in Word or text, unit tests, load tests, and generic tests

Rational Visual Test 4.0 - old test tool for Windows applications

Freeware:
Watir – Web application testing in Ruby – module to automate web applications using Ruby script

Selenium remote control – automated web testing -- Java server with dotNet dll for tests written in C#, Python module, and Ruby module

Selenium Recorder – Firefox extension with record and playback for automating web tests

Selenium IDE – Firefox extension to record, edit, and debug tests

Selenium Core - Selenium uses JavaScript and Iframes to embed a test automation engine in your browser. This technique should work with any JavaScript-enabled browser. Because different browsers handle JavaScript somewhat differently, we usually have to tweak the engine to support a wide range of browsers on Windows, Mac OS X and Linux.

pyWinAuto – Python module for automating windows applications

PAMIE - Python Automation Module for Internet Explorer

WebTest recorded using Fiddler fails: The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false

Used the Fiddler Tool to record a web test over an AJAX enabled webpage.

The error on playback is: Primary Request Failed "500 The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false. Ensure that the property is set to true during an async postback."

Resolved by deleting references to Hidden fields.

Under Form Post Parameters: _VIEWSTATE={{$HIDDEN3.__VIEWSTATE}}
Extract Rule with Context Parameter Name: $HIDDEN4.__VIEWSTATE

Getting Started with Team System videos

The videos can be viewed here: http://www.asp.net/learn/vsts-videos/

He talks fast and I can't remember all the information. This is the 2nd time I have watched them.

Here's a list of what I forgot or missed (so far):

1) add an application config file to the test project then copy the connection string from web.config file

2) a build is required before the test results can be published; create a build, run tests, publish results, then report a bug from the test result

3) edit the local test run config to add an assembly to code coverage; right click to view the code coverage report after the tests are run

4) The Extraction Rule to extract hidden fields is automatically added to a recorded web test; this could cause a test to fail, so you might need to delete unnecessary rows from the request under the Form Post Parameters folder

5) The Test Manager is used to edit the vsmdi file; this file is stored in XML format; select Test>Windows>Test Manager to view Test Manager

6) Select Test>Windows>Test View to run one or more tests; search for tests using the Keyword Filter; group tests by solution, project, type or other fields

7) Searching in the test view does not edit the test run settings

8) use witexport and witimport to modify the work items on team foundation server; WIT stands for Work Item Type (or use the Process Editor instead)

Web Tests for AJAX enabled pages

The Fiddler2 utility is used to capture HTTP requests on AJAX enabled pages: http://www.fiddler2.com/fiddler/help/WebTest.asp

Run the Fiddler2 utility, open the site in IE and browse pages, select requests in Fiddler, and select File>Save Sessions>as Visual Studio Web Test. Then add existing item to the Test Project. Open the Test in VSTS, select the top node, right click and select Generate Code.

Visual Studio Team System - Recorded Web Tests

Something usually needs modified in the recorded web test. When I replay them it usually fails. Here's one of those times:

The error message says "1 dependant requests failed" The test results show an X next to the request that failed. Edit the recorded web test to find the request. Right click, select Properties, and change "Parse Dependent Requests" to False.

Microsoft Defy All Challenges

These ads are pretty funny. Mine is pretty stupid.

The defyallchallenges.com site lets you edit your own Machinima videos. View and vote for mine, so I'll move up in the rankings--or create your own.

Here's the link: http://defyallchallenges.com/?mixID=1027