Automation testing tools

Today, having an automated testing tool is as important as writing code to develop a feature. Be it for unit testing, functional testing, non-functional or automated UI testing. Most important out of them are unit testing tools which act as first and foremost quality assurance parameter while development. Rest tools help to ensure quality for deliverables at later stages.

These days I am reading a book titled “Testing ASP.NET applications” by Jeff McWherter and Ben Hall. In this book I found very interesting information about evolution of automated testing tools we use today. Here I will talk about that and also, I will add few more rings to this chain of evolution from my experience.

Basically, testing tools were there since developers started wring code. But real evolution started around 15 years back when automated testing was recognized as a separate strict discipline along with development. After that, tools evolved as the development processes evolved. Over a period of time testing tools have got standardized but I am sure that this evolution is not going to stop ever.

Following is the list of automated testing frameworks and tools. As they serve different purposes at various stages in development processes, I will classify them into following categories.

• Test Driven Development Tools
• Behavior driven Development Tools
• Functional Testing Tools
• Non-functional testing Tools
• Static Code Analysis Tools
• Dynamic Code Analysis Tools

Test Driven Development Tools

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle. Unit testing is tightly associated with this process and other practices from agile software development approaches such as Extreme Programming or Scrum. Following are some popular tools under this category.

sUnit created by Kent Beck for Small Talk, sUnit has become known as the “mother of testing frameworks.” Many popular unit testing frameworks such as jUnit and nUnit are ports of sUnit.

jUnit a port of sUnit for Java created by Kent Beck and Erich Gamma in late 1998. jUnit helped bring automated unit testing into the main stream.

nUnit saw light of day in late 2000, all the great things about jUnit were ported to .NET allowing C# developers to write jUnit style unit tests against their C# code.

qUnit is the unit test running for the jQuery Framework. In May 2008, qUnit was promoted to a top-level application in the jQuery project. qUnit allows web developers to run unit tests on JavaScript.

MSTest is also a unit testing framework introduced by Microsoft integrated with Visual Studio IDE.

MbUnit provides advanced unit testing support with advanced fixtures to enable developers and testers to test all aspects of their software.

csUnit is a free and open source unit testing tool for the .NET Framework.

Behavior Driven Development Tools

These tools encourage collaboration between developers, QA and non-technical or business participants in a software project. BDD is primarily an agile development technique. For more details on this visit this wiki page. It is quite a new technique and has evolved over last few years. Following are some very important tools in this category

NBehave is Based on Dan North’s initial vision of rbehave and utilizing the behavioral domain specific language (DSL) of Behavior Driven Design (BDD) we created the NBehave framework. The primary goal of NBehave is a framework for defining and executing application requirement goals. These characterizations are modeled after the Behavioral Driven Design (BDD) terms Story, Scenario, Given, When, Then. Relying on a syntax that is lightweight and targeted at product owners (a few “quotes” mostly), the code becomes an executable and self-describing requirements document.

RSpec is a Behaviour-Driven Development tool for Ruby programmers. BDD is an approach to software development that combines Test-Driven Development, Domain Driven Design, and Acceptance Test-Driven Planning. RSpec helps you do the TDD part of that equation, focusing on the documentation and design aspects of TDD.

Along with this there is a huge list of tools available under this category.

Functional Testing Tools

Functional testing is a type of black box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered. Functional testing tools automate what manual testers do. Of course they cannot replace a manual tester.

Watir: In May 2002, the Web Application Testing in Ruby Watir (pronounced “Water”), a library to automate browser acceptance tests in Ruby, is released.

Selenium: Selenium provides a suite of tools for automated user interface testing of web applications. In 2004, Jason Huggins of Thoughtworks created the core “JavaScriptTestRunner” mode for automation testing of a time and expense system.

WatiN: In May 2006, Watir, the popular browser acceptance testing framework, is ported to .NET as the WatiN (pronounced “Watt in”) project.

WebAii: Free web testing framework that will help you automate AJAX and Silverlight applications. Benefit from rich API, LINQ support, browser abstraction, and of course special wrappers for Telerik RadControls. Enjoy integration with various unit testing frameworks such as VS Unit testing, NUnit, MbUnit and XUnit.

White: White is a framework for automating rich client applications based on Win32, WinForms, WPF, Silverlight and SWT (Java) platforms. It is .NET based and does not require the use of any proprietary scripting languages. Tests/automation programs using White can be written with whatever .NET language

Visual Studio Coded UI Test: In 2010 Microsoft has come up with its own offering for writing automated UI tests. It provides a common platform for writing coded UI test both for windows and web clients. This feature is available only with ultimate edition of Visual studio 2010.

Mercury’s QTP and WinRunner: Mercury Functional Testing™ combines Mercury’s functional testing products, Mercury QuickTest Professional™ (QTP) and Mercury WinRunner®, to deliver a complete solution for functional test, GUI test, and regression test automation — with support for practically every software application and environment.

IBM Rational Robot is a general purpose test automation tool for QA teams who want to perform functional testing of client/server applications.

TestPartner and QARun: Compuware offers automated functional and regression testing tools. When testing complex applications based on Microsoft, Java and web-based technologies, TestPartner helps reassure you that testing efforts are complete. For other application types, QARun gives developers and testers the automation capabilities needed to create and execute test scripts, verify tests and analyze test results.

Non-Functional Testing Tools

Non-functional testing is the testing of a software application for its non-functional requirements. Which includes  Baseline testing, Compatibility testing, Compliance testing, Documentation testing, Endurance testing, Load testing, Localization testing and mangi testing, Performance testing, Recovery testing, Resilience testing, Security testing, Scalability testing, Stress testing, Usability testing, Volume testing. So, if not all, tools under this category should be able to cover most of them. Following are

Microsoft Visual Studio Load Test: This is a comprehensive non-functional testing tool integrated with Microsoft visual studio. It has to be used in conjunction with other tests, mostly “web test” to perform testing. Detailed information about load test can be found here.

Load Runner : HP LoadRunner can emulate hundreds or thousands of concurrent users to put the application through the rigors of real-life user loads, while collecting information from key infrastructure components (Web servers, database servers etc.) The results can then be analyzed in detail, to explore the reasons for particular behavior. For more details click here.

Web Capacity Analysis Tool: (WCAT) is a lightweight HTTP load generation tool primarily designed to measure the performance of a web server within a controlled environment.  WCAT can simulate thousands of concurrent users making requests to a single web site or multiple web sites.  The WCAT engine uses a simple script to define the set of HTTP requests to be played back to the web server. Extensibility is provided through plug-in DLLs and a standard, simple API.

Other performance testing tools: here.

 

Static Code Analysis tools

Static code analysis is performed without actually executing programs built from that software. In this process code is tested against quality metrics set by development team. Static code analysis includes code reviews, walkthrough and inspections. Here I will highlight the most common tasks that are covered by the static code analysis. They are:

• Finding probable bugs
• Locating the “dead” code
• Detecting performance issues
• Improving code structure and maintainability
• Conforming to coding guidelines and standards
• Conforming to specifications

Here is the list of static cod analysis tools.

Dynamic Code Analysis tools

Dynamic program analysis is the analysis of computer software that is performed by executing programs built from that software system on a real or virtual processor. For dynamic program analysis to be effective, the target program must be executed with sufficient test inputs to produce interesting behavior. Dynamic code analysis tools helps to expose various serious defects in code like

• Race conditions / Deadlocks
• Exceptions
• Resource and memory leaks
• Security attach vulnerabilities
• Code coverage

Most Performance analysis tools use dynamic program analysis techniques. These tools are also used in conjunction with Non-Functional testing tools to drill down the root cause for the performance issues highlighted while using non functional testing tools.

 

In this blog I have highlighted only few tools. But on the way I have provided  lot of hyper links which will lead you to the ocean of tools belonging to a particular category.