Solid Optimizer Project

In order a project to be reliable it has to have proper QA. SolidOpt provides system that simplifies the testing of its parts. SolidOpt Dev Team has adopted the one of the important branches of extreme programming and namely the test-driven development. Our experience shows this is one of the most suitable approaches in development of open source projects.

Test-Driven Development

In our case a relatively small test (show) cases can outline the required improvements in functionality. In other words, the test case is our atomic functional point. Given that we have to provide a set of tools that help the creation, execution and validation of test cases. Even a non-developer must be able to create a test case, a test case expected results and run them, understanding if there is any discrepancy.

A test case has to be:
• Easy to construct
• Easy to understand
• Easy to run
• Easy to debug

We use NUnit, which gives us the building blocks that we need to achieve the requirements. Most of our tests have to test transformations. A transformation takes a representation and produces another one. There are an input and an output. The input is given in a text file (aka test case) and the output of the transformation (result) is has to be compared against an expected result.

The described scenario almost covers 99% of the cases that we have. This allows us to extract out the common implementation in a test "driver".

Test Driver

The test driver is in a separate library which can be easily referenced and reused. The driver supports running test case(s), comparing the expected against the "seen" results and writing out useful details if the test failed.

Test Cases

Test cases are the input files that we pass in as an argument to the tested transformation. Sometimes they have to be preprocessed first. For example, we might need to compile the test case into assembly and then pass it in to the transformation method. Here is a typical test case that needs preprocessing:

// RUN: "@ILASM@" /DLL @TEST_CASE@
// XFAIL:
.assembly TestCase {}

.class public auto ansi beforefieldinit TestCase
       extends [mscorlib]System.Object
{
  // public static void Main() {
  //   Math.Abs(-1);
  // }
 .method public hidebysig static void Main() cil managed
 {
   .locals init (bool local0)
   IL_0000: ldc.i4.m1
   IL_0001: call int16 [mscorlib]System.Math::Abs(int16)
   IL_0006: pop
   IL_0007: ret
 }
}

In the example above, there are two levels of preprocessing. Firstly, the build system expands the build variables between the @ symbols (see our custom cmake variables). For instance @ILASM@ is replaced with the full path to the ILASM executable. Secondly, the test driver preprocesses the so called test case directives. A test case directive starts with a comment(//) followed by a name (RUN) and colon(:). In the shown snippet there are two directives - RUN and XFAIL. RUN directive signals the test driver that before running the test it needs to call ilasm compiler to compile the source file into an assembly. XFAIL directive signals the test driver that this case is expected to fail.

SolidTest

See this presentation.

Continuous Integration

Our CMake build system allows easy integration of CTest and CDash. We have a few deployed virtual machines, running different platforms. Currently Unix and Windows. We are working towards enabling full and incremental builds on them.