RADE

C++ Unit Test Manager

Testing Your Frameworks

How to write, deliver, and run test programs in the CAA V5 environment
Technical Article

Abstract

Like for development tasks, tests activities can take place in your environment that provides an infrastructure for writing, delivering, and running test programs associated with CAA V5 frameworks. This article explains the structure of the test frameworks and how to write test programs, and what are the interests of delivering them in the CAA V5 file tree.

Why Should You Use the CAA V5 Test Infrastructure?

CAA V5 supplies an infrastructure to test your frameworks and applications. This infrastructure is fully integrated in the development environment you use to develop your frameworks and applications and takes advantage of the same tools and mechanisms. In addition, it allows you to automate the test program execution.

[Top]

Tests Are Related to Frameworks

Test programs are written for a specific purpose (i.e. test a specific program!). As a consequence it seems to be obvious that they should be related to, and delivered with the application frameworks they are intended to test. This the CAA V5 file tree provides test frameworks that can contain any kind of programs and that contains, as any framework, an IdentityCard file specifying the tested application frameworks.

[Top]

Tests Can Be Developed and Shared As Well As Programs

Test programs can be as complex as any program, thus the same problems can be encountered: concurrent engineering, parallel development, and so on. Using the test framework, developers can:

[Top]

A Predefined Environment Eases the Development of Test Programs

There are often two problems to address when developing test programs: on one hand, you must think of the way you'll test your application, and on the other hand you must set up the runtime environment and automate these tests. This second aspect has been studied and some solutions can be brought in the CAA V5 scope:

[Top]

Tests Can Be Run Automatically

The fact that test frameworks can be delivered along with application frameworks and that their test programs can be run automatically allow the automation of some test based certifications before code delivery.

[Top]

How to Write Test Programs

The basis of CAA V5 tests is composed of programs that use the interfaces and classes exported by the application frameworks, and shells that are containers for those programs. Shells are used as containers because they can be run with Unix, and withWindows since the Korn Shell is a prerequisite for CAA V5 tools. Test programs and shells are delivered in test frameworks.

[Top]

What Is a Test Framework?

The test framework is one of the three types of CAA V5 frameworks, it provides a predefined structure to store both test programs and related data. A test framework is composed of:

The FunctionTests directory contains predefined directories but, inside them, the user is free to create his/her own structure of files and directories. Here are these directories and what should be stored/find in each of them:

TestCases This is the directory where shells must be stored. Shells bear .sh as suffix and their purpose is to firstly set up some test specific environment variables, secondly to run the program and finally to return an appropriate error code regarding the program return code and outputs.
InputData You can store in it any data that could be used as input for your programs. There is no particular suffix defined for those data.
Output This directory is the repository of all outputs and data that are produced during test program execution. For instance, if a program generates a file, this should be generated in this directory.
OutputRef This directory is used to store any "reference" data which allows shells to compare what is produced by programs (output messages, files, and so on) with what should be produced (if they worked!). For example, if you want to test a program which automate some modifications on a CATIA model, you can first perform these modifications "by hand" and save the result in a file you store in the OutputRef directory, then make a comparison (in the shell) between this model and the one produced by the test program in the Output/$OS directory.
SwitchTestCases This is the directory where switch test program shells are stored. These test programs are used to decide whether to switch the current level without replaying all the test objects. They should be representative of the framework
TestCases This is the directory where common test programs shells are stored. Shells bear .sh as suffix and their purpose is to firstly set up some test specific environment variables, secondly to run the program and finally to return an appropriate error code regarding the program return code and output.

The next section shows you the way to reference these directories and to write portable shells.

[Top]

Referencing External Data

Shell variables are automatically set according to the current test environment (i.e., the workspace from where the tests are run). You can use them to access your data whatever the absolute path to them may be [2].

[Top]

Basic Shell Samples

The minimum that can be written.

my_program
exit $?
 As you can see, the minimum consists of first running the program and secondly exiting with the program return code.

How to use predefined environment variables.

#------
export TEST=YES
export DICO=$ADL_ODT_IN
#-----
my_program <$ADL_ODT_IN/in1_mod1
rc=$?
if [ $rc != 0 ]
 then
   exit $rc
fi
diff $ADL_ODT_OUT/out1_mod1 $ADL_ODT_REF/out1_mod1
rc=$?
#------
exit $rc
 This example is a little bit more complex. Let's detail it:
  • The first line is a comment (starts with '#').
  • The two next lines show 2 variables that are set for a test purpose - no matter this purpose is, but just note that the second variable is set with a particular directory using another variable
  • Then note that the program my_program is run with input data from ADL_ODT_IN directory. The program my_program must handle the outputs redirection to a file under ADL_ODT_OUT directory. In this example, this file is named out1_mod1.
  • If the program fails, the shell exits directly using the program return code as error code. Otherwise, additional checking may be performed. Here, the traces of the program execution are compared with a reference trace file stored in the ADL_ODT_REF directory.

    NOTE : In the diff command, do not use the .traces file (generated by mkodt) to compare traces of the program execution with the reference file; redirect the outputs to a file whose name is unique (for example, ODTName.out where ODTName is the ODT shell name ) and then, compare this file to the reference file.

In addition to these samples, some rules can help to write shells which can be executed on both Unix and Windows platforms [3].

[Top]

Tests Depend on What You Have to Test

In this section we would like to give you some advice about how to tests your programs regarding their types. We've distinguished three types of "things" to test: a library, an executable and finally a new part of the CAA V5 application.

[Top]

Testing a Library

A library is not used alone, but you must write a program to access its services and then to be able to test them. So given a library, you can write one or more programs that you deliver as modules of a test framework and one shell per program.

Fig.x: The Links Between Application and Test Frameworks
The application framework CAAApplicationframe includes shared libraries or DLLs that need to be tested, such as CAAAfrDLL1.m.

The associated test framework CAAApplicationFrame.tst tests them using the test programs CAAAfrTestModule1.m and CAATestModule2.m, that are linked to the shared libraries or DLLs to test using their Imakefile.mk file:
BUILT_OBJECT_TYPE=LOAD MODULE
LINK_WITH=CAAAfrDLL1

The shells call the test programs, that are usually modules or EXEs
CAAAfrTestModule2
exit $?

The test framework identity card must refer to its application framework:
AddPrereqComponent("CAAApplicationFrame", "Public");

[Top]

Testing a Batch Program

Testing a program is quite the same that testing a library (API) except the fact that there isn't any additional program to write. All the test data should be located somewhere under the FunctionTests directory.

[Top]

[Top]

How to Run Tests

As mentioned in previous sections, tests are not directly executed by the user but their execution is controlled by a particular command: mkodt. Its purpose is to prepare the test environment by setting variables needed at runtime according to the local user environment. It is available with "mkmk" commands.

[Top]

Inside a Workspace

Tests always take place in a workspace and are used to check what is developed in this workspace, thus only programs and shared libraries that are visible in a workspace can be tested by test objects in this workspace. However, note that the concatenation feature brings visibility on data managed among multiple workspaces.

Any test can be run either being under a workspace - the current working directory is under the workspace root directory - or by referencing a workspace. Both modes are specified through mkodt options [2]. One test object or all test objects can be run at user's convenience.

We'd just like to give you an idea on how you can put in place some automatic tests in order to certify the quality of your software. Here is an example of an organization where several people are working on the same software, each of them owning a workspace from which he/she delivers pieces of software. Regarding the workspace hierarchy, we can see that all modifications must go through the project workspace. This project workspace is a place where automatic tests should be run each time there are some new modifications to promote outside the scope of this organization.

[Top]

Controlling Test Execution

A shell can include specific commands to control some aspects of its execution. These commands are:

ODTInit [4] Runs an initialization shell script
SetOdtParam [5] Sets ODT parameter values
WhichDir [6] Finds a directory or a file in a workspace

Top]

In Short

Like program source files are managed in application frameworks, the test framework provides a predefined organization of files and directories (modules, FunctionTests, ...) in which:

[Top]

References

[1] mkmk
[2] mkodt
[3] Writing Shell Dos and Don'ts
[4] ODTInit
[5] SetOdtParam
[6] WhichDir

[Top]

History
Version: 1 [May 2000] • Document created

[Top]

Copyright © 2000, Dassault Systèmes. All rights reserved.