Description
CANTATA WORKS IN YOUR ENVIRONMENT
Cantata installs on Windows® and Linux® host operating systems, with a Built-on-Eclipse® IDE or as an Eclipse-Ready® plug-in set. It supports GNU GCC & gpp and Microsoft Developer Studio® compilers, and is also integrated with an extensive set of embedded development toolchains.
- IDEs and Cross-Compilers
- Real Time Operating Systems
- Debugger
- Build / Continuous Integration
- Software Configuration Management
- Requirements management
To confirm how your specific tools and platforms are supported, please contact QA Systems.
Deploy to your specific target
The built in Cantata deployment wizard and powerful editor provide unlimited deployment of Cantata to target platforms without any license restrictions. The wizard makes selection of the basic settings simple, and a dedicated multi-mode editor with context sensitive help, gives comprehensive step by step assistance to fine tune any settings.
Cantata deployments consist of libraries and configuration options, set up for each specific target so it is binary compatible with your code. Once deployed, it can be used by any engineer for their Cantata tests on that platform.
Cantata users can deploy Cantata to a target themselves, or if preferred this is available as a professional service. For a free of charge assessment for any target environment to confirm Cantata compatibility, please contact QA Systems or your supplier.
Easy and flexible testing on target
Cantata tests (platform independent test scripts in C/C++ and target platform specific deployments) are built as C/C++ executables, downloaded and run on a target just as you would with your own code on a:
- Simulator
- Emulator
- Physical Target Board
Cantata can use whichever target communication mechanism is available to download the test executable and upload the various functional and code coverage test results.
The process is completely automated using Cantata Makefiles, test scripts and platform deployments for easy and flexible on-target testing from the GUI or CLI.
Easy tool integration
You build your embedded software to run on one or more target platforms, and that is how your development toolchain is set up.
Cantata is pre-configured as a special purpose tool suite to easily slot straight into an existing development toolchain, making it easy it set up your testing on those targets.
Cantata is installed from the cloud to your workstation using standard InstallAnywhere® and Eclipse capabilities.
The built-in Cantata deployment wizard provides unlimited deployment to and use on embedded targets, without any license restrictions. A deployment zip file is generated to store the Cantata library and configuration settings for each specific target.
Multiple target configurations can be shared across teams and test scripts. To support tool certification needs, Cantata target deployments can be verified with a target platform specific test suite as part of the deployment. A deployment registration checksum ensures control of test executions with the relevant target configuration.
Cantata unit or integration tests are built as executables and can be run under de-buggers for fine-grain diagnostics.
All Cantata inputs and outputs can be stored in SCM systems, and the in-built Cantata Makefiles can replace or work with build and continuous integration tools.
Cantata also integrates with common requirements management tools using CSV, Excel or ReqIF® data interchange. For more information see the Cantata Requirements Traceability capability.
How each part of Cantata fits into your environment:
| Executables | GUI | Libraries | |
|---|---|---|---|
| Cantata Component | 32 or 64-bit executables for Windows or Linux.
|
GUI plug-ins for test creation, editing and results diagnostics. | C and C++ libraries for test framework. |
| Easy Installation | Installed on host native operating system. | Built-on-Eclipse® IDE installed or Eclipse-Ready® plugins added via Eclipse Install New Software.
|
Wizard driven deployment of library, configured for each target platform (cross-compiler / RTOS and chipset). |
| Easy Integration | Out of the box correct bit type operations for your host OS.
|
Out of the box use of Cantata within IDE and on CLI. | Selectable zip file configuration for each target, keeping test scripts platform independent. |
INTEGRATED TEST FRAMEWORK
Cantata is a dynamic testing tool for unit and integration testing C and C++. The test framework provides a high degree of automation for the generation, execution and analysis of tests that fits into your development workflow.
Its integrated test framework automation enable developers to comply with the testing requirements of software safety standards and to implement industry best practices.
The Cantata test framework empowers you to select the most effective and efficient ways to develop and manage tests to suit your workflow
The Cantata test framework can also be easily extended or integrated directly with continuous integration and static analysis tools for further synchronization and automation of testing activities.
The Cantata integrated test framework makes it easy to:
- Set up testing projects to fit your workflow
- Generate tests
- Edit tests
- Control build and run on target
- Automate continuous integration and regression testing
- Produce diagnostics and reports
- Generate certification compliance evidence
Cantata provides an integrated test framework (test scripts and a supporting library) which enables any combination of testing styles for both unit and scalable integration testing. Tests can be edited in a GUI or directly as C/C++ code, and run as executables on both host and multiple target platforms.
User code is driven by portable test scripts with target compatible libraries, and built as a single test executable to run on multiple host or target platforms. Instrumentation is used for white-box access and code coverage, so production code is never modified for testing.
Setting up testing projects to fit your workflow
Cantata testing Eclipse projects can be set up in 5 flexible ways to best suit your normal workflow:
- Use an existing Eclipse project
- Create a workspace above the source code
- Create a project in or above the source code
- Link the source directory into a project
- Copy the source code into a project
Testing projects can be set up for bottom-up, top down or isolation unit testing approaches, and for either Cantata or externally driven integration testing.
Generating tests
Cantata tests scripts are generated from one or more selected source code files, and can contain several styles of test cases:
- Single test case per function in the code
- Multiple table driven test cases, using multiple user defined input values
- Robustness tests, using multiple pre-defined input values
- AutoTest (see AutoTest capability for details)
The source code is parsed to obtain a comprehensive model (.csi file) of the code, so structured tests can be generated from it. C/C++ test scripts are generated within test projects with corresponding Cantata Makefiles for automated regression testing.
Editing tests
Test scripts and Makefiles can be edited using a synchronised graphical Test Script Manager or any C/C++ code editor.
Initial default values are set for inputs, expected outputs, and expected call order for fast iterative test case editing. Function calls are controlled (simulated or intercepted) in the test script.
See the Unique call control capability for details.
Controlling build and run on target
All Cantata related files are stored in the “Cantata” directory at the top-level of a project. This allows the test files to co-exist with your source files in a self-contained hierarchy and to be easily managed using version control tools.
Tests are built with the code under test, using your standard build mechanism on host as binary compatible executables, and downloaded to run on a target platform via the IDE or command line.
Cantata can be deployed to an extensive range of targets (see Cantata Works in your environment). The flexible deployment and use of different build and I/O options, means Cantata works seamlessly with your target development environment, and allows you to test the code with software, hardware or processor in the loop.
Platform independent tests can be re-used over multiple targets, through controlling multiple Cantata target deployment configurations.
Diagnosing and reporting
Cantata provides powerful drill-down Eclipse views for pinpoint diagnostics of functional test and integrated code coverage results. Where even finer grained diagnostics are required, test execution can be stepped through under the control of a de-bugger.
Configurable reports are generated RTF, XML and HTML along with certification ready ASCII text.
Diagnostics and reports can also be enhanced and aggregated over multiple user through the optional Cantata Team Reporting add-on.
Producing certification evidence
Cantata comes with comprehensive tool safety standard specific guidance for the building, execution and management of tests for testing to software safety standards.
Cantata certification ready functional test and code coverage results are automatically produced in asci text format, directly from the target platform.
Together with the appropriate tool certification / qualification kit, and target deployment evidence, this provides all the necessary evidence of software testing required for device software certification submission.
WHY INVEST IN CANTATA
Industry leaders use Cantata to produce high quality, safety standard compliant or business critical embedded software, in the most cost effective way.
If embedded software is the main differentiator in your products, or critical to the running of your business, you want to deliver software reliability through cost effective testing.
Industry leaders across all safety related and business critical sectors, have already invested in Cantata. Every day they are getting a competitive advantage over those yet to benefit from automated unit and integration testing.
When doing nothing is the higher cost option.
To continue testing the hard way with manual tests, in-house or legacy tools, can be a lot more expensive than buying a suitable tool that makes your testing effective and efficient.
Investing in Cantata makes doing unit and integration testing easier, enabling organisations to:
Cut the cost of standards compliance
Reduce the risk of software failure
Lower testing costs
Shorten time to market
Why unit test with Cantata?
By testing units (defined as the smallest testable part of an application) Cantata pinpoints failures more precisely than larger system tests on complex applications. This means that errors can be fixed quickly and cheaply. Complex applications will always be more difficult and expensive to integrate and system test when the component units do not work correctly, so testing the units reduces these downstream costs and makes them more predictable.
- Automated test framework allows developers to fully test units on target, as soon as code is written.
- Complex applications can be verified by unit tests, which are cheaper than system tests to develop and maintain.
- Test case generation, flexible editing, powerful diagnostics, and tool integrations make developers efficient testers.
- Continuous integration and provides an efficient safety net against breaking existing code.
- Unit tests with integrated code coverage, quantify project progress.
Why integration test with Cantata
Cantata enables developers to test the interactions between units working as a group. Cantata integration tests can be performed with software (SiL), hardware (HiL) or processor (PiL) in the loop, with no technical limitation on scalability.
Cantata makes integration testing more effective
-
Correct internal and external sequences can be easily verified
-
Choice of entry point into integrated group, gives more test scenario flexibility
-
All inputs and outputs between units (parameters and data can be automatically checked
-
Interfaces between calls can be intercepted and checked, not just simulated with potentially false assumptions
-
User code can be easily added to C/C++ test script at a wide range of test control points
How can Cantata cut the cost of standards compliance?
Cantata meets all the testing requirements of software safety standards. Providing a single solution for dynamic unit and integration testing on host and target platforms.
Certification of development tools can be a heavy compliance cost burden. Cantata has been independently certified by SGS-TÜV SAAR GmbH providing a tool certification kit with everything needed out-of-the-box, available free of charge.
Compliance with the dynamic test requirements of software safety standards is complex and time consuming. Cantata provides comprehensive guidance on how to use its powerful feature set to accelerate meeting all standard specific test requirements.
How does Cantata reduce the risk of software failure?
Product recalls and infection of the wider brand and corporate reputations, can far exceed the development cost of individual applications. Unit testing is the most thorough way to test application code and prevent bugs in shipping devices.
Project over-runs can be mitigated by shifting verification earlier in the lifecycle to unit testing, reducing the risks of delays when integrating proven components.
Fitness for purpose litigation against companies and individuals is now an increasing risk. Where companies fail to employ accepted industry practices like thorough unit testing with Cantata, they cannot use the “state of the art” legal defence against such litigation.
How will Cantata lower my testing costs?
Testing earlier lowers costs by minimizing re-work dependencies on the whole code stream. Developers can immediately identify defects with Cantata unit and integration testing as soon as each component is available.
The high cost of unit and integration testing standard compliant software can be dramatically lowered through automation. From framework set-up through test case generation and execution on host/target platforms to diagnostics and results production, Cantata automation makes these cost effective.
Integrating tools into a toolchain can have hidden testing costs. Cantata’s tight integration with cross-compilation environments and intuitive C/C++ code tests in Eclipse GUI or code editors, make it easy to slot into any toolchain, and fast for developers to learn.
How does Cantata shorten my time to market?
Industry leaders recognise the need to ship faster without endangering quality. Cantata tests provide two key time advantages for Development Managers. Team collaboration and efficiency is improved with structured consistent tests and certification ready reports. Integration times are more predictable and shorter when integrating tested individual software components.
Why Cantata is better than any ‘free’ open source tool?
Unlike open source tools Cantata is specifically addresses the safety and business critical technical needs of end users, whilst providing ease of use and powerful diagnostics to make unit / integration testing as efficient as possible. Commercial grade tool quality is guaranteed with significant functional and productivity advantages over open source tools, whilst still offering very high levels of customization. Professional technical support is provided as well as comprehensive documentation, so time is not wasted trying to make the tool do what you need it to.
Cantata is pre-certified for many safety related standards; open source tools can be qualified or certified as suitable or usable in a safety related context. However the costs associated with such an exercise (defining requirements for the tool, developing and running tests against those and demonstrating the tool operates correctly in the end users environment), are prohibitive.
Existing investment in legacy open source test scripts implemented in C/C++ (e.g. CppUnit and CxxTest), generically known as xUnit can be re-used and embedded inside Cantata tests. This has the advantage of retaining all the existing test cases, whilst supplementing them with the more powerful features of Cantata.
AutoTest automatically generates passing Cantata unit tests to exercise all the C source code. It can be used on new or legacy projects, making it easy to:
- Configure automatic test generation
- Identify code testability issues
- Generate tests with full code coverage
- Plug ‘edge case’ gaps in coverage
- Create a safety net of baseline regression tests
- Upgrade from legacy test tools to Cantata
Autotest Video
What is Cantata AutoTest?
Cantata AutoTest creates complete unit test scripts or adds test cases to existing Cantata scripts. It parses the C source code to determine all possible paths through the code as required by a structural code coverage metric target:
- 100% function Entry-points
- 100% Statements
- 100% Decisions
- 100% Unique Cause MC/DC
An algorithm creates test case vectors which exercise all required code paths, using the Cantata powerful white-box capabilities to set data, parameters and control function call interfaces.
The test vectors drive the code, and check the parameters passed between functions, values of accessible global data, order of calls and return values.
AutoTest generated cases, are editable in the same ways as user generated cases, and each test case has a description of what path through the code it was created to exercise, making them easy to maintain.
What does AutoTest help me do?
CONFIGURE AUTOMATIC TEST GENERATION
AutoTest generation options can be easily configured via the GUI or command line:
-
Code coverage required (Rule Set)
-
How function calls are simulated or intercepted
-
Whether to access global static variables
-
Isolation unit test or cluster integration test static functions
-
What data to check (i.e. global data, function call arguments or return values);
-
Whether global data is modified during function calls to force paths;
-
Generation max limits on time, paths, arrays etc.
Selecting a code coverage Rule Set will determine the depth (i.e. number of test cases needed to achieve the target). The style is determined by use of the AutoTest specific and standard Cantata Workspace code generation options.
Isolation unit tests or cluster integration tests, using black-box or white-box testing approaches, provide flexibility to fit with whatever initial testing thoroughness and test maintainability for code refactoring is required.
IDENTIFY CODE TESTABILITY ISSUES
Where AutoTest is unable to generate tests, or the tests are incomplete, the AutoTest Generation Report provides useful insight into any testability issues within the source code.
Warning messages are written to the AutoTest Generation Report, showing where and why it was not possible to generate test cases, indicating possible problems in the source code or the ability to dynamically test it. It can therefore be a valuable supplement to code review, even if AutoTests are not dynamically run.
GENERATE TESTS WITH FULL CODE COVERAGE
The Cantata AutoTest Generation Report identifies the expected code coverage which will be achieved by executing the generated tests. As AutoTest cases are initially generated to pass, the only failing checks will be where a Coverage Rule Set checks achieved coverage against a target.
Achieving full code coverage is of course only part of the testing objectives. AutoTest generated test cases can be reviewed and where suitable mapped to requirements (see Cantata Requirements Traceability capability for more details). In this case, AutoTest dramatically reduces the developer effort to select test case vectors to verify the requirements.
PLUG EDGE CASE GAPS IN COVERAGE
Where complex code is being tested, even detailed requirements, and thorough robustness tests may leave untested execution paths in the code, because the different combinations of input vectors required for full coverage are difficult to identify. For these ‘edge cases’, AutoTest can generate test cases to plug any gaps in code coverage if the code is not redundant.
AutoTest can be used on just selected functions to generate a focused set of test cases achieving 100% coverage. Cantata automatic code coverage optimisation can at a single click delete/disable any test cases which do not add to the overall code coverage achieved.
CREATE A BASELINE REGRESSION TEST SAFETY NET
Legacy code that relies on system testing is very expensive and inefficient to modify, but writing a comprehensive set of unit tests from scratch is simply not commercially viable. Cantata AutoTest automatically generates a suite of unit tests for all selected code. These tests can act as a baseline safety net against future regression errors as code changes. This reduces reliance on expensive and time consuming system tests as well as being more thorough and pin-pointing errors precisely.
Cantata AutoTest generates a set of Cantata Makefiles, these can be easily integrated into any open source or commercial CI tools to enable fully automated testing during development, at every integration.
UPGRADE FROM LEGACY TEST TOOLS
Even where other unit testing tools are already in use, there can be real advantages in upgrading to the modern powerful test facilities available in Cantata. This especially true where legacy tools are not well supported or are not qualified for safety standards tool certification needs.
Cantata AutoTest provides two highly automated routes for upgrading unit test tools:
For a code base with an accepted level of quality, existing tests can simply be replaced by an AutoTest generated set of Cantata unit tests.
For legacy unit tests written in C (e.g. xUnit tools), existing tests can be incorporated into Cantata test scripts, retaining investment in legacy test cases and then enhanced with additional Cantata capabilities including AutoTest.
How does AutoTest work?
AutoTest can be invoked using a Test Script Wizard, or it can generate additional test cases in an existing test script in the Test Script Manager view. For larger code-bases it may be more convenient to run AutoTest from the command line.
AutoTest parses the source code and uses Cantata options to generate:
- AutoTest Generation Report, which provides both summary and detailed information on the tests that have been generated.
- Cantata test scripts (or additional test cases to existing scripts)
- Cantata Makefiles to build the test executable
The generated tests are then built and run, producing the standard set of Cantata results and reports.
AUTOTEST GENERATION ALGORITHM
Cantata AutoTest creates complete Cantata unit test scripts from source code using an algorithm that automatically generates test case vectors that exercise code paths, check data, check parameters and record call orders. The code paths used to generate the test cases are determined by selecting the metric types (Entry-point, Statement, Decision or unique cause MC/DC) in a code coverage Rule Set.
Preferences are available to influence the AutoTest generation algorithm, including the required level of code coverage, the method of controlling calls, what types of data to check, how data may be modified to force execution paths, time limits, path limits, and many more. These preferences affect the style of the Cantata tests, and limit the test generation scope.
Tests can be more or less thorough, and can use white-box or black-box approaches. The decision on what style of unit tests to generate with Cantata AutoTest will depend on regression plan objectives and how code changes. For example, allowing flexibility for code refactoring of static functions without breaking black-box tests.
AUTOTEST WARNING AND ERROR MESSAGES
Where Cantata AutoTest generation results in incomplete tests or encounters suspect code quality, warning messages are written to the AutoTest Generation Report. These warning messages indicate possible problems in the source code or its dynamic testability, so they can be of significant value.
Cantata AutoTest will report:
- unreachable code, where it cannot generate test cases to exercise all parts of the software under test
- potential crash scenarios such as null pointers, an out-of-bounds array index, decisions using uninitialised variables, and division by zero
- type truncation, where type casting might result in data loss
- uninitialised data use, such as a field within a large structure
- non-constant static data within a function, as its value will be unknown at the start of a test case
- non-returning functions, where a function does not return control to the caller – although Cantata does have the capability to work around this for testing purposes.
- implicit function declarations, where a function is called before it has been declared
Cantata code coverage capabilities can be used integrated with Cantata tests, or with any external test driver.
It provides an objective measurement of test execution completeness, to supplement requirements based and robustness tests.
Cantata code coverage makes it easy to:
- Set how required coverage is measured
- Diagnose coverage results
- Filter coverage results
- Generate reports and certification evidence
What is code coverage?
Code Coverage is a dynamic analysis technique to measure which parts of the code have been executed by tests. The logic structure within the code can be identified by individual constructs (syntax elements). The measurement of these individual constructs is recorded using structural code coverage metrics.
Code coverage analysis is used to identify testing gaps where code is not exercised by requirements driven and robustness driven testing approaches.
Coverage analysis is required for compliance with all international software safety standards, with higher safety levels requiring achievement of more rigorous coverage metrics.
What does Cantata Code Coverage help me do?
DIAGNOSE COVERAGE RESULTS
Cantata diagnostics present coverage from complete project code-trees, drilling down to individual code constructs within each line of code helping you to pin-point gaps and direct test effort efficiently.
Automatic test case coverage optimisation aids test case vector selection from large data sets, and reduces regression testing overhead.
FILTER COVERAGE RESULTS
Cantata provides unparalleled filtering of code coverage data by selection of:
- Coverage Metrics
- Projects
- Test Cases
- Inheritance Contexts
- User-defined Contexts
- Coverage Files
Cantata can also trace code coverage throughout the execution of individual test cases.
GENERATE REPORTS
In addition to the powerful diagnostics of code coverage results within the Eclipse® GUI, Cantata generates flexible user configurable reports in XML, HTML and safety standards certification ready ASCII text reports.
Reports and certification evidence embed the integrated checks on the achievement of the required coverage metric targets.
How does Cantata Code Coverage Work
Cantata Coverage uses instrumentation (via the GUI or CLI) on a copy of the source code, and coverage data is recorded for both graphical diagnosis and certification ready evidence.
Coverage analysis can be used integrated with Cantata tests or used to measure execution from other tests, and data from multiple tests can be easily combined.
Analysis is made simpler by customisable coverage Rule Sets, which automate all the code instrumentation, data reporting and required coverage checking, for all the main software safety standards.
What is Cantata requirements traceability?
Cantata provides full bi-directional traceability relationships between the user’s requirements or test plans and Cantata test scripts and test cases, as required by all the software safety standards.
Requirements traceability applies to many more items in your software development than just Cantata unit and integration tests. That is why Cantata provides added value to your existing traceability mechanism and does not replace it. Cantata Trace automation makes it easier to:
- Comply with traceability standards
- Perform requirements driven testing
- Trace existing Cantata tests to requirements
- Synchronize with requirements management tools on open standards interfaces:
Microsoft Excel®, CSV UTF-8, and ReqIF XML.
Cantata Trace imports sets of requirements or test plans from popular requirements management tools directly into Cantata. An intuitive drag-and-drop interface makes it easy to assign bi-directional trace relationships between requirements and test scripts and/or test cases. All traced information is stored on a central Cantata server. The requirements traceability associations are then exported back to the requirements management tool together with the status of each test and code coverage results.
Which Requirements Tools does Cantata integrate with?
Cantata can be integrated with any tool using a CSV, or ReqIF 1.x (Requirements Interchange Format) interface.
Cantata Trace has been integrated with the popular tools below used for requirements management and traceability. For requirements in text documents (e.g. Microsoft Word®) traceability can be performed using Micosoft Excel® or via tools to export in ReqIF format. If your preferred tool is not already integrated with Cantata Trace, please contact us to arrange an integration.
IBM® Rational® DOORS®
Version 9.6.1.4 and higher using ReqIF import/export
PTC® Integrity™
Version 10.5 and higher using ReqIF import/export
Polarion® ALM™
Version 2016 SR1 onwards using ReqIF import/export
Microsoft® Excel®
Version 15.0 and higher using .xls or .xlsx workbooks
Spreadsheets
Such as Libre Office Calc using CSV UTF-8 files.
intland codeBeamer®
Version CB-8.0.0-RC and higher using ReqIF
What does Cantata Trace help me to do?
COMPLY WITH TRACEABILITY STANDARDS
Bi-directional traceability is required by all major software safety related standards to demonstrate requirements coverage, i.e. that all requirements have associated passing tests for the corresponding code implementation. In the reverse direction, knowing requirements are verified by a given passing test, aids regression test management.
Cantata Trace imports requirements or test plans, provides a drag-and-drop linking with Cantata test, and exports back the links, execution pass/fail status of each test, and levels of code coverage obtain by a script. Cantata Trace makes it simple to provide a fully compliant bi-directional requirements traceability process that fits with your existing tools.
EASY TO USE REQUIREMENTS DRIVEN TESTING
With Cantata Trace you can view your requirements imported directly into the Cantata Eclipse IDE alongside tests, avoiding the need to refer to external documents or access other tools when creating requirements driven tests.
Imported and selected requirements, or test plans, are displayed in a table list for trace association, with each requirements detail (unique ID, requirements text, images and links) available in a built-in HML view.
As test cases are implemented, the requirements which they verify can simply be traced (i.e. associated with) by drag-and-drop in bi-directional views.
FASTER TRACEABILITY FOR EXISTING TESTS
Cantata Trace can be used to assign requirements or test plans to existing Cantata test scripts and test cases, making it faster to generate traceability data and verify that existing tests fully verify all the requirements.
Existing tests to which traceability can be added include:
- Tests created using a previous version of Cantata.
- Tests automatically generated by Cantata AutoTest, which when reviewed satisfy a requirement.
- Tests written by a 3rd party, where requirements traceability was incomplete
- Tests re-used from another tool inside Cantata (e.g. to imported from an open source xUnit tool, or converted from RTRT).
Existing tests can be traced (i.e. associated with requirements or test plans) by drag-and-drop in bi-directional views.
EASY ROUND-TRIP SYNCHRONIZATION WITH REQUIREMENTS MANAGEMENT TOOLS
Importing and exporting data for round trip synchronisation of requirements with tests is done using CSV UTF 8, Microsoft Excel® or ReqIF XML files.
For CSV and Excel, Cantata Trace provides spreadsheet header mappings for External ID, Descriptions and for the various exported results. For ReqIF data interchange, it is a one-time set up of the ReqIF interface in the requirements management tool.
Synchronisation can be done either from the Cantata Trace UI, or scripted via the Cantata server Trace REST API for automation with continuous integration builds.
How does Cantata Trace work?
MANAGING REQUIREMENTS IMPORT
Cantata Trace can import requirements (or test plan information), directly from popular requirements management tools. Sets of requirements data (unique ID, requirements text, images and links) are imported in CSV UTF-8, Microsoft Excel®, or ReqIF formats from a requirement management tool. The data is stored as a non-editable copy on a Cantata server database.
A Requirement Set can be all or a subset of the requirements, named appropriately on import, and users can further filter a set for their own work. Within a data file to be imported, specific Excel Workbook or ReqIF Specifications can also be selected.
The mapping of ID and Description fields for display in Cantata Trace, plus other data for round trip alignment when exporting, can be set up as ReqIF XML specifications or spreadsheet header mappings.
MANAGING CHANGES TO REQUIREMENTS
Whenever a set of requirements is imported, a date stamp is assigned to it. When changes or additions to the requirements are made in the requirements management tool, the updated requirements can simply be re-imported into Cantata. A “Copy Assigns” function retains any existing Cantata associations from a previously imported set.
When trace associations are assigned between requirements and tests, these relationships are automatically stored on the Cantata server, so engineers are able to see in real time all traced associations between the latest requirements and tests.
Cantata Trace provides colour coded visibility of changed requirements since last the last time that a set of requirements was imported. Together with selecting only changed requirements that have changed. This makes it easy to focus tracing on the just changed requirements.
MANAGING REQUIREMENTS EXPORT
Exporting from Cantata Trace is controlled by the Trace Admin functions. The same file formats are supported as for import, and the user can configure what information should be exported and which coverage options are relevant.
Cantata Trace exports the requirements (ID and Description) the traced links between requirements or test plans and Cantata test scripts / test cases, the execution pass/fail status of each test, and levels of code coverage obtain by a test script.
Cantata Trace data can also be exported when required using the GUI, or as an automated script export via the REST API as part of a continuous integration test run.
Testing what happens when function calls from the Software Under Test (SUT) are made, is integral to all levels of dynamic testing. Cantata call interface control automatically generates the necessary test framework for these interfaces, making it easy to:
- Simulate the call interface between the SUT and called objects (whether included in the build or not)
- Intercept the call interface between the SUT and called objects included in the build
- Set and check data (accessible to the SUT) when function calls occur
- Set and check return values back to the SUT or raise exceptions
- Inject errors to the SUT
- Check the order of calls made
- Control each instance of a function call – so that different behaviour can be set/checked for each occurrence
Automatic Wrapping to intercept (not just simulate) calls is unique to Cantata, providing unparalleled call control for both unit isolation and integration testing.
What is Cantata Call Control?
Cantata call interface controls are automatically generated for all function calls made by the Software Under Test (SUT) e.g. to other functions in the same or other files, 3rd party object code, operating system / compiler library, or target firmware / hardware. Users have complete control over the appropriate style of Cantata call control to be generated for each call made from each function:
- Stubs to simulate calls to objects not included in the build
- Isolates to simulate specific calls to objects included in the build.
- Wrappers to simulate or intercept specific calls to objects included in the build.
- Dependency Resolver to automatically remove undefined link references
Each type of Cantata call control is written in C/C++ as appropriate, giving a test control point in the execution flow to set and check data available to the SUT (e.g. global data or hardware registers) and parameters/returns. As each control type can also have multiple instances, different behavior and call order can be set or verified each time a call is made in each test case scenario.
Isolation of C++ for testing from the remainder of the code base, can result in a lengthy daisy-chain effect of undefined references at link time when building test executables. Cantata Dependency Resolver determines the missing references and automatically adds the code required to resolve all the undefined references (methods or variables).
The various Cantata call controls give testers the flexibility to choose simulation or interception as needed to drive and verify the code.
What is Cantata Wrapping?
Wrapping is a unique automatic accessibility instrumentation that intercepts or simulates function calls. It goes far beyond traditional simulations (e.g. stubs, mocks, fakes etc), providing more flexibility for testing interfaces when objects are integrated together.
Automatically generated Wrappers intercept calls before the call (BEFORE Wrappers), after the call (AFTER Wrappers), or can replace the original call with a simulation (REPLACE Wrappers).
Where a Cantata test script is used, BEFORE, AFTER and REPLACE sections of the Wrapper are written to the test script. Wrapping can also be used outside a Cantata test script to provide call interface control with other test drivers.
What does Wrapping help me do?
INTERCEPT & SIMULATE CALLS AT UNIT TEST
Unit testing verifies one or more functions without the rest of the system. Traditional call controls (e.g. stubs, mocks, fakes dummies) only simulate called objects excluded from the compiled software unit under test.
Cantata Wrapping enables control over specific function calls both inside and outside that compilation boundary. Specific calls to functions can be controlled with:
- Simulate Wrapper to replace the call
- Intercept Wrapper to set / check the interface before and after the call
Each Wrapper also allows setting/checking of all accessible data when the calls are made.
Wrapping uniquely gives automated control inside the compilation boundary (e.g. functions calling other functions in the same file or class), and also greater tester choice in defining what is included in the unit testing boundary for isolation or cluster unit testing.
INTERCEPT & SIMULATE CALLS AT INTEGRATION & SYSTEM TEST
Integration & systems testing verifies the actual interaction between multiple units, so the more units are integrated, the less useful replacing those units with simulations becomes. Cantata Wrapping at integration & system testing retains all the benefits of Wrapping at unit testing, but it becomes even more useful to simulate or intercept specific calls when testing the actual interactions.
Cantata Wrapping interceptions are available for any C/C++ function call made from anywhere within the software under test to:
- Other software components in the build
- External software (including 3rd party or object code)
- Operating system or standard library functions
- Firmware or hardware
Cantata Wrapping interceptions allow the tester to check actual interface behavior, rather than use test scenarios based on assumptions in a simulation. Interception of calls provides unparalleled capability inject errors and verify actual behavior of actual interactions (e.g. failures with operating systems, 3rd party code, firmware and hardware) which are otherwise impossible or too expensive to test.
How does Wrapping work?
Cantata Wrapping uses accessibility instrumentation (via the GUI or CLI) on a copy of the source code to intercept function calls from the Software Under Test (SUT).
Anything called by a C/C++ function call can be Wrapped (e.g. another function, a 3rd party object, a database etc). A Wrapper has an identical function prototype to the called object being replaced, but testability instrumentation allows the called object to be included in the test build without causing linker errors.
The BEFORE, REPLACE and AFTER Wrapper sections insert programmable additional C/C++ functions, which then call: another Wrapper section, the called object or pass back to the SUT. Each Wrapper section contains Cantata directives to record call order, modify or check parameters / exceptions and all data accessible to the SUT or called object, and returns.
EDITING CANTATA CALL CONTROLS
Cantata automatically generates each Wrapper (containing BEFORE, AFTER and REPLACE sections) into the test script with a default BEFORE-AFTER pairing instance. This means that called objects can be Wrapped without forcing the tester to specify the return value.
Additional Wrapper instances (in either Before-After or Before-Replace pairings) may be added to check all parameter values passed to and from the called object. When additional instances are added, Cantata automatically offers checks of actual against expected values for all the parameters passed over the interface.
A function call to an object, can have all calls Wrapped or just selected calling functions.
Wrappers are implemented as C/C++ into a Cantata test script, and can be edited in the Cantata Call Interface Control GUI, or directly using any C/C++ editor.
CONTROLLING CALLS IN TEST CASES
Cantata call interface controls can use multiple instances stubs, Wrappers or isolates to provide different behaviors on different occasions the call is made. These controlled calls and instances are used in a Cantata test script to set the expected order of calls and controlled behavior for each test case.
Expected Call Sequences are very flexible (linear, fixed size loops, parallel etc.), and can be set for specific orders or any time matches for each controlled call. The order of expected calls in each test case can be edited in the Cantata GUI (test script manager) or directly in any C/C++ editor.
FLEXIBLE TESTING TECHNIQUES
Cantata is a dynamic testing tool for C/C++ unit and integration testing. Its flexible set of test automation capabilities enable developers to comply with the testing requirements of software safety standards and to implement industry best practices.
Cantata test automation empowers you to select the most effective and efficient combination of dynamic testing techniques to suit your C and C++ code implementation styles.
Cantata tests can be extended directly in any code editor, extending the flexibility to anything possible in the C/C++ programming language.
Cantata automation accelerates:
- Requirements driven testing
- Robustness testing
- Structural testing
- Black-box testing
- White-box testing
- Procedural and state machine testing
- Object oriented testing
- Isolation unit testing
- Integration testing
Focused Flexibility
No single tool suite can support every style of testing, and Cantata is no exception. Cantata is focused on the unit and integration testing requirements of standard compliant dynamic testing C and C++ code on embedded systems.
With the Cantata flexible toolset you can choose which combination of automated testing capabilities are right to drive your unit tests, integration tests or to enhance your higher level tests.
Cantata capabilities by test technique
REQUIREMENTS DRIVEN TESTING
Cantata supports requirements driven testing to verify that the software does what it should do (derived from low level requirements, specifications or models).
The Cantata test harness drives the code under test via a test script calling a function with selected inputs (parameters and data) and automating the checking of actual against expected behaviour.
Requirements imported into the Cantata IDE make it easier to define test cases and trace requirements with test scripts or test cases.
ROBUSTNESS TESTING
Cantata supports dynamic robustness testing to verify that code copes with abnormal conditions using techniques recommended by all software safety standards such as:
- abnormal system initialization
- invalid inputs & data
- failure mode handling
- boundary values
- arithmetic edge conditions
- timing constraints
- memory allocation
- error seeding
Robustness testing is made easy with Cantata Robustness Rule Sets of pre-defined values for basic data types, in looping test cases. All accessible global data is also automatically checked for inadvertent changes. Cantata test scripts and wrapping make it easy to create abnormal test pre-conditions and inject errors during test execution. (See also Cantata Call Control)
STRUCTURAL TESTING
Structural testing identifies source code untested by requirements driven and robustness tests, and addresses those gaps.
Cantata code coverage (used within or outside Cantata tests) pin-points any such unexecuted code elements in specific or all contexts. Gaps can be resolved by adding new requirements based test cases, documenting reasons why the code should not be executed in a particular context, or removing redundant code.
Cantata AutoTest automatically generates whole test scripts or additional test cases to achieve 100% structural coverage for C code.
BLACK BOX TESTING
Black-box testing verifies code only via its public interfaces, and does not require knowledge of internal code implementation for checking correct behavior.
Cantata brings power to this technique with user-selected or pre-defined parameterised looping tests, so the same functions can be called with large data sets of input parameter or data values. The sequence of expected calls and expected return values can vary according to the input data sets, alongside global data outputs checked throughout the test run.
Cantata table driven test data sets can also be imported/exported via CSV. A GUI combinatorial effect calculator aids test vector selection. Cantata automatic coverage optimisation identifies the precise set of test case vectors needed to reach coverage targets.
WHITE BOX TESTING
White-box testing verifies code by granting access to the encapsulated code internals (private / static data and functions).
Cantata grants testers direct access to these without polluting production code with conditional compilation, by using fully automatic accessibility instrumentation only during testing. Test scripts can directly call private or static functions and set / check parameters and data declared private or static to a file or local function.
Cantata white-box tests can set up trigger conditions, and directly check values or internal behaviour more efficiently than with black-box tests.
Cantata call control can also be used on calls inside the compilation boundary for white-box testing.
PROCEDURAL AND STATE MACHINE TESTING
Cantata procedural tests verify the processing logic or algorithms in functions using appropriate sets of parameter / data inputs, and checking that actual call order, data read/writes and return values are as expected. Cantata parses the source code to generate test scripts with the necessary test hooks on function parameters and accessible data for users to set and check.
State machines are tested by setting trigger conditions and creating events, to verify correct state transitions. Cantata white-box testing makes this particularly efficient through direct access to local function static data storing the machine state.
Cantata user defined context code coverage can also be used to verify code has been tested in different states.
OBJECT ORIENTED TESTING
Cantata OO style tests are implemented as classes for testing methods, templates, or clusters of classes. They feature automated:
- test case re-use via a parallel inheritance hierarchy
- test class inheritance structure for inherited classes
- concrete implementation of abstract base classes (ABCs) or pure virtual methods (PVMs)
To break the class dependency chain and make C++ isolation testing easy, Cantata automatically finds and resolves dependencies on undefined references that are not directly called by the software under test.
ISOLATION UNIT TESTING
Cantata does not dictate what code item is tested as a unit, or how much each unit is isolated (decoupled) from the rest of the system. Complete flexibility is given by automatic configurable generation of stubs and isolates to simulate, and wrappers to intercept calls made both externally and internally by the unit under test. The correct internal processing and correct use of interfaces (parameters and data), can be verified with Cantata as an isolated executable as soon as a unit compiles, without the need for the rest of the system.
Cantata generates an initial executable test case for each function / method in the selected source file(s) under test. Cantata unit tests call a function, initializing all input / output parameters, accessible data and expected call order. Test cases are independent of each other and the platform on which they run, avoiding a daisy-chain of dependencies.
INTEGRATION TESTING
Cantata driven integration tests are just large unit tests, but with a cluster of multiple units built as a single executable. Such a tests can verify the same internal behavior of unit, but more importantly the interactions between units such as:
- inter-unit memory allocation
- order of data read / writes
- order of calls between units
Cantata integration tests provide the power to choose test script driver entry-points, with both internal and external simulations (stubs and isolates) or interceptions (wrappers) for flexible use of top-down, bottom-up or all-as-one integration testing styles. Cantata wrappers can also be used where Cantata is not the test diver, adding further control over integration tests
Reviews
There are no reviews yet.