Developing Tests for Application Profiles

Introduction

This document offers guidance about how to develop and execute a test suite that verifies an implementation of some OGC application profile ([definitions-profiles]) that is based on one or more existing standards. It is primarily intended for developers and other technical professionals who seek to test implementations that claim conformance to an OGC profile. For more general information about how to develop OGC conformance tests, see the TEAM Engine documentation site.

A profile always has one or more dependencies that must be heeded when developing an implementation or a conformance test suite. As indicated in [wms-profiles], a test suite for any WMS profile will include tests covering one or more conformance classes in the OGC WMS conformance test suite. The DGIWG profile requires conformance at the "Queryable WMS" level; the associated test suite also includes tests that cover the specific requirements of that profile.

A test suite for a profile generally selects one or more conformance classes (or levels) ([definitions-conformance-classes]) from the relevant set of base specifications; these tests then implicitly become part of the dependent test suite. The base tests are invoked in the course of running the profile-specific tests.

The source code can be included directly. The manner in which this is done depends on how a test suite is implemented (TestNG or CTL). The type of the base test suites determines the type of the profile test suite: if the base test suite is implemented in TestNG the profile test suite must also be implemented in TestNG. The same applies to CTL.

A test suite that covers the requirements of an application profile is accessed and executed just like any other test suite. It appears in the listing of available suites, and it can be selected and run in the same manner. See the following figure: Profile-specific test suites.

Profile-specific test suites
Figure 1. Profile-specific test suites

TestNG

Most of the OGC conformance test suites developed in recent years are based on the TestNG framework. The framework provides a simple declarative mechanism for incorporating tests from other suites in a test suite definition, which is an XML document that conforms to this document type definition.

Note

While an XML file (conventionally named testng.xml) is often used to define a test suite, it is also possible to assemble the runtime definition programmatically using the TestNG API. The XmlSuite class is the entry point for doing this (it is correlated with the top-level <suite> tag).

The root <suite> element contains one or more child <test> elements that correspond to conformance classes. These conformance classes may be drawn from other test suites. For example, the WFS 2.0 test suite uses two conformance classes from the GML 3.2 test suite. The following listing shows how this dependency is declared. Precondition is that the base test suites are available in class path. If Maven is used the dependencies of the base test suites must be included in the pom.xml

Importing conformance classes from other test suites
<suite name="wfs20-1.25" verbose="0" configfailurepolicy="continue">
  <parameter name="wfs"  value=""/>
  <parameter name="fid"  value=""/>

  <test name="All GML application schemas">
    <classes>
      <class name="org.opengis.cite.iso19136.general.XMLSchemaTests" />
      <class name="org.opengis.cite.iso19136.general.GeneralSchemaTests" />
      <class name="org.opengis.cite.iso19136.general.ModelAndSyntaxTests" />
      <class name="org.opengis.cite.iso19136.general.ComplexPropertyTests" />
    </classes>
  </test>
  <test name="GML application schemas defining features and feature collections">
    <classes>
      <class name="org.opengis.cite.iso19136.components.FeatureComponentTests" />
    </classes>
  </test>
  <test name="Simple WFS">
    <packages>
      <package name="org.opengis.cite.iso19142.simple" />
    </packages>
  </test>
  <!-- remaining WFS tests -->
</suite>

The <test> sets are run in document order by default, which is usually the preferred behavior. If any tests fail in a base conformance class, this can cause profile-specific tests to be skipped since there is already evidence of non-conformance.

How to

  1. Create a new maven project

  2. Include the dependencies of the base test suites

  3. Create a testng.xml file including all tests form the base test suites

  4. Implement the additional tests for the profile

  5. Add the additional tests to the existing testng.xml

Examples

CTL

The OGC Compliance Test Language (CTL) specifies an XML grammar for defining a test suite. Some older OGC test suites were implemented using CTL scripts. In recent years test suites have been developed using the TestNG framework, and this is the recommended approach for new test suites. But some current profiles are based on older OGC standards for which only CTL-based test suites exist.

Tests may be organized into separate packages denoted by the ctl:package element; the main package must contain a ctl:suite element that identifies the starting test. The ctl:profile element may be used to formally define a test group that corresponds to a conformance class. A profile must refer to its base test group (usually the main test suite), and it must also identify the starting test. The sample listing below displays a profile definition for the GET method binding in the WMTS test suite.

Profile for the GET method binding in WMTS 1.0
<profile xmlns="http://www.occamlab.com/ctl" name="wmts:server.profile.kvp.get">
  <title>WMTS 1.0 Server Compliance Test Profile for KVP GET binding</title>
  <description>Verifies that a WMTS 1.0 server implementation complies with conformance classes for KVP GET binding.</description>
  <defaultResult>Pass</defaultResult>
  <base>wmts:server.suite.base</base>
  <starting-test>wmts:server.profile.kvp.get.main</starting-test>
</profile>

The <base> element refers to the suite or profile (by name) that this profile depends on. This establishes an implicit ordering, such that the root suite is run first and test execution follows the dependency graph.

It is required to include the required resources of the base test suites in the profile test suite.

How To

  1. Create a new maven project

  2. Create a new CTL script as base entry point for the test suite

  3. Include the required resources of the base test suites

  4. Implement the additional tests for the profile

Example

No example available yet.

Reporting

Test suites of application profiles support the same output formats as a non profile test suites (see User Guide).

To define the mandatory tests of the profile, conformance classes have to be encoded as described in Encoding Conformance Classes. Several report formats (e.g. EARL, HTML) consider this configuration.