OGC API - Features Conformance Test Suite

Scope

This executable test suite (ETS) verifies that an OGC API - Features instance conforms to OGC API - Features - Part 1: Core (OGC 17-069r3), OGC API - Features - Part 2: Coordinate Reference Systems by Reference (OGC 18-058) and related standards. Conformance testing is a kind of "black box" testing that examines externally visible characteristics or behaviors of the SUT and is independent of any implementation details.

Several conformance classes are defined in the principal specifications. The ones listed below are covered by this test suite:

Core
- The Core specifies requirements that all OGC API - Features instances have to fulfill.
Coordinate Reference Systems by Reference
- Optional conformance class to fulfill requirements of OGC API - Features - Part 2: Coordinate Reference Systems by Reference specification.

Test requirements

The documents listed below stipulate requirements that must be satisfied by a conforming implementation.

OGC API - Features - Part 1: Core

There are some requirements the test suite expects to be satisfied by the service under test:

General:
- The API landing page response must be available in JSON.
- The API definition response must be available in OpenAPI 3.0.
- The conformance classes response must be available in JSON.
- The feature collections metadata response and feature collection metadata responses must be available in JSON.
- Feature collections and features must be available in GeoJSON.
Coordinate Reference Systems by Reference:
- CRS definitions must start with urn:ogc:def:crs:EPSG or http://www.opengis.net/def/crs/EPSG (Tests of A.2.1. and A.2.2. of Abstract Test Suite).

Reserved identifiers

The collection identifier dUmmYColLection is reserved and must not be used by a service under test.

Test suite structure

The test suite definition file (testng.xml) is located in the root package, org.opengis.cite.ogcapifeatures10. A conformance class corresponds to a <test> element, each of which includes a set of test classes that contain the actual test methods. The general structure of the test suite is shown in Table 1.

Table 1. Test suite structure
Conformance class	Test classes
Core	org.opengis.cite.ogcapifeatures10.conformance.core.general.* org.opengis.cite.ogcapifeatures10.conformance.core.landingpage.* org.opengis.cite.ogcapifeatures10.conformance.core.apidescription.* org.opengis.cite.ogcapifeatures10.conformance.core.conformance.* org.opengis.cite.ogcapifeatures10.conformance.core.collections.* org.opengis.cite.ogcapifeatures10.conformance.crs.discovery.* org.opengis.cite.ogcapifeatures10.conformance.crs.query.*

The Javadoc documentation provides more detailed information about the test methods that constitute the suite.

Implementation details

Whenever features are tested by a single test, the number of features is limited to one.
The following limits for tests apply:
- Paging iterations: 3
- Number of tested collections, if no limit is specified: 20
- Number of tested CRSs: 20
- A limit of 10 is set for all requests to collection items (features), except if a limit is already set by the test.
- For tests for correct matched numbers of features a maximum of 10000 is set. Tests are skipped for collections that contain more features.

How to run the tests

The options for running the test suite are summarized below.

As entry point the API landing page of an OGC API - Features instance is expected.

1. Integrated development environment (IDE)

Use a Java IDE such as Eclipse, NetBeans, or IntelliJ. Clone the repository and build the project.

Set the main class to run: org.opengis.cite.ogcapifeatures10.TestNGController

Arguments: The first argument must refer to an XML properties file containing the required test run arguments. If not specified, the default location at $ {user.home}/test-run-props.xml will be used.

You can modify the sample file in src/main/config/test-run-props.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties version="1.0">
  <comment>Test run arguments</comment>
  <entry key="iut">https://www.ldproxy.nrw.de/rest/services/kataster/</entry>
</properties>

The TestNG results file (testng-results.xml) will be written to a subdirectory in ${user.home}/testng/ having a UUID value as its name.

2. Command shell (console)

One of the build artifacts is an "all-in-one" JAR file that includes the test suite and all of its dependencies; this makes it very easy to execute the test suite in a command shell:

java -jar ets-ogcapi-features10-${version}-aio.jar [-o|--outputDir $TMPDIR] [-h|--generateHtmlReport] [test-run-props.xml]

3. Docker

This test suite comes with a Dockerfile which can be used to easily setup the OGC test harness with the test suite.

You need to retrieve the image from Docker Hub by running the following

docker run -p 8081:8080 ogccite/ets-ogcapi-features10

Then, open http://localhost:8081/teamengine/ to see below

Create a new session from the list

Choose 'OGC' organization then Specification 'OGC API - Features - 1.0'

Choose the endpoint URL to test if you query all the endpoints (it could take too long depending of the available datasets for the endpoint)

ogc api features conformance test suite ogccite 05

Wait test execution and see the results from the validator

result ogc api features test suite ogccite 06

HTTP calls to localhost are not working with Docker. You need to make calls using host IP (ipconfig on Windows or ifconfig on Unix based OS like Linux or Mac)

Details can be found on Create Docker Image and create and start Docker Container.

4. OGC test harness

Use TEAM Engine, the official OGC test harness. The latest test suite release are usually available at the beta testing facility. You can also build and deploy the test harness yourself and use a local installation.

Using the REST API

Using the REST API of the TEAM Engine requires some run arguments. These are summarized in Table 2. The Obligation descriptor can have the following values: M (mandatory), O (optional), or C (conditional).

Table 2. Test run arguments
Name	Value domain	Obligation	Description
iut	URI	M	A URI that refers to the landing page of the service under test. Ampersand ('&') characters must be percent-encoded as '%26'.
noofcollections	Integer	O	Number of collections under test. -1 means that all collections should be tested. Default: 3