This page discusses query testing capabilities provided by Stardog to perform regression testing for correctness and performance using declarative test definitions.
<details open markdown="block"> <summary> Page Contents </summary> 1. TOC </details>Testing the performance and correctness of software code is an indispensable part of software development process. When developing a graph-based solution, a similar capability is required for testing the performance and correctness of queries as the data models and the queries evolve over time. This kind of testing can certainly be done by using your choice of Stardog API with a software testing library. However, Stardog also provides a way to declaratively define your tests in a test definition file, automatically create test definitions from your existing queries and run these tests via CLI command and/or using the API.
A test definition is an RDF file commonly serialized using the Turtle syntax. A test definition file contains a list of tests and each test is for testing a single query against a single database with configuration parameters. Typically, most of the configuration parameters are defined at the file level but each test might define its configuration parameters as shown in the following example:
:SelectAlbums a :Test ;
:server "https://express.stardog.cloud" ;
:database "stardog-tutorial-music" ;
:queryString "SELECT ?album { ?album a :Album }" ;
:resultCount 1037 ;
:warmups 2;
:runs 3;
:expectedTime 100 .
As seen a single test definition is a set of properties attached to the same subject. The local name of the subject is used as the test name and
each test definition should have the Test type declaration shown in the first line of this example.
We explain what the configuration parameters mean and how they can be organized in definition files in the following sections.
Each test definition file can optionally contain a configuration section at the beginning that specifies the configuration options that will be used
by default. The default value are attached to a special subject named defaults but otherwise look exactly like how they would look like when
attached to regular tests.
:defaults
:server "https://express.stardog.cloud" ;
:database "stardog-tutorial-music" ;
:warmups 2;
:runs 3 .
Typically, server and database information will be shared between the tests and defined in the default configuration section. The default values will be used for all the tests unless the test overrides these configuration options.
The credentials to access the server are not specified as part of the test definitions in order to avoid exposing sensitive information in plain files. The credentials are specified while running tests and can take advantage of features like password files.
There are three different ways to specify the query for a test and only one should be used for each query.
First option is to specify a query string inline in the test definition file using the queryString property:
:SelectAlbums a :Test ;
:queryString "SELECT ?album { ?album a :Album }" ;
...
Multi-line query strings can be specified using the triple-quote syntax of Turtle:
:SelectAlbums a :Test ;
:queryString """SELECT ?album
WHERE {
?album a :Album
}""" ;
...
When the query string is sent to the database it will use the stored namespaces from the database. The query string may include additional prefix declarations too. But the prefix declarations defined in the test file will not be applied to the query.
The second option is to specify a path to a query using the queryFile property:
:SelectAlbums a :Test ;
:queryFile "selectAlbums.sparql" ;
...
The property value is a path for the query file. If a relative path is specified then it will be resolved using the path of the test definition
file. That is, in the above example selectAlbums.sparql should exist in the same directory as the definition file. Paths may point to subdirectories
or parent directories as needed.
The third option is to specify the name of a stored query using the queryStored
property:
:SelectAlbums a :Test ;
:queryStored "selectAlbums" ;
...
The stored query with the given name should exist in the server and should be executable against the database.
There are two different ways to check the correctness of a query.
The first way is to save the expected query results into a file and specify the file path in the test definition using the resultFile property:
:SelectAlbums a :Test ;
:resultFile "selectAlbums.srx" ;
...
The file path is again relative to the test definition file similar to queryFile explained above. The file format will be selected based on the
extension of the file name. The extensions .srx or .xml can be used if the query results are in the SPARQL/XML format and the extensions
.srj or .json can be used if the query results are in the SPARQL/JSON format. The query results should not contain any bnodes and very large
result sets, e.g. more than million bindings, should be avoided.
The second option is to specify the number of expected results using the resultCount property:
:SelectAlbums a :Test ;
:resultCount 1037 ;
...
This option is not as reliable because a query may return the same number of results but the contents of the query results might not be same. But in some cases such as the very large result set use case mentioned above this would be a reasonable option.
The result count for queries are described as follows:
SELECT: The number of bindings, i.e. rows, returnedCONSTRUCT and DESCRIBE: The number of unique triples returnedASK: Count is 1 if the query returns true and 0 if falsePATHS: The number of paths returnedUpdate queries (COPY, CLEAR, INSERT, ...) do not return any results; they update the database. The resultCount or resultFile values
specified for update queries are simply ignored. since an update query can never fail in a test by definition. The correctness of update queries can be
checked by adding a read query after the update
query in the test definitions. The tests defined in a file are executed in the order they have been defined in the file so update and read queries
can be organized in a way to allow this.
Testing the performance of a query is done by running the query multiple times (preferably after a number of warmup runs) and comparing the average
execution time with the expected query answering time. Expected query time is specified in milliseconds using the expectedTime property:
:SelectAlbums a :Test ;
:expectedTime 100 ;
...
It is expected that the query execution times fluctuate slightly between runs so a slight difference compared to the expected time does not
necessarily indicate a slowdown. For this reason, there is a notion of a "failure threshold" that is take into account before a test is considered
to fail. Threshold is expressed as a percentage point (integer value) and the default failure threshold value is 10. For example, if the
expectedTime for a test is 100ms as in the above example then the test will fail if average query execution is more than 110ms. The percentage can
be set to a value higher than 100 if the query execution time is known to fluctuate highly especially for queries that complete very quickly.
The default warmups are 0 and the default number of runs 1 which is geared towards correctness testing more than performance testing. You should increase these values if you will do performance testing. By default, auto test creator uses 3 warmups and 2 runs but based on query characteristics more warmups or runs might be needed. Running the queries multiple times and observing query execution times is the most reliable way to determine optimal number of warmups and runs.
By default, update queries are executed only once regardless of the warmups and runs setting. It is typically not meaningful to run an update
query multiple times since the first execution applies the changes and subsequent executions have no effect. But in some cases it might make sense
to run update queries multiple queries
Test definitions can be grouped in different files and a test definition file can be included in another one using the include property:
[] :include "album-test.ttl" , "artist-test.ttl" , "path-test.ttl" .
If a test file defines defaults then those defaults will be applied to the included test file and overridden by any defaults defined in the
included files. Included test files will be run in the order they have been defined in the file. A test file can contain both include statements
and test definitions. Again execution ordering will follow the definition order.
The RDF vocabulary used for test definition is defined in the namespace tag:stardog:api:test: but there is no need to declare this namespace in the
test definition file because the test reader utility uses this default namespace while parsing the test definition files.
There is only a single class Test defined in the vocabulary and all the properties except for include are attached to test instances. The special
test defaults is used to define default properties that will apply to all the tests as explained above.
| Property | Value | Description |
| server | string | Stardog server URL for running the test |
| database | string | Database against which the test query will be executed |
| reasoning | string/boolean | Either a boolean value to turn on/off reasoning with default schema, or a reasoning schema name to use |
| queryFile | string | Path to the file that contains the SPARQL query that will be tested |
| queryString | string | SPARQL query string that wil lbe tested |
| queryStored | string | Name of the store query that will be tested |
| resultFile | string | Path to the file that contains the expected results of the query |
| resultCount | integer | Number of results the query is expected to return|
| resultOrdered | boolean | If the results expected results should be checked in the given order |
| expectedTime | integer | Expected query answering time in milliseconds |
| failureThreshold | integer | Percentage points the average query execution may vary from expected time |
| warmups | integer | Number of times query will be executed before computing the average execution time |
| runs | integer | Number of times the query will be executed to compute the average execution time |
| ignore | boolean | If true the test will be ignored and skipped |
Tests can be run with the following command:
$ stardog test run test-file.ttl
The command will print progress information as test are rerunning and print a summary of test results:
+--------------------+------------------------------+-------------------------------------------------------+
| Result | Test | Message |
+--------------------+------------------------------+-------------------------------------------------------+
| PASSED | query1 | |
| FAILED_ERROR | query2 | java.nio.file.NoSuchFileException: query2.sparql |
| FAILED_TIMING | query2 | Query Time Expected : 184 Actual: 226 (Slowdown: 22%) |
| FAILED_CORRECTNESS | query4 | Query Results Expected : 1036 Actual: 1037 |
| PASSED | query5 | |
+--------------------+------------------------------+-------------------------------------------------------+
Finished running 5 tests in 00:00:01.389
PASSED: 2
FAILED_CORRECTNESS: 1
FAILED_ERROR: 1
FAILED_TIMING: 1
As shown in the output above there are four possible outcomes for a test:
PASSED: Test passedFAILED_CORRECTNESS: The results returned by the query did not match expected resultsFAILED_TIMING: The average execution timing exceeded expected query time more than the threshold percentageFAILED_ERROR: An error occurred during the execution of a testThe test run command by default uses the admin/admin credentials to connect to the server specified in the test definition file. A different
username and password can be provided just like any other CLI command:
$ stardog test run -u myuser -p mypass test-file.ttl
The test run command provides additional options, e.g. to print query plans. Please check out the help page for a complete list of options supported.
If you have a set of SPARQL queries saved in files then you can use the stardog test create command to automatically create a test definition using
the following command:
$ stardog test create myDb path/to/queries
The command will recursively traverse the directory, find the query files matching the provided glob expression (by default files with extension
.sparql or .rq), execute each query against the provided database and generate one test definition file for each directory. The name of the
test definition file will be same as the directory name. The test file can be renamed as long as it is not used in an include statement. Test
definition file will contain one test for each query file found. By default, the command runs each query 3 times as a warmup and then runs the query
times to compute the expected time. The results for each query is saved in a file in the same directory as the query file.
Here is another example creating a test file for the SPARQL queries from the Stardog tutorials against the publicly accessible Stardog Express server:
$ stardog test create -u anonymous -p anonymous --only-counts --glob "[0|1]*.sparql" https://express.stardog.cloud:5820/stardog-tutorial-music ../stardog-tutorials/sparql/
@prefix : <tag:stardog:api:test:> .
:defaults :server "https://express.stardog.cloud:5820/" ;
:database "stardog-tutorial-music" ;
:warmups 3 ;
:runs 2 .
:01a-albums a :Test ;
:queryFile "01a-albums.sparql" ;
:resultCount 1037 ;
:expectedTime 98 .
:01b-albums a :Test ;
:queryFile "01b-albums.sparql" ;
:resultCount 1037 ;
:expectedTime 91 .
:02-albums-artists a :Test ;
:queryFile "02-albums-artists.sparql" ;
:resultCount 1039 ;
:expectedTime 95 .
:03-albums-solo-artists a :Test ;
:queryFile "03-albums-solo-artists.sparql" ;
:resultCount 604 ;
:expectedTime 89 .
:04a-albums-dates a :Test ;
:queryFile "04a-albums-dates.sparql" ;
:resultCount 1115 ;
:expectedTime 105 .
:04b-albums-dates a :Test ;
:queryFile "04b-albums-dates.sparql" ;
:resultCount 1115 ;
:expectedTime 102 .
:05-albums-dates-sorted a :Test ;
:queryFile "05-albums-dates-sorted.sparql" ;
:resultCount 1115 ;
:expectedTime 126 .
:06-albums-dates-limited a :Test ;
:queryFile "06-albums-dates-limited.sparql" ;
:resultCount 2 ;
:expectedTime 83 .
:07a-albums-dates-filtered a :Test ;
:queryFile "07a-albums-dates-filtered.sparql" ;
:resultCount 1000 ;
:expectedTime 126 .
:07b-albums-dates-filtered a :Test ;
:queryFile "07b-albums-dates-filtered.sparql" ;
:resultCount 1000 ;
:expectedTime 120 .
:07c-albums-dates-filtered a :Test ;
:queryFile "07c-albums-dates-filtered.sparql" ;
:resultCount 1000 ;
:expectedTime 121 .
:08a-albums-years-duplicates a :Test ;
:queryFile "08a-albums-years-duplicates.sparql" ;
:resultCount 1115 ;
:expectedTime 103 .
:08b-albums-years-distinct a :Test ;
:queryFile "08b-albums-years-distinct.sparql" ;
:resultCount 60 ;
:expectedTime 85 .
:09-albums-dates.minmax a :Test ;
:queryFile "09-albums-dates.minmax.sparql" ;
:resultCount 1 ;
:expectedTime 83 .
:10-albums-count a :Test ;
:queryFile "10-albums-count.sparql" ;
:resultCount 1 ;
:expectedTime 77 .
:11a-albums-dates-grouped a :Test ;
:queryFile "11a-albums-dates-grouped.sparql" ;
:resultCount 60 ;
:expectedTime 86 .
:11b-albums-duplicate-dates a :Test ;
:queryFile "11b-albums-duplicate-dates.sparql" ;
:resultCount 66 ;
:expectedTime 90 .
:12-albums-dates-subselect a :Test ;
:queryFile "12-albums-dates-subselect.sparql" ;
:resultCount 1 ;
:expectedTime 83 .
:13-artists-union a :Test ;
:queryFile "13-artists-union.sparql" ;
:resultCount 308 ;
:expectedTime 84 .
:14a-songs-length a :Test ;
:queryFile "14a-songs-length.sparql" ;
:resultCount 3640 ;
:expectedTime 132 .
:14b-songs-optional-length a :Test ;
:queryFile "14b-songs-optional-length.sparql" ;
:resultCount 3749 ;
:expectedTime 123 .
:14c-songs-unbound-length a :Test ;
:queryFile "14c-songs-unbound-length.sparql" ;
:resultCount 109 ;
:expectedTime 85 .
:14d-songs-no-length a :Test ;
:queryFile "14d-songs-no-length.sparql" ;
:resultCount 109 ;
:expectedTime 87 .
:15a-cowriters a :Test ;
:queryFile "15a-cowriters.sparql" ;
:resultCount 6782 ;
:expectedTime 152 .
:15b-cowriters-sequence-path a :Test ;
:queryFile "15b-cowriters-sequence-path.sparql" ;
:resultCount 6782 ;
:expectedTime 149 .
:15c-cowriters-mccartney a :Test ;
:queryFile "15c-cowriters-mccartney.sparql" ;
:resultCount 9 ;
:expectedTime 77 .
:15d-cowriters-recursive-path a :Test ;
:queryFile "15d-cowriters-recursive-path.sparql" ;
:resultCount 996 ;
:expectedTime 160 .
:15e-songs-optional-path a :Test ;
:queryFile "15e-songs-optional-path.sparql" ;
:resultCount 44 ;
:expectedTime 77 .
:15f-songs-alternative-path a :Test ;
:queryFile "15f-songs-alternative-path.sparql" ;
:resultCount 480 ;
:expectedTime 83 .
:16a-cowriters-paths a :Test ;
:queryFile "16a-cowriters-paths.sparql" ;
:resultCount 40525 ;
:expectedTime 765 .
:16b-cowriters-paths a :Test ;
:queryFile "16b-cowriters-paths.sparql" ;
:resultCount 614 ;
:expectedTime 116 .
:16c-cowriters-paths a :Test ;
:queryFile "16c-cowriters-paths.sparql" ;
:resultCount 609 ;
:expectedTime 182 .
:17-bands-writers-ask a :Test ;
:queryFile "17-bands-writers-ask.sparql" ;
:resultCount 1 ;
:expectedTime 83 .
:18a-beatles-describe a :Test ;
:queryFile "18a-beatles-describe.sparql" ;
:resultCount 7 ;
:expectedTime 81 .
:18b-bands-describe a :Test ;
:queryFile "18b-bands-describe.sparql" ;
:resultCount 75 ;
:expectedTime 97 .
:19a-bands-construct a :Test ;
:queryFile "19a-bands-construct.sparql" ;
:resultCount 240 ;
:expectedTime 82 .
:19b-bands-members-construct a :Test ;
:queryFile "19b-bands-members-construct.sparql" ;
:resultCount 208 ;
:expectedTime 81 .
Note that, we are specifying the server URL as part of the connection string with the database name. We have also changed the credentials used to
connect to the server since the default admin/admin credentials are not allowed for Stardog Express. Update queries are not allowed against Stardog
express. Stardog tutorial contains two update queries (20-bands-members-insert.sparql and 21-songs-length-delete.sparql) so we provided a glob
expression that tells the command to only include SPARQL files that start with the character 0 or 1 effectively excluding the update queries.
Finally, the --only-counts tell the command to not save query results in files and instead use the resultCount property to only record expected
number of results.
If we want to create a test file that only checks correctness and not performance we can use the --no-timings option and run each query only once
without the warmups:
$ stardog test create -u anonymous -p anonymous --warmups 0 --runs 1 --no-timings --glob "[0|1]*.sparql" https://express.stardog.cloud:5820/stardog-tutorial-music ../stardog-tutorials/sparql/
@prefix : <tag:stardog:api:test:> .
:defaults :server "https://express.stardog.cloud:5820/" ;
:database "stardog-tutorial-music" .
:01a-albums a :Test ;
:queryFile "01a-albums.sparql" ;
:resultFile "01a-albums_results.srx" .
:01b-albums a :Test ;
:queryFile "01b-albums.sparql" ;
:resultFile "01b-albums_results.srx" .
:02-albums-artists a :Test ;
:queryFile "02-albums-artists.sparql" ;
:resultFile "02-albums-artists_results.srx" .
:03-albums-solo-artists a :Test ;
:queryFile "03-albums-solo-artists.sparql" ;
:resultFile "03-albums-solo-artists_results.srx" .
:04a-albums-dates a :Test ;
:queryFile "04a-albums-dates.sparql" ;
:resultFile "04a-albums-dates_results.srx" .
:04b-albums-dates a :Test ;
:queryFile "04b-albums-dates.sparql" ;
:resultFile "04b-albums-dates_results.srx" .
:05-albums-dates-sorted a :Test ;
:queryFile "05-albums-dates-sorted.sparql" ;
:resultFile "05-albums-dates-sorted_results.srx" .
:06-albums-dates-limited a :Test ;
:queryFile "06-albums-dates-limited.sparql" ;
:resultFile "06-albums-dates-limited_results.srx" .
:07a-albums-dates-filtered a :Test ;
:queryFile "07a-albums-dates-filtered.sparql" ;
:resultFile "07a-albums-dates-filtered_results.srx" .
:07b-albums-dates-filtered a :Test ;
:queryFile "07b-albums-dates-filtered.sparql" ;
:resultFile "07b-albums-dates-filtered_results.srx" .
:07c-albums-dates-filtered a :Test ;
:queryFile "07c-albums-dates-filtered.sparql" ;
:resultFile "07c-albums-dates-filtered_results.srx" .
:08a-albums-years-duplicates a :Test ;
:queryFile "08a-albums-years-duplicates.sparql" ;
:resultFile "08a-albums-years-duplicates_results.srx" .
:08b-albums-years-distinct a :Test ;
:queryFile "08b-albums-years-distinct.sparql" ;
:resultFile "08b-albums-years-distinct_results.srx" .
:09-albums-dates.minmax a :Test ;
:queryFile "09-albums-dates.minmax.sparql" ;
:resultFile "09-albums-dates.minmax_results.srx" .
:10-albums-count a :Test ;
:queryFile "10-albums-count.sparql" ;
:resultFile "10-albums-count_results.srx" .
:11a-albums-dates-grouped a :Test ;
:queryFile "11a-albums-dates-grouped.sparql" ;
:resultFile "11a-albums-dates-grouped_results.srx" .
:11b-albums-duplicate-dates a :Test ;
:queryFile "11b-albums-duplicate-dates.sparql" ;
:resultFile "11b-albums-duplicate-dates_results.srx" .
:12-albums-dates-subselect a :Test ;
:queryFile "12-albums-dates-subselect.sparql" ;
:resultFile "12-albums-dates-subselect_results.srx" .
:13-artists-union a :Test ;
:queryFile "13-artists-union.sparql" ;
:resultFile "13-artists-union_results.srx" .
:14a-songs-length a :Test ;
:queryFile "14a-songs-length.sparql" ;
:resultFile "14a-songs-length_results.srx" .
:14b-songs-optional-length a :Test ;
:queryFile "14b-songs-optional-length.sparql" ;
:resultFile "14b-songs-optional-length_results.srx" .
:14c-songs-unbound-length a :Test ;
:queryFile "14c-songs-unbound-length.sparql" ;
:resultFile "14c-songs-unbound-length_results.srx" .
:14d-songs-no-length a :Test ;
:queryFile "14d-songs-no-length.sparql" ;
:resultFile "14d-songs-no-length_results.srx" .
:15a-cowriters a :Test ;
:queryFile "15a-cowriters.sparql" ;
:resultFile "15a-cowriters_results.srx" .
:15b-cowriters-sequence-path a :Test ;
:queryFile "15b-cowriters-sequence-path.sparql" ;
:resultFile "15b-cowriters-sequence-path_results.srx" .
:15c-cowriters-mccartney a :Test ;
:queryFile "15c-cowriters-mccartney.sparql" ;
:resultFile "15c-cowriters-mccartney_results.srx" .
:15d-cowriters-recursive-path a :Test ;
:queryFile "15d-cowriters-recursive-path.sparql" ;
:resultFile "15d-cowriters-recursive-path_results.srx" .
:15e-songs-optional-path a :Test ;
:queryFile "15e-songs-optional-path.sparql" ;
:resultFile "15e-songs-optional-path_results.srx" .
:15f-songs-alternative-path a :Test ;
:queryFile "15f-songs-alternative-path.sparql" ;
:resultFile "15f-songs-alternative-path_results.srx" .
:16a-cowriters-paths a :Test ;
:queryFile "16a-cowriters-paths.sparql" ;
:resultFile "16a-cowriters-paths_results.srx" .
:16b-cowriters-paths a :Test ;
:queryFile "16b-cowriters-paths.sparql" ;
:resultFile "16b-cowriters-paths_results.srx" .
:16c-cowriters-paths a :Test ;
:queryFile "16c-cowriters-paths.sparql" ;
:resultFile "16c-cowriters-paths_results.srx" .
:17-bands-writers-ask a :Test ;
:queryFile "17-bands-writers-ask.sparql" ;
:resultFile "17-bands-writers-ask_results.srx" .
:18a-beatles-describe a :Test ;
:queryFile "18a-beatles-describe.sparql" ;
:resultFile "18a-beatles-describe_results.nq" .
:18b-bands-describe a :Test ;
:queryFile "18b-bands-describe.sparql" ;
:resultFile "18b-bands-describe_results.nq" .
:19a-bands-construct a :Test ;
:queryFile "19a-bands-construct.sparql" ;
:resultFile "19a-bands-construct_results.nq" .
:19b-bands-members-construct a :Test ;
:queryFile "19b-bands-members-construct.sparql" ;
:resultFile "19b-bands-members-construct_results.nq" .