IntegrityCheck

In modelling, especially when complexity rises, errors are easily made.

The GeoDMS contains ways of assisting modellers in tracking down and solving errors, think for instance of Unit metric consistency.

Another useful feature is the configuration of integrity checks for data items.

An IntegrityCheck is used to check if an (intermediate) result meets certain requirements, for instance that all values need to be within a certain range or that no missing data may occur.

If an IntegrityCheck fails, the data of the resulting items can usually still be requested in a table view, in order to find out what is wrong. The data can not be exported.

A check guards the items below it

An IntegrityCheck is enforced for every item below the item carrying it: requesting a nested item also validates the checks of its ancestors, and a failing ancestor check fails the requested item. A typical example is the version guard shown under Recent version below, which protects every item of the configuration.

What counts as “below” is tree position, not item kind. A Container and a domain unit are the usual carriers, but an Attribute can have sub-items too — a total broken up into its components, or a categorical attribute with further categorisations underneath it — and a check on such an attribute guards those sub-items in exactly the same way.

This also covers items read from a storage, that is, items with a StorageName and no calculation rule. The guard travels with every use of the imported data, so a calculation or table that consumes such an item fails when the check fails, exactly as for calculated items. Requesting the imported item itself validates it in the validation phase, after reading, so that the data of a failed item can still be inspected, as described above.

A check on an item read from a storage may also refer to that item itself, by name or as this, just as a check on a calculated item may:

parameter<string> x : StorageName = "%SourceDataDir%/x.txt", StorageType = "str"
, IntegrityCheck = "strlen(x) > 0";

The stored item is read once, and the check is evaluated on the data read.

A check travels along ExplicitSuppliers

A check also guards the items that declare the checked item, or an item below it, as an [[ExplicitSuppliers ExplicitSupplier]]. Declaring a supplier means “evaluate this first”, and the checks that apply to that supplier (its own, its ancestors’, and, transitively, those of its own ExplicitSuppliers) apply to the declaring item as well: they are woven into its calculation like ancestor checks, and a failing one fails the declaring item and everything computed from it.
parameter<uint32> run_step := ..., IntegrityCheck = "this == 0";
parameter<string> result   : StorageName = "...", ExplicitSuppliers = "run_step";

Here a failing run_step check fails result in every use, even though result’s own definition never mentions run_step. Each guard is deduplicated: an item that also references the supplier in its expression already carries the guard through that reference and is not wrapped again.

The check gates the delivery of the declaring item’s result, not the start of its calculation: the condition and the item’s own expression are evaluated as sibling suppliers.

Two rules to keep in mind when writing such a check:

  • The check may not refer to items inside the subtree it guards: that closes a cycle through the guarded items and is reported as a circular dependency. Let it refer to items outside the guarded subtree, for instance a parameter configured next to it. This is what makes the break-up case awkward: total with sub-items comp1 and comp2 cannot carry IntegrityCheck = "total == total/comp1 + total/comp2", because the components would have to be calculated in order to check them and carry that same check themselves. This is refused whichever item is requested. Moving the check up to the parent does not help, since that parent contains the same items. Express such a cross-item assertion as a separate boolean item that carries the check on itself:

    parameter<bool> total_adds_up := total == total/comp1 + total/comp2
    ,	IntegrityCheck = "total_adds_up == True";
    

    The boolean has no sub-items of its own, and referring to the item that carries the check is the supported self-reference, so nothing reaches into a guarded subtree.

  • The condition is calculated once, not once per guarded item, and each guard is enforced once per calculation, no matter how often the guarded items reference each other. When the check fails, the failure is reported on the item(s) directly under the guard; items computed from those fail because their source failed, and their failure reason names that source. So when tracking down a “Validation (Integrity Check) Failed” error, follow the named source item to the check that actually fired.
  • The checks travel with the calculation rather than beside it: an ancestor’s condition is folded into the calculation of the items below it, so it is scheduled and computed like any other supplier, once, and its verdict reaches every consumer of the guarded item.

If you write an expression in the IntegrityCheck, you can use ‘this’ to refer to itself. This is preferred.

container ReadOpbrengsten_perOP  := 
	for_each_neidv(OP/name
		, 'ReadOpbrengsten_perOP_UNCHECKED/'+OP/name
		, '(all(IsNull(this)))'
		, AdminDomain
		, EUR
	);

examples

No missing data

attribute<city> city_rel (neighborhood) := rlookup(city_code, regions/city/city_code)
, IntegrityCheck = "IsDefined(city_rel)";

The Attribute uses the Rlookup function to find the relation to the regions/city Domain unit.
The configured IntegrityCheck checks if for each neighborhood a city is found.

In such an IntegrityCheck it is allowed to refer to the item for which the IntegrityCheck is configured, the GeoDMS has a workaround to work with such self references.

Recent version

container root: IntegrityCheck = "GeoDmsVersion() >= 7.123"

This IntegrityCheck at the root container of the configuration checks if the configuration is opened in a GeoDMS version 7.123 or later.

This guard protects the whole configuration: every item under the root carries it, so a version older than the one required is refused whatever item is requested. That works only in a version that weaves a check into the items below the checked item. A version older than 20.14.0 evaluates the guard only when the root container itself is requested, which a model run never does, so the guard does not keep it out.

not on an item stored in an MMD

An IntegrityCheck can not be configured on an item that is written to an MMD storage: writing such an item is refused, naming the storage holder to configure the check on instead. See MMD for what the storage records about the units its attributes refer to.

see also

  • an integrity check can also be used in a For_each loop.