Do

syntax

do(item, string)

definition

do(item, string) results in the string Argument, expanded in the context of the item argument exactly as Expand does, but only after the data of item has been calculated. Where expand only reads the placeholders, do first demands item as data.

That makes do a way to state an order between calculations: the result is the string, the work is the first argument.

applies to

any Tree item with data as item: a Data item, a Unit or a Parameter
a string Parameter or Data item as string

since version

14 (2023)

description

Because the string result of one do can be the string argument of the next, do calls nest into a chain:

parameter<string> chain := do(a, do(b, do(c, 'done')));

This is a single calculation with three arguments. When it is calculated, a, b and c are all demanded at once, the leftmost first, and they may run side by side on the worker threads. That differs from ExplicitSuppliers, which updates the listed suppliers one after another, each completely before the next is started (see ExplicitSuppliers). A chain is therefore the way to ask for concurrency between independent calculations, and the leftmost argument is the one that gets started first.

Two conditions apply:

The links must be data items. A container as first argument, such as the result of Parse_xml, only creates its result tree; nothing in it is calculated. Name an attribute of it instead.
The chain must be demanded as data. A string parameter that is only updated, for instance as a GeoDMSRun item, calculates nothing. Give it a storage, StorageName with StorageType = "str", so that its update writes it and the write demands it.

A chain holds every argument until the chain itself is done, so the memory of a chain grows with the number of links. Cut a long series into batches, a chain per batch.

example: parsing file sets concurrently

The BAG import parses the national XML extract in file sets, and writes each file set to its own storage. A storage is written by the ExplicitSuppliers mechanism, and the suppliers of one item are written in sequence: on their own, the file sets are parsed one after the other. A chain over the parsed attributes, named before the storage in each driver, parses them side by side, and the storages are still written by the driver:

// one chain per batch of four file sets, first file set leftmost
unit<uint32> batch := range(uint32, 0u, (#xml/pand/fileset + 3u) / 4u);
Template ChainT
{
	parameter<uint32> b;
	unit<uint32> members := select_with_attr_by_cond(xml/pand/fileset, (xml/pand/fileset/Values - 1u) / 4u == b);
	parameter<string> expr :=
		'do(' + AsList('xml/pand/PerFileSet/fs_' + string(members/Values) + '/ParsedXML/Objecten_pand/Objecten_identificatie', ', do(')
		+ ', ''done''' + AsList(const(')', members), '');
	parameter<string> chain := = expr, StorageName = "= StoreDir + '/pand/chain_b' + string(b) + '.str'", StorageType = "str";
}
container chains := for_each_ne('b_' + string(id(batch)), 'ChainT(' + string(id(batch)) + ')');

// a driver per file set names its batch's chain, and then its storage
Template VerwerkFileSet
{
	parameter<uint32> fileset_rel;
	parameter<string> fileset_expr := 'chains/b_' + string((fileset_rel - 1u) / 4u) + '/chain;xml/pand/PerFileSet/fs_' + string(fileset_rel) + '/pand';
	parameter<string> result       := 'OK', ExplicitSuppliers = "= fileset_expr";
}

The first driver of a batch commits the chain, which parses the four file sets on four threads; the drivers then write the four storages, each finding its parse ready. Measured on the full pand extract of 2026-08-08 (12.2 GB of XML, 50 file sets): 156 s against 310 s without the chain, at a peak of 8.4 GB against 2.4 GB. One chain over all 50 file sets instead of batches parses no faster and holds all 50 results at once, 68 GB.

Two forms that do not work, and why:

A chain over the storages themselves, do(fs_1/pand, do(fs_2/pand, ...)): an item named in a calculation rule is updated in the same sequence as an explicit supplier, and, unlike an explicit supplier, a named storage does not carry its sub-items along, so nothing is written to it. Chain the parsed attributes, and let ExplicitSuppliers write the storages.
A chain over the parse_xml containers: it parses nothing, see the conditions above.