MMD

The MMD format is a new storage type developed as the successor to the FSS storage. It uses memory-mapped files, which makes it significantly faster than any other storage currently available. An MMD file includes metadata about the value type of each attribute and can optionally auto-declare these attributes in your configuration when reading the file.

Compared to fss, the MMD format consolidates data into one or two files per attribute, instead of producing a separate file for each tile. This avoids the creation of thousands of small files for large datasets and improves both file-system efficiency and loading performance.

Writing an MMD file can work the same way as writing a shapefile: you export a unit to disk. Reading an MMD file follows the same pattern and integrates seamlessly into existing workflows.

unit<uint64> store_mmd := Result
,	StorageName = "somestoragename.mmd"
{
	attribute<string>                                   OrgName                     := Result/OrgName;
	attribute<string>                                   DestName                    := Result/DestName;
	attribute<s>                                        Traveltime                  := Result/Traveltime;
	attribute<ct>                                       Price                       := Result/Price;
	attribute<ct>                                       Price_augm                  := Result/Price_augm;
	attribute<string>                                   ModeUsed                    := Result/ModeUsed;
	attribute<dam>                                      TravelDist_Bus              := Result/TravelDist_Bus;
	attribute<dam>                                      Traveldist_Metro            := Result/Traveldist_Metro;
	attribute<dam>                                      Traveldist_Tram             := Result/Traveldist_Tram;
	attribute<dam>                                      Traveldist_Rail             := Result/Traveldist_Rail;
	attribute<dam>                                      Traveldist_Ferry            := Result/Traveldist_Ferry;
	attribute<s>                                        W_time                      := Result/W_time, Descr = "Waiting at home";
	attribute<s>                                        V_time                      := Result/V_time;
	attribute<s>                                        PT_time                     := Result/PT_time;
	attribute<s>                                        N_time                      := Result/N_time;
}

unit<uint64> read_mmd
: StorageName = "somestoragename.mmd"
, StorageReadOnly = "true"
{

}

However, an MMD storage can also store multiple tables and attributes with different domains. Domains can be stored in the MMD, as shown in the example above, or be defined elsewhere and reused when reading the MMD data. When domains are stored in the MMD and then read back, they are unrelated to the original domain. When defined elsewhere, the user is responsible for not changing the domain between producing the MMD and using it.

Furthermore, the user is responsible for first producing MMD data before using it.

configuring a write

The item that carries StorageName is the storage holder, and everything below it is part of the store. Two shapes are in use:

• a unit as holder, the shape of the example above: one table, its stored attributes below it;
• a container as holder, when the store is to hold several tables. Each table is a unit below the holder, with its own attributes:

container store
:	StorageName = "somestoragename.mmd"
{
	unit<uint8> status := src/status
	{
		attribute<string> Label := src/status/Label;
	}
	unit<uint32> nummeraanduiding := src/na
	{
		attribute<string> identificatie := src/na/identificatie;
		attribute<string> postcode      := src/na/postcode;
	}
}

An attribute is written because it has a calculation rule and sits below the holder; it needs no property to say so. Its domain is the nearest unit above it, and it is written with the dotted domain (.) — see Attribute and the dictionary section below. A grouping container may sit between a unit and its attributes.

What follows from “everything below the holder is part of the store”:

• DisableStorage below a holder that is being written is refused, not honoured (see below);
• an IntegrityCheck belongs on the holder, not on a stored item below it: on a stored item it suppresses the dictionary and leaves an .mmd that cannot be read back;
• an item that should not be stored is configured outside the holder, and referred to from within it.

Write the store by requesting the holder, for instance GeoDmsRun.exe /Lrun.log config.dms /store. A reading configuration declares the storage and nothing else; see how a reader is expected to declare the storage.

the dictionary

Next to the data files, an MMD folder holds a 0Dictionary.dms describing what was written: the value type of each attribute and the Range of each Unit that is stored in the MMD. A domain that is stored in the MMD can only be read back when its range is recorded there, so this entry is what makes such a domain usable.

The dictionary describes what the write actually produced, so it is written when there is a write. Requesting only part of a store writes only that part: the dictionary still declares the whole configured store, while the folder holds files for the attributes that were requested. Before GeoDMS 20.20.0 a store could fail to appear altogether, with no folder and no 0Dictionary.dms and a run that still exited 0, when every value it declared was already available elsewhere in the same run; such a store is now opened and written like any other.

Since GeoDMS 20.21.0 the dictionary also carries, for every item below the holder, the properties that say how the item is shown rather than what it computes: Descr, Label, DialogType, and DialogData and cdf when every item they name lies inside the store. So a visualisation style written into a store, a BrushColor attribute of a classification with DialogType = "BrushColor" or a constant parameter<uint32> PenColor on a table, is a style again when the store is read, and a table written with DialogType = "Map", DialogData = "geometry" opens as a map from the reader’s tree. Before, such an attribute came back as a plain uint32 attribute and a reader, which may declare nothing under its holder, had no way to give a stored table a style. A DialogData or cdf that names an item outside the store, by an absolute path or by a relative one that leaves it, is left out of the dictionary: merged under the reader’s holder it would name something else or nothing, and a reference to nothing is an error in the map view. The holder’s own properties are not carried: the reader declares its holder itself. A dictionary written by an older version lacks these entries, and an older GeoDMS reads a dictionary that has them as it reads any configuration.

For a two-dimensional unit the range is written with tagged coordinates since GeoDMS 20.14.0:

unit<ipoint> sub2d:
	range = "[xy(0; 0), xy(5; 4)) "

Dictionaries written by older versions use the untagged {row, col} form and keep being read correctly; see XY order. Note that a dictionary written by 20.14.0 or later is not readable by an older GeoDMS if it contains a two-dimensional unit.

An attribute whose domain is the nearest unit above it is written with the dotted domain (.). That was always so for a direct child of the store holder; since GeoDMS 20.20.0 it also holds for an attribute in a grouping container below the holder, which earlier versions wrote as (..), the form that is only right at the depth it was written at. On reading, (.) binds to the nearest unit above the attribute (see Attribute), so such a group needs no domain in the writing configuration either:

unit<uint32> store := src, StorageName = "store.mmd"
{
	attribute<float64> area := src/area;

	container meta                              // domain store, without spelling it
	{
		attribute<string> name := src/name;
	}
}

A unit declared outside the dictionary keeps its absolute path, as below. A dictionary written by GeoDMS 20.20.0 or later that holds such a group is not readable by an older GeoDMS, which stops at the container and reports Cannot find Domain unit ..

a non-default tiling is recorded as a rule

A Range says how many elements a domain has, not how they were cut into [[tile|tiles]], and the stored files are laid out per tile: each tile starts at a new mapping boundary, so the size of a .dat file follows from the tiling and not from the element count alone. A reader that is given only a range therefore tiles it the way it tiles any range - 65,536 elements per tile - which is not the tiling the writer used.

Since GeoDMS 20.20.0 a tiling that a reader cannot arrive at by itself is recorded in the dictionary as a calculation rule instead of a Range:

unit<uint32> vbo:
	 = TiledUnit(convert(uint32(1000000 ), cat_range(uint32(), uint32(0 ), uint32(9570537 ))))
{
	attribute<string> identificatie(.);
}

The rule is printed from the tiling itself, so it is spelled the way the engine writes an expression rather than the way you would type it: uint32() is the Value type unit, cat_range is the categorical range, and convert(v, u) is v[u]. An irregular tiling comes out as the two-argument TiledUnit(lower_bounds, upper_bounds) over union_data lists.

The rule reproduces the range as well, so no Range is written beside it. A domain whose tiling is the one a reader derives from the range - the default tiling, and a small domain such as a uint8 code list - keeps the Range entry it always had, so the ordinary store is unchanged.

Before this, a store over a TiledUnit could not be read back at all. What a reader saw depended on how many tiles the writer had made:

tiles written	fixed-size attributes (uint32, dpoint, …)	sequence attributes (string, polygons)
more than one	FileTileArray Error, a size mismatch	the same
exactly one	read correctly	Check Failed Error: orgFreeChunk.second >= chunk.second

In a table view the columns were added and then failed one by one, leaving the cells empty; in a map view the layer was added, nothing was drawn and the layer was marked failed, which shows on the layer’s coordinate-unit entry rather than on the coordinate unit itself.

A store written before 20.20.0 does not gain the entry by being read: it must be rewritten. The tiling is not recoverable from the files, so the reader has nothing to restore it from and fails as it did before. Until such a store is rewritten, the alternative is to drop the TiledUnit from the writing configuration, since an untiled domain of the same size reads back correctly however the reader tiles it.

what the dictionary records about units defined elsewhere

A stored attribute may refer to a unit that is not stored in the MMD — a values unit such as meter, or a domain that the configuration declares itself. Such a unit is referenced by name, and that name is resolved again, in the reading configuration, against whatever is declared there under that name. When that declaration changed since the data was written, the reader used to bind the new element type to the old bytes silently: the case that produced garbage, or a crash, when an fpoint values unit had become dpoint.

Since GeoDMS 20.14.0 the dictionary therefore records what the bytes were written against, as an IntegrityCheck on the dictionary root:

container store:
	IntegrityCheck = "LowerBound(dom) == uint32(0) && UpperBound(dom) == uint32(50) && PropValue(vu, 'ValueType') == 'float32'"

• for an external domain the extent is recorded, when it is known at the moment the dictionary is written. The bound literals carry the value type (uint32(0)), so they pin the type as well and no separate value-type restriction is written for that unit — since GeoDMS 20.17.0; earlier versions wrote both;
• the value type is recorded for every other external unit: a values unit, and a domain whose extent could not be established (an unknown range, or 64-bit integral bounds);
• every unit declared outside the dictionary is named by its absolute path, in the restrictions and in the attribute declarations alike — since GeoDMS 20.17.0. A reader merges the dictionary into a container of its own naming, so any relative spelling resolves against the reader’s tree rather than the writer’s: a ../RegioUnit cannot be resolved there at all (Unknown identifier '../RegioUnit'), the dots notation ... silently lands on whatever sits at that position (LowerBound Error: … arg1 of type TreeItem), and a bare name binds to whatever the reader declares under that name. Earlier versions wrote whichever spelling the writing configuration happened to use. Units inside the dictionary keep their relative form, which is the only one that survives the merge.

Units stored inside the MMD describe themselves and get no entry. Because the restriction sits on the root of the dictionary, it guards every attribute read through it, so a reading configuration that has changed one of these units fails loudly instead of reinterpreting the stored bytes.

every attribute the dictionary declares has a file of its own

Two items of one store can resolve to the same thing: a whole table that is an alias of another (unit<uint8> status_standplaats := status_ligplaats;, because two object types share a code list), an attribute that repeats another one’s source (unit<uint32> adres := nummeraanduiding;, so adr/postcode and na/postcode are one item), or simply two items carrying the same expression.

Since GeoDMS 20.20.0 each of them is written to a file of its own, so the store never declares data it does not hold. An attribute whose calculation rule names anything outside the store is materialised. The one exception is a rule whose identifiers all name items inside the same store: that rule is recorded in the dictionary and the reader re-applies it, so the store holds no bytes for it. See a store-local rule is kept as a rule below.

Before this, such content was written once, under the name of whichever item claimed it first, while the dictionary went on declaring all of them. Writing exited 0 and reported nothing; the store failed on the first use of a missing item, possibly on another machine and months later:

[E] [[/store/stand/Label]] Data not found in .MMD storage folder

Whole tables were affected as well as single attributes: an aliased table got no folder at all. A store written by an older GeoDMS that contains such an alias is incomplete on disk and must be rewritten; this release fixes the writer, not the stores it has already produced. Stores written by 20.20.0 stay readable by older versions, because they contain nothing an older reader does not understand – only more files than it would have written itself.

an aliased table is a domain of its own once read back

What a store does not preserve is the relationship between the two. adr and na above come from one unit in the writing configuration, but the store holds two tables and the dictionary declares two independent domains, so combining them elementwise after reading fails:

Domain mismatch between Domain of first argument (/store/adr : uint32)
and Domain of second argument (/store/na : uint32) (different CheckedKeyExpr)

That is by design: the store records what each table contains, not that they were once derived from the same unit. A reading configuration that needs them related must relate them itself, for instance by declaring one domain and reading both sets of attributes into it.

a store-local rule is kept as a rule

Since GeoDMS 20.20.0, an attribute whose rule names only items that are themselves in the store is written to the dictionary as that rule, and gets no file. The reader re-applies it over the attributes it has just read:

container store: StorageName = "store.mmd"
{
	unit<uint32> dom := src
	{
		attribute<int32> own     := src/v * 2i;        // names src, outside -> stored
		attribute<int32> derived := own + 1i;          // names own only     -> rule, no file
		attribute<int32> twice   := src/v * 2i + 1i;   // names src again    -> stored
	}
}

and the dictionary then reads:

attribute<int32> own(.);
attribute<int32> derived(.):
	 = own + 1i;
attribute<int32> twice(.);

derived and twice compute the same values, and the store treats them differently: the test is on the identifiers a rule names, not on what it computes or resolves to. A rule naming anything outside the store is stored, even when an equivalent store-local formulation exists, because only the names can be re-resolved in a reader that does not have the writing configuration.

Two things keep an attribute stored even though its rule is store-local:

• an identifier that names a store item by an absolute path (/project/store/own). It is store-local, but it would resolve outside the store in a reader, where the store sits elsewhere;

• KeepData in effect on the attribute. The rule is exactly that: if the flag is in effect on the attribute, its values are stored, whatever its rule names. KeepData says this data is worth keeping, and a store is a place where it is kept.

  KeepData is not read per item in isolation, and this is the part worth being precise about. It propagates down the tree from wherever it is set, and KeepData = "False" propagates down in the same way, switching it off again for that subtree. “In effect on the attribute” therefore means set on it, or inherited from an ancestor with no "False" in between.

  Where it came from does not matter, including from above the storage holder. A KeepData set on any ancestor of a store reaches every attribute in it, so the whole store is written out in full and no rule is kept as a rule:

container project: KeepData = "True"          // set anywhere above the holder ...
{
	container store: StorageName = "store.mmd"
	{
		unit<uint32> dom := src
		{
			attribute<int32> own     := src/v * 2i;
			attribute<int32> derived := own + 1i;   // ... so this is STORED too, not kept as a rule
		}
	}
}

To get the rule-only behaviour back under such an ancestor, set KeepData = "False" on the store holder or on the attribute itself; that propagates down from there and switches the flag off for the subtree below it.

This saves disk, not time: an attribute that something has interest in is still calculated during the write, its values are simply not stored. And it changes what an older GeoDMS can do with the store: a dictionary carrying a calculation rule is not understood by a reader from before 20.20.0.

the same content may be written to more than one store

Two .mmd stores fed from the same source item each get their own copy. Since GeoDMS 20.20.0 both are written; before, whichever store was written first got the file and the second declared an attribute it did not have. A store whose attributes were all aliases of items produced elsewhere was not created at all, with no folder and no dictionary, and the run still exited 0.

the length of a stored array is checked against the domain

A stored attribute is a keyless positional array: element i of the file is element i of the domain, and nothing in the file itself says which domain that was. Since GeoDMS 20.16.0 the reader compares the length of every stored array with the length its domain requires, and refuses the store when the two differ:

FileTileArray Error: stored array 'C:/.../store.mmd/val' holds 40 bytes, but the domain it is read
into requires exactly 36 bytes (9 elements in 1 tile(s)). The array was written for a different
domain; recreate the storage or restore the domain it was written with.

Before that, a file holding more elements than the domain was accepted silently: the surplus was dropped and the remaining values were handed out as if they belonged to the current domain. Every value could then be attached to the wrong element while all totals stayed plausible, so nothing downstream looked wrong. A file holding fewer elements did fail, but as a Windows error from the memory mapping (CreateFileMapping ... ErrorCode 5: Access is denied) that named neither the domain nor the sizes.

For attributes of variable-length values (strings, sequences) the check covers the index array; the accompanying .seq file has no length that follows from the domain.

This check is independent of the dictionary restrictions above and does not rely on them, so it also guards stores whose dictionary carries no restriction — every .mmd written before 20.14.0. Expect it to fire on the first run after upgrading when a domain was changed after its .mmd was written: such a store must be rewritten.

The same check is what a store over a TiledUnit written before 20.20.0 trips on, for a different reason: not a changed domain but a tiling the dictionary never recorded. See a non-default tiling is recorded as a rule above.

how a reader is expected to declare the storage

A reading configuration names the storage and nothing else: the sub-items come from the dictionary. The reading container may be named freely: the dictionary carries the name of the container that wrote the store, and in the usual decoupling pattern the two differ — the writing project may not even be the same project. Since GeoDMS 20.17.0 that difference is no longer reported as a warning; it is a minor trace naming both, so the event log of a normal run stays clean.

Two shapes are refused rather than half-honoured:

• sub-items declared by the reader under a read-only MMD holder — the dictionary defines those, and a second declaration can disagree with it;
• an IntegrityCheck on the read holder, which would collide with the restrictions the dictionary puts there.

An IntegrityCheck is refused on the writing side too, on any item below a storage holder that is being written: it would suppress the dictionary altogether, leaving an .mmd that cannot be read back. Configure the check on the storage holder instead.

DisableStorage below a storage holder is refused

Everything below the storage holder is part of the store. Since GeoDMS 20.19.3 an item below a holder that is being written and that carries DisableStorage = "True" makes the write fail, naming the item:

[[/store/meta/domain]] DisableStorage is not supported below the MMD storage C:/.../store.mmd that is being written:
every item below its holder /store is part of the store. Move this item outside the holder or drop the property.

Before that, such an item was silently left out of the dictionary, and any attribute declared on a DisableStorage’d unit was written against a domain the dictionary did not carry. A template’s case parameter with DisableStorage = "True" fared worse: the property was lost on instantiation, so the parameter unit was written as a stored domain of its own, with a copy of the argument attribute beside it. The store read back without complaint; only an expression combining, say, store/meta/UPPER with the store itself failed, with a Domain mismatch naming store/meta/domain. Expect the refusal to fire on the first write after upgrading for a configuration that instantiates such a template below an MMD holder: put the instantiation outside the holder, or declare the group’s attributes on the holder’s own domain, (..), which since GeoDMS 20.20.0 they take by themselves when they name no domain, and let the template do without a domain parameter.

The refusal concerns what a configuration declares. A unit whose calculation result carries sub-items of its own, such as unit<uint32> sub := select_with_attr_by_org_rel(...), still shows the result’s org_rel below sub; that inherited item is kept out of the store as before and does not trip the refusal.

an empty attribute reads back

An attribute of a domain with zero elements is stored as a zero-length file. Since GeoDMS 20.19.3 it reads back as an empty array, like any other. Before that, reading its values failed on the file mapping:

FileMapHandle('<store>.mmd/sub/s') ErrorCode 1006: The volume for a file has been externally altered so that the opened file is no longer valid.

while the count of the domain, which comes from the dictionary, read normally. Any store with an optional sub-table hit this as soon as one instance had no rows, which is a normal outcome and not an error. The stores themselves were written correctly; a store written by an older version reads back on 20.19.3 without being rewritten.

since

The format was available as early as 2024, but its initial problems have been resolved in version 18.1.2.