MMD
The MMD format is a storage type developed as the successor to the FSS storage. It uses memory-mapped files, which makes it significantly faster than any other storage that GeoDMS offers. An MMD file includes metadata about the value type of each attribute and can optionally auto-declare these attributes in your configuration when reading the file.
Compared to fss, the MMD format consolidates data into one or two files per attribute, instead of producing a separate file for each tile. This avoids the creation of thousands of small files for large datasets and improves both file-system efficiency and loading performance.
Writing an MMD file can work the same way as writing a shapefile: you export a unit to disk. Reading an MMD file follows the same pattern and integrates seamlessly into existing workflows.
unit<uint64> store_mmd := Result
, StorageName = "somestoragename.mmd"
{
attribute<string> OrgName := Result/OrgName;
attribute<string> DestName := Result/DestName;
attribute<s> Traveltime := Result/Traveltime;
attribute<ct> Price := Result/Price;
attribute<ct> Price_augm := Result/Price_augm;
attribute<string> ModeUsed := Result/ModeUsed;
attribute<dam> TravelDist_Bus := Result/TravelDist_Bus;
attribute<dam> Traveldist_Metro := Result/Traveldist_Metro;
attribute<dam> Traveldist_Tram := Result/Traveldist_Tram;
attribute<dam> Traveldist_Rail := Result/Traveldist_Rail;
attribute<dam> Traveldist_Ferry := Result/Traveldist_Ferry;
attribute<s> W_time := Result/W_time, Descr = "Waiting at home";
attribute<s> V_time := Result/V_time;
attribute<s> PT_time := Result/PT_time;
attribute<s> N_time := Result/N_time;
}
unit<uint64> read_mmd
: StorageName = "somestoragename.mmd"
, StorageReadOnly = "true"
{
}
However, an MMD storage can also store multiple tables and attributes with different domains. Domains can be stored in the MMD, as shown in the example above, or be defined elsewhere and reused when reading the MMD data. When domains are stored in the MMD and then read back, they are unrelated to the original domain. When defined elsewhere, the user is responsible for not changing the domain between producing the MMD and using it.
Furthermore, the user is responsible for first producing MMD data before using it.
configuring a write
The item that carries StorageName is the storage holder, and everything below it is part of the store. Two shapes are in use:
- a unit as holder, the shape of the example above: one table, its stored attributes below it;
- a container as holder, when the store is to hold several tables. Each table is a unit below the holder, with its own attributes:
container store
: StorageName = "somestoragename.mmd"
{
unit<uint8> status := src/status
{
attribute<string> Label := src/status/Label;
}
unit<uint32> nummeraanduiding := src/na
{
attribute<string> identificatie := src/na/identificatie;
attribute<string> postcode := src/na/postcode;
}
}
An attribute is written because it has a calculation rule and sits below the holder; it needs no property to say so. Its domain is the nearest unit above it, and it is written with the dotted domain (.) — see Attribute and the dictionary section below. A grouping container may sit between a unit and its attributes.
What follows from “everything below the holder is part of the store”:
- DisableStorage below a holder that is being written is refused, not honoured (see below);
- an
IntegrityCheckbelongs on the holder, not on a stored item below it: on a stored item it suppresses the dictionary and leaves an.mmdthat cannot be read back; - an item that should not be stored is configured outside the holder, and referred to from within it.
Write the store by requesting the holder, for instance GeoDmsRun.exe /Lrun.log config.dms /store. A reading configuration declares the storage and nothing else; see how a reader is expected to declare the storage.
the dictionary
Next to the data files, an MMD folder holds a 0Dictionary.dms describing what was written: the value type of each attribute and the Range of each Unit that is stored in the MMD. A domain that is stored in the MMD can only be read back when its range is recorded there, so this entry is what makes such a domain usable.
The dictionary describes what the write actually produced, so it is written when there is a write. Requesting only part of a store writes only that part: the dictionary still declares the whole configured store, while the folder holds files for the attributes that were requested. A store whose values were all calculated elsewhere in the same run is opened and written like any other.
A store may be written in parts over several runs, one item per run in a batch or item by item in the GUI. Each run keeps what the dictionary said of the units it does not calculate, their range or TiledUnit rule, and of the store-local rules it does not write (see a store-local rule is kept as a rule), so the dictionary keeps describing the files the earlier runs left. A unit keeps the entry of the run that last calculated it; when its configuration has changed since, write its attributes again, so that the files and the entry agree.
The same holds for the sub-items of a container that gets them from its calculation rule, such as a Template instance, the result of For_each or a copy of a function made with instantiate. A run that does not need such a container does not apply its rule, so the container has no sub-items in that run; the dictionary then declares them as the previous dictionary did, with their value types, domains, store-local rules and ranges, and with the check on the extent of a domain outside the store that they use (see what the dictionary records about units defined elsewhere), so a reader still finds the files that the earlier run wrote. Nothing is calculated for this, so a run that writes only another part of the store does not depend on that rule. A run can only keep what the previous dictionary says: when that dictionary declares such a container without its sub-items, request the holder once, so that the whole store is written again. Until the container is written again, its sub-items keep the declaration of the run that last applied its rule; when that rule has changed, write the container again.
The dictionary also carries, for every item below the holder, the properties that say how the item is shown rather than what it computes: Descr, Label, DialogType, and DialogData and cdf when every item they name lies inside the store. So a visualisation style written into a store, a BrushColor attribute of a classification with DialogType = "BrushColor" or a constant parameter<uint32> PenColor on a table, is a style again when the store is read, and a table written with DialogType = "Map", DialogData = "geometry" opens as a map from the reader’s tree. A reader may declare nothing under its holder, so this is the way to give a stored table a style. A DialogData or cdf that names an item outside the store, by an absolute path or by a relative one that leaves it, is left out of the dictionary: merged under the reader’s holder it would name something else or nothing, and a reference to nothing is an error in the map view. The holder’s own properties are not carried: the reader declares its holder itself.
The dictionary writes a categorical unit, such as the result of unique, a subset or a union, with cat_range, so that it reads back categorical. That matters for how the values units of attributes that refer to it are checked.
For a two-dimensional unit the range is written with tagged coordinates:
unit<ipoint> sub2d:
range = "[xy(0; 0), xy(5; 4)) "
An attribute whose domain is the nearest unit above it is written with the dotted domain (.), also when it sits in a grouping container below the holder. On reading, (.) binds to the nearest unit above the attribute (see Attribute), so such a group needs no domain in the writing configuration either:
unit<uint32> store := src, StorageName = "store.mmd"
{
attribute<float64> area := src/area;
container meta // domain store, without spelling it
{
attribute<string> name := src/name;
}
}
A unit declared outside the dictionary keeps its absolute path, as below.
a non-default tiling is recorded as a rule
A Range says how many elements a domain has, not how they were cut into [[tile|tiles]], and the stored files are laid out per tile: an array file holds the tiles one after another, which in a grid decides where each cell lies, and the .seq file of a string or geometry attribute keeps a table with an entry per tile. A reader that is given only a range therefore tiles it the way it tiles any range, with 65,536 elements per tile, which is not the tiling the writer used.
A tiling that a reader cannot arrive at by itself is recorded in the dictionary as a calculation rule instead of a Range:
unit<uint32> vbo:
= TiledUnit(convert(uint32(1000000 ), cat_range(uint32(), uint32(0 ), uint32(9570537 ))))
{
attribute<string> identificatie(.);
}
The rule is printed from the tiling itself, so it is spelled the way the engine writes an expression rather than the way you would type it: uint32() is the Value type unit, cat_range is the categorical range, and convert(v, u) is v[u]. An irregular tiling comes out as the two-argument TiledUnit(lower_bounds, upper_bounds) over union_data lists.
The rule reproduces the range as well, so no Range is written beside it. A domain whose tiling is the one a reader derives from the range, which holds for the default tiling and for a small domain such as a uint8 code list, gets a Range entry, as in any ordinary store.
When the holder of the store is itself such a unit, the reader’s holder gets the rule: unit<uint32> lees : StorageName = "...", StorageReadOnly = "True"; becomes the TiledUnit, and a reference to the holder itself, such as id(lees), const(x, lees) or range(lees, ...), gets that tiling.
what the dictionary records about units defined elsewhere
A stored attribute may refer to a unit that is not stored in the MMD: a values unit such as meter, or a domain that the configuration declares itself. Such a unit is referenced by name, and that name is resolved again, in the reading configuration, against whatever is declared there under that name. When that declaration has changed since the data was written, a reader without a check would bind the new element type to the old bytes: an fpoint values unit that has become dpoint would give garbage, or a crash.
The dictionary therefore records what the bytes were written against, as an IntegrityCheck on the dictionary root:
container store:
IntegrityCheck = "LowerBound(dom) == uint32(0) && UpperBound(dom) == uint32(50) && PropValue(vu, 'ValueType') == 'float32'"
- for an external domain the extent is recorded, when it is known at the moment the dictionary is written; for a sub-item that the dictionary carries over from the previous one (see the dictionary), the extent that the previous dictionary recorded is kept when the run does not know it. When the run cannot find that domain at all, because it lies in a container outside the store whose rule the run did not apply either, the check that the previous dictionary put on it is kept as it was. The same holds for a unit outside the store that the rule of such a sub-item names (see a store-local rule is kept as a rule). The bound literals carry the value type (
uint32(0)), so they pin the type as well and no separate value-type restriction is written for that unit; - the value type is recorded for every other external unit: a values unit, and a domain whose extent could not be established (an unknown range, or 64-bit integral bounds);
- every unit declared outside the dictionary is named by its absolute path, in the restrictions and in the attribute declarations alike. A reader merges the dictionary into a container of its own naming, so a relative spelling would resolve against the reader’s tree rather than the writer’s: a
../RegioUnitcannot be resolved there at all (Unknown identifier '../RegioUnit'), the dots notation...silently lands on whatever sits at that position (LowerBound Error: … arg1 of type TreeItem), and a bare name binds to whatever the reader declares under that name. Units inside the dictionary keep their relative form, which is the only one that survives the merge.
Units stored inside the MMD describe themselves and get no entry. Because the restriction sits on the root of the dictionary, it guards every attribute read through it, so a reading configuration that has changed one of these units fails loudly instead of reinterpreting the stored bytes.
every attribute the dictionary declares has a file of its own
Two items of one store can resolve to the same thing: a whole table that is an alias of another (unit<uint8> status_standplaats := status_ligplaats;, because two object types share a code list), an attribute that repeats another one’s source (unit<uint32> adres := nummeraanduiding;, so adr/postcode and na/postcode are one item), or simply two items carrying the same expression.
Each of them is written to a file of its own, so the store never declares data it does not hold. An attribute whose calculation rule names anything outside the store is materialised. The one exception is a rule whose identifiers all name items that the same store holds: that rule is recorded in the dictionary and the reader re-applies it, so the store holds no bytes for it. See a store-local rule is kept as a rule below.
The same holds for an attribute whose values are not produced as an array: a value from const() or id(), which compute a tile only when it is asked for, or an expression that comes down to one of them, such as union_data(., id(src)) on a holder with the same range as src. Such an attribute is kept as a rule when its rule names only items of the store and units (see below), and gets its file when it depends on data outside the store, as const(x, .) does for a parameter x outside it.
an aliased table is a domain of its own once read back
What a store does not preserve is the relationship between the two. adr and na above come from one unit in the writing configuration, but the store holds two tables and the dictionary declares two independent domains, so combining them elementwise after reading fails:
Domain mismatch between Domain of first argument (/store/adr : uint32)
and Domain of second argument (/store/na : uint32) (different CheckedKeyExpr)
That is by design: the store records what each table contains, not that they were once derived from the same unit. A reading configuration that needs them related must relate them itself, for instance by declaring one domain and reading both sets of attributes into it.
a store-local rule is kept as a rule
An attribute whose rule names only items that are themselves in the store is written to the dictionary as that rule, and gets no file. The reader re-applies it over the attributes it has just read:
container store: StorageName = "store.mmd"
{
unit<uint32> dom := src
{
attribute<int32> own := src/v * 2i; // names src, outside -> stored
attribute<int32> derived := own + 1i; // names own only -> rule, no file
attribute<int32> twice := src/v * 2i + 1i; // names src again -> stored
}
}
and the dictionary then reads:
attribute<int32> own(.);
attribute<int32> derived(.):
= own + 1i;
attribute<int32> twice(.);
derived and twice compute the same values, and the store treats them differently: the test is on the identifiers a rule names, not on what it computes or resolves to. A rule naming data outside the store is stored, even when an equivalent store-local formulation exists, because only the names can be re-resolved in a reader that does not have the writing configuration.
A Unit outside the store is not data in this sense. A unit outside the store that a rule names is treated like an outside unit that is the domain or values unit of a stored attribute: the dictionary writes it by its path, the holder gets an IntegrityCheck on its extent (see what the dictionary records about units defined elsewhere), and declaring it is the reader’s responsibility. With sourcedata/pand outside the store:
attribute<uint32> nr_verdiepingen (sourcedata/pand) := uint32(sourcedata/pand/hoogte * 0.4f); // data: names an outside attribute
attribute<bool> is_hoogbouw (sourcedata/pand) := nr_verdiepingen > 4; // rule
attribute<uint32> c_lit (sourcedata/pand) := const(7, sourcedata/pand); // rule
attribute<sourcedata/pand> i_pand (sourcedata/pand) := id(sourcedata/pand); // rule
attribute<uint32> c_seven (sourcedata/pand) := const(seven, sourcedata/pand); // data: seven is outside
A Parameter outside the store is data like any other attribute, so c_seven gets a file. Define the parameter inside the store instead and it is stored as a single value, after which const(seven, sourcedata/pand) is kept as a rule. A store can therefore name, in its rules and in the holder’s IntegrityCheck, an outside unit that no attribute has as domain or values unit, const(1, None) with None declared outside the store for instance; the reader then has to declare that unit as well, or every item of the store fails on the check.
Two things keep an attribute stored even though its rule is store-local:
- an identifier that names a store item by an absolute path (
/project/store/own). It is store-local, but it would resolve outside the store in a reader, where the store sits elsewhere; -
KeepData in effect on the attribute. The rule is exactly that: if the flag is in effect on the attribute, its values are stored, whatever its rule names.
KeepDatasays this data is worth keeping, and a store is a place where it is kept.KeepDatais not read per item in isolation, and this is the part worth being precise about. It propagates down the tree from wherever it is set, andKeepData = "False"propagates down in the same way, switching it off again for that subtree. “In effect on the attribute” therefore means set on it, or inherited from an ancestor with no"False"in between.Where it came from does not matter, including from above the storage holder. A
KeepDataset on any ancestor of a store reaches every attribute in it, so the whole store is written out in full and no rule is kept as a rule:
container project: KeepData = "True" // set anywhere above the holder ...
{
container store: StorageName = "store.mmd"
{
unit<uint32> dom := src
{
attribute<int32> own := src/v * 2i;
attribute<int32> derived := own + 1i; // ... so this is STORED too, not kept as a rule
}
}
}
To get the rule-only behaviour back under such an ancestor, set KeepData = "False" on the store holder or on the attribute itself; that propagates down from there and switches the flag off for the subtree below it.
“In the store” means held by the store, as data or as a rule, and every identifier of the rule counts, wherever it stands in it. That a name lies below the storage holder is not enough, so three kinds of rule are stored, because a reader cannot re-apply them:
- a rule that names the
org_relof a holder calculated by select_with_org_rel or select_with_attr_by_org_rel, or any other sub-item of the holder’s own calculated result. It sits below the holder, but the dictionary does not declare it, and the reader’s holder has no calculation rule that could produce it; - a name to the right of the arrow
->, which is looked up in the values unit of the left operand: inv := org_rel -> vthevthat is read isSrc/v, outside the store. An arrow into a values unit that the store holds,rel -> xwithxan attribute of a unit in the same store, is kept as a rule; - a name inside
scope(..), which is how the copied columns of select_with_attr_by_org_rel and select_with_attr_by_cond refer to their source:collect_by_org_rel(org_rel, scope(.., Src/v))andcollect_by_cond(., scope(.., Src/sel), scope(.., Src/v)). A holder calculated by either operator therefore stores every copied column.
Keeping a rule as a rule saves disk, not time: an attribute that something has interest in is still calculated during the write, its values are simply not stored.
the same content may be written to more than one store
Two .mmd stores fed from the same source item each get their own copy. Both are written, and so is a store whose attributes are all aliases of items produced elsewhere.
the length of a stored array is checked against the domain
A stored attribute is a keyless positional array: element i of the file is element i of the domain, and nothing in the file itself says which domain that was. The reader therefore compares the length of every stored array with the length its domain requires, and refuses the store when the two differ:
FileTileArray Error: stored array 'C:/.../store.mmd/val' holds 40 bytes, but the domain it is read
into requires exactly 36 bytes (9 elements in 1 tile(s)). The array was written for a different
domain; recreate the storage or restore the domain it was written with.
Without this check, a file holding more elements than the domain would hand out its values as if they belonged to the current domain, so that every value could be attached to the wrong element while all totals stay plausible.
The check is independent of the dictionary restrictions above and does not rely on them, so it also guards a store whose dictionary carries no restriction. When a domain has changed after its store was written, the check refuses the store, which must then be written again.
A stored array holds exactly the bytes of its values, the tiles one after another.
For attributes of variable-length values (strings, sequences) the check covers the index array; the accompanying .seq file has no length that follows from the domain. Its size is the subject of the next section.
the names of the files in a store
The data file of an item is named after its path relative to the holder, with the extension .mmddata, and each unit or container below the holder is a folder: the attribute nummeraanduiding/postcode of the example above is the file nummeraanduiding\postcode.mmddata, and the values of a string or geometry attribute are in a second file with .seq added, nummeraanduiding\postcode.mmddata.seq. The holder itself, when it is an attribute or parameter, has the file @main.mmddata.
No item name contains a dot, so an item can change between attribute and container from one write to the next: the file dom\a.mmddata of the attribute dom/a and the folder dom\a of a container dom/a can be in the store side by side. The file or folder of the earlier form stays, as everything does that a write does not write itself, and is not read.
A store written by a version before 20.29.0 has data files without extension; see files written by older versions.
the size of a string or geometry attribute on disk
An attribute of variable-length values, a string, an arc or a polygon, is stored in two files: the index, with the begin and end of each element, and <attribute>.seq, with the values themselves. The .seq file holds what its elements use and nothing more. When the writing left room unused, which is the usual case, the file is written afresh at the end of the write, with the tiles in their order, so two writes of the same values give the same file, byte for byte.
The index takes 16 bytes per element, against 8 in FSS, so a variable-length attribute in an MMD store is 8 bytes per element larger than in fss. Attributes of fixed-size values (uint32, dpoint, …) take the same space in both.
If the .seq file cannot be cut back at the end of a write, for instance because another program holds it open, the write succeeds with a warning and the file keeps the size it had.
a write that does not finish leaves no data
An attribute gets its file in the store only when its write has finished. While it is written, its files have a temporary name with .~ in it, and the files of the previous version are removed when the new write begins. So a write that does not finish leaves no file for that attribute: an operator that fails halfway, a full disk, a run that is stopped or killed. A reader then fails on that attribute and names the store and the missing file:
MMD storage C:/.../store.mmd has no data for m: the file C:/.../store.mmd/m is missing.
The write of this attribute did not finish, or never started; write the storage again.
This has further consequences:
- The previous values of an attribute are gone as soon as its rewrite begins, also when that rewrite fails. A reader then gets the error above, not the old values under a dictionary that no longer describes them.
- A store holds a file
0Lock. The process that writes a store holds a lock on it until it ends or closes its configuration, and a second process that starts writing the same store fails at once:MMD Error: storage C:/.../store.mmd is being written by another process, which holds C:/.../store.mmd/0Lock. A store is written by one process at a time; try again when that process has finished or closed its configuration.A process that has just been stopped can hold the lock for a moment longer, until Windows has finished ending it. A GeoDMS that only reads the store does not take this lock.
- A file with
.~in its name is what a stopped write left behind. The next write of the store removes it; it is never read. - On Windows, a rewrite waits, with retries, while another program reads the previous version of an attribute. The GeoDMS that writes the store may itself still use the previous version, for a computation it started before the rewrite: it keeps reading the old data until it is done, and the rewrite does not wait for that.
- The dictionary is replaced as a whole when it is updated, so a reader never reads half of one.
the Linux build
The Linux build (.l) writes and reads an MMD store as the Windows build does, and the two write the same files, so a store written on Windows reads correctly on Linux.
how a reader is expected to declare the storage
A reading configuration names the storage and nothing else: the sub-items come from the dictionary. The reading container may be named freely: the dictionary carries the name of the container that wrote the store, and in the usual decoupling pattern the two differ, as the writing project may not even be the same project. That difference is reported as a minor trace naming both, not as a warning, so the event log of a normal run stays clean.
Two shapes are refused rather than half-honoured:
- sub-items declared by the reader under a read-only MMD holder — the dictionary defines those, and a second declaration can disagree with it;
- an IntegrityCheck on the read holder, which would collide with the restrictions the dictionary puts there.
An IntegrityCheck is refused on the writing side too, on any item below a storage holder that is being written: it would suppress the dictionary altogether, leaving an .mmd that cannot be read back. Configure the check on the storage holder instead.
DisableStorage below a storage holder is refused
Everything below the storage holder is part of the store. An item below a holder that is being written and that carries DisableStorage = "True" makes the write fail, naming the item:
[[/store/meta/domain]] DisableStorage is not supported below the MMD storage C:/.../store.mmd that is being written:
every item below its holder /store is part of the store. Move this item outside the holder or drop the property.
A template’s case parameter with DisableStorage = "True" passes the property on to the instantiated parameter, so a configuration that instantiates such a template below an MMD holder is refused as well. Put the instantiation outside the holder, or declare the group’s attributes on the holder’s own domain, (..), which they also take by themselves when they name no domain, and let the template do without a domain parameter.
The refusal concerns what a configuration declares. A unit whose calculation result carries sub-items of its own, such as unit<uint32> sub := select_with_attr_by_org_rel(...), still shows the result’s org_rel below sub; that inherited item is kept out of the store and does not trip the refusal.
an empty attribute reads back
An attribute of a domain with zero elements is stored as a zero-length file, and reads back as an empty array, like any other. A store with an optional sub-table therefore reads back when one instance has no rows, which is a normal outcome and not an error.
files written by older versions
A store keeps the form of the version that wrote it until it is written again, and writing it again with the current version gives it the form described above.
A dictionary written before 20.14.0 carries no restrictions on the units defined elsewhere; the check of the length of each stored array guards such a store all the same. It writes the range of a two-dimensional unit in the untagged {row, col} form, which is read correctly; see XY order. A dictionary written before 20.17.0 names a unit defined elsewhere in whatever spelling the writing configuration used, so a relative spelling resolves against the reader’s tree, as described in what the dictionary records about units defined elsewhere.
A store written before 20.19.3 with an item with DisableStorage below its holder can be inconsistent: the item is missing from the dictionary, and an attribute declared on such a unit is written against a domain that the dictionary does not carry. The case parameter of a template with that property is stored as a domain of its own, with a copy of the argument attribute beside it. Such a store reads back without complaint, but an expression that combines, say, store/meta/UPPER with the store itself fails with a Domain mismatch naming store/meta/domain.
A store over a TiledUnit written before 20.20.0 has no record of its tiling, and the tiling cannot be recovered from the files, so the store cannot be read back. It fails the length check, and what a reader sees depends on how many tiles the writer made:
| tiles written | fixed-size attributes (uint32, dpoint, …) | sequence attributes (string, polygons) |
|---|---|---|
| more than one | FileTileArray Error, a size mismatch | the same |
| exactly one | read correctly | Check Failed Error: orgFreeChunk.second >= chunk.second |
In a table view the columns are added and then fail one by one, leaving the cells empty; in a map view the layer is added, nothing is drawn and the layer is marked failed, which shows on the layer’s coordinate-unit entry rather than on the coordinate unit itself. Such a store must be written again. A version before 20.20.0 writes a store that reads back only when its domain has no TiledUnit: an untiled domain of the same size reads back correctly however the reader tiles it.
In a store written before 20.20.0, items that resolve to the same content, such as an aliased table, an attribute that repeats the source of another one or two items with the same expression, hold that content once, under the name of whichever item claimed it first, while the dictionary declares all of them. An aliased table has no folder at all, and of two stores fed from the same source item only the one written first has the file. Such a store fails on the first use of a missing item:
[E] [[/store/stand/Label]] Data not found in .MMD storage folder
It is incomplete on disk and must be written again. A dictionary written before 20.20.0 also writes an attribute in a grouping container below the holder with the domain (..), the form that is only right at the depth it was written at.
A store written by a version from 20.20.0 and before 20.23.1 can keep an attribute of one of the three kinds listed under a store-local rule is kept as a rule as a rule that a reader cannot re-apply. The store has no data for that attribute, and reading it fails:
[E] [[/Read/v]] Calculation Rule Parser Error: reference 'org_rel' not found (as left operand of the arrow operator)
[E] [[/Read/v]] Unknown identifier 'org_rel'
Such a store must be written again. In those versions, a rule that names the source unit, Src/v[org_rel] instead of org_rel -> v, avoids the problem; in later versions that is not needed, and does no harm. A store written before 20.20.0 holds the data of such an attribute and reads correctly.
A dictionary written before 20.23.1 lacks the properties that say how an item is shown, so a style attribute reads back as a plain uint32 attribute. It also writes range for every unit, so a categorical unit reads back non-categorical.
An array file written before 20.24.0 is sized as if every tile but the last started on a memory page of 4,096 bytes of its own, so it ends in zeros wherever a tile does not fill whole pages: in a grid whose tiles at the right and bottom edges are cut off (3.7 percent on top of the values for a 1000 × 1000 grid), and in a domain with a TiledUnit or an irregular tiling, up to 4 KB per tile. The values lie back to back all the same, and the reader accepts the padded size. A one-dimensional domain with the default tiling has only whole tiles before its last, so its files are the same in both layouts.
A .seq file written before 20.24.0 keeps the room that the writing did not use. The values of an attribute grow in steps while it is written, each step doubling the room, and the file keeps the room of the last step: up to twice the size of the values. In a domain of more than one tile, a tile whose values outgrow their place is moved to a new place in the file, and the old place stays behind with the bytes it held. Which tile moves where depends on the order in which the tiles were written, so two such stores with equal values can differ in size and in content, and can only be compared by reading them. The values are not affected, and the store keeps its size until it is written again.
A store whose write by a version before 20.24.0 failed or was stopped looks complete, because such a version creates the files of an attribute at their full size when its write begins, and writes the dictionary before any data. A reader reads such a store without any message and gets zeros, empty strings or false wherever the write did not come, and a rewrite that failed does the same to a store that was complete. When a run of such a version that wrote a store failed or was stopped, write that store again.
A Linux build of 20.23.1 or earlier cannot write a string or geometry attribute to an MMD store, whatever the tiling of its domain. Every such write fails with
mmap(.../s.seq, offset=10, size=20):
ErrorCode 22: Invalid argument
and leaves an incomplete index and .seq file behind, so a store that such a build tried to write is incomplete and must be written again.
A store written in parts, over several runs, before 20.25.0 has a dictionary that describes only what the last run calculated. A reader fails on the units of the earlier runs with FileTileArray Error: stored array ... requires exactly 0 bytes (4294967294 elements in 0 tile(s)), or on a store-local rule with has no data for .... A store written before 20.25.0 can also lack the file of an attribute whose values were not produced as an array, while its dictionary declares it as data; and it can lack both the rule and the file of a store-local rule whose expression another item of the same run had already computed, such as an attribute waarde := id(.) of a unit with the same range as src, written after an attribute that uses id(src). A reader fails on both kinds of attribute, on the second with has no data for .... Such a store must be written again.
A store written before 20.29.0 names a data file after its item, without extension: the attribute dom/w is the file dom\w, and its values, for a string, dom\w.seq. Such a store is still read. A reader looks for dom\w.mmddata first and takes dom\w when that is not there, with this warning, once per store and session:
Deprecated: MMD storage C:/.../store.mmd was written by GeoDMS 20.28 or earlier, which named a data file after its item without an extension; dom/w is read from C:/.../store.mmd/dom/w instead of C:/.../store.mmd/dom/w.mmddata. Write the storage again to convert it: a write gives each data file the extension .mmddata and removes the file without it. GeoDms 21 will not read data files without the extension. This warning is given once per storage per session.
GeoDMS 21 will not read data files without extension. Writing the store again converts it attribute by attribute: each attribute that is written gets its file with the extension, and its file without, with its .seq, is removed. An attribute that a partial write does not write keeps its old file and is read from there.
One change cannot be written into such a store. When the attribute dom/a has become a container, its old file dom\a is in the way of the folder dom\a that the items of the container need, and the write fails:
MMD Error: the file C:/.../store.mmd/dom/a is in the way of the folder that C:/.../store.mmd/dom/a/b.mmddata needs. It holds the data of an attribute that GeoDMS 20.28 or earlier wrote, and that attribute is a container now; remove that file, or the whole store, and write the storage again.
The dictionary has been replaced by then, so until the store is written again a reader fails on the attributes of the container with the message for a missing file.
reading a store with an older version
A version before 20.14.0 cannot read a dictionary that contains a two-dimensional unit. A version before 20.16.0 does not check the length of a stored array, and reads a file with more elements than its domain without a message. GeoDMS 20.16.0 up to 20.23.1 expect the padding described above, and refuse a store that 20.24.0 or later wrote for a domain whose tiles do not fill whole pages, with the length error; write such a store with the oldest version that has to read it. A version before 20.19.3 fails to read the values of an attribute of a domain with zero elements, on the file mapping, while the count of the domain, which comes from the dictionary, reads normally:
FileMapHandle('<store>.mmd/sub/s') ErrorCode 1006: The volume for a file has been externally altered so that the opened file is no longer valid.
Such a store is correct, and reads back with a later version without being written again.
A version before 20.29.0 looks for the data file of an attribute without extension, and so cannot read a store, or an attribute of a store, that 20.29.0 or later wrote. It fails on each such attribute with a message that takes the store for an unfinished write:
MMD storage C:/.../store.mmd has no data for dom/w: the file C:/.../store.mmd/dom/w is missing. The write of this attribute did not finish, or never started; write the storage again.
To give a store to such a version, write it with that version.
A version before 20.20.0 does not understand a dictionary that carries a calculation rule, and stops at a grouping container whose attributes have the domain (.), reporting Cannot find Domain unit .. The extra files that 20.20.0 and later write for items that resolve to the same content do not hinder an older version: it only finds more files than it would have written itself. A version before 20.23.1 reads the properties that say how an item is shown as it reads any configuration. A version before 20.24.0 reads the .seq files of a later version, since the layout of the files is the same, and neither reads nor writes 0Lock. A Linux build of 20.23.1 or earlier misreads a store written on Windows as soon as a tile starts at a place in its file that is not a multiple of 4,096 bytes: a fixed-size attribute, such as a dpoint of a TiledUnit, fails with the mmap error above, and a string or geometry attribute of a domain of more than one tile reads back wrong values without any error, and some of these reads end the program. A version before 20.25.0 reads the attributes of a store whose holder is a tiled unit, but not a reference to the holder itself: id(lees) and const(x, lees) fail with Check Failed Error: trd, and range(lees, ...) and poly2allgrids over its polygons stop GeoDMS. A tiled unit below a container holder is read correctly.
since
The format was available as early as 2024, but its initial problems have been resolved in version 18.1.2.