Module 2C, GeoDMS own data formats

learning objective: understanding the GeoDMS native storage formats (MMD and FSS) and when to use them as intermediate storage in a GeoDMS project

introduction

In Modules 2A and 2B you learned how to read and write data using standard formats like shapefiles and GeoTiff. This module covers a different kind of storage: the GeoDMS native formats, designed specifically for fast intermediate storage between models and model steps.

The GeoDMS currently supports two native formats:

• MMD (Memory Mapped Data): the modern format, available from GeoDMS v18 onwards. Recommended for all new projects.
• FSS (Fast Storage Structure): the older format, still widely used in existing projects.

Both formats are GeoDMS-only. They are currently not intended for delivering data to external users. For that, use shapefiles, GeoTiff, CSV or GeoPackage.

When would you use a native format? A typical use case is a preprocessing pipeline: a preparation script reads raw input files (large shapefiles, XML, CSV), performs complex transformations, and writes the results to MMD or FSS. The main model script then reads from the native format, which is significantly faster than re-reading and re-processing the original files on every model run.

There is also a second use case for storing data in the configuration itself. Small amounts of structured data (such as colour palettes, class labels, or lookup tables) can be stored directly in .dms configuration files using the inline array syntax you saw in Module 1d. This is practical for small, stable reference data that belongs together with the model logic.

MMD

MMD uses memory-mapped files, a technique where the operating system maps file contents directly into the process’s memory address space. This makes reading and writing significantly faster than any other storage format currently available in the GeoDMS.

Advantages of MMD over FSS:

• Much faster reading, especially for large datasets.
• Stores value type metadata inside the file itself.
• Can auto-declare attributes when reading: if you leave the unit body empty, the GeoDMS reads the attribute structure from the file automatically.
• Consolidates data into one or two files per attribute, instead of one file per tile. This avoids the creation of thousands of small files for large tiled datasets, which improves both file system efficiency and loading performance.

MMD is available from GeoDMS v18 onwards. For projects that need to run on older versions, use FSS instead.

writing MMD

Writing MMD follows the same pattern as writing a shapefile (see Module 2B): configure a domain unit with a StorageName ending in .mmd, and list the attributes to export as subitems with expressions:

unit<uint32> store_mmd := Result
,   StorageName = "%LocalDataProjDir%/result.mmd"
{
   attribute<string>  OrgName     := Result/OrgName;
   attribute<string>  DestName    := Result/DestName;
   attribute<s>       Traveltime  := Result/Traveltime;
   attribute<ct>      Price       := Result/Price;
   attribute<string>  ModeUsed    := Result/ModeUsed;
}

Trigger the write by double-clicking the unit in the GeoDMS GUI, or by right-clicking it in the TreeView and choosing Update Subtree (Ctrl+T).

reading MMD

Reading MMD has a unique feature: you can leave the unit body empty, and the GeoDMS will auto-declare all attributes stored in the file based on the metadata inside the .mmd file. Including their value types!

unit<uint32> read_mmd
:   StorageName     = "%LocalDataProjDir%/result.mmd"
,   StorageReadOnly = "True"
{

}

All attributes written to the file become available as subitems of read_mmd automatically. This is especially useful for large files with many attributes, where writing out all declarations by hand would be tedious and fragile.

You can also declare attributes explicitly if you only need a subset, but then use the SyncMode property:

unit<uint32> read_mmd
:   StorageName     = "%LocalDataProjDir%/result.mmd"
,   StorageReadOnly = "True"
,   SyncMode        = "None"
{
   attribute<string> OrgName;
   attribute<s>      Traveltime;
}

FSS

FSS (Fast Storage Structure) is the older GeoDMS native format, still widely used in existing projects. It stores each attribute as a separate binary file inside a folder named after the unit. Sub-containers map to subfolders within that folder.

FSS does not need an explicit StorageType: the .fss extension is enough for the GeoDMS to select the right StorageManager automatically.

reading FSS

unit<uint32> FSS_stations
:   StorageName     = "%ProjDir%/Data/IC_Stations_2019.fss"
,   StorageReadOnly = "True"
{
   attribute<rdc>    geometry;
   attribute<string> label;
}

The configuration looks similar to reading a shapefile, but without StorageType. The value types declared here must match exactly with what was written to the FSS file. A type mismatch causes an error at read time. Use the write configuration as your reference.

You do not need to declare every attribute stored in the FSS file, only the ones you actually use.

writing FSS

unit<uint32> write_stations
:   StorageName = "%LocalDataProjDir%/stations.fss"
{
   attribute<rdc>    geometry := SourceData/stations/geometry;
   attribute<string> label    := SourceData/stations/label;
}

FSS storage must always be configured on a unit, not a container. Configuring a StorageName on a container item will cause an error.

Sub-containers create subfolders inside the FSS folder:

unit<uint32> write_municipalities
:   StorageName = "%LocalDataProjDir%/municipalities.fss"
{
   attribute<rdc>     geometry     := SourceData/municipalities/geometry;
   attribute<string>  name         := SourceData/municipalities/name;

   container indicators
   {
      attribute<float32> pop_density (..) := Results/municipalities/pop_density;
      attribute<float32> mean_income  (..) := Results/municipalities/mean_income;
   }
}

The (..) notation refers to the grandparent (write_municipalities) as the domain unit for the attributes inside the subcontainer.

which format to use?

	MMD	FSS
Available from	GeoDMS v18	All versions
Read speed	Very fast (memory-mapped)	Fast
Auto-declare on read	Yes	No
Files per attribute	1 or 2	1 per tile
Recommended for	New projects	Legacy / older GeoDMS versions

If you are starting a new project on GeoDMS v18 or later, use MMD. If you are working with an existing FSS-based project or need compatibility with older GeoDMS versions, stick with FSS.

strategic decoupling

The preprocessing pipeline from the introduction has a name in GeoDMS practice: strategic decoupling. Instead of one long calculation chain from raw source files to final results, you deliberately cut the chain in two: a preparation part that writes stable intermediates to MMD or FSS once, and a model part that reads those files back and calculates onwards from there.

The word strategic matters. Once written, the file on disk is just a data source: the GeoDMS does not track that it was once derived from your raw inputs. There is no automatic invalidation. If the source data or the preprocessing expressions change, the files keep their old contents until you re-run the write step yourself. Refreshing a decoupled intermediate is a deliberate step in your workflow, not something that happens behind your back.

That makes decoupling a good fit for some parts of a project and a poor fit for others:

• Do decouple stable, expensive preprocessing: source data integration, conversions of large input files, transformations that rarely change. You write them once, and every subsequent run starts from the fast native format.
• Do not decouple the parts you are actively developing. As long as an expression still changes daily, you would have to remember to re-run the write step after every edit, and a forgotten refresh means silently calculating with stale data.

Within a single session you do not need decoupling at all: the GeoDMS tracks dependencies, calculates lazily and never computes the same result twice. Decoupling pays off between runs, and between colleagues who share the same prepared data.

For the reasoning behind this design and how production models organise their write and read containers, see strategic decoupling on the GeoDMS wiki.

try it yourself!

• Download the project here if you have not done so already, and unzip it to C:/prj/GeoDMSAcademy.
• Open exercise.dms (in the GeoDMS_Academy/data_sources/cfg subfolder) in your text editor.

The exercise file already contains the source data configurations from Module 2A. Now do the following:

1. Add a Preprocessing container that writes key attributes from the municipality CSV and the COROP shapefile to MMD (or FSS if you are on GeoDMS v17 or earlier). Include at least the geometry and one numeric attribute for each dataset.
2. Trigger the export in the GeoDMS GUI. Verify the output files appear in your LocalData folder.
3. Add a Model container that reads back the same data from the MMD/FSS files instead of from the original source files. Verify the data is identical.
4. Strategic decoupling in action: add an extra attribute to the Preprocessing container, but do not trigger the export yet. Notice that the Model container keeps reading the old file contents; nothing recalculates automatically. Only after you re-trigger the write (as in step 2) does the new attribute reach the read side. This deliberate refresh is exactly the point of the strategic decoupling section above.

The reference solution is in result.dms in the same cfg subfolder.

---

Go to previous module: Module 2B, Storing different data sources

Go to next module: Module 2D, GeoDMS and Python