Module 2B, Storing different data sources

learning objective: configuring data export in a GeoDMS project

introduction

Storing results from a GeoDMS model follows the same pattern as reading data: you configure StorageName and StorageType on a tree item. The key differences from loading are:

  • StorageType uses a write variant, such as gdalwrite.vect instead of gdal.vect.
  • The item must have an expression (:=) that defines what to write.
  • The StorageReadOnly property is not set (it defaults to false for write configurations).
  • The data is only written when you trigger the result item in the GeoDMS GUI, for example by double-clicking it or by using Update Subtree (Ctrl+T). Simply having the configuration present does not write anything.

Results are typically written to the %LocalDataProjDir% folder so they do not end up alongside your source data.

exporting a shapefile

Take the COROP polygon data from Module 2A as the source:

// Reading (from Module 2A):
unit<uint32> Corop
:   StorageName     = "%ProjDir%/Data/CBS_COROP_2012.shp"
,   StorageType     = "gdal.vect"
,   StorageReadOnly = "True"
{
   attribute<rdc>    geometry  (poly);
   attribute<string> statnaam;
}

To export a new shapefile with the geometry and region name:

// Writing:
unit<uint32> CoropExport := Corop
,   StorageName = "%LocalDataProjDir%/CBS_COROP_export.shp"
,   StorageType = "gdalwrite.vect"
{
   attribute<rdc>    geometry  (poly) := Corop/geometry;
   attribute<string> label            := Corop/statnaam;
}

What is happening here:

  • unit<uint32> CoropExport := Corop creates a new domain unit with the same number of rows as Corop. The := Corop expression is what ties this unit to the source data and what triggers the export when the item is activated in the GUI.
  • Each attribute listed as a subitem is written to the output file. The geometry attribute with composition (poly) is written to the .shp and .shx files. All non-geometry attributes are written to the .dbf file.
  • StorageType = "gdalwrite.vect" tells the GeoDMS to write rather than read. Using gdal.vect here would cause an error because the StorageReadOnly property would need to be True for a read, which conflicts with having an expression.

Triggering the export: double-click CoropExport in the GeoDMS GUI tree view, or right-click and choose “Update Subtree”. The GeoDMS will compute all expressions, then write the result to the configured file.

If the source data (Corop) is in a different container, for example SourceData/Corop, adjust the references accordingly:

unit<uint32> CoropExport := SourceData/Corop
,   StorageName = "%LocalDataProjDir%/CBS_COROP_export.shp"
,   StorageType = "gdalwrite.vect"
{
   attribute<rdc>    geometry  (poly) := SourceData/Corop/geometry;
   attribute<string> label            := SourceData/Corop/statnaam;
}

other export formats

The same pattern applies to other GDAL write formats. Change the StorageType and file extension accordingly:

Format StorageType Extension
ESRI Shapefile gdalwrite.vect .shp
GeoPackage gdalwrite.vect .gpkg
CSV gdalwrite.vect .csv
Parquet gdalwrite.vect .parquet
GeoTiff gdalwrite.grid .tif

For GeoTiff output, the domain unit must be a two-dimensional grid unit (same as when reading). The grid projection information is written automatically based on the domain unit’s coordinate configuration.

New formats are added with new GeoDMS releases. For example, the Apache Arrow format (.arrow, .feather) was added as an efficient alternative to CSV for large tabular datasets.

exporting a Parquet file

For tabular results, Apache Parquet is an efficient alternative to CSV: the data is compressed and the value type of each column is stored in the file itself, so a reading process gets correctly typed columns without extra metadata files. The write configuration is the same as for a shapefile, with a .parquet extension in the StorageName:

unit<uint32> MunicipalityExport := SourceData/municipality
,   StorageName = "%LocalDataProjDir%/municipality.parquet"
,   StorageType = "gdalwrite.vect"
{
   attribute<string> name           := SourceData/municipality/municipalityname;
   attribute<nr_inh> nr_inhabitants := SourceData/municipality/nr_inhabitants;
}

Attributes of all value types are written, except value types of the point group: a geometry attribute cannot be written to a Parquet file. Use Parquet for tabular data and a spatial format such as GeoPackage for geometries.

Parquet is also the recommended format for handing tabular data over to Python (pandas, pyarrow); see Module 2D, GeoDMS and Python for the full exchange story.

export for reporting

Configured StorageManagers are the right tool for reproducible data deliveries, but for a report you often just need a map as an image and a table in Excel. The GeoDMS GUI has direct tools for this; no configuration is needed.

map view to image

  • Open a map view and click the copy tool in the toolbar: it copies the visible contents of the viewport as an image to the clipboard. Paste it directly into Word or PowerPoint. The layer control (the legend area) is not part of the viewport; copy its contents separately with the layer control copy tool.
  • For print quality output, use the export tool (the save button) in the map view toolbar: it exports the viewport to bitmap file(s) in your %LocalDataProjDir% folder, using the current region of interest. The files are named after the exported item with a row and column suffix, because large or high resolution exports are split into multiple .bmp tiles. Settings such as DPI, paper size and scale can be configured in a container named ExportSettings; without one, defaults are used (600 dpi). Legends and map titles are not included in these bitmaps; compose the final figure in your report software. See Export viewports for all parameters.

table to Excel

  • Open a table view and click the copy tool: it copies the (selected) data as semicolon delimited text to the clipboard, ready to paste into a spreadsheet. The export tool (the save button) saves the whole table as a semicolon delimited .csv file in your %LocalDataProjDir% folder, named after the exported item.
  • Alternatively, right-click a data item or container in the TreeView and choose Export Primary Data. In the dialog you select the folder, the file name and the format (CSV for tables); the resulting file name(s) are shown before you press Export. See Export primary data for a description of this dialog.

try it yourself!

Use the project from Module 2A (GeoDMS_Academy/data_sources/cfg/exercise.dms).

  1. Configure reading the NS_stations_2019_RD point shapefile (already done in Module 2A if you completed that exercise).
  2. Add an Export container to the configuration.
  3. Inside Export, configure writing the station data to a new shapefile in %LocalDataProjDir%. Include at least the geometry and the station name attribute.
  4. Trigger the export in the GeoDMS GUI by double-clicking the export unit. Verify the output file appears in your LocalData folder.
  5. Optional: add a second export unit that writes the station name attribute (without the geometry) to a .parquet file. Trigger it and verify the file appears.
  6. Optional: open a map view of the stations and copy it to the clipboard with the copy tool, then paste it into a document. Do the same for the table with the table copy tool and paste the result into a spreadsheet.

The reference solution is in result.dms in the same cfg subfolder. See the Export container. The optional steps are not part of the reference solution.

Go to previous module: Module 2A, Loading different data sources

Go to next module: Module 2C, GeoDMS own data formats