Module 2A, Loading different data sources

learning objective: configuring data sources to read non-spatial, vector and grid data in a GeoDMS project

introduction

In this module you will learn how to read the most common data formats into a GeoDMS configuration. All reading uses the same three-property pattern introduced in Module 2: StorageName, StorageType and StorageReadOnly. What changes between formats is the StorageType value and sometimes how the resulting attributes are declared.

the SourceData convention

A widely used convention in GeoDMS projects is to place all data-reading configuration inside a dedicated SourceData container:

container SourceData
{
   unit<uint32> municipality 
   : StorageName = "%ProjDir%/Data/municipality.csv"
   , StorageType = "gdal.vect"
   , StorageReadOnly = "True"
   { 
     ... 
   }

   unit<uint32> provinces    
   : StorageName = "%ProjDir%/Data/provinces.shp"
   , StorageType = "gdal.vect"
   , StorageReadOnly = "True"
   { 
     ... 
   }
}

Keeping source data separate from calculated results makes configurations easier to read and maintain. It also makes it immediately clear which items are inputs and which are derived.

The placeholder %ProjDir% refers to the project directory configured in Module 0. See Folders and Placeholders for the full list of available placeholders.

non-spatial data: CSV

CSV files are the most common non-spatial source in GeoDMS projects. They are read using the gdal.vect StorageManager.

Suppose you have a file municipality.csv with columns municipalityname, municipalitycode and nrinhabitants. The configuration looks like this:

unit<uint32> municipality
:   StorageName     = "%ProjDir%/Data/municipality.csv"
,   StorageType     = "gdal.vect"
,   StorageReadOnly = "True"
{
   attribute<string>  municipalityname;
   attribute<string>  municipalitycode;
   attribute<nr_inh>  nr_inhabitants   := nrinhabitants[nr_inh];
}

A few things to note:

• By default, all columns in the CSV file are read as string. For municipalityname and municipalitycode that is fine.
• For nrinhabitants we want a numeric values unit with a metric. We configure nr_inhabitants explicitly and convert using [nr_inh] (the unit conversion shorthand introduced in Module 1b). The source column name nrinhabitants is used in the expression; the resulting attribute is given the cleaner name nr_inhabitants.
• If you do not declare an attribute explicitly, the GeoDMS still reads all columns from the file as string attributes, accessible under their column names. Explicit declarations let you rename, reinterpret, or convert specific columns.

Other non-spatial formats (.dbf, .xml, .txt) are read in the same way with gdal.vect. For databases such as PostGIS, different StorageManagers apply; see the links at the end of this page.

non-spatial data: Parquet

For larger tables, Apache Parquet is an efficient alternative to CSV. Parquet is a columnar, binary format: the data is compressed and the value type of each column is stored in the file itself. Reading uses the same gdal.vect StorageManager; the .parquet extension is enough for the GeoDMS to select the matching GDAL driver:

unit<uint32> municipality_parquet
:   StorageName     = "%ProjDir%/Data/municipality.parquet"
,   StorageType     = "gdal.vect"
,   StorageReadOnly = "True"
{
}

Two differences from CSV:

• Columns arrive with the value type stored in the file (for example uint32 or float64) instead of as string, so type conversions are rarely needed.
• The unit body can stay empty: all columns from the file are read and become available under their own names. Declare attributes explicitly only if you want to rename or convert them.

Parquet is also the recommended format for exchanging tabular data with Python (pandas, pyarrow). That story is covered in Module 2D, GeoDMS and Python.

spatial data: vector

Spatial data comes in two fundamental types: vector and grid. Vector data stores explicit coordinates in a feature attribute; grid data uses a two-dimensional domain where each cell has an implicit position.

shapefiles and other vector formats

Vector data is often read from ESRI Shapefiles. The configuration is identical to CSV, with the addition of a geometry attribute for the coordinates:

unit<uint32> PC4
:  StorageName     = "%ProjDir%/Data/pc4.shp"
,  StorageType     = "gdal.vect"
,  StorageReadOnly = "True"
{
   attribute<rdc> geometry;
}

The geometry attribute is the feature attribute: it holds the spatial coordinates and tells the GeoDMS GUI that this domain unit can be shown in a map. We strongly recommend always naming this attribute geometry. If you use a different name, you must additionally configure DialogType = "Map" and DialogData = "your_attribute_name" to make maps work correctly.

The values unit of the feature attribute must be a two-dimensional coordinate type. In this example rdc is the Dutch national coordinate system (RD New, EPSG:28992). See How to configure a coordinate system for how to define coordinate system units.

Geometry types

The geometry composition type depends on the shape type in your file:

attribute<rdc> geometry;        // points (default, no composition keyword needed)
attribute<rdc> geometry (arc);  // lines / polylines
attribute<rdc> geometry (poly); // polygons

All other attributes in a shapefile (the .dbf columns) are also read automatically as string. Declare them explicitly if you want to rename or convert them:

unit<uint32> CBS_COROP
:  StorageName     = "%ProjDir%/Data/CBS_COROP_2012.shp"
,  StorageType     = "gdal.vect"
,  StorageReadOnly = "True"
{
   attribute<rdc>    geometry  (poly);
   attribute<string> statnaam;
   attribute<uint32> aant_inw  := aant_inwoners[uint32];
}

For other vector formats (GeoPackage, FileGeoDatabase, PostGIS), the StorageType changes but the pattern is the same. See:

• GeoPackage
• FileGeoDatabase
• PostGIS

spatial data: grid

In grid data, the domain unit is two-dimensional: each element represents a cell identified by its row and column position. There is no explicit geometry attribute; the spatial location of each cell is derived from the projection information stored in the file or a sidecar file.

Most grid data in GeoDMS projects is read from GeoTiff files:

unit<ipoint> land_use_2015
:   StorageName     = "%ProjDir%/Data/bbg2015_100m_10k.tif"
,   StorageType     = "gdal.grid"
,   StorageReadOnly = "True"
,   DialogData      = "rdc_base"
{
   attribute<uint8> GridData;
}

Some differences from vector data:

• The domain unit value type is, for example, ipoint (a two-dimensional integer point), not uint32. The number of cells (rows and columns) is derived from the file itself.
• StorageType is gdal.grid instead of gdal.vect.
• DialogData = "rdc_base" tells the GeoDMS which coordinate system unit to use when interpreting the projection information. rdc_base is the standard RD New base unit. This line is required for the map view to work correctly.
• The GridData attribute refers to the actual cell values in the tiff file. If no domain unit is specified for it, the parent unit (land_use_2015) is used as the domain unit automatically.

If you need to combine multiple grid attributes from different files on the same grid domain, you can define the grid domain explicitly and reuse it for each file. See Grid Domain for details.

try it yourself!

• Download the project here if you have not done so already, and unzip it to C:/prj/GeoDMSAcademy.
• Open exercise.dms (in the GeoDMS_Academy/data_sources/cfg subfolder) in your text editor.

In the data subfolder you will find:

• gemeente.csv: a CSV file with municipality attributes
• NS_stations_2019_RD.shp: point shapefile with train stations
• OSM_Motorways_NL.shp: arc shapefile with motorway segments
• CBS_COROP_2012.shp: polygon shapefile with COROP regions
• bbg2015_100m_10k.tif: a GeoTiff with land-use classification data

Configure all five files in the SourceData container in exercise.dms. Then open the GeoDMS GUI and verify your work:

• Make a table of the CSV data (double-click the domain unit or an attribute).
• Make map views of the three shapefiles (double-click the geometry attribute).
• Make a map view of the grid data (double-click GridData).

Use %ProjDir% in all StorageName paths to keep them portable.

The reference solution is in result.dms in the same cfg subfolder. See the SourceData container there.

---

Go to previous module: Module 2, Loading and storing data sources

Go to next module: Module 2B, Storing different data sources