Parquet
Apache Parquet is a columnar, binary file format for tabular data. Unlike Csv, it stores the Value type of each column in the file itself and compresses the data, which makes it both faster to read/write and smaller on disk.
Parquet (and the related Arrow format) is the recommended format for exchanging tabular data with Python (e.g. with pandas / pyarrow): no separate .csvt is needed to communicate value types and the columnar layout is read efficiently on both sides.
The GeoDMS reads and writes Parquet through the GDAL library, using the same Gdal.vect / Gdalwrite.vect StorageManagers as for Csv.
Read
The following example shows how to read a .parquet file with the Gdal.vect StorageManager.
unit<uint32> woningvoorraad
: StorageName = "%LocalDataProjDir%/Python/temp/woningvoorraad_startgebouwopties.parquet"
, StorageType = "gdal.vect"
, StorageReadOnly = "True"
{
}
All attributes from the Parquet file are read. In contrast to Csv, the Value type of each attribute is taken from the file, so in most cases no Conversion functions are needed.
If the extension is not .parquet, the driver can be set explicitly, see GDAL:
parameter<string> GDAL_Driver : ['Parquet'];
reading data produced elsewhere back into an existing domain
When a Parquet file is produced by an external process (for instance a Python script, see below), reading it back creates a new Domain unit. This new domain does not automatically match an already configured domain, even when it has the same number of rows. There are several ways to relate the imported data to an existing domain results:
- If the rows correspond one-to-one and in the same order as
results, give the read domain a calculation rule referring to the existing domain:
unit<uint32> startgebouwopties_python := results
, StorageName = "%LocalDataProjDir%/Python/temp/woningvoorraad_startgebouwopties.parquet"
, StorageType = "gdal.vect"
, StorageReadOnly = "True";
- Alternatively, configure the attributes explicitly against the existing domain:
container startgebouwopties_python
: StorageName = "%LocalDataProjDir%/Python/temp/woningvoorraad_startgebouwopties.parquet"
, StorageType = "gdal.vect"
, StorageReadOnly = "True"
{
attribute<Classifications/GebouwOptie> GebouwOptie (results); // read against the existing domain results
}
- Or copy the values one-to-one with Union_data (an error is raised on too few or too many rows):
attribute<Classifications/GebouwOptie> GebouwOptie_rel (results) :=
union_data(results, startgebouwopties_python/GebouwOptie);
- If the relation is not one-to-one, relate both domains on an external key with Rlookup and read the values with Lookup. This is the safest route when selections or row orders may diverge (e.g. for editable stam-tables to which rows can be added):
attribute<startgebouwopties_python> py_rel (results) := rlookup(results/externe_key, startgebouwopties_python/externe_key);
attribute<Classifications/GebouwOptie> GebouwOptie_rel (results) := py_rel -> GebouwOptie;
GeoParquet written by geopandas or pyarrow
A GeoParquet file whose geometry column carries the geoarrow.wkb extension type, as files written by geopandas with pyarrow do, is read as fast as one written by GDAL. For such a column the Parquet driver of GDAL would determine the geometry type of the layer by reading every geometry when the file is opened, and the GeoDMS opens a file more than once during a read, which would make such a read take about twice as long. The GeoDMS therefore sets the GDAL configuration option OGR_PARQUET_COMPUTE_GEOMETRY_TYPE=NO on every read with Gdal.vect, and GDAL then reports the geometry type of such a layer as unknown. Files written by GDAL, such as those of Gdalwrite.vect and ogr2ogr, are not affected: GDAL takes their type from the GeoParquet metadata.
For a file with the geoarrow.wkb column this means:
- a geometry that is not configured gets its Composition from the first feature that has a geometry;
- a configured geometry is not compared with the geometry type in the file, so a configured composition that does not match the file is not reported when the file is opened;
- the attributes geometry_z and geometry_m are not added for coordinates with Z or M values.
Set the option to YES yourself to have GDAL determine the geometry type by reading every geometry when the file is opened, at the cost of the slower read:
container Percelen
: StorageName = "%SourceDataDir%/BRP/brp_2025.parquet"
, StorageType = "gdal.vect"
, StorageReadOnly = "True"
{
parameter<string> GDAL_ConfigurationOptions := 'OGR_PARQUET_COMPUTE_GEOMETRY_TYPE=YES';
unit<uint32> brp_2025
{
attribute<rdc_meter> geometry (poly);
}
}
A value for this option set as an environment variable is also respected.
Write
The following example shows how to write a .parquet file with the Gdalwrite.vect StorageManager.
unit<uint32> input_export := src/woningen
, StorageName = "%LocalDataProjDir%/Python/temp/input_startgebouwopties.parquet"
, StorageType = "gdalwrite.vect"
, StorageReadOnly = "False"
{
attribute<uint32> pand_id := src/woningen/pand_id;
attribute<float32> oppervlakte := src/woningen/oppervlakte;
}
Attributes of all value types are written. The value types are stored in the file, so a reading process gets correctly typed columns without an accompanying .csvt.
One attribute with a value type of the point group, a point, arc or polygon geometry, becomes the geometry column of the file, which GDAL writes as GeoParquet: the geometries as WKB, the coordinate system in the metadata of the file, and an extra column geometry_bbox with the bounding box of each row. A unit with a second such attribute is not written; gdalwrite.vect then fails with multiple geometry fields are unsupported in GeoDMS.
use for data exchange with Python
A typical pattern is to let the GeoDMS write a Parquet input file, run a Python script on it, and read the Parquet output back. Running the script and reading its output are ordered automatically by using the ExitCode of Exec_ec in the StorageName of the output; writing the input, however, has to be a GeoDmsRun call of its own, because exec_ec runs at meta-information time and a GeoDMS write happens later. See Exec_ec for the mechanism and a worked end-to-end example.
The same applies to R: see R for applying a model saved with saveRDS to a GeoDMS data matrix, and to Julia: see Julia.
Arrow
Arrow is Parquet’s sibling: the same GDAL drivers write and read it, it carries the Value type of each column just as Parquet does, and it is uncompressed, which makes it the faster of the two for a file that is written and read again immediately. Set the driver explicitly when reading, since the .arrow extension is not recognised by every GDAL build:
parameter<string> GDAL_DRIVER := 'Arrow';
Prefer Arrow when the files are also read by Julia (Arrow.Table) or R (arrow::read_ipc_file); prefer Parquet when they are archived or shipped, where the compression matters. See Julia for a pipeline built on Arrow.