UpdateAll and cache internals

This page is for developers of the model. It describes how batch\UpdateAll.cmd finds its engine, which items its sections ask for, how it runs the output step and which files it leaves in batch\log. It also lists the caches with their names, and how the model writes and checks the signature of a cache. How to run the model with UpdateAll.cmd is on How to run the model.

Engine

UpdateAll.cmd takes the GeoDmsRun.exe it uses from, in this order: the environment variable GEODMS_EXE, for example another engine or a test build; the local build C:\dev\GeoDMS_2026\bin\Release\x64\GeoDmsRun.exe; and the installed GeoDMS of its GEODMS_VERSION line (20.27.0), in the folder that the installer writes to the registry. It stops when it finds none. Any engine must meet GeoDMS software. For runs of many hours a copy of the build folder (without the .pdb files) is safer than the build folder itself: a rebuild during the run then neither fails on files in use nor replaces DLLs under the running process.

The items of the sections

UpdateAll.cmd asks the model for these items with GeoDmsRun. When to run a section, and when to run it again, is under The sections on How to run the model.

  • sources: SourceData/Infrastructure/OSM/Make_Final_Network and GTFS/LoadFeeds/Write. See Cache names for where the caches are.
  • legs: PublicTransport_Prep/Write_PrePostTransport, in one GeoDmsRun: the parquet caches of the base sets of legs (PrePostTransport/<base set>/Write), Org2Stops_Walking, Org2Stops_Cycling, Stops2Dest_Cycling and, when origins and destinations differ, Stops2Dest_Walking. It writes only the base sets without a cache with the current signature (BaseSets_ToDo), and nothing with Advanced/PrePostTransportDecoupled FALSE.
  • chains: the check ChecksBeforeRunning/PT_CheckBlocks with OVprijzen/PriceCoverage/Table_OK_DOVA (the step stops when the price coverage is not complete), then the stop blocks without a cache, in portions of 20 blocks (batch\WriteChainBlocks.ps1). With the environment variable CHAINCACHE_SINGLE it writes one cache instead, PublicTransport_Prep/ChainGeneration_Single/Write (with Advanced/PT_Chains_PerBlock FALSE).
  • cars: PublicTransport_Prep/Direct/CarPerMoment/<moment>/Write_Car for the named moments. Without moments, the moments the run reads (MakeUnlinkedData/Step4_DirectCar/Moments): the departure moments (TD07h00m …) for the time-dependent car trip on TomTom, and MaxSpeed on OSM; see step 5.
  • output: the output per origin block and departure moment, in portions of blocks, each in its own GeoDmsRun; see The output step.

The output step

The output section runs in five steps:

  1. The model writes the pre- and post-transport caches of the base sets without one, as the legs section does (with all that section has done it already, and -nolegs skips it), so that the portions read them instead of each computing them again.
  2. The model writes the output folder to batch\log\output_dir.txt (NetworkSetup/GenerateOutput/OutputDir_ForBatch) and checks the longest path (NetworkSetup/GenerateOutput/Output_Sig/PathGate).
  3. The script deletes the files that the model writes in that folder: tt_* and signature.* in the folder, the files in PerBlock, Mediaan and Tellingen, and od_tellingen.* and unieke_od_tellingen.*. It does so only when the folder lies under a folder Output, or Output_<O>_<D> for an OD pair.
  4. batch\WriteOutputBlocks.ps1 asks the model for the list of blocks (Generate_Output/OutputBlocks_List, written to batch\log\output_blocks.txt), which also writes or checks the signature of the output folder. It then runs one GeoDmsRun per portion of Advanced/Components/BlocksPerPortion blocks (see Model parameters), so that memory is released between the portions. With MergeOutput each portion merges the components per block (ComponentMerge/PerBlock/<block>/Fence/Generate).
  5. batch\OdTellingen.py adds up the counts per block (see Counts next to the output).

With the environment variable OUTPUT_SINGLE, steps 4 and 5 are one request for all blocks, NetworkSetup/GenerateOutput/Generate_Output. While the file batch\pause_before_output.flag exists, the output section waits before it starts, without a computing process.

Logs and helper files

The portions of the chain and output steps write their console output to UpdateAll_chains_b<nnn>.txt and UpdateAll_output_b<nnn>.txt. batch\ReportUpdateAll.py [step ...] summarises the last run of each step: start, end, duration, number of result rows, the peak CommitCharge and PeakLiveLarge from the memory summary at the end of the log, the size of the cache and the error lines. The UpdateAll_* files are not in git. Some steps first ask the model for a value, which it writes to a text file in batch\log: car_moments.txt (the moments of the run, for cars), chain_blocks_todo.txt (the stop blocks without a cache, for chains), and output_dir.txt, output_blocks.txt and output_blocks_per_portion.txt (for output). They are not in git either.

Cache names

A cache that exists is read, so its name is its cache key: a setting that changes the content must change the name, or the model silently reads the old cache. The caches of a run (the pre- and post-transport, chain and car caches) and each output folder therefore have a signature: the text with all settings that set its content. The source caches are named after their edition instead: the feed date and the OSM extract date with its revision. The name keeps a readable part and, for the caches, a hash of 8 characters over the signature (#118). This keeps the paths within the 259 characters that Windows allows; GeoDMS does not ask for long paths. How the model writes and checks the signatures is on UpdateAll and cache internals. The names below are relative to %LocalDataProjDir%.

Chain cache

  • Name: IntermediateResults/PT_Chains_<date>_<time window>_<hash>: a folder of block caches, or one .mmd with Advanced/PT_Chains_PerBlock FALSE.
  • Signature: PublicTransport_Prep/PT_Chains_Signature, in signature.txt in the cache folder.

Car cache per moment

  • Name: IntermediateResults/DirectCar_<network>_<moment>_<origin set>_<destination set>_<hash>.mmd, for example DirectCar_TT2025-12_TD07h00m_b2025_b2025_3dc99b97.mmd. The name starts with the road network (NetTag: TT and the TomTom edition, TT2025-12 for yr2025_12, or OSM and the extract date): a car cache depends on the edition, not on the reference date, so Y2025 and Y2026 share one. The sets are short (Impl/OrgSet_PathTag, DestSet_PathTag: b2025 for buurt2025); the signature keeps the full names.
  • Signature: PublicTransport_Prep/Direct/CarPerMoment/<moment>/Signature, in signature.txt in the cache folder: network edition, day selection, MaxCarTime, origin and destination sets, speed cap, Pareto tag, the window, start time and search of a time-dependent cache, and revision.

Pre- and post-transport cache per base set

  • Name: IntermediateResults/Legs_<Set>toStops_Walking_<hash>.parquet, Legs_<Set>toStops_Cycling_<hash>.parquet and Legs_Stops2<Set>_Cycling_<hash>.parquet, with <Set> from Impl/OrgSet_NameTag or DestSet_NameTag (for example B2025).
  • Signature: PublicTransport_Prep/PrePostTransport/<base set>/Signature, in <name>.signature.txt next to the cache: type, network and speed, maximum time, origin and destination sets, reference date and feed, numbers of places and stops, format and revision.

GTFS cache per feed

  • Name: GTFS/<GTFS_file_date>.mmd, made from the csv files in %NetworkModel_Dir%/Infrastructuur/GTFS/<GTFS_file_date>/.
  • Signature: none; the feed date is the name.

OSM network cache

  • Name: OSM/Final_Network_<extract date><revision>.mmd, made from the Geofabrik extract in %NetworkModel_Dir%/Infrastructuur/OSM.
  • Signature: none; the extract date and OSM/CacheRevision are the name.

Output

  • Name: Output/<date>_<OutputLabel>_<component>[_fb]/, or <date>_<OutputLabel>_Merged<MergeRunNr>[_fb]/ for a merge. It holds signature.txt, PerBlock/tt_<moment>_<n>of<N>.parquet or .csv, Tellingen/tellingen_<n>of<N>.csv, od_tellingen.csv and unieke_od_tellingen.csv; with Advanced/MakeMedian also Mediaan/tt_<date>_mediaan_<n>of<N>; and from the merge items tt_<date>_<moment>.csv and tt_<date>.csv.
  • Signature: NetworkSetup/GenerateOutput/Output_Signature, in signature.txt in the folder: origin set, PT_Output_ParamTag, exported columns and format, and the signatures of the chain cache and the car cache. For a merge it is the signature of NetworkSetup/ComponentMerge_Def, with the signatures of the merged components.

One OD pair

The model can also compute a single origin-destination pair and keep the route of every option on the front, so that the options can be shown on a map, read as a table and plotted as price against travel time; see Routes of an OD pair. It does so when OD_Origin and OD_Destination are set in ModelParameters.dms, and then puts its caches in IntermediateResults_<O>_<D> and its output in Output_<O>_<D>, for example IntermediateResults_BU04020501_BU0363KP03. The caches and output of the whole model stay as they are. Without such a run these folders do not exist.

How the signature of a cache is checked

The hash (Templates/StringHash_T, in Templates/Signature_T.dms) is a polynomial hash modulo 2^31 - 1 written in plain integer arithmetic in the configuration, because GeoDMS has no string hash; it is stable across versions and platforms, as a cache key must be. So caches for different settings live side by side. The signature of an MMD cache is in its own folder (signature.txt and signature.xml, next to the block caches or next to 0Dictionary.dms), so that IntermediateResults lists only the caches. A pre- and post-transport cache is a single parquet file and keeps its signature next to it, in <name>.signature.txt. Templates/Signature_T handles the signature file:

  • a writer (ChainGeneration_PerBlock/<block>/Write, ChainGeneration_Single/Write, PT_Chains_Blocks/ToDo_List, Write_Car, PrePostTransport/<base set>/Write, the output per block) has <signature>/ForWriting among its ExplicitSuppliers. It writes the signature when the file does not exist yet, stops when the file holds another signature, and stops when the longest file path would exceed ModelParameters/Advanced/MaxPathLength (259), before anything is computed;
  • a reader (ChainGeneration_Single/Read, the block readers PT_Chains_Blocks/PerBlock/<block>/Read, Read_Car, PrePostTransport/<base set>/Read, the readers and merges of the output, FindMedian) builds its storage name from <signature>/ForReading and stops when the signature is missing or differs.

A hash collision therefore gives an error instead of a wrong cache. For the output, another set of parameters under the same OutputLabel gives that error too; choose another label, or let batch\UpdateAll.cmd output empty the folder.

A change in the code is not in the signature by itself. For that the signatures carry revisions: PT_Chains_Revision, CarNetwork_Revision, the revision of the pre- and post-transport caches (_parquet_r2 in their signature) and, for the OSM network cache, OSM/CacheRevision. Raise the revision when a code change alters the content of a cache. The chain cache is read only for the departure moments of PT_DepartureHours and PT_DepartureMinutes, because its time window is in its name.