rrt/docs/README.md

410 lines
27 KiB
Markdown

# RT3 Reverse-Engineering Handbook
This handbook is the project bootstrap for reverse-engineering and rewriting Railroad Tycoon 3.
It is written for future us first: enough structure to resume work quickly, without pretending the
project is already mature.
## Canonical Target
- Canonical executable: `rt3_wineprefix/drive_c/rt3/RT3.exe` (patch 1.06)
- Reference executable: `rt3_wineprefix/drive_c/rt3_105/RT3.exe` (patch 1.05)
- Canonical SHA-256: `01b0d2496cddefd80e7e8678930e00b13eb8607dd4960096f527564f02af36d4`
- Reference SHA-256: `9e96b0695cb722a700f99c8dce498d34da7235e562b1e275bcc1764f8c9b7eb1`
## Documents
- Stable docs
- `setup-workstation.md`: toolchain baseline and local environment setup.
- `re-workflow.md`: how to analyze the binary, record findings, and export reusable artifacts.
- `function-map.md`: canonical schema and conventions for function-by-function mapping.
- `control-loop-atlas.md`: compatibility index for the split atlas, preserving legacy anchors.
- `control-loop-atlas/`: canonical section files for the atlas narrative.
- `subsystem-views/`: curated cross-cut subsystem views over the atlas.
- `runtime-rehost-plan.md`: bottom-up runtime replacement plan and milestone breakdown.
- Active queue and history
- `rehost-queue.md`: active implementation and research queue.
- `rehost-queue/`: archived queue snapshots and related preserved worklog material.
- `history/progress-history.md`: preserved high-detail milestone narrative from the former root README.
- Artifact indexes
- `../artifacts/exports/rt3-1.06/README.md`: canonical 1.06 export manifest.
- `../artifacts/captures/README.md`: committed capture inventory and policy.
## Repo Conventions
- `docs/`: stable project guidance and durable design notes.
- `tools/py/`: committed Python helpers for analysis and validation.
- `artifacts/exports/`: committed derived outputs that can be regenerated.
- `artifacts/captures/`: committed logs, sample binaries, and retained capture evidence.
- `artifacts/tmp/`: scratch-only local working area; do not commit new files here.
- Local-only state stays untracked: `.venv/`, Ghidra projects, Rizin databases, crash dumps, and other
bulky/generated working files.
## Current Baseline
The current technical milestone is a repeatable loop-mapping workflow for the 1.06 executable.
Before injection work or deep file-format work, we capture:
- executable hashes and PE metadata
- section layout, imports, and notable strings
- a starter subsystem inventory plus a control-loop atlas
- focused address and string context exports for branch-deepening passes
- a reusable CLI RE kit for branch dossiers where the atlas needs deeper grounding
- a stable curated function ledger in `artifacts/exports/rt3-1.06/function-map.csv`
Current coverage is broad enough to support future sessions without rediscovery, especially in:
- CRT startup and bootstrap handoff
- shell frame, layout, presentation, deferred-message, and frontend overlay flow
- Multiplayer.win UI, chat, session-event, and transport ownership
- map/scenario load and text-export paths
- shared support layers such as intrusive queues, vectors, hashed stores, and tracked heaps
README maintenance rule:
- Keep this section at subsystem level only.
- Do not mirror per-pass function additions here.
- Detailed mapping progress belongs in `artifacts/exports/rt3-1.06/function-map.csv` and the derived branch artifacts under `artifacts/exports/rt3-1.06/`.
Current local tool status:
- Ghidra is installed at `~/software/ghidra`
- `~/software/ghidra/ghidraRun` launches successfully in an interactive shell
- Rizin is installed and available on `PATH`
- `winedbg` works with `rt3_wineprefix`
- RT3 launches under `/opt/wine-stable/bin/wine` when started from `rt3_wineprefix/drive_c/rt3`
## Next Focus
The atlas milestone is broad enough that the next implementation focus has already shifted downward
into runtime rehosting. The current runtime baseline now includes deterministic stepping, periodic
trigger dispatch, normalized runtime effects, staged event-record mutation, fixture execution,
state-diff tooling, tracked save-slice documents for captured-runtime inputs, overlay import
documents that combine captured snapshots with save-derived state, and a packed-event persistence
bridge that now reaches per-record summaries and selective executable import.
The highest-value next passes are now:
- preserve the atlas and function map as the source of subsystem boundaries while continuing to
avoid shell-first implementation bets
- keep using overlay imports as the context bridge when selectively executable packed rows still
need runtime context that current save slices and raw save inspection do not yet persist
- treat broader real grouped-descriptor recovery as the active packed-event frontier now that the
first company-scoped batch already parses, summarizes, and executes through the ordinary runtime
path when overlay context resolves its symbolic company scope: descriptor `2` `Company Cash`,
descriptor `13` `Deactivate Company`, and descriptor `16` `Company Track Pieces Buildable`
- descriptors `1` `Player Cash` and `14` `Deactivate Player` now join that executable real batch
through the same ordinary runtime path, backed by the minimal player runtime and overlay-import
context
- the first chairman-targeted real grouped rows now execute too through that same path when the
hidden grouped target-subject lane resolves to grounded chairman scope ordinals `0..3`:
`condition_true_chairman`, `selected_chairman`, `human_chairmen`, and `ai_chairmen`; wider
chairman ordinals stay parity-only under `blocked_chairman_target_scope`
- chairman runtime ownership is broader now too: selected-chairman condition rows for chairman
cash, holdings value, net worth, and purchasing power import through the same service path, and
the first grounded company governance issue batch now executes too via book-value-per-share,
investor-confidence, and management-attitude thresholds; wider chairman target ordinals remain
frontier
- checked-in save-slice documents can now carry explicit company rosters and chairman profile
tables too, so the current company-targeted and chairman-targeted descriptor/condition batches
can execute from standalone save-slice fixtures without overlay snapshots when that context is
present; raw `.gms` inspection/export now reconstructs those company/chairman collections
directly from save-side tagged record families, while the fixed save-side `0x32c8` world block
still provides selected company/chairman ids plus the grounded campaign-override, issue-`0x37`
value/multiplier, and chairman-slot / role-gate analysis bytes; the current raw-save frontier is narrower now:
company/chairman identity, active flags, links, chairman cash, chairman holdings, chairman
purchasing power, company debt, company track-laying capacity, and collection counts are
grounded, while broader company
finance/governance scalar lanes plus controller-kind reconstruction still remain conservative
defaults until their raw offsets are pinned more strongly; the offline analysis command
`runtime inspect-save-company-chairman <save.gms>` now dumps those remaining raw record
candidates directly from the rehosted parser, including fixed-world chairman slot / role-gate
context, the grounded fixed-world issue-`0x37` pair, the fixed-dword world finance
neighborhood rooted at `[world+0x0d]` with the saved calendar tuple and absolute counter, the separate six-float economic
tuning band, derived holdings-at-share-price and cached purchasing-power totals,
context, company dword candidate windows, and richer chairman qword cache views; the current
rehosted company-side owner state now also includes a typed market/cache map carrying saved
outstanding-shares, support/share-price/cache words, salary lanes, calendar words, connection
latches, and the first grounded stat-band root windows at `[company+0x0cfb]`, `[company+0x0d7f]`,
and `[company+0x1c47]` for each live company, so later finance/stat-family readers can attach to
owned runtime data instead of one more guessed save offset; the first runtime-side `0x2329`
stat-family reader seam is now also rehosted for slots `0x0d` and `0x1d`, and the saved
stat-band windows themselves now carry 32 dwords per root; the matching world-side issue reader
seam is now rehosted for the grounded `0x37` lane, and the adjacent raw issue-byte strip
`0x37..0x3a` now also flows through save-slice/runtime restore state for later credit / prime
/ management readers; selected-company summaries now expose the
unassigned share pool derived from outstanding shares minus chairman-held shares for later annual
finance logic; that same owned company market state now also backs a bundled annual-finance
reader seam for assigned shares, public float, and rounded cached share price; the fixed-world
finance neighborhood is now widened to 17 dwords rooted at `[world+0x0d]` so later issue-family
readers can target a broader owned restore-state window, and the saved absolute counter now
flows through normal runtime restore state instead of staying on shell-context metadata; that
same world owner surface now also carries the packed year word and partial-year progress lane
behind annual-finance recent-history weighting; the same annual-finance state now also
feeds a shared company market reader for stock-capital, salary, bonus, and the full two-word
current/prior issue-calendar tuples, and now derives elapsed years since founding, last dividend,
and last bankruptcy for later annual finance-policy rehosting; live bond-slot count now travels through that same owned annual-finance
state for the stock-capital branch gate, and the grounded bond table now also contributes both
the largest live bond principal and the chosen highest-coupon live bond principal into that same
owner-state surface, plus the highest live coupon rate for the stock-capital approval ladder;
the same fixed-world save block now also carries the raw stock, bond,
bankruptcy, and dividend finance-policy bytes, and the first annual creditor-pressure branch now
executes as a pure runtime reader over that owner state instead of remaining atlas-only; the
later deep-distress bankruptcy fallback now runs on that same save-native cash and trailing-
profit seam; the annual bond, stock-repurchase, and stock-capital issue branches now do too,
and periodic boundary service now also commits the bond-issue branch through that owned
stat-post and live-bond seam instead of leaving it as a reader-only finance lane; the rehosted
company market reader now carries the first bundled annual net-profit surface too; the annual
dividend-adjustment branch now does as well through the
shared year-or-control-transfer reader and board-approved dividend ceiling helper; the same
owner seam now also carries the fixed-world building-density
growth setting plus the linked chairman personality byte, which is enough to run the annual
stock-repurchase gate headlessly as another pure reader; periodic boundary service now also
chooses one annual-finance action per active company and already commits the shellless
creditor-pressure-bankruptcy, deep-distress-bankruptcy, dividend-adjustment, stock-repurchase,
stock-issue, and annual bond-restructure branches against owned runtime state, with bankruptcy now following
the grounded “halve live bond debt and stamp the year” path rather than a liquidation shortcut;
the same live bond-slot owner
surface now also carries save-native maturity years into annual bond policy summaries, derives the current live coupon burden
directly from owned bond slots, and now also commits the shellless “repay matured live bonds,
compact the table, then issue the exact staged count” path during periodic service; the same
service surface now also carries the per-cycle retired-versus-issued principal totals that feed
the later debt-news family plus the issued-share and repurchased-share counts behind the later
equity-offering and buyback news tails; runtime summaries now also expose the grounded
retired-versus-issued relation directly, and annual finance service now maps that same relation
onto the exact debt headline selectors `2882..2886`
while persisting the last emitted annual-finance news events as structured runtime-owned records
carrying company id, selector label, action label, and the grounded debt/share payload totals
- shellless calendar advance now also drives that annual seam directly: `StepCount` and `AdvanceTo`
invoke periodic-boundary service automatically on year rollover instead of requiring a second
manual service pass to make the annual finance stack run
- that stepped world-time path now also refreshes the derived selected-year gap scalar owner lane
`[world+0x4ca2]`, so later selected-year and periodic-boundary world work can build on runtime
state instead of a frozen load-time scalar
- that same save-native world restore surface now also carries the grounded locomotive-policy bytes
and cached available-locomotive rating from the fixed world block, so the `All
Steam/Diesel/Electric Locos Avail.` descriptor strip now writes through owner state instead of
living only as mirrored world flags
- the selected-year seam now follows the same rule: the checked-in `0x00433bd0` year ladder now
drives a derived selected-year bucket scalar in runtime restore state, and the economic-tuning
mirror `[world+0x0bde]` now rebuilds from tuning lane `0` instead of freezing one stale
load-time word
- that same selected-year owner family now also rebuilds the direct bucket trio
`[world+0x65/+0x69/+0x6d]`, the complement trio `[world+0x71/+0x75/+0x79]`, and the scaled
companion trio `[world+0x7d/+0x81/+0x85]` from the bucket scalar instead of preserving stale
save-time residue
- the same save-native company direct-record seam now also carries the full outer periodic-company
side-latch trio rooted at `0x0d17/0x0d18/0x0d56`, including the preferred-locomotive
engine-type chooser byte beside the city-connection and linked-transit finance gates
- that same seam now also resolves the base world route-preference byte at `[world+0x4c74]`, the
effective electric-only override fed by `0x0d17`, and the matching `1.4x` versus `1.8x`
route-quality multiplier as a normal runtime reader instead of a pure atlas note
- that same seam now also owns the first route-preference mutation path directly: beginning the
electric-only periodic-company override rewrites the world route-preference byte to the
effective company preference, ending it restores the base world byte, and runtime service state
now carries both the active and last applied override
- save inspection now also separates the shared `0x5209/0x520a/0x520b` save family correctly: the
smaller direct `0x1d5` collection is the live train family and now exposes a live-entry
directory rooted at metadata dword `16`, while the actual region collection is the larger
non-direct `Marker09` family; the tagged placed-structure header `0x36b1/0x36b2/0x36b3`
remains grounded alongside them, so the remaining city-connection / linked-transit blocker is
record-body reconstruction rather than missing save-side collection identity
- the project rule on the remaining closure work is now explicit too: when one runtime-facing field
is still ambiguous, prefer rehosting the owning source state or real reader/setter family first
instead of guessing another derived leaf field from neighboring raw offsets; the checked-in
`docs/rehost-queue.md` file is the control surface for that work loop, and after each commit the
next queue item should run unless the queue is empty, a real blocker remains that cannot be
advanced by any further non-hook work without guessing, or approval is needed; `final`
responses are stop-only there too, so active work should continue under `commentary` updates
rather than placeholder status replies
- `rrt-runtime` now also exposes higher-layer probe commands for the current blocked frontier:
`runtime inspect-periodic-company-service-trace <save.gms>`,
`runtime inspect-region-service-trace <save.gms>`, and
`runtime inspect-infrastructure-asset-trace <save.gms>`. These reports make the current
shellless city-connection / linked-transit blockers explicit as missing owner seams rather than
generic “still unresolved” residue.
- a checked-in `EventEffects` export now exists at
`artifacts/exports/rt3-1.06/event-effects-table.json`, and a checked-in semantic closure layer
now exists at `artifacts/exports/rt3-1.06/event-effects-semantic-catalog.json`
- recovered descriptor rows now land on explicit semantic frontier buckets such as
`blocked_shell_owned_descriptor`, `blocked_evidence_blocked_descriptor`, and
`blocked_variant_or_scope_blocked_descriptor` instead of generic unmapped-descriptor residue
- the first recovered governance descriptor tranche now executes through the generic
company-governance scalar effect surface: descriptor `56` `Credit Rating` and descriptor `57`
`Prime Rate`
- adjacent recovered finance/control-transfer descriptors such as `55` `Stock Prices` and `58`
`Merger Premium` now land on explicit shell-owned descriptor parity instead of generic unmapped
descriptor residue, with tracked fixtures now pinning finance, scenario-outcome, and
control-transfer shell rows explicitly
- the recovered whole-game scalar economy/performance strip `59..104` now has a bounded runtime
landing surface too: representative rows execute into `RuntimeState.world_scalar_overrides`
through stable normalized keys such as `world.build_stations_cost` and
`world.track_maintenance_cost`
- the runtime-variable strip `39..54` now executes too through bounded event-owned scalar maps on
world/company/player/territory state, and the matching ordinary-condition strip now gates
imported records through those same runtime-owned variable maps; these variables are still
runtime-owned only in the current model and are not yet reconstructed from raw saves
- the grounded aggregate cargo-economics descriptors now execute too: descriptor `105`
`All Cargo Prices` and descriptors `177..179` `All Cargo Production` / `All Factory Production`
/ `All Farm/Mine Production` land on bounded event-owned cargo override state, and the grounded
named cargo-production strip `180..229` now lands on named cargo production overrides too
- the named cargo-price strip `106..176` now lands on named cargo price overrides too; the
checked-in static selector reconstruction is now explicit because the broader 1.06 CargoTypes
corpus has `51` names, the Cargo106 `cargoSkin` corpus has `70`, and the rehosted offline
selector builder now closes the 71-row price strip as `cargoSkin` plus the core `Rock`
carry-over; a new
checked-in offline cargo-source report at
`artifacts/exports/rt3-1.06/economy-cargo-sources.json` now parses both `CargoTypes` and the
`Cargo106.PK4` `cargoSkin` descriptors through rehosted code, normalizes localized
`~####Name` tokens into visible names, builds a merged live cargo registry, and now derives
exact named cargo-production and named cargo-price selectors from the checked-in bindings;
dedicated CLI inspector commands now expose both grounded selectors directly, while the same
report still shows the nine excluded CargoTypes-only industrial names that sit outside the
71-row price strip
- the add-building strip `503..519` is now explicitly classified as recovered shell-owned parity,
with tracked fixture coverage, instead of generic unresolved descriptor residue
- widen real packed-event executable coverage descriptor by descriptor after identity, target mask,
and normalized effect semantics are all grounded, not just after row framing is parsed
- the first grounded condition-side unlock now exists for negative-sentinel `raw_condition_id = -1`
company scopes, and the first ordinary nonnegative condition batch now executes too: numeric
thresholds for company finance, company track, aggregate territory track, and company-territory
track
- exact named-territory binding now executes too, while named-territory no-match cases remain the
explicit binding blocker frontier
- real descriptors `8` `Economic Status`, `9` `Confiscate All`, and `15` `Retire Train` now join
the executable batch through the same ordinary runtime path, backed by the opaque economic-status
lane and the minimal event-owned train roster
- descriptor `3` `Territory - Allow All` now executes as company-to-territory access rights through
the same ordinary runtime path; shell purchase-flow parity remains out of scope, and mixed
supported/unsupported real rows still stay parity-only
- whole-game ordinary-condition execution now exists too: special-condition thresholds,
candidate-availability thresholds, and economic-status-code thresholds now gate imported runtime
records, and the packed-event frontier now reports explicit unmapped world-condition and
world-descriptor buckets
- that whole-game condition batch is now metadata-driven too: special-condition label ids,
economic-status, and the generic `%1 Avail.` candidate-availability template plus candidate-name
side strings all decode through checked-in world-condition metadata instead of fixture-only ids
- the first real whole-game grouped-descriptor batch is now metadata-driven too: checked-in
descriptor metadata covers special-condition and candidate-availability setters, and descriptor
`110` `Disable Stock Buying and Selling` now executes too through the checked-in keyed runtime
flag `world.disable_stock_buying_and_selling`
- that world-toggle path now covers a broader recovered boolean scenario-rule band too:
descriptors `111..138` now decode through checked-in metadata into either keyed `world_flags`
or the bounded `world_restore.limited_track_building_amount` scalar for finance/trading,
construction, and governance restrictions
- the late recovered world-toggle band now executes too where current evidence is equally strong:
`Use Bio-Accelerator Cars`, `Disable Cargo Economy`, `Disable Train Crashes`,
`Disable Train Crashes AND Breakdowns`, and `AI Ignore Territories At Startup`
- whole-game ordinary-condition coverage is broader now too: checked-in world-flag condition ids
can lower into `world_flag_equals` gates for boolean equality/inequality forms, so real packed
rows can gate whole-game effects on existing `world_flags`
- the tracked parity save-slice now keeps its remaining non-imported residue as structured
`real_packed_v1` parity records, with the first captured leftover now pinned to the unresolved
upper locomotives-page availability tail and moved onto explicit descriptor parity
- the next recovered locomotives-page descriptor batch is partially executable too:
descriptors `454..456` (`All Steam/Diesel/Electric Locos Avail.`) now lower through checked-in
metadata into keyed `world_flags`, while the wider locomotive availability/cost scalar bands now
split cleanly between the grounded lower executable prefix and the explicit unresolved tails
- raw `.smp` inspection/export now reconstructs the persisted save-side named locomotive table and
derives a minimal locomotive catalog from its row order, so save-slice documents can carry both
`RuntimeState.named_locomotive_availability` and the catalog context needed for descriptor
lowering
- recovered scalar locomotive availability and locomotive-cost descriptors now import through that
save-native or embedded `RuntimeState.locomotive_catalog` context into the ordinary
`named_locomotive_availability` and `named_locomotive_cost` runtime maps
- the full lower locomotive bands now execute through that save-native ordinal catalog too:
availability `241..351` and cost `352..451`; the checked `29`-save `.gms + .gmx`
`locomotive-catalog-tail-census.json` export now fixes the last save-stable static descriptor
boundary at ordinal `58` (`VL80T`) and leaves descriptor `452` plus the upper bands
`457..474` / `475..502` as external-corpus or dynamic blockers rather than active repo-local
static work
- cargo-production `230..240` and territory-access-cost `453` now execute too through minimal
world-side scalar landing surfaces: slot-indexed `cargo_production_overrides` and
`world_restore.territory_access_cost`
- recipe-book probing now derives a save-native `cargo_catalog` too, so save-slice documents carry
stable cargo slot labels and token-stem evidence into runtime state without requiring a separate
cargo simulation layer
- world-scalar ordinary-condition coverage now matches those runtime surfaces too: checked-in
metadata lowers named locomotive availability, named locomotive cost, named cargo-production
slot thresholds, aggregate cargo production, factory/farm-mine/other cargo production,
limited-track-building-amount, and territory-access-cost rows into explicit runtime condition
gates
- cargo slot classification is now checked in and save-native too, so the remaining cargo frontier
is broader descriptor/condition breadth rather than classification or save/import plumbing
- the company/chairman frontier has moved too: checked-in save-slice documents can now carry that
context natively, so the next work on that axis is broader recovery and eventual raw save
reconstruction rather than overlay-only ownership
- keep in mind that the current local `.gms + .gmx` corpus does carry packed-event collections,
but the checked `29`-save locomotive census still finds no descriptor carriers in `452` or
`457..502`, so further locomotives-page closure now needs either broader save evidence or
dynamic captures
- use `rrt-hook` primarily as optional capture or integration tooling, not as the first execution
environment
- keep `docs/runtime-rehost-plan.md` current as the runtime baseline and next implementation slice
change
Regenerate the initial exports with:
```bash
python3 tools/py/collect_pe_artifacts.py \
rt3_wineprefix/drive_c/rt3/RT3.exe \
artifacts/exports/rt3-1.06
```
Regenerate the startup-focused Ghidra exports with:
```bash
python3 tools/py/export_startup_map.py \
rt3_wineprefix/drive_c/rt3/RT3.exe \
artifacts/exports/rt3-1.06
```
Regenerate the checked-in `EventEffects` table export with:
```bash
python3 tools/py/extract_event_effects.py \
rt3_wineprefix/drive_c/rt3/RT3.exe \
rt3_wineprefix/drive_c/rt3/Data/Language/RT3.lng \
artifacts/exports/rt3-1.06/event-effects-table.json
```
Regenerate the checked-in `EventEffects` semantic catalog with:
```bash
python3 tools/py/build_event_effect_semantic_catalog.py \
artifacts/exports/rt3-1.06/event-effects-table.json \
artifacts/exports/rt3-1.06/event-effects-semantic-catalog.json
```
That default export now walks two roots:
- `entry:0x005a313b`
- `bootstrap:0x00484440`
For a focused branch-deepening pass, regenerate the analysis context exports with:
```bash
python3 tools/py/export_analysis_context.py \
rt3_wineprefix/drive_c/rt3/RT3.exe \
artifacts/exports/rt3-1.06 \
--addr 0x00444dd0 \
--addr 0x00508730 \
--addr 0x00508880 \
--string gpdLabelDB \
--string gpdCityDB \
--string 2DLabel.imb \
--string 2DCity.imb \
--string "Geographic Labels"
```
For the pending-template dispatch-store branch, regenerate the new branch dossier with:
```bash
python3 tools/py/rt3_rekit.py \
pending-template-store \
rt3_wineprefix/drive_c/rt3/RT3.exe \
artifacts/exports/rt3-1.06
```
That dossier is now a targeted follow-up tool, not the default first pass.