236 lines
12 KiB
Markdown
236 lines
12 KiB
Markdown
# klamath-rs
|
|
|
|
A fast GDSII parser and writer in Rust, with Apache Arrow integration for efficient data handling.
|
|
|
|
## Overview
|
|
|
|
`klamath-rs` is designed to handle large GDSII files with high performance. It uses Apache Arrow as its primary in-memory data representation, allowing for efficient memory usage and easy integration with other tools in the Arrow ecosystem (like pandas, polars, or DuckDB via Python bindings).
|
|
|
|
## Documentation
|
|
|
|
- [Arrow Schema Documentation](ARROW_SCHEMA.md): Detailed information about the Arrow schema used for GDSII data.
|
|
|
|
## Features
|
|
|
|
- Fast parsing of GDSII files.
|
|
- Efficient memory representation using Apache Arrow.
|
|
- Support for common GDSII elements: SREF, AREF, Boundary, Path, Text, etc.
|
|
- Optimized "batch" representations for rectangles and boundaries sharing the same layer.
|
|
- C-compatible FFI for integration with other languages (e.g., Python).
|
|
|
|
REAL8 writing uses normalized GDS encodings and represents every in-range `f64`
|
|
exactly. Finite magnitudes below `2^-260` underflow to canonical zero; nonfinite
|
|
values and magnitudes at or above `2^252` are rejected by the encoder.
|
|
|
|
## Usage
|
|
|
|
`read_path` exports a library through the Arrow C Data Interface. Allocate
|
|
initialized `ArrowArray` and `ArrowSchema` output structures, then check the
|
|
returned status: `0` means success and `-1` means failure. Only import the Arrow
|
|
outputs after success; the caller owns their release callbacks. Failed calls
|
|
leave the outputs unchanged. See `test.py` for a Python CFFI example.
|
|
|
|
`last_error_message()` returns a UTF-8, null-terminated error string, or null
|
|
after a successful parser call. Errors belong to the calling thread. Copy the
|
|
message before making another parser call on that thread, and before the thread
|
|
exits; calls on other threads do not invalidate it. Parser panics that unwind
|
|
are converted into errors.
|
|
|
|
For partial reads, call `scan_bytes` with a borrowed GDSII byte buffer to obtain
|
|
cell ranges and references without decoding geometry. Pass the desired ranges
|
|
to `read_cells_bytes` to load their full contents. Both calls use the same Arrow
|
|
outputs and status/error handling as `read_path`:
|
|
|
|
```c
|
|
int scan_bytes(const uint8_t *data, size_t size,
|
|
struct ArrowArray *array, struct ArrowSchema *schema);
|
|
int read_cells_bytes(const uint8_t *data, size_t size,
|
|
const uint64_t *ranges, size_t range_count,
|
|
struct ArrowArray *array, struct ArrowSchema *schema);
|
|
```
|
|
|
|
`ranges` contains alternating inclusive start and exclusive end offsets, with
|
|
two native-endian `uint64_t` values per cell. Each range must cover one complete
|
|
cell. A null `ranges` pointer is accepted when `range_count` is zero. Likewise,
|
|
null `data` is accepted for zero bytes and produces an ordinary parse error for
|
|
the empty input. Nonempty buffers must be readable for the duration of the call;
|
|
range values must be aligned. Exported Arrow data owns its storage independently
|
|
of the input. See [Arrow Schema Documentation](ARROW_SCHEMA.md) for the scan
|
|
schema, offset definitions, and ID assignment.
|
|
|
|
## Layout review CLI
|
|
|
|
The review CLI is a standalone Rust executable:
|
|
|
|
```sh
|
|
cargo build --bin klamath
|
|
target/debug/klamath dump chip.gds
|
|
target/debug/klamath dump chip.gds --records
|
|
target/debug/klamath diff old.gds new.gds --hierarchy ascii
|
|
target/debug/klamath diff old.gds new.gds --hierarchy mermaid --max-changes 20
|
|
target/debug/klamath hierarchy chip.gds --format mermaid --depth 4
|
|
target/debug/klamath diff old.gds new.gds --top CHIP --layer 1/0
|
|
target/debug/klamath diff old.gds new.gds --show-layer 1/0
|
|
target/debug/klamath diff old.gds new.gds --format json
|
|
target/debug/klamath diff old.gds new.gds --format json --hierarchy graph
|
|
target/debug/klamath name-layers chip.gds named.gds --cell CHIP --carrier 900/7 --name 1:0:WG
|
|
```
|
|
|
|
The object dump is deterministic and omits source offsets. Record dumps preserve
|
|
record order and show offsets, tags, sizes, and values, including records the
|
|
object reviewer does not support. They can inspect an invalid metadata declaration
|
|
without invoking object-level validation. Zero padding after ENDLIB is accepted.
|
|
|
|
The fixed `klamath-object-v2` comparison profile:
|
|
|
|
- Matches cells by name and layers by numeric pair. The default scope is the
|
|
entire library; explicit selectors are recorded in each report.
|
|
- Ignores library/cell timestamps, cell order, element order, and property order.
|
|
- Normalizes boundary/box starting vertex and winding, retaining other vertices.
|
|
- Preserves duplicate objects and duplicate properties as multisets.
|
|
- Compares paths, labels, references, array endpoints/counts, properties, and all
|
|
retained flags exactly. Explicit defaults and their omission remain distinct;
|
|
SREFs and single-element AREFs remain distinct. REAL8 encodings are retained.
|
|
- Reports library fields, including UNITS, separately. It does not rescale files
|
|
into a shared grid, infer cell renames, or perform geometric Boolean operations.
|
|
- Uses bounding boxes as coordinate envelopes, not evidence of equality. Path
|
|
envelopes do not include stroke width; text/reference envelopes are coordinates,
|
|
not rendered or recursively transformed bounds.
|
|
- Returns an incomplete result for unsupported records, future metadata versions,
|
|
unresolved references, or cycles. These cannot produce an equality claim.
|
|
|
|
`diff` exits with **0** for equality under this profile, **1** for differences,
|
|
and **2** for invalid input or incomplete comparison. `--max-changes` limits the
|
|
number of object details displayed, not the comparison. Source offsets are
|
|
included in changed-object reports for tracing changes back to record dumps.
|
|
|
|
`diff --format json` emits a versioned report with typed before/after evidence,
|
|
structured diagnostics, and independent comparison status and completeness.
|
|
JSON includes all changes by default. Explicit display filters and `--max-changes`
|
|
record omitted evidence without changing totals or exit status. `--hierarchy graph`
|
|
adds the complete scoped reference graph, with symbolic array multiplicities;
|
|
`--depth` applies to text output only. Timestamp-only changes still return 0.
|
|
Input and argument errors also produce JSON when requested. See the
|
|
[report contract](docs/diff-reports.md) and
|
|
[version 1 JSON Schema](docs/diff-report-v1.schema.json).
|
|
|
|
Unmatched objects are paired for explanation only when candidates are unique by
|
|
type/layer/bounding box, then by type/layer among the remaining objects.
|
|
References additionally match by target. Ambiguous groups stay additions and
|
|
removals. Reports explain reference movement, array-count changes, path widths,
|
|
and polygons changed within the same bounds. Full before/after records remain
|
|
visible, and duplicate counts remain significant. Unchanged layer names appear
|
|
once as a legend for changed objects on those layers.
|
|
|
|
### Comparison scope and display filters
|
|
|
|
`--top NAME` compares that cell and its reachable subtree independently in each
|
|
revision. `--cell NAME` compares the exact cell, including its reference records,
|
|
without comparing descendant contents. Combine the two to select a cell within
|
|
a subtree. A selector absent from both revisions is an error; absence on one
|
|
side is an addition/deletion.
|
|
|
|
`--layer L/T` selects objects on a numeric layer/type pair, plus references whose
|
|
targets use that layer transitively. Cells with no relevant content are omitted.
|
|
Unresolved references, cycles, and unsupported records with unknown layer effect
|
|
remain incomplete results. Unsupported objects on another known layer can be
|
|
excluded. Library fields, including units, remain compared. Unsupported layer-name
|
|
version diagnostics are always retained, since those declarations can affect any
|
|
layer or cell in the library. Scoped dumps and
|
|
hierarchy views accept the same selectors. Scope selection parses the complete
|
|
input: malformed record grammar and conflicting metadata are still errors.
|
|
|
|
`--show-cell NAME` and `--show-layer L/T` filter object details in a diff report
|
|
without changing comparison status, totals, metadata changes, or hierarchy.
|
|
References have no numeric layer and are hidden by `--show-layer`. The comparison
|
|
scope and any display-only filter are printed explicitly. The Rust API accepts
|
|
one combined `Scope`; applying another scope to an already scoped document is
|
|
rejected rather than losing reference coverage information.
|
|
|
|
Hierarchy output distinguishes directly changed cells, affected ancestors, and
|
|
unchanged targets of modified references. It includes edges from both revisions,
|
|
so removed branches remain visible. Counts are direct edge multiplicities,
|
|
including arrays, rather than expanded placements. Shared definitions are marked
|
|
as already shown in text and appear once in Mermaid. `--depth` limits displayed
|
|
descendants only. The CLI emits Mermaid source, requiring no diagram renderer.
|
|
|
|
Layer names are read from labels in an existing cell. A label with the exact
|
|
string `KLAMATH_LAYER_NAMES_V1` identifies an annotation carrier `(layer, texttype)`
|
|
pair within that cell. All other labels on that same pair in that same cell must
|
|
be `layer:datatype:name` declarations, for example `1:0:WG`. The carrier pair is
|
|
not fixed; choose a suitable annotation layer in the design's technology.
|
|
Declarations split at the first two colons, allowing colons in names. Numeric
|
|
fields are unsigned 16-bit decimal integers. Names are nonempty UTF-8 strings.
|
|
Repeated identical declarations are accepted; conflicting declarations fail.
|
|
Names are collected across the library. Recognized metadata labels are omitted
|
|
from object comparison; mappings are compared separately and printed once per
|
|
numeric pair, so renaming a layer does not rewrite every shape description.
|
|
|
|
`name-layers INPUT OUTPUT --cell NAME --carrier L/T --name L:T:NAME ...` writes
|
|
this convention. It consolidates existing recognized declarations into one legend
|
|
in the chosen existing cell, retains existing names not explicitly updated, and
|
|
overrides the named pairs. The chosen carrier must contain no ordinary design
|
|
objects in that cell. Unrelated records, timestamps, unknown records and trailing
|
|
padding are copied byte-for-byte. No structures or references are added. Output
|
|
must be a new file: the CLI refuses to overwrite an existing destination, including
|
|
the input. Invalid declarations, conflicting arguments, oversized labels and
|
|
future metadata versions fail before output creation.
|
|
|
|
The record-preserving review adapter retains information normalized or omitted
|
|
by the Arrow geometry reader; existing Arrow APIs are unchanged. Structured Rust
|
|
change records feed the text and hierarchy renderers. LayoutEditor's `LayerNames`
|
|
extension, layout extraction/remapping, and geometric Boolean comparison remain
|
|
outside the current CLI.
|
|
|
|
### Git integration
|
|
|
|
Install the executable on PATH with `cargo install --path . --bin klamath`.
|
|
The templates in [contrib/git/config](contrib/git/config) and
|
|
[contrib/git/attributes](contrib/git/attributes) provide two alternative drivers.
|
|
For structural summaries and affected hierarchy, add this attribute:
|
|
|
|
```gitattributes
|
|
*.gds diff=klamath
|
|
```
|
|
|
|
Configure the native external driver in Git config:
|
|
|
|
```gitconfig
|
|
[diff "klamath"]
|
|
command = klamath git-diff --hierarchy ascii --
|
|
binary = true
|
|
trustExitCode = false
|
|
```
|
|
|
|
The adapter consumes Git's old/new temporary files directly, handles additions,
|
|
deletions and renames, and quotes display paths. It returns success after rendering
|
|
either equality or differences, as expected by the external-driver protocol.
|
|
Invalid or incomplete comparisons retain a nonzero exit and diagnostics. Scope
|
|
and display options can precede the final `--` in the configured command. Existing
|
|
repositories using `diff=layout` can put the same command under `[diff "layout"]`.
|
|
|
|
For ordinary Git line diffs, use canonical textconv instead:
|
|
|
|
```gitattributes
|
|
*.gds diff=klamath-text
|
|
```
|
|
|
|
Configure the driver in Git config:
|
|
|
|
```gitconfig
|
|
[diff "klamath-text"]
|
|
textconv = klamath dump
|
|
binary = true
|
|
cachetextconv = true
|
|
```
|
|
|
|
This is a human review representation, not an applicable GDS patch. Invalidate
|
|
Git's textconv cache when changing the dump profile/tool version. For automated
|
|
equality checks, use `klamath diff` and its exit status directly.
|
|
|
|
`tests/review.rs` contains ten initial fixture pairs, checked-in canonical/text/
|
|
Mermaid goldens, metadata byte-preservation checks, scope and report regressions,
|
|
and actual Git integration tests. It also checks near-limit polygon records,
|
|
2048-cell hierarchies and arrays representing over a billion placements without
|
|
expanding them. Run `cargo test` and `cargo test --release` for debug and optimized
|
|
validation. No KLayout installation or Python package is required.
|