Introduce Pheon GDSII parsing and layout review tools
Implement native GDSII reading and writing with Arrow interchange, hierarchy scans and selective cell loading. Validate numeric packing and array pitches, and preserve normalized REAL8 encodings exactly within the supported range. Add standalone dump, structural diff, hierarchy, layer-name annotation and Git diff commands. Provide scoped text and JSON reports, deterministic comparison, and record-preserving metadata edits. Package the shared library and optional Python Arrow readers as pheon using CFFI ABI mode and Python-independent platform wheels. Include locked source builds, native API checks, cross-platform artifact workflows, documentation and regression coverage. Require Rust 1.88 or newer and Python 3.11 or newer. Validation: 87 native tests, 14 binding tests on Python 3.11/3.13/3.14, and 71 installed companion integration tests pass. Full, scanned and selected Arrow layouts match the previous schemas.
This commit is contained in:
parent
e0a634f476
commit
b3346b40a4
1 changed files with 12956 additions and 1 deletions
337
README.md
Normal file
337
README.md
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
# Pheon
|
||||
|
||||
A fast GDSII parser and writer in Rust, with Apache Arrow integration for efficient data handling.
|
||||
|
||||
Pheon provides native GDSII codecs and a standalone layout review CLI. The Rust
|
||||
crate, Python package and executable are all named `pheon`. Python Klamath and
|
||||
Masque remain separate consumers. The former `klamath_rs_ext` import and
|
||||
`klamath` executable are not provided as aliases.
|
||||
|
||||
## Overview
|
||||
|
||||
`pheon` is designed to handle large GDSII files with high performance. It uses Apache Arrow as its primary in-memory data representation, allowing for efficient memory usage and easy integration with other tools in the Arrow ecosystem (like pandas, polars, or DuckDB via Python bindings).
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Arrow Schema Documentation](ARROW_SCHEMA.md): Detailed information about the Arrow schema used for GDSII data.
|
||||
|
||||
## Features
|
||||
|
||||
- Fast parsing of GDSII files.
|
||||
- Efficient memory representation using Apache Arrow.
|
||||
- Support for common GDSII elements: SREF, AREF, Boundary, Path, Text, etc.
|
||||
- Optimized "batch" representations for rectangles and boundaries sharing the same layer.
|
||||
- C-compatible FFI for integration with other languages (e.g., Python).
|
||||
|
||||
REAL8 writing uses normalized GDS encodings and represents every in-range `f64`
|
||||
exactly. Finite magnitudes below `2^-260` underflow to canonical zero; nonfinite
|
||||
values and magnitudes at or above `2^252` are rejected by the encoder.
|
||||
|
||||
## Usage
|
||||
|
||||
`read_path` exports a library through the Arrow C Data Interface. Allocate
|
||||
initialized `ArrowArray` and `ArrowSchema` output structures, then check the
|
||||
returned status: `0` means success and `-1` means failure. Only import the Arrow
|
||||
outputs after success; the caller owns their release callbacks. Failed calls
|
||||
leave the outputs unchanged. `pheon/_ffi.h` declares the C API; `test.py`
|
||||
demonstrates the supported Python reader.
|
||||
|
||||
`last_error_message()` returns a UTF-8, null-terminated error string, or null
|
||||
after a successful parser call. Errors belong to the calling thread. Copy the
|
||||
message before making another parser call on that thread, and before the thread
|
||||
exits; calls on other threads do not invalidate it. Parser panics that unwind
|
||||
are converted into errors.
|
||||
|
||||
For partial reads, call `scan_bytes` with a borrowed GDSII byte buffer to obtain
|
||||
cell ranges and references without decoding geometry. Pass the desired ranges
|
||||
to `read_cells_bytes` to load their full contents. Both calls use the same Arrow
|
||||
outputs and status/error handling as `read_path`:
|
||||
|
||||
```c
|
||||
int scan_bytes(const uint8_t *data, size_t size,
|
||||
struct ArrowArray *array, struct ArrowSchema *schema);
|
||||
int read_cells_bytes(const uint8_t *data, size_t size,
|
||||
const uint64_t *ranges, size_t range_count,
|
||||
struct ArrowArray *array, struct ArrowSchema *schema);
|
||||
```
|
||||
|
||||
`ranges` contains alternating inclusive start and exclusive end offsets, with
|
||||
two native-endian `uint64_t` values per cell. Each range must cover one complete
|
||||
cell. A null `ranges` pointer is accepted when `range_count` is zero. Likewise,
|
||||
null `data` is accepted for zero bytes and produces an ordinary parse error for
|
||||
the empty input. Nonempty buffers must be readable for the duration of the call;
|
||||
range values must be aligned. Exported Arrow data owns its storage independently
|
||||
of the input. See [Arrow Schema Documentation](ARROW_SCHEMA.md) for the scan
|
||||
schema, offset definitions, and ID assignment.
|
||||
|
||||
## Python package and validation
|
||||
|
||||
Python 3.11 or newer is required. Source builds require Rust 1.88 or newer;
|
||||
installing a prebuilt wheel requires no Rust compiler. The shared library uses
|
||||
a C ABI rather than Python's or Rust's native ABI, so one platform wheel serves
|
||||
multiple Python minor versions. CFFI, NumPy and optional PyArrow must themselves
|
||||
support the selected interpreter. A Rust application uses the crate normally
|
||||
and compiles it with its own compatible toolchain. The checkout selects the
|
||||
stable toolchain locally through `rust-toolchain.toml`, without changing your
|
||||
global default. CI checks both Rust 1.88 and current stable.
|
||||
|
||||
```sh
|
||||
python3 -m pip install 'pheon[arrow]'
|
||||
```
|
||||
|
||||
```python
|
||||
from pheon import arrow
|
||||
|
||||
library = arrow.read_path('chip.gds') # one-row StructArray; also accepts gzip
|
||||
scan = arrow.scan_bytes(gds_bytes)
|
||||
cells = scan.field('cells')[0].values
|
||||
ranges = list(zip(cells.field('struct_start_offset').to_pylist(),
|
||||
cells.field('struct_end_offset').to_pylist()))
|
||||
selected = arrow.read_cells_bytes(gds_bytes, ranges[:2])
|
||||
```
|
||||
|
||||
`read_bytes` reads a complete borrowed buffer. Full, scanned and selected Arrow
|
||||
results own their storage independently of that buffer. `PheonError` reports
|
||||
native parse failures; malformed range shapes and types raise Python value/type
|
||||
errors before calling native code. Integer packing needs no PyArrow import.
|
||||
|
||||
### Discovery and companion packages
|
||||
|
||||
Klamath's `native` extra installs `pheon`; Masque's `arrow` extra installs
|
||||
`pheon[arrow]`. Klamath imports Pheon's packing helpers, while Masque calls
|
||||
`pheon.arrow` and translates parse failures to its existing error type.
|
||||
|
||||
The Python loader selects the library bundled with the imported `pheon` package,
|
||||
or the exact file selected by `PHEON_LIBRARY`. It checks C ABI version 1 and
|
||||
Arrow layout schema version 1 before exposing native functions. Incompatible
|
||||
or unversioned libraries fail with an import error. Those versions are independent
|
||||
of package releases and the CLI's JSON report schema.
|
||||
|
||||
`cargo install --path . --locked --bin pheon` installs the CLI, not the Python
|
||||
package or a discoverable shared library. Klamath does not search PATH or Cargo
|
||||
directories for it. To reuse an external native build, install the Python
|
||||
components too and set `PHEON_LIBRARY`; an editable source install is useful
|
||||
during development. The old library override variables are not recognized.
|
||||
|
||||
The Python packing helpers require NumPy and CFFI, both declared as runtime
|
||||
dependencies. Setuptools invokes `cargo build --release --lib` and bundles the
|
||||
C ABI shared library in a platform-specific wheel. Its version comes from
|
||||
`Cargo.toml`. The C declarations are in `pheon/_ffi.h`; the loader uses
|
||||
CFFI ABI mode, so no Python extension compilation or Maturin is required.
|
||||
|
||||
```sh
|
||||
python3 -m pip install .
|
||||
```
|
||||
|
||||
To build a source archive and a wheel with the `build` frontend and installed
|
||||
Setuptools/Wheel tooling:
|
||||
|
||||
```sh
|
||||
python3 -m build --no-isolation
|
||||
```
|
||||
|
||||
Source archives include a Cargo lockfile, and wheel builds use `--locked` so
|
||||
building from an archive retains the resolved Rust dependency versions. Wheels
|
||||
are tagged for the build platform and use the Python-independent `py3-none` ABI.
|
||||
The CLI continues to build directly with Cargo and has no Python dependency.
|
||||
|
||||
For source-tree development, point the CFFI loader at a Cargo-built library:
|
||||
|
||||
```sh
|
||||
cargo build --release --lib
|
||||
PHEON_LIBRARY="$PWD/target/release/libpheon.so" \
|
||||
python3 -m unittest discover -s tests/python -v
|
||||
```
|
||||
|
||||
Use `libpheon.dylib` on macOS or `pheon.dll` on Windows. Without
|
||||
the explicit override, the loader uses the library bundled inside the package.
|
||||
After installing a wheel, run its tests outside the source checkout to exercise
|
||||
the installed package:
|
||||
|
||||
```sh
|
||||
python3 -I -m unittest discover -s /absolute/path/to/pheon/tests/python -v
|
||||
```
|
||||
|
||||
The Python tests cover multidimensional and noncontiguous packing, input
|
||||
preservation, range errors, and the C declarations for Arrow exports and release
|
||||
callbacks, input lifetimes and ABI mismatches. Run `cargo test` and
|
||||
`cargo test --release` for native validation. See [release validation](docs/releases.md)
|
||||
for wheel builds, platform repair and testing the same artifact on multiple
|
||||
Python versions.
|
||||
|
||||
## Layout review CLI
|
||||
|
||||
The review CLI is a standalone Rust executable:
|
||||
|
||||
```sh
|
||||
cargo build --bin pheon
|
||||
target/debug/pheon dump chip.gds
|
||||
target/debug/pheon dump chip.gds --records
|
||||
target/debug/pheon diff old.gds new.gds --hierarchy ascii
|
||||
target/debug/pheon diff old.gds new.gds --hierarchy mermaid --max-changes 20
|
||||
target/debug/pheon hierarchy chip.gds --format mermaid --depth 4
|
||||
target/debug/pheon diff old.gds new.gds --top CHIP --layer 1/0
|
||||
target/debug/pheon diff old.gds new.gds --show-layer 1/0
|
||||
target/debug/pheon diff old.gds new.gds --format json
|
||||
target/debug/pheon diff old.gds new.gds --format json --hierarchy graph
|
||||
target/debug/pheon name-layers chip.gds named.gds --cell CHIP --carrier 900/7 --name 1:0:WG
|
||||
```
|
||||
|
||||
The object dump is deterministic and omits source offsets. Record dumps preserve
|
||||
record order and show offsets, tags, sizes, and values, including records the
|
||||
object reviewer does not support. They can inspect an invalid metadata declaration
|
||||
without invoking object-level validation. Zero padding after ENDLIB is accepted.
|
||||
|
||||
The fixed `klamath-object-v2` comparison profile:
|
||||
|
||||
- Matches cells by name and layers by numeric pair. The default scope is the
|
||||
entire library; explicit selectors are recorded in each report.
|
||||
- Ignores library/cell timestamps, cell order, element order, and property order.
|
||||
- Normalizes boundary/box starting vertex and winding, retaining other vertices.
|
||||
- Preserves duplicate objects and duplicate properties as multisets.
|
||||
- Compares paths, labels, references, array endpoints/counts, properties, and all
|
||||
retained flags exactly. Explicit defaults and their omission remain distinct;
|
||||
SREFs and single-element AREFs remain distinct. REAL8 encodings are retained.
|
||||
- Reports library fields, including UNITS, separately. It does not rescale files
|
||||
into a shared grid, infer cell renames, or perform geometric Boolean operations.
|
||||
- Uses bounding boxes as coordinate envelopes, not evidence of equality. Path
|
||||
envelopes do not include stroke width; text/reference envelopes are coordinates,
|
||||
not rendered or recursively transformed bounds.
|
||||
- Returns an incomplete result for unsupported records, future metadata versions,
|
||||
unresolved references, or cycles. These cannot produce an equality claim.
|
||||
|
||||
`diff` exits with **0** for equality under this profile, **1** for differences,
|
||||
and **2** for invalid input or incomplete comparison. `--max-changes` limits the
|
||||
number of object details displayed, not the comparison. Source offsets are
|
||||
included in changed-object reports for tracing changes back to record dumps.
|
||||
|
||||
`diff --format json` emits a versioned report with typed before/after evidence,
|
||||
structured diagnostics, and independent comparison status and completeness.
|
||||
JSON includes all changes by default. Explicit display filters and `--max-changes`
|
||||
record omitted evidence without changing totals or exit status. `--hierarchy graph`
|
||||
adds the complete scoped reference graph, with symbolic array multiplicities;
|
||||
`--depth` applies to text output only. Timestamp-only changes still return 0.
|
||||
Input and argument errors also produce JSON when requested. See the
|
||||
[report contract](docs/diff-reports.md) and
|
||||
[version 1 JSON Schema](docs/diff-report-v1.schema.json).
|
||||
|
||||
Unmatched objects are paired for explanation only when candidates are unique by
|
||||
type/layer/bounding box, then by type/layer among the remaining objects.
|
||||
References additionally match by target. Ambiguous groups stay additions and
|
||||
removals. Reports explain reference movement, array-count changes, path widths,
|
||||
and polygons changed within the same bounds. Full before/after records remain
|
||||
visible, and duplicate counts remain significant. Unchanged layer names appear
|
||||
once as a legend for changed objects on those layers.
|
||||
|
||||
### Comparison scope and display filters
|
||||
|
||||
`--top NAME` compares that cell and its reachable subtree independently in each
|
||||
revision. `--cell NAME` compares the exact cell, including its reference records,
|
||||
without comparing descendant contents. Combine the two to select a cell within
|
||||
a subtree. A selector absent from both revisions is an error; absence on one
|
||||
side is an addition/deletion.
|
||||
|
||||
`--layer L/T` selects objects on a numeric layer/type pair, plus references whose
|
||||
targets use that layer transitively. Cells with no relevant content are omitted.
|
||||
Unresolved references, cycles, and unsupported records with unknown layer effect
|
||||
remain incomplete results. Unsupported objects on another known layer can be
|
||||
excluded. Library fields, including units, remain compared. Unsupported layer-name
|
||||
version diagnostics are always retained, since those declarations can affect any
|
||||
layer or cell in the library. Scoped dumps and
|
||||
hierarchy views accept the same selectors. Scope selection parses the complete
|
||||
input: malformed record grammar and conflicting metadata are still errors.
|
||||
|
||||
`--show-cell NAME` and `--show-layer L/T` filter object details in a diff report
|
||||
without changing comparison status, totals, metadata changes, or hierarchy.
|
||||
References have no numeric layer and are hidden by `--show-layer`. The comparison
|
||||
scope and any display-only filter are printed explicitly. The Rust API accepts
|
||||
one combined `Scope`; applying another scope to an already scoped document is
|
||||
rejected rather than losing reference coverage information.
|
||||
|
||||
Hierarchy output distinguishes directly changed cells, affected ancestors, and
|
||||
unchanged targets of modified references. It includes edges from both revisions,
|
||||
so removed branches remain visible. Counts are direct edge multiplicities,
|
||||
including arrays, rather than expanded placements. Shared definitions are marked
|
||||
as already shown in text and appear once in Mermaid. `--depth` limits displayed
|
||||
descendants only. The CLI emits Mermaid source, requiring no diagram renderer.
|
||||
|
||||
Layer names are read from labels in an existing cell. A label with the exact
|
||||
string `KLAMATH_LAYER_NAMES_V1` identifies an annotation carrier `(layer, texttype)`
|
||||
pair within that cell. All other labels on that same pair in that same cell must
|
||||
be `layer:datatype:name` declarations, for example `1:0:WG`. The carrier pair is
|
||||
not fixed; choose a suitable annotation layer in the design's technology.
|
||||
Declarations split at the first two colons, allowing colons in names. Numeric
|
||||
fields are unsigned 16-bit decimal integers. Names are nonempty UTF-8 strings.
|
||||
Repeated identical declarations are accepted; conflicting declarations fail.
|
||||
Names are collected across the library. Recognized metadata labels are omitted
|
||||
from object comparison; mappings are compared separately and printed once per
|
||||
numeric pair, so renaming a layer does not rewrite every shape description.
|
||||
|
||||
`name-layers INPUT OUTPUT --cell NAME --carrier L/T --name L:T:NAME ...` writes
|
||||
this convention. It consolidates existing recognized declarations into one legend
|
||||
in the chosen existing cell, retains existing names not explicitly updated, and
|
||||
overrides the named pairs. The chosen carrier must contain no ordinary design
|
||||
objects in that cell. Unrelated records, timestamps, unknown records and trailing
|
||||
padding are copied byte-for-byte. No structures or references are added. Output
|
||||
must be a new file: the CLI refuses to overwrite an existing destination, including
|
||||
the input. Invalid declarations, conflicting arguments, oversized labels and
|
||||
future metadata versions fail before output creation.
|
||||
|
||||
The record-preserving review adapter retains information normalized or omitted
|
||||
by the Arrow geometry reader; existing Arrow APIs are unchanged. Structured Rust
|
||||
change records feed the text and hierarchy renderers. LayoutEditor's `LayerNames`
|
||||
extension, layout extraction/remapping, and geometric Boolean comparison remain
|
||||
outside the current CLI.
|
||||
|
||||
### Git integration
|
||||
|
||||
Install the executable on PATH with `cargo install --path . --bin pheon`.
|
||||
The templates in [contrib/git/config](contrib/git/config) and
|
||||
[contrib/git/attributes](contrib/git/attributes) provide two alternative drivers.
|
||||
For structural summaries and affected hierarchy, add this attribute:
|
||||
|
||||
```gitattributes
|
||||
*.gds diff=pheon
|
||||
```
|
||||
|
||||
Configure the native external driver in Git config:
|
||||
|
||||
```gitconfig
|
||||
[diff "pheon"]
|
||||
command = pheon git-diff --hierarchy ascii --
|
||||
binary = true
|
||||
trustExitCode = false
|
||||
```
|
||||
|
||||
The adapter consumes Git's old/new temporary files directly, handles additions,
|
||||
deletions and renames, and quotes display paths. It returns success after rendering
|
||||
either equality or differences, as expected by the external-driver protocol.
|
||||
Invalid or incomplete comparisons retain a nonzero exit and diagnostics. Scope
|
||||
and display options can precede the final `--` in the configured command. Existing
|
||||
repositories using `diff=layout` can put the same command under `[diff "layout"]`.
|
||||
|
||||
For ordinary Git line diffs, use canonical textconv instead:
|
||||
|
||||
```gitattributes
|
||||
*.gds diff=pheon-text
|
||||
```
|
||||
|
||||
Configure the driver in Git config:
|
||||
|
||||
```gitconfig
|
||||
[diff "pheon-text"]
|
||||
textconv = pheon dump
|
||||
binary = true
|
||||
cachetextconv = true
|
||||
```
|
||||
|
||||
This is a human review representation, not an applicable GDS patch. Invalidate
|
||||
Git's textconv cache when changing the dump profile/tool version. For automated
|
||||
equality checks, use `pheon diff` and its exit status directly.
|
||||
|
||||
`tests/review.rs` contains ten initial fixture pairs, checked-in canonical/text/
|
||||
Mermaid goldens, metadata byte-preservation checks, scope and report regressions,
|
||||
and actual Git integration tests. It also checks near-limit polygon records,
|
||||
2048-cell hierarchies and arrays representing over a billion placements without
|
||||
expanding them. Run `cargo test` and `cargo test --release` for debug and optimized
|
||||
validation. No KLayout installation or Python package is required.
|
||||
Loading…
Reference in a new issue