337 lines
17 KiB
Markdown
337 lines
17 KiB
Markdown
|
|
# Pheon
|
||
|
|
|
||
|
|
A fast GDSII parser and writer in Rust, with Apache Arrow integration for efficient data handling.
|
||
|
|
|
||
|
|
Pheon provides native GDSII codecs and a standalone layout review CLI. The Rust
|
||
|
|
crate, Python package and executable are all named `pheon`. Python Klamath and
|
||
|
|
Masque remain separate consumers. The former `klamath_rs_ext` import and
|
||
|
|
`klamath` executable are not provided as aliases.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
`pheon` is designed to handle large GDSII files with high performance. It uses Apache Arrow as its primary in-memory data representation, allowing for efficient memory usage and easy integration with other tools in the Arrow ecosystem (like pandas, polars, or DuckDB via Python bindings).
|
||
|
|
|
||
|
|
## Documentation
|
||
|
|
|
||
|
|
- [Arrow Schema Documentation](ARROW_SCHEMA.md): Detailed information about the Arrow schema used for GDSII data.
|
||
|
|
|
||
|
|
## Features
|
||
|
|
|
||
|
|
- Fast parsing of GDSII files.
|
||
|
|
- Efficient memory representation using Apache Arrow.
|
||
|
|
- Support for common GDSII elements: SREF, AREF, Boundary, Path, Text, etc.
|
||
|
|
- Optimized "batch" representations for rectangles and boundaries sharing the same layer.
|
||
|
|
- C-compatible FFI for integration with other languages (e.g., Python).
|
||
|
|
|
||
|
|
REAL8 writing uses normalized GDS encodings and represents every in-range `f64`
|
||
|
|
exactly. Finite magnitudes below `2^-260` underflow to canonical zero; nonfinite
|
||
|
|
values and magnitudes at or above `2^252` are rejected by the encoder.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
`read_path` exports a library through the Arrow C Data Interface. Allocate
|
||
|
|
initialized `ArrowArray` and `ArrowSchema` output structures, then check the
|
||
|
|
returned status: `0` means success and `-1` means failure. Only import the Arrow
|
||
|
|
outputs after success; the caller owns their release callbacks. Failed calls
|
||
|
|
leave the outputs unchanged. `pheon/_ffi.h` declares the C API; `test.py`
|
||
|
|
demonstrates the supported Python reader.
|
||
|
|
|
||
|
|
`last_error_message()` returns a UTF-8, null-terminated error string, or null
|
||
|
|
after a successful parser call. Errors belong to the calling thread. Copy the
|
||
|
|
message before making another parser call on that thread, and before the thread
|
||
|
|
exits; calls on other threads do not invalidate it. Parser panics that unwind
|
||
|
|
are converted into errors.
|
||
|
|
|
||
|
|
For partial reads, call `scan_bytes` with a borrowed GDSII byte buffer to obtain
|
||
|
|
cell ranges and references without decoding geometry. Pass the desired ranges
|
||
|
|
to `read_cells_bytes` to load their full contents. Both calls use the same Arrow
|
||
|
|
outputs and status/error handling as `read_path`:
|
||
|
|
|
||
|
|
```c
|
||
|
|
int scan_bytes(const uint8_t *data, size_t size,
|
||
|
|
struct ArrowArray *array, struct ArrowSchema *schema);
|
||
|
|
int read_cells_bytes(const uint8_t *data, size_t size,
|
||
|
|
const uint64_t *ranges, size_t range_count,
|
||
|
|
struct ArrowArray *array, struct ArrowSchema *schema);
|
||
|
|
```
|
||
|
|
|
||
|
|
`ranges` contains alternating inclusive start and exclusive end offsets, with
|
||
|
|
two native-endian `uint64_t` values per cell. Each range must cover one complete
|
||
|
|
cell. A null `ranges` pointer is accepted when `range_count` is zero. Likewise,
|
||
|
|
null `data` is accepted for zero bytes and produces an ordinary parse error for
|
||
|
|
the empty input. Nonempty buffers must be readable for the duration of the call;
|
||
|
|
range values must be aligned. Exported Arrow data owns its storage independently
|
||
|
|
of the input. See [Arrow Schema Documentation](ARROW_SCHEMA.md) for the scan
|
||
|
|
schema, offset definitions, and ID assignment.
|
||
|
|
|
||
|
|
## Python package and validation
|
||
|
|
|
||
|
|
Python 3.11 or newer is required. Source builds require Rust 1.88 or newer;
|
||
|
|
installing a prebuilt wheel requires no Rust compiler. The shared library uses
|
||
|
|
a C ABI rather than Python's or Rust's native ABI, so one platform wheel serves
|
||
|
|
multiple Python minor versions. CFFI, NumPy and optional PyArrow must themselves
|
||
|
|
support the selected interpreter. A Rust application uses the crate normally
|
||
|
|
and compiles it with its own compatible toolchain. The checkout selects the
|
||
|
|
stable toolchain locally through `rust-toolchain.toml`, without changing your
|
||
|
|
global default. CI checks both Rust 1.88 and current stable.
|
||
|
|
|
||
|
|
```sh
|
||
|
|
python3 -m pip install 'pheon[arrow]'
|
||
|
|
```
|
||
|
|
|
||
|
|
```python
|
||
|
|
from pheon import arrow
|
||
|
|
|
||
|
|
library = arrow.read_path('chip.gds') # one-row StructArray; also accepts gzip
|
||
|
|
scan = arrow.scan_bytes(gds_bytes)
|
||
|
|
cells = scan.field('cells')[0].values
|
||
|
|
ranges = list(zip(cells.field('struct_start_offset').to_pylist(),
|
||
|
|
cells.field('struct_end_offset').to_pylist()))
|
||
|
|
selected = arrow.read_cells_bytes(gds_bytes, ranges[:2])
|
||
|
|
```
|
||
|
|
|
||
|
|
`read_bytes` reads a complete borrowed buffer. Full, scanned and selected Arrow
|
||
|
|
results own their storage independently of that buffer. `PheonError` reports
|
||
|
|
native parse failures; malformed range shapes and types raise Python value/type
|
||
|
|
errors before calling native code. Integer packing needs no PyArrow import.
|
||
|
|
|
||
|
|
### Discovery and companion packages
|
||
|
|
|
||
|
|
Klamath's `native` extra installs `pheon`; Masque's `arrow` extra installs
|
||
|
|
`pheon[arrow]`. Klamath imports Pheon's packing helpers, while Masque calls
|
||
|
|
`pheon.arrow` and translates parse failures to its existing error type.
|
||
|
|
|
||
|
|
The Python loader selects the library bundled with the imported `pheon` package,
|
||
|
|
or the exact file selected by `PHEON_LIBRARY`. It checks C ABI version 1 and
|
||
|
|
Arrow layout schema version 1 before exposing native functions. Incompatible
|
||
|
|
or unversioned libraries fail with an import error. Those versions are independent
|
||
|
|
of package releases and the CLI's JSON report schema.
|
||
|
|
|
||
|
|
`cargo install --path . --locked --bin pheon` installs the CLI, not the Python
|
||
|
|
package or a discoverable shared library. Klamath does not search PATH or Cargo
|
||
|
|
directories for it. To reuse an external native build, install the Python
|
||
|
|
components too and set `PHEON_LIBRARY`; an editable source install is useful
|
||
|
|
during development. The old library override variables are not recognized.
|
||
|
|
|
||
|
|
The Python packing helpers require NumPy and CFFI, both declared as runtime
|
||
|
|
dependencies. Setuptools invokes `cargo build --release --lib` and bundles the
|
||
|
|
C ABI shared library in a platform-specific wheel. Its version comes from
|
||
|
|
`Cargo.toml`. The C declarations are in `pheon/_ffi.h`; the loader uses
|
||
|
|
CFFI ABI mode, so no Python extension compilation or Maturin is required.
|
||
|
|
|
||
|
|
```sh
|
||
|
|
python3 -m pip install .
|
||
|
|
```
|
||
|
|
|
||
|
|
To build a source archive and a wheel with the `build` frontend and installed
|
||
|
|
Setuptools/Wheel tooling:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
python3 -m build --no-isolation
|
||
|
|
```
|
||
|
|
|
||
|
|
Source archives include a Cargo lockfile, and wheel builds use `--locked` so
|
||
|
|
building from an archive retains the resolved Rust dependency versions. Wheels
|
||
|
|
are tagged for the build platform and use the Python-independent `py3-none` ABI.
|
||
|
|
The CLI continues to build directly with Cargo and has no Python dependency.
|
||
|
|
|
||
|
|
For source-tree development, point the CFFI loader at a Cargo-built library:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
cargo build --release --lib
|
||
|
|
PHEON_LIBRARY="$PWD/target/release/libpheon.so" \
|
||
|
|
python3 -m unittest discover -s tests/python -v
|
||
|
|
```
|
||
|
|
|
||
|
|
Use `libpheon.dylib` on macOS or `pheon.dll` on Windows. Without
|
||
|
|
the explicit override, the loader uses the library bundled inside the package.
|
||
|
|
After installing a wheel, run its tests outside the source checkout to exercise
|
||
|
|
the installed package:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
python3 -I -m unittest discover -s /absolute/path/to/pheon/tests/python -v
|
||
|
|
```
|
||
|
|
|
||
|
|
The Python tests cover multidimensional and noncontiguous packing, input
|
||
|
|
preservation, range errors, and the C declarations for Arrow exports and release
|
||
|
|
callbacks, input lifetimes and ABI mismatches. Run `cargo test` and
|
||
|
|
`cargo test --release` for native validation. See [release validation](docs/releases.md)
|
||
|
|
for wheel builds, platform repair and testing the same artifact on multiple
|
||
|
|
Python versions.
|
||
|
|
|
||
|
|
## Layout review CLI
|
||
|
|
|
||
|
|
The review CLI is a standalone Rust executable:
|
||
|
|
|
||
|
|
```sh
|
||
|
|
cargo build --bin pheon
|
||
|
|
target/debug/pheon dump chip.gds
|
||
|
|
target/debug/pheon dump chip.gds --records
|
||
|
|
target/debug/pheon diff old.gds new.gds --hierarchy ascii
|
||
|
|
target/debug/pheon diff old.gds new.gds --hierarchy mermaid --max-changes 20
|
||
|
|
target/debug/pheon hierarchy chip.gds --format mermaid --depth 4
|
||
|
|
target/debug/pheon diff old.gds new.gds --top CHIP --layer 1/0
|
||
|
|
target/debug/pheon diff old.gds new.gds --show-layer 1/0
|
||
|
|
target/debug/pheon diff old.gds new.gds --format json
|
||
|
|
target/debug/pheon diff old.gds new.gds --format json --hierarchy graph
|
||
|
|
target/debug/pheon name-layers chip.gds named.gds --cell CHIP --carrier 900/7 --name 1:0:WG
|
||
|
|
```
|
||
|
|
|
||
|
|
The object dump is deterministic and omits source offsets. Record dumps preserve
|
||
|
|
record order and show offsets, tags, sizes, and values, including records the
|
||
|
|
object reviewer does not support. They can inspect an invalid metadata declaration
|
||
|
|
without invoking object-level validation. Zero padding after ENDLIB is accepted.
|
||
|
|
|
||
|
|
The fixed `klamath-object-v2` comparison profile:
|
||
|
|
|
||
|
|
- Matches cells by name and layers by numeric pair. The default scope is the
|
||
|
|
entire library; explicit selectors are recorded in each report.
|
||
|
|
- Ignores library/cell timestamps, cell order, element order, and property order.
|
||
|
|
- Normalizes boundary/box starting vertex and winding, retaining other vertices.
|
||
|
|
- Preserves duplicate objects and duplicate properties as multisets.
|
||
|
|
- Compares paths, labels, references, array endpoints/counts, properties, and all
|
||
|
|
retained flags exactly. Explicit defaults and their omission remain distinct;
|
||
|
|
SREFs and single-element AREFs remain distinct. REAL8 encodings are retained.
|
||
|
|
- Reports library fields, including UNITS, separately. It does not rescale files
|
||
|
|
into a shared grid, infer cell renames, or perform geometric Boolean operations.
|
||
|
|
- Uses bounding boxes as coordinate envelopes, not evidence of equality. Path
|
||
|
|
envelopes do not include stroke width; text/reference envelopes are coordinates,
|
||
|
|
not rendered or recursively transformed bounds.
|
||
|
|
- Returns an incomplete result for unsupported records, future metadata versions,
|
||
|
|
unresolved references, or cycles. These cannot produce an equality claim.
|
||
|
|
|
||
|
|
`diff` exits with **0** for equality under this profile, **1** for differences,
|
||
|
|
and **2** for invalid input or incomplete comparison. `--max-changes` limits the
|
||
|
|
number of object details displayed, not the comparison. Source offsets are
|
||
|
|
included in changed-object reports for tracing changes back to record dumps.
|
||
|
|
|
||
|
|
`diff --format json` emits a versioned report with typed before/after evidence,
|
||
|
|
structured diagnostics, and independent comparison status and completeness.
|
||
|
|
JSON includes all changes by default. Explicit display filters and `--max-changes`
|
||
|
|
record omitted evidence without changing totals or exit status. `--hierarchy graph`
|
||
|
|
adds the complete scoped reference graph, with symbolic array multiplicities;
|
||
|
|
`--depth` applies to text output only. Timestamp-only changes still return 0.
|
||
|
|
Input and argument errors also produce JSON when requested. See the
|
||
|
|
[report contract](docs/diff-reports.md) and
|
||
|
|
[version 1 JSON Schema](docs/diff-report-v1.schema.json).
|
||
|
|
|
||
|
|
Unmatched objects are paired for explanation only when candidates are unique by
|
||
|
|
type/layer/bounding box, then by type/layer among the remaining objects.
|
||
|
|
References additionally match by target. Ambiguous groups stay additions and
|
||
|
|
removals. Reports explain reference movement, array-count changes, path widths,
|
||
|
|
and polygons changed within the same bounds. Full before/after records remain
|
||
|
|
visible, and duplicate counts remain significant. Unchanged layer names appear
|
||
|
|
once as a legend for changed objects on those layers.
|
||
|
|
|
||
|
|
### Comparison scope and display filters
|
||
|
|
|
||
|
|
`--top NAME` compares that cell and its reachable subtree independently in each
|
||
|
|
revision. `--cell NAME` compares the exact cell, including its reference records,
|
||
|
|
without comparing descendant contents. Combine the two to select a cell within
|
||
|
|
a subtree. A selector absent from both revisions is an error; absence on one
|
||
|
|
side is an addition/deletion.
|
||
|
|
|
||
|
|
`--layer L/T` selects objects on a numeric layer/type pair, plus references whose
|
||
|
|
targets use that layer transitively. Cells with no relevant content are omitted.
|
||
|
|
Unresolved references, cycles, and unsupported records with unknown layer effect
|
||
|
|
remain incomplete results. Unsupported objects on another known layer can be
|
||
|
|
excluded. Library fields, including units, remain compared. Unsupported layer-name
|
||
|
|
version diagnostics are always retained, since those declarations can affect any
|
||
|
|
layer or cell in the library. Scoped dumps and
|
||
|
|
hierarchy views accept the same selectors. Scope selection parses the complete
|
||
|
|
input: malformed record grammar and conflicting metadata are still errors.
|
||
|
|
|
||
|
|
`--show-cell NAME` and `--show-layer L/T` filter object details in a diff report
|
||
|
|
without changing comparison status, totals, metadata changes, or hierarchy.
|
||
|
|
References have no numeric layer and are hidden by `--show-layer`. The comparison
|
||
|
|
scope and any display-only filter are printed explicitly. The Rust API accepts
|
||
|
|
one combined `Scope`; applying another scope to an already scoped document is
|
||
|
|
rejected rather than losing reference coverage information.
|
||
|
|
|
||
|
|
Hierarchy output distinguishes directly changed cells, affected ancestors, and
|
||
|
|
unchanged targets of modified references. It includes edges from both revisions,
|
||
|
|
so removed branches remain visible. Counts are direct edge multiplicities,
|
||
|
|
including arrays, rather than expanded placements. Shared definitions are marked
|
||
|
|
as already shown in text and appear once in Mermaid. `--depth` limits displayed
|
||
|
|
descendants only. The CLI emits Mermaid source, requiring no diagram renderer.
|
||
|
|
|
||
|
|
Layer names are read from labels in an existing cell. A label with the exact
|
||
|
|
string `KLAMATH_LAYER_NAMES_V1` identifies an annotation carrier `(layer, texttype)`
|
||
|
|
pair within that cell. All other labels on that same pair in that same cell must
|
||
|
|
be `layer:datatype:name` declarations, for example `1:0:WG`. The carrier pair is
|
||
|
|
not fixed; choose a suitable annotation layer in the design's technology.
|
||
|
|
Declarations split at the first two colons, allowing colons in names. Numeric
|
||
|
|
fields are unsigned 16-bit decimal integers. Names are nonempty UTF-8 strings.
|
||
|
|
Repeated identical declarations are accepted; conflicting declarations fail.
|
||
|
|
Names are collected across the library. Recognized metadata labels are omitted
|
||
|
|
from object comparison; mappings are compared separately and printed once per
|
||
|
|
numeric pair, so renaming a layer does not rewrite every shape description.
|
||
|
|
|
||
|
|
`name-layers INPUT OUTPUT --cell NAME --carrier L/T --name L:T:NAME ...` writes
|
||
|
|
this convention. It consolidates existing recognized declarations into one legend
|
||
|
|
in the chosen existing cell, retains existing names not explicitly updated, and
|
||
|
|
overrides the named pairs. The chosen carrier must contain no ordinary design
|
||
|
|
objects in that cell. Unrelated records, timestamps, unknown records and trailing
|
||
|
|
padding are copied byte-for-byte. No structures or references are added. Output
|
||
|
|
must be a new file: the CLI refuses to overwrite an existing destination, including
|
||
|
|
the input. Invalid declarations, conflicting arguments, oversized labels and
|
||
|
|
future metadata versions fail before output creation.
|
||
|
|
|
||
|
|
The record-preserving review adapter retains information normalized or omitted
|
||
|
|
by the Arrow geometry reader; existing Arrow APIs are unchanged. Structured Rust
|
||
|
|
change records feed the text and hierarchy renderers. LayoutEditor's `LayerNames`
|
||
|
|
extension, layout extraction/remapping, and geometric Boolean comparison remain
|
||
|
|
outside the current CLI.
|
||
|
|
|
||
|
|
### Git integration
|
||
|
|
|
||
|
|
Install the executable on PATH with `cargo install --path . --bin pheon`.
|
||
|
|
The templates in [contrib/git/config](contrib/git/config) and
|
||
|
|
[contrib/git/attributes](contrib/git/attributes) provide two alternative drivers.
|
||
|
|
For structural summaries and affected hierarchy, add this attribute:
|
||
|
|
|
||
|
|
```gitattributes
|
||
|
|
*.gds diff=pheon
|
||
|
|
```
|
||
|
|
|
||
|
|
Configure the native external driver in Git config:
|
||
|
|
|
||
|
|
```gitconfig
|
||
|
|
[diff "pheon"]
|
||
|
|
command = pheon git-diff --hierarchy ascii --
|
||
|
|
binary = true
|
||
|
|
trustExitCode = false
|
||
|
|
```
|
||
|
|
|
||
|
|
The adapter consumes Git's old/new temporary files directly, handles additions,
|
||
|
|
deletions and renames, and quotes display paths. It returns success after rendering
|
||
|
|
either equality or differences, as expected by the external-driver protocol.
|
||
|
|
Invalid or incomplete comparisons retain a nonzero exit and diagnostics. Scope
|
||
|
|
and display options can precede the final `--` in the configured command. Existing
|
||
|
|
repositories using `diff=layout` can put the same command under `[diff "layout"]`.
|
||
|
|
|
||
|
|
For ordinary Git line diffs, use canonical textconv instead:
|
||
|
|
|
||
|
|
```gitattributes
|
||
|
|
*.gds diff=pheon-text
|
||
|
|
```
|
||
|
|
|
||
|
|
Configure the driver in Git config:
|
||
|
|
|
||
|
|
```gitconfig
|
||
|
|
[diff "pheon-text"]
|
||
|
|
textconv = pheon dump
|
||
|
|
binary = true
|
||
|
|
cachetextconv = true
|
||
|
|
```
|
||
|
|
|
||
|
|
This is a human review representation, not an applicable GDS patch. Invalidate
|
||
|
|
Git's textconv cache when changing the dump profile/tool version. For automated
|
||
|
|
equality checks, use `pheon diff` and its exit status directly.
|
||
|
|
|
||
|
|
`tests/review.rs` contains ten initial fixture pairs, checked-in canonical/text/
|
||
|
|
Mermaid goldens, metadata byte-preservation checks, scope and report regressions,
|
||
|
|
and actual Git integration tests. It also checks near-limit polygon records,
|
||
|
|
2048-cell hierarchies and arrays representing over a billion placements without
|
||
|
|
expanding them. Run `cargo test` and `cargo test --release` for debug and optimized
|
||
|
|
validation. No KLayout installation or Python package is required.
|