Organize documentation into focused user guides
This commit is contained in:
parent
b3346b40a4
commit
a384bb2f89
1 changed files with 798 additions and 681 deletions
358
README.md
358
README.md
|
|
@ -1,337 +1,57 @@
|
|||
# Pheon
|
||||
|
||||
A fast GDSII parser and writer in Rust, with Apache Arrow integration for efficient data handling.
|
||||
Pheon reads GDSII layouts into Apache Arrow and provides command-line tools for
|
||||
inspecting and comparing layout revisions. It is written in Rust, with Python
|
||||
bindings and a C API.
|
||||
|
||||
Pheon provides native GDSII codecs and a standalone layout review CLI. The Rust
|
||||
crate, Python package and executable are all named `pheon`. Python Klamath and
|
||||
Masque remain separate consumers. The former `klamath_rs_ext` import and
|
||||
`klamath` executable are not provided as aliases.
|
||||
- Inspect cells, records, and reference hierarchies.
|
||||
- Compare layout objects, with text or JSON reports and Git integration.
|
||||
- Read geometry into Arrow, or scan the hierarchy before loading selected cells.
|
||||
|
||||
## Overview
|
||||
Comparisons ignore timestamps and element order while preserving duplicates and
|
||||
other object details. They compare object fields and coordinates exactly; see the
|
||||
[comparison rules](docs/comparison.md) for supported normalization and limits.
|
||||
|
||||
`pheon` is designed to handle large GDSII files with high performance. It uses Apache Arrow as its primary in-memory data representation, allowing for efficient memory usage and easy integration with other tools in the Arrow ecosystem (like pandas, polars, or DuckDB via Python bindings).
|
||||
## Try the command-line tools
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Arrow Schema Documentation](ARROW_SCHEMA.md): Detailed information about the Arrow schema used for GDSII data.
|
||||
|
||||
## Features
|
||||
|
||||
- Fast parsing of GDSII files.
|
||||
- Efficient memory representation using Apache Arrow.
|
||||
- Support for common GDSII elements: SREF, AREF, Boundary, Path, Text, etc.
|
||||
- Optimized "batch" representations for rectangles and boundaries sharing the same layer.
|
||||
- C-compatible FFI for integration with other languages (e.g., Python).
|
||||
|
||||
REAL8 writing uses normalized GDS encodings and represents every in-range `f64`
|
||||
exactly. Finite magnitudes below `2^-260` underflow to canonical zero; nonfinite
|
||||
values and magnitudes at or above `2^252` are rejected by the encoder.
|
||||
|
||||
## Usage
|
||||
|
||||
`read_path` exports a library through the Arrow C Data Interface. Allocate
|
||||
initialized `ArrowArray` and `ArrowSchema` output structures, then check the
|
||||
returned status: `0` means success and `-1` means failure. Only import the Arrow
|
||||
outputs after success; the caller owns their release callbacks. Failed calls
|
||||
leave the outputs unchanged. `pheon/_ffi.h` declares the C API; `test.py`
|
||||
demonstrates the supported Python reader.
|
||||
|
||||
`last_error_message()` returns a UTF-8, null-terminated error string, or null
|
||||
after a successful parser call. Errors belong to the calling thread. Copy the
|
||||
message before making another parser call on that thread, and before the thread
|
||||
exits; calls on other threads do not invalidate it. Parser panics that unwind
|
||||
are converted into errors.
|
||||
|
||||
For partial reads, call `scan_bytes` with a borrowed GDSII byte buffer to obtain
|
||||
cell ranges and references without decoding geometry. Pass the desired ranges
|
||||
to `read_cells_bytes` to load their full contents. Both calls use the same Arrow
|
||||
outputs and status/error handling as `read_path`:
|
||||
|
||||
```c
|
||||
int scan_bytes(const uint8_t *data, size_t size,
|
||||
struct ArrowArray *array, struct ArrowSchema *schema);
|
||||
int read_cells_bytes(const uint8_t *data, size_t size,
|
||||
const uint64_t *ranges, size_t range_count,
|
||||
struct ArrowArray *array, struct ArrowSchema *schema);
|
||||
```
|
||||
|
||||
`ranges` contains alternating inclusive start and exclusive end offsets, with
|
||||
two native-endian `uint64_t` values per cell. Each range must cover one complete
|
||||
cell. A null `ranges` pointer is accepted when `range_count` is zero. Likewise,
|
||||
null `data` is accepted for zero bytes and produces an ordinary parse error for
|
||||
the empty input. Nonempty buffers must be readable for the duration of the call;
|
||||
range values must be aligned. Exported Arrow data owns its storage independently
|
||||
of the input. See [Arrow Schema Documentation](ARROW_SCHEMA.md) for the scan
|
||||
schema, offset definitions, and ID assignment.
|
||||
|
||||
## Python package and validation
|
||||
|
||||
Python 3.11 or newer is required. Source builds require Rust 1.88 or newer;
|
||||
installing a prebuilt wheel requires no Rust compiler. The shared library uses
|
||||
a C ABI rather than Python's or Rust's native ABI, so one platform wheel serves
|
||||
multiple Python minor versions. CFFI, NumPy and optional PyArrow must themselves
|
||||
support the selected interpreter. A Rust application uses the crate normally
|
||||
and compiles it with its own compatible toolchain. The checkout selects the
|
||||
stable toolchain locally through `rust-toolchain.toml`, without changing your
|
||||
global default. CI checks both Rust 1.88 and current stable.
|
||||
From a source checkout, install the CLI with Rust 1.88 or newer:
|
||||
|
||||
```sh
|
||||
python3 -m pip install 'pheon[arrow]'
|
||||
cargo install --path . --locked --bin pheon
|
||||
pheon dump chip.gds
|
||||
pheon diff before.gds after.gds
|
||||
pheon hierarchy chip.gds --format mermaid
|
||||
```
|
||||
|
||||
`diff` returns 0 for equality, 1 for differences, and 2 for invalid or incomplete
|
||||
comparisons. Start with the [CLI guide](docs/cli.md), then set up
|
||||
[Git integration](docs/git.md) if you review layouts in a repository.
|
||||
|
||||
## Read layouts in Python
|
||||
|
||||
Python 3.11 or newer is required. From a source checkout:
|
||||
|
||||
```sh
|
||||
python -m pip install '.[arrow]'
|
||||
```
|
||||
|
||||
Run Python outside the source checkout to use the installed package:
|
||||
|
||||
```python
|
||||
from pheon import arrow
|
||||
|
||||
library = arrow.read_path('chip.gds') # one-row StructArray; also accepts gzip
|
||||
scan = arrow.scan_bytes(gds_bytes)
|
||||
cells = scan.field('cells')[0].values
|
||||
ranges = list(zip(cells.field('struct_start_offset').to_pylist(),
|
||||
cells.field('struct_end_offset').to_pylist()))
|
||||
selected = arrow.read_cells_bytes(gds_bytes, ranges[:2])
|
||||
layout = arrow.read_path("chip.gds")
|
||||
print(layout.field("cell_names")[0].as_py())
|
||||
```
|
||||
|
||||
`read_bytes` reads a complete borrowed buffer. Full, scanned and selected Arrow
|
||||
results own their storage independently of that buffer. `PheonError` reports
|
||||
native parse failures; malformed range shapes and types raise Python value/type
|
||||
errors before calling native code. Integer packing needs no PyArrow import.
|
||||
Source installation needs Rust; prebuilt Python wheels bundle the native library.
|
||||
The [Python guide](docs/python.md) covers installation, selected reads, and integer
|
||||
packing. The [Arrow reference](docs/arrow-schema.md) describes the returned data.
|
||||
|
||||
### Discovery and companion packages
|
||||
## Learn more
|
||||
|
||||
Klamath's `native` extra installs `pheon`; Masque's `arrow` extra installs
|
||||
`pheon[arrow]`. Klamath imports Pheon's packing helpers, while Masque calls
|
||||
`pheon.arrow` and translates parse failures to its existing error type.
|
||||
Browse the [documentation](docs/index.md) for layer names, native integration,
|
||||
JSON reports, development, and releases. For writing, Pheon provides low-level
|
||||
GDSII record helpers in Rust.
|
||||
|
||||
The Python loader selects the library bundled with the imported `pheon` package,
|
||||
or the exact file selected by `PHEON_LIBRARY`. It checks C ABI version 1 and
|
||||
Arrow layout schema version 1 before exposing native functions. Incompatible
|
||||
or unversioned libraries fail with an import error. Those versions are independent
|
||||
of package releases and the CLI's JSON report schema.
|
||||
|
||||
`cargo install --path . --locked --bin pheon` installs the CLI, not the Python
|
||||
package or a discoverable shared library. Klamath does not search PATH or Cargo
|
||||
directories for it. To reuse an external native build, install the Python
|
||||
components too and set `PHEON_LIBRARY`; an editable source install is useful
|
||||
during development. The old library override variables are not recognized.
|
||||
|
||||
The Python packing helpers require NumPy and CFFI, both declared as runtime
|
||||
dependencies. Setuptools invokes `cargo build --release --lib` and bundles the
|
||||
C ABI shared library in a platform-specific wheel. Its version comes from
|
||||
`Cargo.toml`. The C declarations are in `pheon/_ffi.h`; the loader uses
|
||||
CFFI ABI mode, so no Python extension compilation or Maturin is required.
|
||||
|
||||
```sh
|
||||
python3 -m pip install .
|
||||
```
|
||||
|
||||
To build a source archive and a wheel with the `build` frontend and installed
|
||||
Setuptools/Wheel tooling:
|
||||
|
||||
```sh
|
||||
python3 -m build --no-isolation
|
||||
```
|
||||
|
||||
Source archives include a Cargo lockfile, and wheel builds use `--locked` so
|
||||
building from an archive retains the resolved Rust dependency versions. Wheels
|
||||
are tagged for the build platform and use the Python-independent `py3-none` ABI.
|
||||
The CLI continues to build directly with Cargo and has no Python dependency.
|
||||
|
||||
For source-tree development, point the CFFI loader at a Cargo-built library:
|
||||
|
||||
```sh
|
||||
cargo build --release --lib
|
||||
PHEON_LIBRARY="$PWD/target/release/libpheon.so" \
|
||||
python3 -m unittest discover -s tests/python -v
|
||||
```
|
||||
|
||||
Use `libpheon.dylib` on macOS or `pheon.dll` on Windows. Without
|
||||
the explicit override, the loader uses the library bundled inside the package.
|
||||
After installing a wheel, run its tests outside the source checkout to exercise
|
||||
the installed package:
|
||||
|
||||
```sh
|
||||
python3 -I -m unittest discover -s /absolute/path/to/pheon/tests/python -v
|
||||
```
|
||||
|
||||
The Python tests cover multidimensional and noncontiguous packing, input
|
||||
preservation, range errors, and the C declarations for Arrow exports and release
|
||||
callbacks, input lifetimes and ABI mismatches. Run `cargo test` and
|
||||
`cargo test --release` for native validation. See [release validation](docs/releases.md)
|
||||
for wheel builds, platform repair and testing the same artifact on multiple
|
||||
Python versions.
|
||||
|
||||
## Layout review CLI
|
||||
|
||||
The review CLI is a standalone Rust executable:
|
||||
|
||||
```sh
|
||||
cargo build --bin pheon
|
||||
target/debug/pheon dump chip.gds
|
||||
target/debug/pheon dump chip.gds --records
|
||||
target/debug/pheon diff old.gds new.gds --hierarchy ascii
|
||||
target/debug/pheon diff old.gds new.gds --hierarchy mermaid --max-changes 20
|
||||
target/debug/pheon hierarchy chip.gds --format mermaid --depth 4
|
||||
target/debug/pheon diff old.gds new.gds --top CHIP --layer 1/0
|
||||
target/debug/pheon diff old.gds new.gds --show-layer 1/0
|
||||
target/debug/pheon diff old.gds new.gds --format json
|
||||
target/debug/pheon diff old.gds new.gds --format json --hierarchy graph
|
||||
target/debug/pheon name-layers chip.gds named.gds --cell CHIP --carrier 900/7 --name 1:0:WG
|
||||
```
|
||||
|
||||
The object dump is deterministic and omits source offsets. Record dumps preserve
|
||||
record order and show offsets, tags, sizes, and values, including records the
|
||||
object reviewer does not support. They can inspect an invalid metadata declaration
|
||||
without invoking object-level validation. Zero padding after ENDLIB is accepted.
|
||||
|
||||
The fixed `klamath-object-v2` comparison profile:
|
||||
|
||||
- Matches cells by name and layers by numeric pair. The default scope is the
|
||||
entire library; explicit selectors are recorded in each report.
|
||||
- Ignores library/cell timestamps, cell order, element order, and property order.
|
||||
- Normalizes boundary/box starting vertex and winding, retaining other vertices.
|
||||
- Preserves duplicate objects and duplicate properties as multisets.
|
||||
- Compares paths, labels, references, array endpoints/counts, properties, and all
|
||||
retained flags exactly. Explicit defaults and their omission remain distinct;
|
||||
SREFs and single-element AREFs remain distinct. REAL8 encodings are retained.
|
||||
- Reports library fields, including UNITS, separately. It does not rescale files
|
||||
into a shared grid, infer cell renames, or perform geometric Boolean operations.
|
||||
- Uses bounding boxes as coordinate envelopes, not evidence of equality. Path
|
||||
envelopes do not include stroke width; text/reference envelopes are coordinates,
|
||||
not rendered or recursively transformed bounds.
|
||||
- Returns an incomplete result for unsupported records, future metadata versions,
|
||||
unresolved references, or cycles. These cannot produce an equality claim.
|
||||
|
||||
`diff` exits with **0** for equality under this profile, **1** for differences,
|
||||
and **2** for invalid input or incomplete comparison. `--max-changes` limits the
|
||||
number of object details displayed, not the comparison. Source offsets are
|
||||
included in changed-object reports for tracing changes back to record dumps.
|
||||
|
||||
`diff --format json` emits a versioned report with typed before/after evidence,
|
||||
structured diagnostics, and independent comparison status and completeness.
|
||||
JSON includes all changes by default. Explicit display filters and `--max-changes`
|
||||
record omitted evidence without changing totals or exit status. `--hierarchy graph`
|
||||
adds the complete scoped reference graph, with symbolic array multiplicities;
|
||||
`--depth` applies to text output only. Timestamp-only changes still return 0.
|
||||
Input and argument errors also produce JSON when requested. See the
|
||||
[report contract](docs/diff-reports.md) and
|
||||
[version 1 JSON Schema](docs/diff-report-v1.schema.json).
|
||||
|
||||
Unmatched objects are paired for explanation only when candidates are unique by
|
||||
type/layer/bounding box, then by type/layer among the remaining objects.
|
||||
References additionally match by target. Ambiguous groups stay additions and
|
||||
removals. Reports explain reference movement, array-count changes, path widths,
|
||||
and polygons changed within the same bounds. Full before/after records remain
|
||||
visible, and duplicate counts remain significant. Unchanged layer names appear
|
||||
once as a legend for changed objects on those layers.
|
||||
|
||||
### Comparison scope and display filters
|
||||
|
||||
`--top NAME` compares that cell and its reachable subtree independently in each
|
||||
revision. `--cell NAME` compares the exact cell, including its reference records,
|
||||
without comparing descendant contents. Combine the two to select a cell within
|
||||
a subtree. A selector absent from both revisions is an error; absence on one
|
||||
side is an addition/deletion.
|
||||
|
||||
`--layer L/T` selects objects on a numeric layer/type pair, plus references whose
|
||||
targets use that layer transitively. Cells with no relevant content are omitted.
|
||||
Unresolved references, cycles, and unsupported records with unknown layer effect
|
||||
remain incomplete results. Unsupported objects on another known layer can be
|
||||
excluded. Library fields, including units, remain compared. Unsupported layer-name
|
||||
version diagnostics are always retained, since those declarations can affect any
|
||||
layer or cell in the library. Scoped dumps and
|
||||
hierarchy views accept the same selectors. Scope selection parses the complete
|
||||
input: malformed record grammar and conflicting metadata are still errors.
|
||||
|
||||
`--show-cell NAME` and `--show-layer L/T` filter object details in a diff report
|
||||
without changing comparison status, totals, metadata changes, or hierarchy.
|
||||
References have no numeric layer and are hidden by `--show-layer`. The comparison
|
||||
scope and any display-only filter are printed explicitly. The Rust API accepts
|
||||
one combined `Scope`; applying another scope to an already scoped document is
|
||||
rejected rather than losing reference coverage information.
|
||||
|
||||
Hierarchy output distinguishes directly changed cells, affected ancestors, and
|
||||
unchanged targets of modified references. It includes edges from both revisions,
|
||||
so removed branches remain visible. Counts are direct edge multiplicities,
|
||||
including arrays, rather than expanded placements. Shared definitions are marked
|
||||
as already shown in text and appear once in Mermaid. `--depth` limits displayed
|
||||
descendants only. The CLI emits Mermaid source, requiring no diagram renderer.
|
||||
|
||||
Layer names are read from labels in an existing cell. A label with the exact
|
||||
string `KLAMATH_LAYER_NAMES_V1` identifies an annotation carrier `(layer, texttype)`
|
||||
pair within that cell. All other labels on that same pair in that same cell must
|
||||
be `layer:datatype:name` declarations, for example `1:0:WG`. The carrier pair is
|
||||
not fixed; choose a suitable annotation layer in the design's technology.
|
||||
Declarations split at the first two colons, allowing colons in names. Numeric
|
||||
fields are unsigned 16-bit decimal integers. Names are nonempty UTF-8 strings.
|
||||
Repeated identical declarations are accepted; conflicting declarations fail.
|
||||
Names are collected across the library. Recognized metadata labels are omitted
|
||||
from object comparison; mappings are compared separately and printed once per
|
||||
numeric pair, so renaming a layer does not rewrite every shape description.
|
||||
|
||||
`name-layers INPUT OUTPUT --cell NAME --carrier L/T --name L:T:NAME ...` writes
|
||||
this convention. It consolidates existing recognized declarations into one legend
|
||||
in the chosen existing cell, retains existing names not explicitly updated, and
|
||||
overrides the named pairs. The chosen carrier must contain no ordinary design
|
||||
objects in that cell. Unrelated records, timestamps, unknown records and trailing
|
||||
padding are copied byte-for-byte. No structures or references are added. Output
|
||||
must be a new file: the CLI refuses to overwrite an existing destination, including
|
||||
the input. Invalid declarations, conflicting arguments, oversized labels and
|
||||
future metadata versions fail before output creation.
|
||||
|
||||
The record-preserving review adapter retains information normalized or omitted
|
||||
by the Arrow geometry reader; existing Arrow APIs are unchanged. Structured Rust
|
||||
change records feed the text and hierarchy renderers. LayoutEditor's `LayerNames`
|
||||
extension, layout extraction/remapping, and geometric Boolean comparison remain
|
||||
outside the current CLI.
|
||||
|
||||
### Git integration
|
||||
|
||||
Install the executable on PATH with `cargo install --path . --bin pheon`.
|
||||
The templates in [contrib/git/config](contrib/git/config) and
|
||||
[contrib/git/attributes](contrib/git/attributes) provide two alternative drivers.
|
||||
For structural summaries and affected hierarchy, add this attribute:
|
||||
|
||||
```gitattributes
|
||||
*.gds diff=pheon
|
||||
```
|
||||
|
||||
Configure the native external driver in Git config:
|
||||
|
||||
```gitconfig
|
||||
[diff "pheon"]
|
||||
command = pheon git-diff --hierarchy ascii --
|
||||
binary = true
|
||||
trustExitCode = false
|
||||
```
|
||||
|
||||
The adapter consumes Git's old/new temporary files directly, handles additions,
|
||||
deletions and renames, and quotes display paths. It returns success after rendering
|
||||
either equality or differences, as expected by the external-driver protocol.
|
||||
Invalid or incomplete comparisons retain a nonzero exit and diagnostics. Scope
|
||||
and display options can precede the final `--` in the configured command. Existing
|
||||
repositories using `diff=layout` can put the same command under `[diff "layout"]`.
|
||||
|
||||
For ordinary Git line diffs, use canonical textconv instead:
|
||||
|
||||
```gitattributes
|
||||
*.gds diff=pheon-text
|
||||
```
|
||||
|
||||
Configure the driver in Git config:
|
||||
|
||||
```gitconfig
|
||||
[diff "pheon-text"]
|
||||
textconv = pheon dump
|
||||
binary = true
|
||||
cachetextconv = true
|
||||
```
|
||||
|
||||
This is a human review representation, not an applicable GDS patch. Invalidate
|
||||
Git's textconv cache when changing the dump profile/tool version. For automated
|
||||
equality checks, use `pheon diff` and its exit status directly.
|
||||
|
||||
`tests/review.rs` contains ten initial fixture pairs, checked-in canonical/text/
|
||||
Mermaid goldens, metadata byte-preservation checks, scope and report regressions,
|
||||
and actual Git integration tests. It also checks near-limit polygon records,
|
||||
2048-cell hierarchies and arrays representing over a billion placements without
|
||||
expanding them. Run `cargo test` and `cargo test --release` for debug and optimized
|
||||
validation. No KLayout installation or Python package is required.
|
||||
Licensed under [GPL-3.0-only](LICENSE.md).
|
||||
|
|
|
|||
Loading…
Reference in a new issue