From 7ddfc0cd7ff856d06233236003c7fd6cd8ab467b Mon Sep 17 00:00:00 2001 From: Jan Petykiewicz Date: Thu, 2 Apr 2026 20:41:16 -0700 Subject: [PATCH] some docs --- ARROW_SCHEMA.md | 117 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 23 ++++++++++ 2 files changed, 140 insertions(+) create mode 100644 ARROW_SCHEMA.md create mode 100644 README.md diff --git a/ARROW_SCHEMA.md b/ARROW_SCHEMA.md new file mode 100644 index 0000000..f80112b --- /dev/null +++ b/ARROW_SCHEMA.md @@ -0,0 +1,117 @@ +# Arrow Schema Documentation + +This project uses Apache Arrow to represent GDSII (Graphic Data System II) library data. The schema is optimized for memory efficiency and fast access, employing several packed representations for common data types. + +## Top-Level Structure + +The top-level object is a `StructArray` representing the entire GDSII library. + +| Field | Type | Description | +|---|---|---| +| `meters_per_db_unit` | `Float64` | Database unit size in meters. | +| `user_units_per_db_unit` | `Float64` | Number of user units in one database unit. | +| `lib_name` | `Utf8` | The name of the GDSII library. | +| `mod_time` | `FixedSizeList(Int16, 6)` | Last modification time [year, month, day, hour, minute, second]. | +| `acc_time` | `FixedSizeList(Int16, 6)` | Last access time [year, month, day, hour, minute, second]. | +| `cells` | `List(Cell)` | A list of cells (structures) in the library. | +| `cell_names` | `List(Utf8)` | Names of all cells, indexed by their ID. | +| `layers` | `List(UInt32)` | List of packed layer/datatype pairs, indexed by their ID. | + +## Cell Structure + +Each cell in the `cells` list is a `StructArray`. + +| Field | Type | Description | +|---|---|---| +| `id` | `UInt32` | Unique identifier for the cell. | +| `file_offset` | `UInt64` | Byte offset of the cell in the original GDSII file. | +| `srefs` | `List(SRef)` | Simple structure references (no properties). | +| `arefs` | `List(ARef)` | Array references (no properties). | +| `sref_props` | `List(SRefProp)` | Structure references with properties. | +| `aref_props` | `List(ARefProp)` | Array references with properties. | +| `rect_batches` | `List(RectBatch)` | Batches of rectangular boundaries (optimized). | +| `boundary_batches` | `List(BoundaryBatch)` | Batches of generic boundaries. | +| `boundary_props` | `List(BoundaryProp)` | Individual boundaries with properties. | +| `paths` | `List(Path)` | Path elements. | +| `nodes` | `List(BoxNode)` | Node elements. | +| `boxes` | `List(BoxNode)` | Box elements. | +| `texts` | `List(Text)` | Text labels. | + +## Element Types + +### Common Packed Encodings + +- **Layer ID (`UInt32`)**: A packed representation of `(layer, datatype)`. + - `layer = (layer_id >> 16) as i16` + - `datatype = (layer_id & 0xFFFF) as i16` +- **XY Coordinate (`UInt64`)**: A packed representation of a single point `(x, y)`. + - `x = (xy >> 32) as i32` + - `y = (xy & 0xFFFFFFFF) as i32` +- **Array Counts (`UInt32`)**: A packed representation of `(columns, rows)`. + - `columns = (counts >> 16) as i16` + - `rows = (counts & 0xFFFF) as i16` + +### Reference Elements (SRef / ARef) + +References point to other cells in the library. + +**SRef (Simple Reference)**: +- `target`: `UInt32` (Cell ID) +- `invert_y`: `Boolean` (Vertical flip) +- `scale`: `Float64` (Magnification) +- `angle_rad`: `Float64` (Rotation in radians) +- `xy`: `UInt64` (Packed origin point) + +**ARef (Array Reference)**: +- Same as SRef, plus: +- `xy0`: `UInt64` (Packed column basis vector increment) +- `xy1`: `UInt64` (Packed row basis vector increment) +- `counts`: `UInt32` (Packed columns and rows) + +**SRefProp / ARefProp**: +- Same as above, plus a `properties` field. + +### Boundary Elements + +Boundaries are grouped into batches for efficiency when they share properties or are simple rectangles. + +**RectBatch**: +- `layer`: `UInt32` (Packed Layer ID) +- `rects`: `List(Int32)` (Flattened list of `[min_x, min_y, max_x, max_y]` for each rectangle) + +**BoundaryBatch**: +- `layer`: `UInt32` (Packed Layer ID) +- `vertices`: `List(Int32)` (Flattened list of all vertices for all boundaries in the batch) +- `vertex_offsets`: `List(UInt32)` (Start index for each boundary in the `vertices` list) + +**BoundaryProp**: +- Individual boundary with `properties`. + +### Other Elements + +**Path**: +- `layer`: `UInt32` +- `path_type`: `Int16` (Optional) +- `extension_start`: `Int32` (Optional) +- `extension_end`: `Int32` (Optional) +- `width`: `Int32` +- `xy`: `List(Int32)` (centerline coordinates) +- `properties`: `List(Property)` + +**Text**: +- `layer`: `UInt32` +- `presentation_horiz`: `UInt8` +- `presentation_vert`: `UInt8` +- `presentation_font`: `UInt8` +- `path_type`: `Int16` +- `width`: `Int32` +- `invert_y`: `Boolean` +- `mag`: `Float64` +- `angle_deg`: `Float64` +- `xy`: `UInt64` (Packed position) +- `string`: `Utf8` +- `properties`: `List(Property)` + +**Property**: +- `key`: `Int16` +- `value`: `Utf8` diff --git a/README.md b/README.md new file mode 100644 index 0000000..3d5b1b1 --- /dev/null +++ b/README.md @@ -0,0 +1,23 @@ +# klamath-rs + +A fast GDSII parser and writer in Rust, with Apache Arrow integration for efficient data handling. + +## Overview + +`klamath-rs` is designed to handle large GDSII files with high performance. It uses Apache Arrow as its primary in-memory data representation, allowing for efficient memory usage and easy integration with other tools in the Arrow ecosystem (like pandas, polars, or DuckDB via Python bindings). + +## Documentation + +- [Arrow Schema Documentation](ARROW_SCHEMA.md): Detailed information about the Arrow schema used for GDSII data. + +## Features + +- Fast parsing of GDSII files. +- Efficient memory representation using Apache Arrow. +- Support for common GDSII elements: SREF, AREF, Boundary, Path, Text, etc. +- Optimized "batch" representations for rectangles and boundaries sharing the same layer. +- C-compatible FFI for integration with other languages (e.g., Python). + +## Usage + +(To be added)