Skip to content

CHANGELOG.md

Latest commit

 

History

History
57 lines (40 loc) · 3.07 KB

CHANGELOG.md

File metadata and controls

57 lines (40 loc) · 3.07 KB
Raw

Changelog

All notable changes to pymemex are documented here.

The format is based on Keep a Changelog.

[Unreleased]

[0.2.0] - 2026-04-15

Added

  • RecoveryConfig (pydantic v2) — declarative config for legacy → rmcp-memex mapping. Fields: text/id/vector column overrides, namespace (static) or namespace_from_column (per-row), metadata_include, min_importance, permanent_only, include_embeddings.
  • auto_detect_columns() — heuristic column discovery: picks id from ("id", "uuid", "_id"), text from ("text", "summary", "content", "body"), vector as first fixed_size_list<float*>.
  • recover() generator — iterates source dataset, applies filters + namespace injection, yields rmcp-memex-compatible row dicts with fields: id, namespace, text, vector, content_hash, metadata.
  • pymemex recover CLI command — 11 options; writes JSONL or stdout; supports --dry-run stats mode.
  • Tests — tests/test_recover.py (unit + integration + round-trip) + TestRecoverCommand in tests/test_cli.py (CliRunner).

Changed

  • Package positioning: introspection → recovery + simplify + namespace-injection engine. pyproject.toml description and __init__.py docstring updated.

Migration note

Flat legacy datasets (e.g. Python-lance ad-hoc tables without namespace) can now be turned into rmcp-memex-importable JSONL in one command. Target schema includes namespace field required by rmcp-memex mcp_documents.lance.

[0.1.0] - 2026-04-15

Added

  • Initial package scaffold (uv + hatchling + pyproject PEP 621).
  • core.detect_layout() — auto-detects legacy-root vs v2-subtable Lance layout.
  • core.open_dataset() — unified open path for both layouts; returns DatasetInfo.
  • core.list_subtables() — enumerates .lance subtables under v2 db path.
  • schema.inspect_columns() — column-wise schema with vector detection.
  • schema.row_count(), schema.version_summary(), schema.has_vector_column().
  • cli inspect — shows layout, subtables, schema, row count, last 5 versions.
  • cli sample — row sampling with --hide-vectors (default) / --show-vectors.
  • cli tables — subtable listing for v2 layout.
  • BUSL-1.1 license (Change Date: 2030-04-15).

Tests

  • 32 tests, 95% line coverage (pytest + pytest-cov).
  • test_roundtrip.py — end-to-end write-read contract: build known dataset with pyarrow.Table + lance.write_dataset, then verify every pymemex read path (detect_layout, open_dataset, inspect_columns, row_count, full row content equality) returns the exact known input.
  • test_core.py — layout detection (legacy + v2 + multi-table + empty + file).
  • test_schema.py — column types, vector detection (dim inference), row count.
  • test_cli.py — click CliRunner smoke tests for inspect/sample/tables.
  • conftest.py — shared legacy_lance, v2_lance, v2_multi_table_lance fixtures.

Scope note

All operations are read-only. pymemex never mutates the source Lance dataset. Future export and map commands will write to separate output files/paths.