Lacuna data path

Agent projects spend most of their effort upstream of the model call. This pair of repos is the operational answer to that — collect, refine, classify, export — with the JSONL plumbing factored out into a separate, tested library so the pipeline stays composable.

Problem

Move heterogeneous chat / instruct datasets (alpaca, sharegpt, raw chat) from upstream sources (Hugging Face, GitHub) into one consistent JSONL layout the agent project — Lacuna — can train on, with optional LLM-assisted refinement and human-in-the-loop labeling. The hard part is not the model call; it's keeping the ingest and normalization repeatable across new sources.

Constraints

• Single-developer scope, no proprietary infra — everything runs on Prefect, GitHub Actions, RunPod, and Hugging Face.
• Hugging Face / GitHub repos as the wire format, with Git LFS as the typical large-file transport.
• convmerge ships to PyPI and runs on Python 3.10 / 3.11 / 3.12 in CI; its core has zero runtime dependencies (PyYAML / datasets / pyarrow are optional extras).
• GPU work is isolated to a single RunPod worker process — pipeline orchestration stays on CPU.
• data/ directory in lacuna-data-pipeline is intentionally not bundled (only .gitkeep at HEAD); raw datasets are fetched on demand from sources.yaml.

Architecture

Two ingest paths feed one refine → classify → export chain:

• Inputs: sources.yaml + .env (HF_TOKEN, GITHUB_TOKEN, RUNPOD_API_KEY, ANTHROPIC_API_KEY).
• Collect (choose one): Prefect collect_flow does git+LFS clones of HF / GitHub dataset repos OR collect_convmerge_flow uses convmerge fetch (HF snapshot_download / Trees API / git clone) — both write to data/raw.
• Refine: Prefect refine.py walks data/raw → jsonl/ → multi_turn/ + single_turn/ → final/train.jsonl (+ test). Optional refine_llm.py runs Anthropic identity-eval and assistant-rewrite passes.
• Classify: dataset_pipeline batches refined JSONL to the RunPod worker's /classify and /classify/batch HTTP endpoints (FastAPI, scripts/data_worker_api.py).
• Export: export.py converts a Label Studio JSON export (or an existing .jsonl) into messages.jsonl for downstream trainers.
• Library backbone: convmerge ships the four CLI verbs (fetch, normalize, convert, dedupe / turns) the pipeline imports — adapters for alpaca / sharegpt / chat, emitters for messages / alpaca, and four fetch backends (HF datasets, GitHub raw URL, GitHub Trees API, git clone).

Scale (verifiable in the repos at HEAD)

• lacuna-data-pipeline: 7 Prefect flows (collect, collect_convmerge, refine, refine_llm, classify, export, dataset_pipeline), 36 Python files, 234 pinned dependencies in requirements.txt.
• RunPod worker: 5 FastAPI endpoints (/health, /translate, /translate/batch, /classify, /classify/batch) — see scripts/data_worker_api.py.
• convmerge: PyPI v0.3.3, 47 source files / 19 test files, 100 tests passing + 1 skipped (parquet) in CI; 82% line coverage measured locally.
• convmerge surface area: 3 chat adapters (alpaca, sharegpt, chat / auto), 2 emitters (messages, alpaca), 4 fetch backends, 6 optional extras (fetch, fetch-hf, fetch-all, parquet, preset, all).
• CI: GitHub Actions matrix on Python 3.10 / 3.11 / 3.12 (Ruff check + format + pytest); tag-triggered PyPI publish via PYPI_API_TOKEN.

Key decisions

• Two ingest paths kept side-by-side. git + LFS handles small structured dataset repos; convmerge fetch (HF snapshot) handles large gated parquet — picked per-dataset in sources.yaml rather than committing to one transport.
• convmerge core is dependency-free; PyYAML / datasets / pyarrow live in [project.optional-dependencies]. Result: the library installs cleanly inside a classifier-only worker that doesn't need YAML or HF support.
• GPU isolated to one FastAPI process (data_worker_api.py) deployed via runpod_deploy_worker.sh. The pipeline calls it over HTTP — orchestration stays CPU-only and portable.
• Anthropic refinement is opt-in. refine_llm.py (identity-eval, assistant rewrite + translate) is not part of the default refine flow — it's a separate flow you invoke when you want it.
• Public adapter set is intentionally narrow (alpaca, sharegpt, chat). chat is heuristic and routes by record-key inspection; for unusual schemas the docs tell you to pin an explicit adapter rather than over-fit the heuristic.
• Shipping convmerge to PyPI as a sister project (vs. inlining it) means the JSONL plumbing is independently tested, versioned, and reusable outside Lacuna.

What broke / how it was fixed

The documented failure mode — captured in PIPELINE.md — was Git-LFS pointer stubs surviving a failed git lfs pull, leaving collect to write 100-byte text files where parquet should have been. The fix: collect now preflights `git lfs version` before any clone that would use LFS, runs `git lfs install --local` + `git lfs pull` on each clone, and raises if pull fails instead of leaving stubs on disk. There's also a per-dataset opt-out (`git_lfs: false` in sources.yaml) and a manual recovery script (scripts/git_lfs_pull_raw.sh) for already-cloned trees. (No other failure history is reconstructible from the public commit log.)

What I'd change next

• Add a tests/ directory to lacuna-data-pipeline. convmerge is well-tested (82% coverage); the pipeline side has none.
• Replace the 234-line pinned requirements.txt with a lock file (pip-tools or uv). The current pinning makes upgrades cliff-edge, especially for transitive Django / label-studio-sdk pulls.
• Add a typed Pydantic schema for the JSONL records that flow between stages, so refine and classify share one source of truth. Today schema validation only lives inside convmerge.normalize.schema.
• Add a flow-level smoke test (tiny dataset, end-to-end) so collect → refine → classify is exercised on every PR rather than only at run time.
• Surface dataset provenance (source URL, checksum, fetch timestamp) into the final JSONL records, not just the raw clone tree, so downstream training can audit where each row came from.