Finding data is only the first discovery problem. The second is discovering what the data mean. Scientific variety contains information: community definitions and source distinctions must survive synthesis.
A research prototype for human-governed dataset creation: discover observations, preserve source metadata and semantic evidence, review scientific harmonization recipes, then stream approved harmonized outputs into machine representations with source and decision lineage.
AI chooses among tested scientific operations. It does not invent the analysis.
VERDE interprets, selects, parameterizes, compares and proposes within a versioned JSON backbone. A closed operation registry owns the scientific computation. Schema/scientific validation and human review govern execution; approved recipes run with the model off. Model trust boundary.
Research website · Documentation · Current status · Architecture audit
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev,embeddings]'
prototype run --all-tracks --output run/offline
prototype mosaiks-demo --output run/mosaiks-example
prototype operation-demo --output run/operation-trust-example
prototype management-demo --output run/amber-workbench --metadata-records 10000
pytest -q
mkdocs build --strict
prototype workbench --registry run/amber-workbench/registry --port 8787Fixtures and MOSAIKS demonstration inputs are explicitly generated. No source data or models are downloaded by these commands. MOSAIKS consumes harmonized rasters, fits empirical filters on bounded training-only references, and produces numeric shards, compact lineage, separate labels and held-out diagnostics with baselines. Derived texture diagnostics do not establish ecological validity or native LLM understanding.
The local data management workbench connects
existing MOSAIKS outputs to an immutable SQLite artifact/lineage index,
content-addressed files, portable JSONL registries, cohort feedback, resource
plans and actual snapshot/restore tests. Open http://127.0.0.1:8787/ to inspect
the demo. Scale probes measure generated metadata indexing, not global raster
throughput. Three scientific products and remote partner services remain planned.
Package code lives in src/dataset_prototype/; offline tests in tests/;
small synthetic fixtures in data/fixtures/; scientific track configuration in
tracks/; website and technical documentation in docs/.
- Discovery & access: scientific requests → source metadata and explicit access plans.
- Harmonization & review: evidence → frozen recipes, scoped findings, deterministic QA and human decisions.
- Representation: reviewed harmonized raster streams → MOSAIKS vectors and verification.
The orchestrator composes typed, independently callable modules. Food and water tracks share immutable source/core resources while retaining separate assumptions and product identities. The third track remains disabled.
Discovery now records data + metadata + semantic evidence using typed source annotations and concept candidates. Module 2 owns mapping and harmonization. The project composes community semantic resources rather than replacing them with a universal ontology. See the Semantics resource and ontology research framework, dataset benchmarking and working metadata diagnostics. Automatic ontology composition and evidence-reviewed dataset potential/diversity remain planned research, separate from implemented evidence preservation.
Sentinel STAC metadata and bounded COG access are implemented. Historical tiny real pixel reads and real model proposals exist; rejected Sentinel reviews have not produced an approved real MOSAIKS dataset. New real-data paths stream bounded windows without persisting raw or harmonized rasters, including tests.
The isolated OpenClaw 2026.5.18 / Verde adapter provides JSON-only specialist
inference with no model tools or fallback. review-metadata adds evidence-bound
semantic, calibration, support, rights, integrity and quality reviews plus QA synthesis.
Review eligibility never bypasses existing consensus, veto or human gates.
NEON metadata and authenticated-access boundaries exist; normalized observation
processing remains pending. MESA is an explicit unavailable adapter, not a live integration.
Live setup · Streaming · Sentinel case study · Sources and rights
python scripts/generate_figures.py
python scripts/generate_reference.py
python scripts/generate_semantics.py
mkdocs build --strictMarkdown is the source of truth; generated SVG figures and JSON Schemas live beside it. Preserve Material and the existing Pages workflow. See AGENTS.md for scientific, streaming and source-rights governance and design decisions for the reasoning behind the implementation.
CubeDynamics is an external ESIIL dependency,
accessed through seven registered capabilities rather than model-written Python.
Install .[dev,cubedynamics] under Python 3.12 and run
prototype cubedynamics-demo --output run/cubedynamics-example for generated lazy
windows → harmonization → real CubeDynamics → MOSAIKS. Core semantic review remains
independent of this optional package. Integration and limits.
The 38-page pipeline state report explains the scientific story with 34 reproducible figures. Editable sources, evidence and rebuild instructions preserve the distinction between historical real observations, generated execution and planned validation.
The validation findings now separate measured graph structure and live model behavior from provisional semantic assessments and blocked scientific workflows. The pass includes 25 observation specifications, 18 scoped candidate records, 153 pairs, three real vocabulary graphs and 100 live inert Verde calls. No real scientific release or human approval is claimed.
Scientific validation PDF · Reproducible evidence and methods · Sentinel expert package · NEON–Sentinel missing-data/method gates.
The year-by-year infrastructure budget separates TACC primary storage, versioned backups, inference allowances and unawarded compute requests.
The data-location and management report maps provider storage, bounded in-memory processing, local artifacts, model records and release bundles. Editable sources and the data inventory separate current implementation from proposed backup, retention and preservation commitments.