Managing computational analyses across iterations requires knowing how every dataset, transformation, and figure came into existence. In complex scientific projects, a single script modification or an unrecorded dependency change can invalidate weeks of downstream findings. Establishing robust research data versioning lineage ensures that every input file, parameter, intermediate matrix, and output plot remains auditable and reproducible across distributed teams.

In this practical guide, you will learn how to design a two-tiered provenance tracking system. You will use datalad to handle file-level dataset versioning and execution provenance, alongside lamindb to manage biological entities, schema validations, and fine-grained data object lineages.

---

Establishing Research Data Versioning Lineage Across Files and Objects

Scientific computing often suffers from a dual-layer tracking problem. On one level, raw binary assets, container images, and script repositories must be tracked as whole files on disk. On another level, domain-specific data structures—such as single-cell objects containing sample metadata, gene annotations, and embeddings—require internal schema validation and feature-level tracking.

DataLad operates primarily at the filesystem and Git/git-annex layer. It excels at capturing shell command executions, external data references, and large artifact movement without burdening standard version control systems. In contrast, LaminDB operates closer to the data science runtime, parsing datasets into structured entities, registering transformations, and validating metadata ontologies. Combining both tools bridges the gap between infrastructure tracking and domain data governance.

---

Step 1: Isolate the Computational Environment and Inspect Resources

Before tracking data transformations, lock your software dependencies and check system capacity. Conda creates isolated software environments that prevent package conflicts across projects.

  1. Define an environment.yml file declaring the channels and pinned package versions necessary to recreate your environment.
  2. If you need to record the exact builds of all installed packages, run conda env export to generate a comprehensive snapshot.
  3. Before dispatching resource-intensive data processing runs, query your local machine or compute cluster using the get-available-resources skill to verify available CPU cores, memory limits, and storage mount points.

This step prevents mid-pipeline memory exhaustion and ensures that downstream collaborators execute your scripts in an identical software context.

---

Step 2: Initialize a DataLad Dataset for File-Level Tracking

DataLad wraps Git and git-annex, allowing you to track small code files in regular version control while offloading large assets to an annex.

Begin by initializing a top-level DataLad dataset to store your scripts, inputs, and execution logs:

# Create the root dataset
datalad create -c text2git experiment-workspace
cd experiment-workspace

# Verify resource availability before allocating scratch space
# (Optionally invoke system checks via your agent runner)

# Track an initial raw data file or download asset
datalad download-url https://example.org/data/sample_raw.h5 \
  -m "Ingest raw single-cell matrix" \
  -O data/raw/sample_raw.h5

# Save state
datalad save -m "Initialize project dataset with raw data input"

When you process data using DataLad, run commands via datalad run. This command captures the exact input files, the executable command, and the resulting output files directly within a Git commit message, preserving full execution provenance on the filesystem.

---

Step 3: Configure LaminDB for Object-Level and Metadata Lineage

While DataLad records that a file was modified, domain workflows often process complex scientific containers. For example, HDF5 is a binary format for large hierarchical datasets with chunking and compression. AnnData (.h5ad) builds on HDF5 to store annotated single-cell data matrices together with observation and variable metadata. LaminDB provides native support for tracking these internal dimensions, features, and biological labels.

Initialize an instance and record an analysis script using the Python API:

import lamindb as ln
import scanpy as sc

# Initialize or connect to an existing LaminDB instance
ln.connect("my-organization/single-cell-lineage")

# Track the execution context (the script or notebook running this code)
ln.track()

# Load an existing AnnData object stored in an HDF5-backed format
adata_path = "data/raw/sample_raw.h5ad"
adata = sc.read_h5ad(adata_path)

# Normalize and filter
sc.pp.filter_genes(adata, min_cells=3)
sc.pp.normalize_total(adata)
sc.pp.log1p(adata)

# Register the processed AnnData artifact back into LaminDB
output_path = "data/processed/normalized_filtered.h5ad"
adata.write_h5ad(output_path)

artifact = ln.Artifact(
    output_path,
    description="Filtered and log-normalized single-cell matrix",
    type="dataset"
).save()

# Finish the execution tracking step
ln.finish()

By tracking through LaminDB, your database records which script created normalized_filtered.h5ad, which input artifact it derived from, and which feature sets (such as gene identifiers) are contained within the .h5ad file.

---

Step 4: Integrate with Multi-Step Workflow Managers

For automated end-to-end pipelines, manual execution of DataLad or Python scripts is insufficient. Workflow managers such as Snakemake and Nextflow define multi-step analyses as rules or processes with declared inputs and outputs, enabling reruns, parallel execution, and portable pipelines.

To merge your tracking layers within a workflow manager:

  • Define workflow rules where each rule accepts declared inputs and yields declared outputs.
  • In each rule's execution script, invoke ln.track() at the start and ln.finish() at completion to log execution runs into LaminDB.
  • Wrap pipeline invocations with datalad run to commit the entire pipeline state, including workflow manager lockfiles and generated output artifacts, into Git history.

This creates a nested provenance tree: Snakemake or Nextflow orchestrates task dependencies, DataLad seals file changes at the repository boundary, and LaminDB indexes the biological metadata changes inside the files.

---

Checks Before You Trust the Result

Never assume a tracking system captured every side effect. Perform these manual and programmatic verifications:

  1. Verify Annex Integrity: Run datalad status and datalad diff to confirm that no untracked scratch files, temporary caches, or loose binary outputs remain uncommitted.
  2. Inspect Transformation Records: In LaminDB, run ln.Transform.filter() and ln.Run.filter() to confirm that every output artifact links directly to an active parent run and explicit input artifacts.
  3. Validate AnnData Metadata: Check that observation (.obs) and variable (.var) slots inside processed .h5ad files contain valid metadata and do not contain silent null values introduced during normalization.
  4. Test Environment Replay: In a fresh shell, recreate the environment from your environment.yml and test a dry-run execution to confirm that undeclared system libraries were not silently required.
  5. Inspect Storage Backends: Verify that HDF5 chunking and compression configurations match your storage performance requirements, ensuring that downstream access does not incur excessive I/O latency.

---

Implementing a systematic tracking pipeline eliminates ambiguity regarding data origin and ensures that complex multi-step analyses remain verifiable over time. Combining filesystem snapshots with rich biological metadata modeling provides an end-to-end audit trail suitable for collaborative discovery.

These instructions are planning guides rather than benchmarked execution reports. Before implementing these steps in production or executing automation pipelines, visit the datalad, lamindb, and get-available-resources skill documentation pages on ScienceSkills to inspect API references, installation prerequisites, and environment considerations.