Experiment Process / Metadata Tool (epmt)

epmt collects metadata and performance data about shell processes, down to individual threads in individual processes. Currently, epmt is particularly specialized for interfacing with Slurm batch jobs associated with earth modeling workflows, but is generalizable to other computational workflow contexts. It also offers entrypoints to analyzing your data by interfacing with jupyter for easy access to a notebook-style interface.

readthedocs codecov pylint weekly_cache_builds build_conda markdown lint spell check

Workflow Python 3.10 Python 3.11
create_test_conda_env 3.10 3.11
Workflow SQLite PostgreSQL
docker_build_test sqlite postgres
build_and_test_epmt sqlite postgres

Installation

These are not-yet fully functional installations, as epmt was designed in an era where virtual environments were not as ubiquitous as they are today. For full-featured build/installation approaches, consult the Makefile, .github/workflows, and DEVELOPER.md

The conda installation is currently favored as a quick-start for new users.

conda install noaa-gfdl::epmt

From repo checkout

The following creates a whole epmt conda environment with epmt accessible via an editable pip installation.

git clone https://github.com/NOAA-GFDL/epmt.git
cd epmt
conda env create -f environment.yaml
conda activate epmt

If you already have an environment created that you wish to install epmt, and it's already activated:

git clone https://github.com/NOAA-GFDL/epmt.git
cd epmt
pip install src/

Verifying an Installation

The check command is a first-stop sanity-check of your epmt installation. Call it with

epmt check

Verify the version:

epmt -V

Quickstart: Watch epmt work

Try wrapping your commands with epmt start / epmt stop:

epmt start
epmt run ./compute_the_world --debug
epmt stop
epmt submit

Or use the --auto (-a) flag to automate the start/stop cycle:

epmt -a run ./compute_the_world --debug
epmt submit

Further Documentation

For detailed information on configuration, data collection, SLURM integration, database submission, analysis, performance metrics, debugging, and CI/CD, see DEVELOPER.md.