Project Management (v1.0)#

GAT v1.0 introduces a comprehensive project management system that uses the filesystem and git for collaboration. Projects are self-contained directories with YAML configuration files that can be shared across teams. The CLI provides hierarchical subcommands for intuitive scenario and palette management.

Overview#

The project management system uses a two-tier architecture:

User Metadata (~/.config/gat/) - Lightweight references to projects on your system
Project Directories - Self-contained project folders (can be anywhere, typically git repositories)

This design allows teams to share projects via git while keeping user-specific settings separate.

Quick Start#

# Create a new project
gat project init my-analysis --name "My Analysis Project"

# Add a scenario
gat project scenario add sienna base_case \
    --system ../data/system.json \
    --simulation ../data/results.h5

# List all scenarios across all projects
gat project scenario list

# List scenarios in a specific project
gat project scenario list my-analysis

# Show project details
gat project show my-analysis

Project Structure#

When you initialize a project, GAT creates this directory structure:

my-project/
├── gat-project.yaml          # Project configuration
├── scenarios/                # Scenario configurations
│   └── base_case.yaml
├── palettes/                 # Visualization palettes
│   └── default.yaml
├── pipelines/                # Analysis pipelines
├── notebooks/                # Jupyter notebooks
├── outputs/                  # Generated plots (not tracked in git)
├── .gat-cache/              # Cached data (not tracked in git)
├── .gitignore
└── README.md

Commands#

Initialize a Project#

gat project init [PATH] [OPTIONS]

Creates a new GAT project with the standard directory structure.

Arguments:

PATH - Directory path for the project (default: current directory)

Options:

--name TEXT - Project name (defaults to directory name)
--description TEXT - Project description
--gat-version TEXT - Required GAT version (default: 1.0.0)
--no-palettes - Don’t copy user palettes into project
--no-add - Don’t add project to user metadata
--set-default - Set as default project

Examples:

# Initialize in current directory
gat project init .

# Initialize in new directory with name
gat project init ./my-analysis --name "My Analysis"

# Initialize and set as default
gat project init ./my-analysis --set-default

Add an Existing Project#

gat project add PATH [OPTIONS]

Adds an existing project to your user metadata. This creates a lightweight reference that points to the project directory.

Arguments:

PATH - Path to existing project directory

Options:

--id TEXT - Project ID (defaults to directory name)
--set-default - Set as default project

Examples:

# Add a local project
gat project add ./my-project

# Add and set as default
gat project add ./my-project --set-default

# Add with custom ID
gat project add ./my-project --id my_proj

List Projects#

gat project list [OPTIONS]

Lists all projects from your user metadata.

Options:

--verbose, -v - Show detailed information

Examples:

# Compact listing
gat project list

# Detailed listing
gat project list --verbose

Show Project Details#

gat project show PROJECT_ID

Displays detailed information about a project including scenarios, palettes, and configuration.

Arguments:

PROJECT_ID - Project identifier

Example:

gat project show my-analysis

Set Default Project#

gat project set-default PROJECT_ID

Sets a project as the default. Commands that accept --project will use the default if not specified.

Arguments:

PROJECT_ID - Project identifier

Example:

gat project set-default my-analysis

# Now you can omit --project in other commands
gat project add-scenario sienna test --system sys.json --simulation results.h5

Scenario Management#

GAT provides hierarchical subcommands for scenario management under gat project scenario.

Add a Scenario#

gat project scenario add TYPE SCENARIO_ID [OPTIONS]

Adds a scenario configuration to a project.

Arguments:

TYPE - Scenario type: sienna, reeds, or plexos
SCENARIO_ID - Unique identifier for the scenario

Common Options:

--name TEXT - Scenario name (defaults to ID)
--description TEXT - Scenario description
--project TEXT - Project ID (uses default if not specified)

Sienna Options:

--system TEXT - Path to system JSON file (required)
--simulation TEXT - Path to simulation HDF5 file (required, can specify multiple)
--metadata TEXT - Path to GAT metadata JSON file

ReEDS Options:

--path TEXT - Path to ReEDS output directory (required)
--solve-year INTEGER - Solve year to analyze

Plexos Options:

--solution TEXT - Path to Plexos solution file (required)

Examples:

# Add a Sienna scenario
gat project scenario add sienna base_2035 \
    --system ../data/system.json \
    --simulation ../data/results.h5

# Add a Sienna scenario with multiple simulation files
gat project scenario add sienna multi_week \
    --system ../data/system.json \
    --simulation ../data/week1.h5 \
    --simulation ../data/week2.h5 \
    --simulation ../data/week3.h5

# Add with description
gat project scenario add sienna high_load \
    --name "High Load Scenario" \
    --description "Sensitivity with 10% load increase" \
    --system ../data/system.json \
    --simulation ../data/high_load.h5

# Add a ReEDS scenario
gat project scenario add reeds reeds_2035 \
    --path ../reeds_output \
    --solve-year 2035

# Add to specific project
gat project scenario add sienna test \
    --project my_other_project \
    --system ../data/system.json \
    --simulation ../data/results.h5

Note: All paths are automatically resolved to absolute paths when adding scenarios, ensuring clarity about which files are being used regardless of your current working directory.

List Scenarios#

gat project scenario list [PROJECT_ID]

Lists scenarios across all projects or in a specific project.

Arguments:

PROJECT_ID - (Optional) Project identifier. If omitted, lists scenarios from all projects.

Examples:

# List scenarios across all projects
gat project scenario list

# List scenarios in specific project
gat project scenario list my-analysis

Output (all projects):

My Analysis (my-analysis):
  base_2035            [sienna] Base 2035 Scenario
  high_renewable       [sienna] High Renewable Scenario

Planning Study (planning):
  reeds_2050           [reeds] ReEDS 2050 Scenario
  plexos_base          [plexos] Plexos Base Case

Total scenarios: 4

Output (specific project):

Scenarios in 'My Analysis': (2)
  base_2035            [sienna] Base 2035 Scenario
  high_renewable       [sienna] High Renewable Scenario

Show Scenario Details#

gat project scenario show SCENARIO_ID [OPTIONS]

Displays detailed information about a specific scenario including paths and validation.

Arguments:

SCENARIO_ID - Scenario identifier

Options:

--project TEXT - Project ID (uses default if not specified)

Examples:

# Show scenario in default project
gat project scenario show base_2035

# Show scenario in specific project
gat project scenario show base_2035 --project my-analysis

Output:

Scenario: base_2035
Name:        Base 2035 Scenario
Type:        sienna
Description: Baseline scenario for 2035 analysis
Created:     2024-01-15 10:30
Updated:     2024-01-20 14:45

Paths:
  System:      ✓ /data/shared/systems/base_2035.json
  Simulations: (2 file(s))
    - ✓ /data/shared/results/week1.h5
    - ✓ /data/shared/results/week2.h5

✓ All paths validated

Remove a Scenario#

gat project scenario remove SCENARIO_ID [OPTIONS]

Removes a scenario from a project. This deletes the scenario configuration file but does not delete the underlying data files.

Arguments:

SCENARIO_ID - Scenario identifier

Options:

--project TEXT - Project ID (uses default if not specified)
--yes, -y - Skip confirmation prompt

Examples:

# Remove with confirmation
gat project scenario remove old_scenario

# Remove without confirmation
gat project scenario remove old_scenario --yes

# Remove from specific project
gat project scenario remove old_scenario --project my-analysis

Remove a Project#

gat project remove PROJECT_ID [OPTIONS]

Removes a project from your user metadata. This does not delete the project directory, only the reference to it.

Arguments:

PROJECT_ID - Project identifier

Options:

--yes, -y - Skip confirmation prompt

Example:

gat project remove my-analysis
gat project remove old-project --yes

Configuration Files#

Project Configuration (`gat-project.yaml`)#

The main project configuration file:

name: My Analysis Project
description: Comprehensive power system analysis
version: 0.1.0
gat_version: 1.0.0
created_at: '2024-01-15T10:30:00Z'
updated_at: '2024-01-22T15:00:00Z'

default_scenario: base_case
default_palette: default

contributors:
  - Alice Smith
  - Bob Jones

repository_url: git@github.com:team/my-project.git

settings:
  output_dir: ./outputs
  cache_dir: ./.gat-cache
  default_output_format: png

Scenario Configuration (`scenarios/*.yaml`)#

Individual scenario configurations:

name: Base Case 2035
description: Base scenario for 2035 analysis
type: sienna
tags:
  - base
  - '2035'

created_at: '2024-01-15T10:30:00Z'
updated_at: '2024-01-20T14:00:00Z'

system_path: ../data/systems/2035_base.json
simulation_paths:
  - ../data/results/week1.h5
  - ../data/results/week2.h5
  - ../data/results/week3.h5
metadata_path: ../data/gat_metadata.json

Note: All paths in scenario configs are stored as absolute filesystem paths. When adding scenarios via the CLI, relative paths are automatically resolved to absolute paths based on your current working directory.

User Project Reference (`~/.config/gat/projects/*.yaml`)#

Lightweight references in user metadata:

project_id: my-analysis
name: My Analysis Project
path: /home/user/projects/my-analysis
description: Power system analysis
remote_url: git@github.com:team/my-analysis.git
last_accessed: '2024-01-22T15:00:00Z'
is_default: true
tags:
  - transmission
  - planning

Team Collaboration#

Projects are designed to be shared via git:

Using a Shared Project#

# Clone the repository
git clone git@github.com:team/my-analysis.git
cd my-analysis

# Add to your GAT projects
gat project add . --set-default

# Now you can work with the project
gat project show my-analysis

Collaborating#

# Pull latest changes
cd my-analysis
git pull

# Make your changes
gat project scenario add sienna sensitivity --system ../data/sys.json --simulation ../data/sens.h5

# Commit and push
git add scenarios/sensitivity.yaml
git commit -m "Add sensitivity scenario"
git push

Best Practices#

Project Organization#

Keep data separate - Store large data files outside the project directory and use relative paths
Use git - Version control your project configuration and analysis code
Document scenarios - Use descriptive names and descriptions for scenarios
Share palettes - Copy user palettes into projects for team consistency

Naming Conventions#

Project IDs: Use lowercase with hyphens or underscores (e.g., ntp-base, wecc_2035)
Scenario IDs: Use descriptive names with underscores (e.g., base_case, high_load, sensitivity_1)
Palette names: Use descriptive names (e.g., renewable_focus, presentation)

File Structure#

workspace/
├── my-analysis/              # GAT project (git repo)
│   ├── gat-project.yaml
│   ├── scenarios/
│   │   ├── base_case.yaml   # Contains absolute paths to data
│   │   └── high_load.yaml
│   └── ...
└── data/                     # Data directory (not in git)
    ├── systems/
    │   └── 2035_base.json
    └── results/
        ├── week1.h5
        └── week2.h5

Keep data files separate from projects. When adding scenarios, paths are automatically resolved to absolute paths, making it clear which files are being used.

Migration from v0.x#

If you’re using the old data sources approach, you can migrate by:

Create a project for each data source:

gat project init my-project --name "My Project"

Add scenarios from your old data sources:

gat project scenario add sienna base \
    --system /path/to/old/system.json \
    --simulation /path/to/old/results.h5

List all scenarios to verify:
```
gat project scenario list
```

Advanced Usage#

Multiple Environments#

Projects can specify their own virtual environment:

Edit gat-project.yaml:

venv:
  path: .venv
  python_version: '3.11'
  requirements:
    - pandas>=2.0
    - matplotlib>=3.5
    - custom-extension==0.1.0

Custom Metadata#

Projects support arbitrary metadata through YAML:

# gat-project.yaml
name: My Project
custom_field: custom_value
team_settings:
  slack_channel: "#grid-analysis"
  meeting_day: "Tuesday"

Tags for Organization#

Use tags in project references for organization:

# ~/.config/gat/projects/my-project.yaml
project_id: my-project
name: My Project
tags:
  - transmission
  - nlr
  - 2035
  - high-priority

Then filter in your workflows or scripts.

Troubleshooting#

Project not found#

$ gat project show my-project
Error: Project 'my-project' not found

Solution: Add the project to your user metadata:

gat project add /path/to/my-project

Path not found warning#

$ gat project list
my-project    My Project    ✗ missing

Solution: The project directory has been moved or deleted. Either:

Move it back to the original location

Remove and re-add with new path:

gat project remove my-project
gat project add /new/path/to/my-project

Scenario already exists#

$ gat project scenario add sienna test ...
Error: Scenario already exists: test

Solution: Use a different scenario ID or remove the existing one first:

gat project scenario remove test

Project Management (v1.0)

Contents

Project Management (v1.0)#

Overview#

Quick Start#

Project Structure#

Commands#

Initialize a Project#

Add an Existing Project#

List Projects#

Show Project Details#

Set Default Project#

Scenario Management#

Add a Scenario#

List Scenarios#

Show Scenario Details#

Remove a Scenario#

Remove a Project#

Configuration Files#

Project Configuration (`gat-project.yaml`)#

Scenario Configuration (`scenarios/*.yaml`)#

User Project Reference (`~/.config/gat/projects/*.yaml`)#

Team Collaboration#

Using a Shared Project#

Collaborating#

Best Practices#

Project Organization#

Naming Conventions#

File Structure#

Migration from v0.x#

Advanced Usage#

Multiple Environments#

Custom Metadata#

Tags for Organization#

Troubleshooting#

Project not found#

Path not found warning#

Scenario already exists#

See Also#

Project Management (v1.0)

Contents

Project Management (v1.0)#

Overview#

Quick Start#

Project Structure#

Commands#

Initialize a Project#

Add an Existing Project#

List Projects#

Show Project Details#

Set Default Project#

Scenario Management#

Add a Scenario#

List Scenarios#

Show Scenario Details#

Remove a Scenario#

Remove a Project#

Configuration Files#

Project Configuration (gat-project.yaml)#

Scenario Configuration (scenarios/*.yaml)#

User Project Reference (~/.config/gat/projects/*.yaml)#

Team Collaboration#

Sharing a Project#

Using a Shared Project#

Collaborating#

Best Practices#

Project Organization#

Naming Conventions#

File Structure#

Migration from v0.x#

Advanced Usage#

Multiple Environments#

Custom Metadata#

Tags for Organization#

Troubleshooting#

Project not found#

Path not found warning#

Scenario already exists#

See Also#

Project Configuration (`gat-project.yaml`)#

Scenario Configuration (`scenarios/*.yaml`)#

User Project Reference (`~/.config/gat/projects/*.yaml`)#