Migration Guide: Simulation and System Interface Refactor

Migration Guide: Simulation and System Interface Refactor#

This guide helps you update code to use GAT’s refactored simulation and system interfaces.

Note (Phase 10): SiennaSimulationAggregator has been removed — the snippets below labeled “still works” in earlier revisions of this guide are now historical. Use SimulationAggregator(parser_class=SiennaSimulationParser) exclusively.

Table of Contents#

Overview#

GAT’s simulation and system interfaces have been refactored to provide:

  1. Consistent API: Both simulations and systems now expose list_datasets() and get_dataset() methods

  2. Generic Aggregation: New SimulationAggregator works with any parser type

  3. Parallel Loading: Multi-file scenarios load faster with multiprocessing

  4. Plugin-Friendly: Easier to add support for new simulation tools

Timeline: This refactor is live as of version X.Y.Z

Compatibility: Most existing code will continue to work, but some deprecated patterns should be updated.

What Changed#

Simulation Parsers#

New Base Interface#

# New standardized interface
class BaseSimulationParser(ABC):
    @property
    @abstractmethod
    def simulation_models(self) -> List[str]:
        """List available models"""
        
    @abstractmethod
    def list_datasets(self) -> Dict[str, str]:
        """List available datasets"""
        
    @abstractmethod
    def get_dataset(self, key: str) -> pd.DataFrame:
        """Get a dataset"""

Generic Aggregator#

# New generic aggregator (works with any parser)
from gat.simulations import SimulationAggregator

aggregator = SimulationAggregator(
    file_paths=["sim1.h5", "sim2.h5"],
    parser_class=SiennaSimulationParser,
    parallel=True
)

System Parsers#

Dataset Interface Added#

# New methods on BaseSystem
class BaseSystem(ABC):
    def list_datasets(self) -> Dict[str, str]:
        """List available datasets"""
        
    def get_dataset(self, key: str, **kwargs) -> pd.DataFrame:
        """Get a dataset"""
        
    def get_datasets(self, *keys: str) -> Dict[str, pd.DataFrame]:
        """Get multiple datasets"""

Migration Steps#

Step 1: Update Imports#

Before:

from gat.datahelpers.sienna import SiennaSimulationAggregator

After:

from gat.simulations import SimulationAggregator, SiennaSimulationParser

Step 2: Update Aggregator Creation (Optional)#

Before:

from gat.simulations import SiennaSimulationAggregator

aggregator = SiennaSimulationAggregator(
    file_paths=["sim1.h5", "sim2.h5", "sim3.h5"]
)

After (recommended):

from gat.simulations import SimulationAggregator, SiennaSimulationParser

aggregator = SimulationAggregator(
    file_paths=["sim1.h5", "sim2.h5", "sim3.h5"],
    parser_class=SiennaSimulationParser,
    parallel=True  # Enable parallel loading
)

Step 3: Update System Data Access (Optional)#

Before:

system = SiennaSystem("system.json")
generators = system.get_generator_data()
loads = system.get_load_data()

After (recommended):

system = SiennaSystem("system.json")

# New unified interface
generators = system.get_dataset("generators")
loads = system.get_dataset("loads")

# Or get multiple at once
data = system.get_datasets("generators", "loads")

After (compatible):

# Old methods still work
system = SiennaSystem("system.json")
generators = system.get_generator_data()
loads = system.get_load_data()

Step 4: Update Custom Parsers#

If you have custom simulation parsers, update them to follow the new interface:

Before:

class MyParser:
    def get_data(self, dataset_name):
        # Custom implementation
        pass

After:

from gat.simulations import BaseSimulationParser

class MyParser(BaseSimulationParser):
    @property
    def simulation_models(self) -> List[str]:
        return ["default"]
    
    def list_datasets(self) -> Dict[str, str]:
        return {"power": "/results/power"}
    
    def get_dataset(self, key: str) -> pd.DataFrame:
        # Implementation
        pass

Before and After Examples#

Example 1: Loading Multiple Simulation Files#

Before:

from gat.simulations import SiennaSimulationAggregator

# Sienna-specific aggregator
agg = SiennaSimulationAggregator(
    file_paths=["day1.h5", "day2.h5", "day3.h5"]
)

# Set model
agg.selected_model = "UC"

# Get data
datasets = agg.list_datasets()
dispatch = agg.get_dataset("generator_dispatch")

After:

from gat.simulations import SimulationAggregator, SiennaSimulationParser

# Generic aggregator with parallel loading
agg = SimulationAggregator(
    file_paths=["day1.h5", "day2.h5", "day3.h5"],
    parser_class=SiennaSimulationParser,
    parallel=True,  # NEW: Parallel loading
    max_workers=4   # NEW: Control parallelism
)

# Set model (same API)
agg.selected_model = "UC"

# Get data (same API)
datasets = agg.list_datasets()
dispatch = agg.get_dataset(
    "generator_dispatch",
    merge_strategy="left"  # NEW: Control overlap handling
)

Example 2: Single File Parser#

Before:

from gat.simulations import SiennaSimulationParser

parser = SiennaSimulationParser("simulation.h5")
parser.selected_model = "ED"

data = parser.get_dataset("ActivePowerTimeSeriesParameter__RenewableDispatch")

After:

from gat.simulations import SiennaSimulationParser

# API is unchanged!
parser = SiennaSimulationParser("simulation.h5")
parser.selected_model = "ED"

data = parser.get_dataset("ActivePowerTimeSeriesParameter__RenewableDispatch")

Example 3: System Data Access#

Before:

from gat.datahelpers import SiennaSystem

system = SiennaSystem("system.json")

# Get data using specific methods
generators = system.get_generator_data()
solar_gens = system.get_generator_data(category="Solar_PV")

# Get categories
categories = system.list_generator_categories()

After:

from gat.datahelpers import SiennaSystem

system = SiennaSystem("system.json")

# New unified dataset interface
generators = system.get_dataset("generators")
solar_gens = system.get_dataset("generators", category="Solar_PV")

# Or use specific methods (still work)
generators = system.get_generator_data()
solar_gens = system.get_generator_data(category="Solar_PV")

# Categories unchanged
categories = system.list_generator_categories()

# NEW: List all available datasets
datasets = system.list_datasets()
# Returns: {"generators": "generator_data", "loads": "load_data", ...}

# NEW: Get multiple datasets at once
data = system.get_datasets("generators", "loads", "system_info")

Example 4: Creating a Scenario#

Before:

from gat.scenariohandlers import SiennaScenario

scenario = SiennaScenario(
    solution_data=["sim1.h5", "sim2.h5"],
    system_file="system.json"
)

# Access parser
dispatch = scenario.parser.get_dataset("generator_dispatch")

After:

from gat.scenariohandlers import SiennaScenario

# API unchanged - aggregation happens automatically
scenario = SiennaScenario(
    solution_data=["sim1.h5", "sim2.h5"],
    system_file="system.json"
)

# Parser is now using new aggregator internally
# (transparent to user)
dispatch = scenario.parser.get_dataset("generator_dispatch")

Example 5: Custom Parser Plugin#

Before:

# No standard interface - custom implementation
class MySimParser:
    def __init__(self, path):
        self.path = path
    
    def read_dispatch(self):
        return pd.read_csv(self.path)

After:

from gat.simulations import BaseSimulationParser
import pandas as pd

class MySimParser(BaseSimulationParser):
    def __init__(self, path: str):
        super().__init__()
        self.path = path
    
    @property
    def simulation_models(self) -> list[str]:
        return ["default"]
    
    def list_datasets(self) -> dict[str, str]:
        return {
            "generator_dispatch": "dispatch.csv",
            "generator_commit": "commit.csv"
        }
    
    def get_dataset(self, key: str) -> pd.DataFrame:
        datasets = self.list_datasets()
        if key not in datasets:
            raise KeyError(f"Dataset '{key}' not found")
        
        file_path = datasets[key]
        df = pd.read_csv(file_path, index_col=0, parse_dates=True)
        return df

# Now works with generic aggregator!
from gat.simulations import SimulationAggregator

agg = SimulationAggregator(
    file_paths=["sim1/", "sim2/", "sim3/"],
    parser_class=MySimParser,
    parallel=True
)

Breaking Changes#

Minimal Breaking Changes#

The refactor was designed to be backward compatible. Most existing code will continue to work.

Parser Class Must Be Pickleable (for parallel loading)#

If using parallel loading, your parser class must be pickleable:

# ✅ Good - pickleable
class GoodParser(BaseSimulationParser):
    def __init__(self, path: str):
        super().__init__()
        self.path = path
        # Don't store unpickleable objects in __init__

# ❌ Bad - not pickleable
class BadParser(BaseSimulationParser):
    def __init__(self, path: str):
        super().__init__()
        self.file_handle = open(path, 'rb')  # File handles aren't pickleable

Solution: Open files in methods, not __init__():

class GoodParser(BaseSimulationParser):
    def __init__(self, path: str):
        super().__init__()
        self.path = path
    
    def get_dataset(self, key: str) -> pd.DataFrame:
        with open(self.path, 'rb') as f:
            # Use file handle here
            pass

Import Paths Changed#

Some imports have moved:

# ❌ Old (removed in Phase 10)
from gat.datahelpers.sienna import SiennaSimulationAggregator
from gat.simulations import SiennaSimulationAggregator

# ✅ New
from gat.simulations import SimulationAggregator, SiennaSimulationParser

New Features#

1. Parallel File Loading#

Speed up multi-file scenarios with parallel loading:

from gat.simulations import SimulationAggregator, SiennaSimulationParser

# Load 10 files in parallel
agg = SimulationAggregator(
    file_paths=[f"sim_{i}.h5" for i in range(10)],
    parser_class=SiennaSimulationParser,
    parallel=True,      # Enable parallel loading
    max_workers=4       # Use 4 processes
)

Performance: ~4x faster for 10 files on a 4-core system

2. Merge Strategy Control#

Control how overlapping time periods are handled:

# Keep earlier timestamps (typical for multi-stage sims)
data = agg.get_dataset("dispatch", merge_strategy="left")

# Keep later timestamps (typical for rolling forecasts)
data = agg.get_dataset("dispatch", merge_strategy="right")

3. Context Manager Support#

Automatic resource cleanup:

with SiennaSimulationParser("sim.h5") as parser:
    data = parser.get_dataset("dispatch")
# File automatically closed

with SimulationAggregator(files, SiennaSimulationParser) as agg:
    data = agg.get_dataset("dispatch")
# All parsers automatically closed

4. Unified Dataset Interface#

Systems now support the same dataset interface as simulations:

system = SiennaSystem("system.json")

# List available datasets
datasets = system.list_datasets()

# Get single dataset
generators = system.get_dataset("generators")

# Get multiple datasets
data = system.get_datasets("generators", "loads", "system_info")

5. Enhanced Metadata#

Access simulation metadata:

parser = SiennaSimulationParser("sim.h5")

metadata = parser.get_metadata()
# {
#     "start_time": "2024-01-01T00:00:00",
#     "end_time": "2024-12-31T23:00:00",
#     "resolution": "1H",
#     ...
# }

6. Validation#

Check for issues before processing:

parser = SiennaSimulationParser("sim.h5")

warnings = parser.validate()
for warning in warnings:
    print(f"Warning: {warning}")

Checklist#

Use this checklist to ensure your code is updated:

  • Updated imports to use gat.simulations instead of gat.datahelpers

  • Consider using SimulationAggregator instead of tool-specific aggregators

  • Enable parallel=True for multi-file scenarios

  • Use list_datasets() and get_dataset() for consistent API

  • Update custom parsers to inherit from BaseSimulationParser

  • Ensure custom parsers are pickleable (for parallel loading)

  • Test with both single and multiple files

  • Update documentation and examples

Getting Help#

If you encounter issues during migration:

  1. Check the plugin development guide: docs/plugin_development_guide.md

  2. Review examples: Look at SiennaSimulationParser for reference

  3. Open an issue: Report migration problems on GitHub

  4. Ask the team: Reach out on Slack/Discord

Summary#

Most code requires no changes - the refactor is backward compatible.

Recommended updates:

  • Use SimulationAggregator for better performance

  • Enable parallel loading for multi-file scenarios

  • Use unified dataset interface for consistency

Benefits:

  • Faster multi-file loading (parallel processing)

  • Easier to add new simulation tools

  • Consistent API across simulations and systems

  • Better resource management