Development guide

The code lives on GitHub at calliope-project/calliope.

Development takes place in the master branch. Stable versions are tagged off of master with semantic versioning.

Tests are included and can be run with py.test from the project’s root directory.

See the list of open issues and planned milestones for an overview of where development is heading, and join us on Gitter to ask questions or discuss code.

Installing a development version

As when installing a stable version, using conda is recommended.

First, clone the repository:

$ git clone https://github.com/calliope-project/calliope

Using Anaconda/conda, install all requirements, including the free and open source GLPK solver, into a new environment, e.g. calliope_dev:

$ conda env create -f ./calliope/requirements.yml -n calliope_dev
$ source activate calliope_dev

On Windows:

$ conda env create -f ./calliope/requirements.yml -n calliope_dev
$ activate calliope_dev

Then install Calliope itself with pip:

$ pip install -e ./calliope

Creating modular extensions

Constraint generator functions

By making use of the ability to load custom constraint generator functions (see Running a Linear (LP) or Mixed Integer Linear (MILP) model), a Calliope model can be extended by additional constraints easily without modifying the core code.

Constraint generator functions are called during construction of the model with the Model object passed as the only parameter.

The Model object provides, amongst other things:

  • The Pyomo model instance, under the property m
  • The model data under the data property
  • An easy way to access model configuration with the get_option() method

A constraint generator function can add constraints, parameters, and variables directly to the Pyomo model instance (Model.m). Refer to the Pyomo documentation for information on how to construct these model components.

The default cost-minimizing objective function provides a good example:

import pyomo.core as po  # pylint: disable=import-error

def objective_cost_minimization(model):
    """
    Minimizes total system monetary cost.
    Used as a default if a model does not specify another objective.

    """
    m = model.m

    def obj_rule(m):
        return sum(model.get_option(y + '.weight') *
                       sum(m.cost[y, x, 'monetary']
                        for x in m.x) for y in m.y)

    m.obj = po.Objective(sense=po.minimize, rule=obj_rule)
    m.obj.domain = po.Reals

See the source code of the ramping_rate() function for a more elaborate example.

The process of including custom, optional constraints is as follows:

First, create the source code (see e.g. the above example for the ramping_rate function) in a file, for example my_constraints.py

Then, assuming your custom constraint generator function is called my_first_custom_constraint and is defined in my_constraints.py, you can tell Calliope to load it by adding it to the list of optional constraints in your model configuration as follows:

constraints:
    - constraints.optional.ramping_rate
    - my_constraints.my_first_custom_constraint

This assumes that the file my_constraints.py is importable when the model is run. It must therefore either be in the directory from which the model is run, installed as a Python module (see this document on how to create importable and installable Python packages), or the Python import path has to be adjusted according to the official Python documentation.

Subsets

Calliope internally builds many subsets to better manage constraints, in particular, subsets of different groups of technologies. These subsets can be used in the definition of constraints and are used extensively in the definition of Calliope’s built-in constraints. See the detailed definitions in calliope.sets, an overview of which is included here.

Main sets & sub-sets

Technologies:

  • m.y_demand: all demand sources
    • m.y_sd_r_area: if any r_area constraints are defined (shared)
    • m.y_sd_finite_r: if finite resource limit is defined (shared)
  • m.y_supply: all basic supply technologies
    • m.y_sd_r_area: if any r_area constraints are defined (shared)
    • m.y_sd_finite_r: if finite resource limit is defined (shared)
  • m.y_storage: specifically storage technologies
  • m.y_supply_plus: all supply+ technologies
    • m.y_sp_r_area: If any r_area constraints are defined
    • m.y_sp_finite_r: if finite resource limit is defined
    • m.y_sp_r2: if secondary resource is allowed
  • m.y_conversion: all basic conversion technologies
  • m.y_conversion_plus: all conversion+ technologies
    • m.y_cp_2out: secondary carrier(s) out
    • m.y_cp_3out: tertiary carrier(s) out
    • m.y_cp_2in: secondary carrier(s) in
    • m.y_cp_3in: tertiary carrier(s) in
  • m.y_transmission: all transmission technologies
  • m.y_unmet: dummy supply technologies to log
  • m.y_export: all technologies allowing export of their carrier outside the system
  • m.y_purchase: technology defining a ‘purchase’ cost, thus triggering an associated binary decision variable
  • m.y_milp: technology defining a ‘units’ maximum, minimum, or equality, thus triggering an associated integer decision variable

Locations:

  • m.x_transmission: all transmission locations
  • m.x_r: all locations which act as system sources/sinks
  • m.x_conversion: all locations in which there are conversion/conversion_plus technologies
  • m.x_store: all locations in which storage is allowed
  • m.x_export: locations allowing ‘y_export’ technologies to export outside the system
  • m.x_purchase: locations associated with ‘y_purchased’ technologies
  • m.x_milp: locations associated with ‘y_milp’ technologies

Shared subsets

  • m.y_finite_r: shared between y_demand, y_supply, and y_supply_plus. Contains:
    • m.y_sd_finite_r
    • m.y_sp_finite_r
  • m.y_r_area: shared between y_demand, y_supply, and y_supply_plus. Contains:
    • m.y_sd_r_area
    • m.y_sp_r_area

Meta-sets

Technologies:

  • m.y: all technologies, includes:
    • m.y_demand
    • m.y_supply
    • m.y_storage
    • m.y_supply_plus
    • m.y_conversion
    • m.y_conversion_plus
    • m.y_transmission
    • m.y_unmet
    • m.y_export
    • m.y_purchase
    • m.y_milp
  • m.y_sd: all basic supply & demand technologies, includes:
    • m.y_demand
    • m.y_supply
    • m.y_unmet
  • m.y_store: all technologies that have storage capabilities, includes:
    • m.y_storage
    • m.y_supply_plus

Locations:

  • m.x: all locations, includes:
    • m.x_transmission
    • m.x_r
    • m.x_conversion
    • m.x_store
    • m.x_export
    • m.x_purchase
    • m.x_milp

Time functions and masks

Like custom constraint generator functions, custom functions that adjust time resolution can be loaded dynamically during model initialization. By default, Calliope first checks whether the name of a function or time mask refers to a function from the calliope.time_masks or calliope.time_functions module, and if not, attempts to load the function from an importable module:

time:
   masks:
       - {function: week, options: {day_func: 'extreme', tech: 'wind', how: 'min'}}
       - {function: my_custom_module.my_custom_mask, options: {...}}
   function: my_custom_module.my_custom_function
   function_options: {...}

Profiling

To profile a Calliope run with the built-in national-scale example model, then visualize the results with snakeviz:

make profile  # will dump profile output in the current directory
snakeviz calliope.profile  # launch snakeviz to visually examine profile

Use mprof plot to plot memory use.

Other options for visualizing:

  • Interactive visualization with KCachegrind (on macOS, use QCachegrind, installed e.g. with brew install qcachegrind)

    pyprof2calltree -i calliope.profile -o calliope.calltree
    kcachegrind calliope.calltree
    
  • Generate a call graph from the call tree via graphviz

    # brew install gprof2dot
    gprof2dot -f callgrind calliope.calltree | dot -Tsvg -o callgraph.svg
    

Checklist for new release

Pre-release

  • Make sure all unit tests pass
  • Make sure documentation builds without errors
  • Make sure the release notes are up-to-date, especially that new features and backward incompatible changes are clearly marked

Create release

  • Change _version.py version number
  • Update changelog with final version number and release date
  • Commit with message “Release vXXXX”, then add a “vXXXX” tag, push both to GitHub
  • Create a release through the GitHub web interface, using the same tag, titling it “Release vXXXX” (required for Zenodo to pull it in)
  • Upload new release to PyPI: make all-dist
  • Update the conda-forge package:
    • Fork conda-forge/calliope-feedstock, and update recipe/meta.yaml with:
      • Version number: {% set version = "XXXX" %}
      • MD5 of latest version from PyPI: {% set md5 = "XXXX" %}
      • Reset build: number: 0 if it is not already at zero
      • If necessary, carry over any changed requirements from requirements.yml or setup.py
    • Submit a pull request from an appropriately named branch in your fork (e.g. vXXXX) to the conda-forge/calliope-feedstock repository

Post-release

  • Update changelog, adding a new vXXXX-dev heading, and update _version.py accordingly, in preparation for the next master commit

Note

Adding ‘-dev’ to the version string, such as __version__ = '0.1.0-dev', is required for the custom code in doc/conf.py to work when building in-development versions of the documentation.

Previous: Built-in example models | Next: API Documentation