Skip to content

Migrating from v0.6 to v0.7

In Calliope v0.7 we have made many user-facing changes which will mean you cannot load a model definition used in v0.6.

On this page we will list the main changes to help you understand these changes and migrate your existing models to work in v0.7.

We believe these changes will make your life easier in the long run. Some might seem like steps back, as you have to write more YAML for the same definition. However, the resulting definition should be easier to understand when you come back to it in the future, and the changes made to model definition have made the internal code much easier - leading to (hopefully) fewer bugs!

Since v0.7 is in a pre-release phase, if there are changes that you don't agree with or bugs when you try implementing something, please raise an issue or start/join a discussion thread on our GitHub repository.

Changes

Flat technology definition

We have removed the distinction between essentials, constraints and costs. Instead, you define all your technology parameters at the same level.

ccgt:
  essentials:
    name: 'Combined cycle gas turbine'
    color: '#E37A72'
    parent: supply
    carrier_out: power
  constraints:
    resource: inf
    energy_eff: 0.5
    energy_cap_max: 40000
    energy_cap_max_systemwide: 100000
    energy_ramping: 0.8
    lifetime: 25
  costs:
    monetary:
      interest_rate: 0.10
      energy_cap: 750
      om_con: 0.02
ccgt:
  name: 'Combined cycle gas turbine'
  color: '#E37A72'
  base_tech: supply
  carrier_out: power
  source_use_max: inf
  flow_out_eff: 0.5
  flow_cap_max: 40000
  flow_cap_max_systemwide: 100000
  flow_ramping: 0.8
  lifetime: 25
  cost_interest_rate:
    data: 0.10
    index: monetary
    dims: costs
  cost_flow_cap:
    data: 750
    index: monetary
    dims: costs
  cost_flow_in:
    data: 0.02
    index: monetary
    dims: costs

file=/df=data_sources section

file=/df= parameter values as references to timeseries data is replaced with loading tabular data at the top-level using the data_sources key.

Assuming you have these two files:

demand_file.csv:

,node1,node2
2005-01-01 00:00,100,100  # (1)!
2005-01-01 00:00,200,200
...

  1. We're using positive numbers here which reflects our change to positive values for demand data.

supply_file.csv:

,node1,node2
2005-01-01 00:00,10,20
2005-01-01 00:00,1,2
...

techs:
  demand_tech:
    constraints:
      resource: file=demand_file.csv
      force_resource: true
  supply_tech:
    constraints:
      resource: file=supply_file.csv
      force_resource: false
data_sources:
  demand_data:
    source: demand_file.csv
    rows: timesteps
    columns: nodes
    add_dimensions:
      techs: demand_tech
      parameters: sink_equals

  supply_data:
    source: supply_file.csv
    rows: timesteps
    columns: nodes
    add_dimensions:
      techs: supply_tech
      parameters: source_max

Negative → positive demand and carrier consumption values

Demand data are now strictly positive numbers and so are the values of the carrier_con (now flow_in) decision variable.

model.run()model.build() + model.solve()

When running in Python, building and solving your optimisation problem have been split into two steps:

  1. model.build() creates the in-memory Python objects that define optimisation problem components (decision variables, constraints, the objective function, ...). This creates the calliope.Model.backend object, which you can query and use to tweak the optimisation problem before sending it to the solver.

  2. model.solve() sends the built optimisation problem to the configured solver and, provided an optimal solution is available, creates the calliope.Model.results object which contains the optimal results.

model and runconfig.init/.build/.solve

The model configuration is now split based on the stages of going from a model definition to solving your Calliope model:

  • All the options in config.init are applied when you create your model (calliope.Model(...)).
  • All the options in config.build are applied when you build your optimisation problem (calliope.Model.build(...)).
  • All the options in config.solve are applied when you solve your optimisation problem (calliope.Model.solve(...)).

This split means you can change configuration options on-the-fly if you are working in Python by adding them as keyword arguments in your calls:

config:
  init:
    time_subset: ["2005-01", "2005-02"]
  build:
    mode: plan
  solve:
    solver: glpk
import calliope
model = calliope.Model(time_subset=["2005-01", "2005-02"])
model.build(mode="plan")
model.solve(solver="glpk")

locationsnodes

locations (abbreviated to locs in the Calliope data dimensions) has been renamed to nodes (no abbreviation). This allows us to not require an abbreviation and is a disambiguation from the pandas.DataFrame.loc and xarray.DataArray.loc methods.

parentbase_tech + inherit

Technology inheritance has been unlinked from its abstract "base" technology. inherit allows for inheriting attributes from tech_groups while base_tech is fixed to be one of [demand, supply, conversion, transmission, storage].

tech_groups:
  supply_interest_rate:
    essentials:
      parent: supply
    costs:
      monetary:
        interest_rate: 0.1
  conversion_interest_rate:
    essentials:
      parent: conversion
    costs:
      monetary:
      interest_rate: 0.1
techs:
  supply_tech:
    essentials:
      parent: supply_interest_rate
  conversion_tech:
    essentials:
      parent: supply_conversion_rate
tech_groups:
  common_interest_rate:
    cost_interest_rate:
      data: 0.1
      index: monetary
      dims: costs
techs:
  supply_tech:
    base_tech: supply
    inherit: common_interest_rate
  conversion_tech:
    base_tech: conversion
    inherit: common_interest_rate

costs.monetary.flow_capcost_flow_cap

We have changed the nesting structure for defining technology costs so they are flatter and can leverage our multi-dimensional parameter definition.

techs:
  supply_tech:
    costs:
      monetary:
        interest_rate: 0.1
        energy_cap: 10
        om_con: 0.05
      emissions:
        om_con: 0.1
techs:
  supply_tech:
    cost_interest_rate:
      data: 0.1
      index: monetary
      dims: costs
    cost_flow_cap:
      data: 10
      index: monetary
      dims: costs
    cost_flow_in:
      data: [0.05, 0.1]
      index: [monetary, emissions]
      dims: costs

The top-level key links no longer exists. Instead, links are defined as separate transmission technologies in techs, including to/from keys:

techs:
  ac_transmission:
    essentials:
      parent: transmission
    constrains:
      energy_cap_max: 10
  dc_transmission:
    essentials:
      parent: transmission
    constrains:
      energy_cap_max: 5
links:
  X1,X2:
    techs:
      ac_transmission:
      dc_transmission:
techs:
  x1_to_x2_ac_transmission:
    from: X1
    to: X2
    base_tech: transmission
    flow_cap_max: 10
  x1_to_x2_dc_transmission:
    from: X1
    to: X2
    base_tech: transmission
    flow_cap_max: 5

Note

You can use tech_groups to minimise duplications in the new transmission technology definition.

Renaming parameters/decision variables without core changes in function

You may have a already noticed new parameter names being referenced in examples of other changes. We have renamed parameters to improve clarity in their function and to make it clear that although Calliope is designed to model energy systems, its flow representation is suitable for modelling any kind of flow (water, waste, etc.).

Here are the main changes to parameter/decision variable names that are not linked to changes in functionality (those are detailed elsewhere on this page):

  • energy/carrierflow, e.g. energy_cap_max is now flow_cap_max and energy_cap is now flow_cap.
  • prod/conout/in, e.g., carrier_prod is now flow_out.
  • om_prod/concost_flow_out/in.
  • resourcesource_use (for things entering the model) and sink_use (for things leaving the model).
  • resource_areaarea_use.
  • energy_cap_min_useflow_out_min_relative (i.e., the value is relative to flow_cap).
  • parasitic_effflow_out_parasitic_eff.
  • force_asynchronous_prod_conforce_async_flow.
  • existsactive.

Renaming / moving configuration options

Along with changing the YAML hierarchy of model configuration, we have changed the name of configuration options, mainly to create a flat YAML hierarchy or to group settings alphabetically:

  • model.subset_timeconfig.init.time_subset
  • model.time: {function: resample, function_options: {'resolution': '6H'}}config.init.time_resample
  • run.operation.windowconfig.build.operate_window
  • run.operation.horizonconfig.build.operate_horizon
  • run.operation.use_cap_resultsconfig.build.operate_use_cap_results

We have also moved some data out of the configuration and into the top-level parameters section:

  • run.objective_options.cost_classparameters.objective_cost_weights
  • run.bigMparameters.bigM

force_resourcesource_use_equals / sink_use_equals

Instead of defining the binary trigger force_resource to enforce the production/consumption of a resource into/out of the system, you can define the data using the new _equals parameters source_use_equals / sink_use_equals. source_use_equals is used to force the use of a resource in a supply technology. sink_use_equals is used to force the amount of resource that a demand technology must consume.

If you want these resource uses to be upper or lower bounds, use the equivalent _max/_min parameters.

You can find an example of this change above.

units + purchasedpurchased_units

We have rolled the integer decision variable units and the binary purchased into one decision variable purchased_units. To achieve the same functionality for purchased, set purchased_units_max: 1.

Explicitly triggering MILP and storage decision variables/constraints

In v0.6, we inferred that a mixed-integer linear model was desired based on the user defining certain parameters. For example, defining units_max would trigger the integer units decision variable. Defining the purchase cost parameter would trigger the binary purchased decision variable.

Now, you need to explicitly set the method using cap_method:

techs:
  supply_tech:
    constraints:  # triggers `units` integer variable
      units_max: 4
      energy_cap_per_unit: 300
      energy_cap_min_use: 0.2
  conversion_tech:  # triggers `purchased` integer variable
    costs:
      monetary:
        purchase: 2000
techs:
  supply_tech:
    units_max: 4
    flow_cap_per_unit: 300
    flow_in_min_relative: 0.2
    cap_method: integer  # triggers the `purchased_units` integer variable
  conversion_tech:
    cost_purchase:
      data: 2000
      index: monetary
      dims: costs
    cap_method: integer  # triggers the `purchased_units` integer variable

To include a storage buffer in non-storage technologies, you also need to explicitly enable it. To do so, use include_storage: true - simply defining e.g. storage_cap_max and expecting storage decision variables to be triggered is not enough!

Note

You do not need to enable storage with include_storage in storage technologies!

Structure of input and output data within a Calliope model

The concatenated loc::tech and loc::tech::carrier sets have been removed. Model components are now indexed separately over nodes, techs, and carriers (where applicable). Although primarily an internal change, this affects the xarray dataset structure and hence how users access data in model.inputs and model.results.

For example:

model.inputs.energy_cap_max.loc[{"loc_techs": "X::pv"}]
model.inputs.flow_cap_max.loc[{"nodes": "X", "techs": "pv"}]

Note

This change is functionally equivalent to first calling model.get_formatted_array("energy_cap_max") in v0.6, which is no longer necessary in v0.7.

Defining node coordinates

Only geographic coordinates are now allowed (we have removed x/y coordinates) and they can be defined directly as latitude/longitude.

nodes:
  X1:
    coordinates:
      lat: 1
      lon: 2
nodes:
  X1:
    latitude: 1
    longitude: 2

Distance units

Distances between nodes along transmission links will be automatically derived according to the nodes' geographic coordinates, if the user does not set a distance. This was also the case in v0.6. The change in v0.7 is that we default to deriving kilometres, not metres. If you prefer to keep your distance units in metres, set the configuration option: config.init.distance_unit: m

Operate mode inputs

  • To set the capacities in operate mode, you no longer need to set the _max constraints for your technologies (area_use_max, flow_cap_max, etc.); you can specify the decision variables as parameters directly. Therefore, you can define e.g. flow_cap as one of your technology parameters. This is because the additional math applied in operate mode deactivates the decision variables with the same names, paving the way for the parameters to be used in the math formulation instead.

    techs:
      tech1:
        constraints:
        energy_cap_max: 1  # will be translated internally to `energy_cap` by Calliope
        storage_cap_max: 1  # will be translated internally to `storage_cap` by Calliope
    
    techs:
      tech1:
        flow_cap: 1
        storage_cap: 1
    
  • Operate horizon and window periods are based on Pandas time frequencies, not integer number of timesteps. Therefore, 24H is equivalent to 24 in v0.6 if you are using hourly resolution, but is equivalent to 12 in v0.6 if you are using 2-hourly resolution:

    model:
      time: {function: resample, function_options: {'resolution': '6H'}}
      operation:
        window: 2
        horizon: 4
    
    config:
      init:
        time_resample: 6H
      build:
        operate_window: 12H
        operate_horizon: 24H
    

Warning

Although we know that operate mode works on our example models, we have not introduced thorough tests for it yet - proceed with caution!

Per-technology cyclic storage

The configuration option to set cyclic storage globally (run.cyclic_storage) has been moved to a parameter at the technology level. With this change, you can decide if a specific storage technology (or technology with a storage buffer) has cyclic storage enforced or not. As in v0.6, cyclic storage defaults to being on (cyclic_storage: true).

run:
  cyclic_storage: true
techs:
  storage_tech_with_cyclic_storage:
    base_tech: storage
    cyclic_storage: true
  supply_tech_without_cyclic_storage:
    base_tech: supply
    include_storage: true
    cyclic_storage: false

Removals

_equals constraints

parameters such as energy_cap_equals have been removed. You can reimplement them by setting _max and _min parameters to the same value. The benefit of this is that you can switch between fixing the parameter value (previously _equals) and having a range of values (different _min/_max values) by updating parameters in the build optimisation model. With _equals constraints, it would trigger a completely different mathematical formulation, which you could not then tweak - you had to rebuild the optimisation problem entirely.

techs:
  tech1:
    constraints:
      energy_cap_equals: 1
      storage_cap_equals: 2
techs:
  tech1:
    flow_cap_min: 1
    flow_cap_max: 1
    storage_cap_min: 2
    storage_cap_max: 2

Note

The exception to this is source_use_equals/sink_use_equals. These parameters have been introduced, to replace force_resource. They are in the model because these tend to be timeseries parameters, so we want to avoid the memory overhead of repeating the data in _min and _max parameters.

x/y coordinates

We no longer process cartesian node coordinates. Instead, you should define your coordinates using latitude/longitude.

Comma-separated node definitions

Defining duplicate definitions for nodes by chaining their names in the YAML key (node1,node2,node3: ...) is no longer possible. We are trying to minimise the custom elements of our YAML files which allows us to leverage YAML schemas to validate user inputs and to keep our YAML readers more maintainable.

You can now use node_groups to minimise duplicating key-value pairs in your YAML definitions.

supply_plus and conversion_plus technology base classes

We have removed the supply_plus and conversion_plus base technology classes.

Instead, supply_plus can be effectively represented by using supply as the base tech and setting include_storage: true in the model definition.

conversion_plus can be represented by using conversion as the base tech and using lists of carriers in carrier_in and/or carrier_out. To reimplement arbitrary links between carrier "tiers" (in_2, out_2 etc.), you can define your own math, which is a simultaneously more powerful and more human-readable way of defining complex conversion technologies.

carrier key

We now require carrier_in and carrier_out to be explicitly defined for all base techs (only carrier_in for demand and carrier_out for supply technologies). This means you cannot use the alias carrier to define the same inflow/outflow carrier. We do this because it aligns with the internal Calliope data structure (we were always converting carrier to carrier_in/_out) and it makes it clearer to the user that the carrier is the same. This is especially important now that you can define different inflow/outflow carriers for any technology base class.

techs:
  battery:
    essentials:
      parent: storage
      carrier: electricity
  supply_tech:
    essentials:
      parent: supply
      carrier: electricity
techs:
  battery:
    base_tech: storage
    carrier_in: electricity
    carrier_out: electricity
  supply_tech:
    base_tech: supply
    carrier_out: electricity

carrier_tiers and carrier_ratios

Carrier tiers were only used in conversion_plus technologies, yet led to a whole new model dimension. Additionally, carrier_ratios could be easily confused due to the complex nature of their application. With the removal of the conversion_plus base class, we have simplified how multiple carriers in/out are defined. To achieve the same functionality as carrier tiers/ratios offered, you will need to apply your own math.

One form of carrier flow interactions is still possible using only the pre-defined math. This is where there is a choice between inflow/outflow carriers instead of one carrier inflow/outflow requiring the inflow/outflow of another carrier. You can do this with flow efficiencies indexed over carriers rather than using carrier_ratios.

For instance, here's how you represent a reversible heat pump without additional math:

techs:
  heat_pump:
    essentials:
      parent: conversion_plus
      carrier_in: electricity
      carrier_out: [heat, cooling]
    constraints:
      carrier_ratios:
        carrier_out:
          heat: 3
          cooling: 2.5
techs:
  heat_pump:
    base_tech: conversion
    carrier_in: electricity
    carrier_out: [heat, cooling]
    flow_out_eff:
      data: [3, 2.5]
      index: [heat, electricity]
      dims: carriers

Group constraints

One driving reason to implement our own math syntax was to replace our "group constraints". These constraints were becoming more and more complex and it ultimately proved impossible to manage all the different ways users wanted to apply them. We have re-implemented all these constraints as tested additional math snippets, which you can explore in our example gallery.

Configuration options

  • With the change in how timeseries data is defined, we have removed the reference to a timeseries_data_path. Instead, data source filepaths should always be relative to the model.yaml file or they should be absolute paths.
  • We have removed run.relax_constraint alongside removing group constraints.
  • We have removed model.file_allowed, which many users will not even know existed (it was a largely internal configuration option)! Instead, it is possible to index any parameter over the time dimension. It is up to you to ensure the math formulation is set up to handle this change, which may require tweaking existing math.
  • With the removal of time clustering, we have removed model.random_seed and model.time options.

Plotting

It is now no longer possible to plot natively with Calliope. We made this decision due to the wide variety of visualisations that we saw being created outside our plotting module. It has proven impossible to keep our plotting methods agile given the almost infinite tweaks that libraries like matplotlib and plotly allow.

If you want to achieve some of the same plots that were possible with the Calliope v0.6 plotting module, see our example notebooks.

At a later stage, we are planning for a separate visualisation module that will provide similar functionality to the formerly-included plotting.

Clustering

Time masking and clustering capabilities have been severely reduced. Time resampling and clustering are now accessible by top-level configuration keys: e.g., config.init.time_resample: 2H, config.init.time_cluster: cluster_file.csv. Clustering is simplified to only matching model dates to representative days, with those representative days being in the clustered timeseries.

If you want to masking/cluster data you should now leverage other tools. We made this decision due to the complex nature of time clustering. With our former implementation, we were making decisions about the data that the user should have more awareness of and control over. It is also a constantly evolving field, but not the focus of Calliope, so we are liable to fall behind on the best-in-class methods.

If you want cluster the timeseries data yourself, we recommend these tools:

  • tsam: designed specifically for large-scale energy system models.
  • scikit-learn: a general machine learning library that has a clustering module. We were previously using this in our internal clustering.
  • tslearn: a timeseries-focussed machine learning library.

Additions

Storage buffers in all technology base classes

On removing supply_plus, we have opened up the option to have a storage "buffer" for any technology base class. This enables any flow into the technology to be stored across timesteps as it is in a storage technology. We have not yet enabled this for demand technologies, but you could add your own math to enable it.

Warning

Although our math should be set up to handle a storage buffer for a conversion or transmission technology, we do not have any direct tests to check possible edge cases.

Multiple carriers and different carriers in/out in all technology base classes

On removing conversion_plus, we have opened up the option to have different carriers in/out of storage/transmission technologies, and to define multiple carriers in/out of any technology. This means you could define different output carriers for a supply technology, or a different carrier into a storage technology compared to the carrier that comes out.

Warning

Although our math should be set up to handle multiple carriers and different inflow/outflow carriers for non-conversion technologies, we do not have any direct tests to check possible edge cases.

node_groups

node_groups is the equivalent of tech_groups for inheritance of attributes in nodes. This makes up for the removal of grouping node names in keys by comma separation.

So, to achieve this result:

nodes:
  region1:
    techs:
      battery:
      demand_electricity:
      ccgt:
  region2:
    techs:
      battery:
      demand_electricity:
      ccgt:
  region3:
    techs:
      battery:
      demand_electricity:
      ccgt:

We would do:

nodes:
  region1,region2,region3:
    techs:
      battery:
      demand_electricity:
      ccgt:
node_groups:
  standard_tech_list:
    techs:
      battery:
      demand_electricity:
      ccgt:
nodes:
  region1.inherit: standard_tech_list
  region2.inherit: standard_tech_list
  region3.inherit: standard_tech_list

Inflow and outflow efficiencies

Flow efficiencies are now split into inflow (flow_in_eff) and outflow (flow_out_eff) efficiencies. This enables different storage charge/discharge efficiencies to be applied.

Differentiating capacities and efficiencies between carriers

flow_cap (formerly energy_cap) is indexed in the optimisation problem over carriers as well as nodes and techs. This allows capacity constraints to be defined separately for inflows and outflows for conversion technologies. For example:

techs:
  dual_fuel_coal_plant:
    base_tech: conversion
    carrier_in: [coal, biofuel]
    carrier_out: electricity
    flow_cap_max:
      data: [100, 80]
      index: [coal, biofuel]
      dims: carriers
    flow_in_eff:
      data: [0.5, 0.6]
      index: [coal, biofuel]
      dims: carriers

You can "switch off" a constraint for a given carrier by setting its value to null in the indexed parameter data or just not referencing it:

techs:
  dual_fuel_coal_plant:
    base_tech: conversion
    carrier_in: [coal, biofuel]
    carrier_out: electricity
    flow_cap_max:
      data: [100, null]
      index: [coal, biofuel]
      dims: carriers
    flow_in_eff:
      data: [0.5, 0.6]
      index: [coal, biofuel]
      dims: carriers
techs:
  dual_fuel_coal_plant:
    base_tech: conversion
    carrier_in: [coal, biofuel]
    carrier_out: electricity
    flow_cap_max:
      data: 100
      index: coal
      dims: carriers
    flow_in_eff:
      data: [0.5, 0.6]
      index: [coal, biofuel]
      dims: carriers

Warning

If you define multiple inflow/outflow carriers but don't specify the carriers dimension in your parameter definitions, the values will be applied to all carriers. That is,

techs:
  dual_fuel_coal_plant:
    base_tech: conversion
    carrier_in: [coal, biofuel]
    carrier_out: electricity
    flow_cap_max: 100
    flow_in_eff: 0.5

Will be interpreted by Calliope as:

techs:
  dual_fuel_coal_plant:
    base_tech: conversion
    carrier_in: [coal, biofuel]
    carrier_out: electricity
    flow_cap_max:
      data: [100, 100]
      index: [coal, biofuel]
      dims: carriers
    flow_in_eff:
      data: [0.5, 0.5]
      index: [coal, biofuel]
      dims: carriers

Defining parameters outside the scope of nodes and techs

We now have a top-level key: parameters in which you can use our indexed parameter syntax to define any model parameters that you want, without them necessarily being linked to a node/technology. For instance, to define a parameter that applies over the carriers dimension:

parameters:
  my_new_param:
    data: [1, 2, 3]
    index: [heat, electricity, waste]
    dims: carriers

Or to define a single value that you might use to limit the total emissions in your system:

parameters:
  emissions_limit:
    data: 100
    index: emissions
    dims: costs

Indexing parameters over arbitrary dimensions

At the tech level, node level, and the top-level (via the parameters key), you can extend the dimensions a parameter is indexed over.

At the tech level, this allows you to define different values for different carriers. At any level, it allows you to define a brand new model dimension and your values over those. For example, if you want to apply some simplified piecewise constraints:

parameters:
  cost_flow_cap_piecewise_slopes:
    data: [5, 7, 14]
    index: [0, 1, 2]
    dims: pieces
  cost_flow_cap_piecewise_intercept:
    data: [0, -2, -16]
    index: [0, 1, 2]
    dims: pieces

Or if you want to set costs for carrier flows at a node:

nodes:
  region1:
    cost_per_flow:
      data: [1, 2]
      index: [[monetary, electricity], [monetary, heat]]
      dims: [costs, carriers]

Note

  1. Just defining new parameters is not enough to have an effect on the optimisation problem. You also need to define your own math.
  2. Because we process your YAML files to create the nodes and techs dimensions you will find in your Calliope model, you cannot use nodes/techs as dimensions of indexed parameters under the nodes or techs keys. It is possible to refer to nodes and techs as dimensions under the top-level parameters key.

Loading non-timeseries tabular data

With the change in loading timeseries data, we have expanded loading of tabular data to allow any data input. Technically, you can now define all your data in tables (although we would still recommend a mix of YAML and tabular model definition).

See also

data_sources introduction and tutorial.

YAML-based math syntax

We have overhauled our internal mathematical formulation to remove the strong link to the Pyomo library. Now, all components of our internal math are defined in a readable YAML syntax that we have developed.

You can add your own math to update the pre-defined math and to represent the physical system in ways we do not cover in our base math, or to apply new modelling methods and problem types (e.g., pathway or stochastic optimisation)!

See also

Our pre-defined and user-defined math documentation.