Using fully with Python

ecodynelec offers the possibility to being fully used via Python. This tutorial shows the different steps and possibilities, relying solely on operations doable within a python script or notebook. As the configuration of ecodynelec pipeline execution may benefit from more visual interaction, an alternative is suggested in the “Using with a spreadsheet-based configuraion” tutorial.

Initialization

To download and install ecodynelec to being used as a python package, the user is referred to either the getting started tutorial.

Configuration

The configuration of ecodynelec is handled by the parameter module.

from ecodynelec.parameter import Parameter # Import the class to manipulate parameters

Python is an object oriented language. Thus a specific configuration can be built and stored in a Parameter object that will be called my_config in this tutorial. The next cell only initializes the configuration object with the default parameters.

my_config = Parameter() # Initialize a configuration object

Before modifying the configuration, let’s have a look at this default setting

print(my_config)

ctry --> ['AT', 'CH', 'CZ', 'DE', 'FR', 'IT']
target --> CH
start --> 2017-02-01 00:00:00
end --> 2017-02-01 23:00:00
freq --> H
timezone --> UTC
cst_imports --> False
net_exchanges --> False
network_losses --> False
sg_imports --> False
residual_local --> False
residual_global --> False
data_cleaning --> True

Filepath to generation --> None
Filepath to exchanges --> None
Filepath to savedir --> None
Filepath to ui_vector --> None
Filepath to mapping --> None
Filepath to neighbours --> None
Filepath to gap --> None
Filepath to swissGrid --> None
Filepath to networkLosses --> None

Server for useServer --> False
Server for host --> sftp-transparency.entsoe.eu
Server for port --> 22
Server for username --> None
Server for password -->
Server for removeUnused --> False
Server for _remoteGenerationDir --> /TP_export/AggregatedGenerationPerType_16.1.B_C/
Server for _remoteExchangesDir --> /TP_export/PhysicalFlows_12.1.G/

The configuration is composed of 3 parts. The detail about the meaning of each is developed in the input data section. Essentially: * the first block contains the elements to configure the execution itself. These elements are directly available and modifiable with the syntax my_config.element. * the second block deals with all paths to information files, directory containing information, or where to write and save information before, during and after the computation. It is accessible with the syntax my_config.path.element * the third block deals with information related to the ENTSO-E server, as electricity data from the ENTSO-E server is at the center of ecodynelec. More on this topic is covered on the next paragraph and on the dedicated downloading tutorial.

The next cell partly modifies the execution configuration. * First the starting date is modified. Note that objects of the Parameter class will verify if this element is a date, and will raise an error if the format is not recognized. * Then we modify the size of time step (frequency) for the computation. Possibilities are specified in the input data section. * Third in this example, the auto-completing feature is turned off.

## Change the starting date
my_config.start = '2017-02-01 00:00'

## Change the time step
my_config.freq = "15min"

## Change the coutry list
my_config.ctry = ['AT','CH','DE','FR','IT']

## Turn off the auto-complete
my_config.data_cleaning = False

The next cell partly modifies file path configuration. Here we modify the location of directories containing data downloaded from the ENTSO-E database.

# Indicate where to find generation data
my_config.path.generation = "./test_data/generations/"

# Indicate where to find exchange data
my_config.path.exchanges = "./test_data/exchanges/"

Note that, for the generation, exchanges and savedir paths, the specified directory will be created if it does not already exist. For every other file path element, a default file is used if nothing is specified, and an error is returned if the information passed does not correspond to any existing file on your local machine.

Downloading Entso-E data

The downloading tutorial covers the specificities about how to download the ENTSO-E data or include the download as part of the ecodynelec pipeline execution. This feature is not triggered per default and ecodynelec is expecting to find already downloaded ENTSO-E files.

Execution

ecodynelec is build out of a myriad of modules that can be used relatively independently, under the condition that inputs data is shaped the correct way. Fortunately, the entire pipeline starting from a set of parameters and computing down to the calculation of impact metrics.

The usage of this entire pipeline is demonstrated below. This pipeline allows to save results into files (c.f. paragraph on configuration). However results are also always returned for further in-script use. These results are stored in the impacts variable for later paragraphs in this tutorial.

from ecodynelec.pipelines import execute

impacts = execute(config=my_config, is_verbose=True)

Load auxiliary datasets...
Load generation data...
    Generation data.
Data loading: 0.03 sec..
Memory usage table: 0.18 MB
Autocomplete...               5/5)...
=========================
Missing data identified: 8 (0.22%)
                                AT CH DE FR IT
Biomass                          -  -  -  -  -
Fossil Gas                       -  -  -  -  -
Fossil Hard coal                 -  -  -  -  -
Fossil Oil                       -  -  -  -  -
Geothermal                       -  -  -  -  -
Hydro Pumped Storage             -  -  -  8  -
Hydro Run-of-river and poundage  -  -  -  -  -
Hydro Water Reservoir            -  -  -  -  -
Other                            -  -  -  -  -
Other renewable                  -  -  -  -  -
Solar                            -  -  -  -  -
Waste                            -  -  -  -  -
Wind Onshore                     -  -  -  -  -
Nuclear                          -  -  -  -  -
Fossil Brown coal/Lignite        -  -  -  -  -
Fossil Coal-derived gas          -  -  -  -  -
Wind Offshore                    -  -  -  -  -
=========================
Extraction raw generation: 0.13 sec.
    Extraction time: 0.16 sec.
    4/4 - Resample exchanges to 15min steps...
Get and reduce importation data...
    Cross-border flow data.
Data loading: 0.02 sec..
Memory usage table: 0.04 MB
Autocomplete...               ...
=========================
Missing data identified: 1152 (71.38%)
    AT CH  DE  FR  IT
CH   -  -   -   -   -
CZ   -  -   -   -   -
DE   -  -   -   -   -
HU  96  -   -   -   -
IT   -  -   -   -   -
SI  96  -   -   -  96
AT   -  -   -   -   -
FR   -  -   -   -   -
DK   -  -  96   -   -
NL   -  -  96   -   -
PL   -  -  96   -   -
SE   -  -  96   -   -
BE   -  -   -  96   -
ES   -  -   -  96   -
GB   -  -   -  96   -
GR   -  -   -   -  96
MT   -  -   -   -  96
=========================
Extraction raw import: 0.10 sec.
    Extraction time: 0.12 sec.
Resample exchanges to 15min steps...
Gather generation and importation...
Import of data: 0.3 sec
Importing information...
Tracking origin of electricity...
    compute for day 1/1
    Electricity tracking: 1.2 sec.

Compute the electricity impacts...
    Global...
    Carbon intensity...
    Human carcinogenic toxicity...
    Fine particulate matter formation...
    Land use...
Impact computation: 0.0 sec.
Adapt timezone: UTC >> UTC
done.

Outcome and Visualization

The outcome is stored in files and returned for further in-script use. In the previous section, results were stored in the impacts variable. The current section highlights the content returned and shows some basic possibilities for data visualization.

import numpy as np
import pandas as pd

Description of the outcome

The impacts variable contains a collection of tables. This collection is a dict with one Global key, and one other key per impact category:

print(impacts.keys())

dict_keys(['Global', 'Carbon intensity', 'Human carcinogenic toxicity', 'Fine particulate matter formation', 'Land use'])

The Global table is the sum across all technologies for each index, as it is shown for the first few time steps:

display(impacts['Global'].head())
Carbon intensity Human carcinogenic toxicity Fine particulate matter formation Land use
2017-02-01 00:00:00 0.460800 0.030586 0.000353 0.007269
2017-02-01 00:15:00 0.460092 0.030610 0.000353 0.007258
2017-02-01 00:30:00 0.460153 0.030682 0.000353 0.007247
2017-02-01 00:45:00 0.457920 0.030642 0.000348 0.007215
2017-02-01 01:00:00 0.458639 0.030747 0.000349 0.007192

The other tables are, for each impact category, the breakdown into all possible sources:

for i in impacts: # Iterate for all impact categories
    if i=='Global': continue; # Skip the Global, already visualized above.

    print(f"#############\nimpacts for {i}:")
    display( impacts[i].head(3).T ) # Transpose table for readability

#############
impacts for Carbon intensity:
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Carbon intensity_source
Mix_Other 0.006686 0.006624 0.006525
Biomass_AT 0.000330 0.000325 0.000317
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.018306 0.017540 0.017000
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

#############
impacts for Human carcinogenic toxicity:
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Human carcinogenic toxicity_source
Mix_Other 0.000446 0.000442 0.000435
Biomass_AT 0.000023 0.000022 0.000022
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000132 0.000126 0.000122
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

#############
impacts for Fine particulate matter formation:
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Fine particulate matter formation_source
Mix_Other 0.000010 0.000010 0.000010
Biomass_AT 0.000001 0.000001 0.000001
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000006 0.000006 0.000005
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

#############
impacts for Land use:
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Land use_source
Mix_Other 0.000192 0.000191 0.000188
Biomass_AT 0.001014 0.000997 0.000974
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000068 0.000065 0.000063
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

Group per country

The following piece of code suggests a basic visualization of the Carbon intensity category, grouping the results per country of origin of the tracked electricity.

def compute_per_country(results):
    """Function to group results per country"""
    countries = np.unique([c.split("_")[-1] for c in results.columns]) # List of countries

    per_country = []
    for c in countries:
        cols = [k for k in results.columns if k[-3:]==f"_{c}"]
        per_country.append(pd.Series(results.loc[:,cols].sum(axis=1), name=c))

    return pd.concat(per_country,axis=1)

gwp_per_country = compute_per_country(impacts['Carbon intensity']) # Group Carbon intensity index impacts per country
gwp_per_country.plot.area(figsize=(12,4), legend='reverse', color=['r','w','y','b','c','k'],
                          title="Some visualization of the Carbon intensity aggregated per country"); # Build the graph
../_images/graph_CC_country.png

Group per production type

The following piece of code suggests a basic visualization of the Carbon intensity category, grouping the results per technology of origin of the tracked electricity.

def compute_per_type(results):
    """Function to group datasets per type of unit, regardless of the country of origin"""
    unit_list = np.unique([k[:-3] if k[-3]=="_" else k for k in results.columns]) # List the different production units

    per_unit = []
    for u in unit_list:
        cols = [k for k in results.columns if k[:-3]==u] # collect the useful columns
        per_unit.append(pd.Series(results.loc[:,cols].sum(axis=1), name=u)) # aggregate

    return pd.concat(per_unit,axis=1)

es13_per_type = compute_per_type(impacts['Carbon intensity']) # Group Carbon intensity index impacts per country
es13_per_type.plot.area(figsize=(12,8), legend='reverse',
                        title="Some visualization of the Carbon intensity index aggregated per source"); # Build the graph
../_images/graph_CC_source.png