Using fully with Python ======================= ``ecodynelec`` offers the possibility to being fully used via Python. This tutorial shows the different steps and possibilities, relying solely on operations doable within a python script or notebook. As the configuration of ``ecodynelec`` pipeline execution may benefit from more visual interaction, an alternative is suggested in the “`Using with a spreadsheet-based configuraion `__” tutorial. Initialization -------------- To download and install ``ecodynelec`` to being used as a python package, the user is referred to either the `getting started tutorial `__. Configuration ------------- The configuration of ``ecodynelec`` is handled by the ``parameter`` module. .. code:: ipython3 from ecodynelec.parameter import Parameter # Import the class to manipulate parameters Python is an object oriented language. Thus a specific configuration can be built and stored in a ``Parameter`` object that will be called ``my_config`` in this tutorial. The next cell only initializes the configuration object with the default parameters. .. code:: ipython3 my_config = Parameter() # Initialize a configuration object Before modifying the configuration, let’s have a look at this default setting .. code:: ipython3 print(my_config) .. parsed-literal:: ctry --> ['AT', 'CH', 'CZ', 'DE', 'FR', 'IT'] target --> CH start --> 2017-02-01 00:00:00 end --> 2017-02-01 23:00:00 freq --> H timezone --> UTC cst_imports --> False net_exchanges --> False network_losses --> False sg_imports --> False residual_local --> False residual_global --> False data_cleaning --> True Filepath to generation --> None Filepath to exchanges --> None Filepath to savedir --> None Filepath to ui_vector --> None Filepath to mapping --> None Filepath to neighbours --> None Filepath to gap --> None Filepath to swissGrid --> None Filepath to networkLosses --> None Server for useServer --> False Server for host --> sftp-transparency.entsoe.eu Server for port --> 22 Server for username --> None Server for password --> Server for removeUnused --> False Server for _remoteGenerationDir --> /TP_export/AggregatedGenerationPerType_16.1.B_C/ Server for _remoteExchangesDir --> /TP_export/PhysicalFlows_12.1.G/ The configuration is composed of 3 parts. The detail about the meaning of each is developed in the `input data section `__. Essentially: \* the first block contains the elements to configure the execution itself. These elements are directly available and modifiable with the syntax ``my_config.element``. \* the second block deals with all paths to information files, directory containing information, or where to write and save information before, during and after the computation. It is accessible with the syntax ``my_config.path.element`` \* the third block deals with information related to the ENTSO-E server, as electricity data from the ENTSO-E server is at the center of ``ecodynelec``. More on this topic is covered on the next paragraph and on the dedicated `downloading tutorial `__. The next cell partly modifies the execution configuration. \* First the starting date is modified. Note that objects of the ``Parameter`` class will verify if this element is a date, and will raise an error if the format is not recognized. \* Then we modify the size of time step (frequency) for the computation. Possibilities are specified in the `input data section `__. \* Third in this example, the auto-completing feature is turned off. .. code:: ipython3 ## Change the starting date my_config.start = '2017-02-01 00:00' ## Change the time step my_config.freq = "15min" ## Change the coutry list my_config.ctry = ['AT','CH','DE','FR','IT'] ## Turn off the auto-complete my_config.data_cleaning = False The next cell partly modifies file path configuration. Here we modify the location of directories containing data downloaded from the ENTSO-E database. .. code:: ipython3 # Indicate where to find generation data my_config.path.generation = "./test_data/generations/" # Indicate where to find exchange data my_config.path.exchanges = "./test_data/exchanges/" Note that, for the ``generation``, ``exchanges`` and ``savedir`` paths, the specified directory *will be created if it does not already exist*. For every other file path element, *a default file* is used if nothing is specified, and an error is returned if the information passed does not correspond to any existing file on your local machine. Downloading Entso-E data ~~~~~~~~~~~~~~~~~~~~~~~~ The `downloading tutorial `__ covers the specificities about how to download the ENTSO-E data or include the download as part of the ``ecodynelec`` pipeline execution. This feature is not triggered per default and ``ecodynelec`` is expecting to find already downloaded ENTSO-E files. Execution --------- ``ecodynelec`` is build out of a myriad of modules that can be used relatively independently, under the condition that inputs data is shaped the correct way. Fortunately, the entire pipeline starting from a set of parameters and computing down to the calculation of impact metrics. The usage of this entire pipeline is demonstrated below. This pipeline allows to save results into files (c.f. paragraph on `configuration `__). However results are also always returned for further in-script use. These results are stored in the ``impacts`` variable for later paragraphs in this tutorial. .. code:: ipython3 from ecodynelec.pipelines import execute .. code:: ipython3 impacts = execute(config=my_config, is_verbose=True) .. parsed-literal:: Load auxiliary datasets... Load generation data... Generation data. Data loading: 0.03 sec.. Memory usage table: 0.18 MB Autocomplete... 5/5)... ========================= Missing data identified: 8 (0.22%) AT CH DE FR IT Biomass - - - - - Fossil Gas - - - - - Fossil Hard coal - - - - - Fossil Oil - - - - - Geothermal - - - - - Hydro Pumped Storage - - - 8 - Hydro Run-of-river and poundage - - - - - Hydro Water Reservoir - - - - - Other - - - - - Other renewable - - - - - Solar - - - - - Waste - - - - - Wind Onshore - - - - - Nuclear - - - - - Fossil Brown coal/Lignite - - - - - Fossil Coal-derived gas - - - - - Wind Offshore - - - - - ========================= Extraction raw generation: 0.13 sec. Extraction time: 0.16 sec. 4/4 - Resample exchanges to 15min steps... Get and reduce importation data... Cross-border flow data. Data loading: 0.02 sec.. Memory usage table: 0.04 MB Autocomplete... ... ========================= Missing data identified: 1152 (71.38%) AT CH DE FR IT CH - - - - - CZ - - - - - DE - - - - - HU 96 - - - - IT - - - - - SI 96 - - - 96 AT - - - - - FR - - - - - DK - - 96 - - NL - - 96 - - PL - - 96 - - SE - - 96 - - BE - - - 96 - ES - - - 96 - GB - - - 96 - GR - - - - 96 MT - - - - 96 ========================= Extraction raw import: 0.10 sec. Extraction time: 0.12 sec. Resample exchanges to 15min steps... Gather generation and importation... Import of data: 0.3 sec Importing information... Tracking origin of electricity... compute for day 1/1 Electricity tracking: 1.2 sec. Compute the electricity impacts... Global... Carbon intensity... Human carcinogenic toxicity... Fine particulate matter formation... Land use... Impact computation: 0.0 sec. Adapt timezone: UTC >> UTC done. Outcome and Visualization ------------------------- The outcome is stored in files and returned for further in-script use. In the previous section, results were stored in the ``impacts`` variable. The current section highlights the content returned and shows some basic possibilities for data visualization. .. code:: ipython3 import numpy as np import pandas as pd Description of the outcome ~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``impacts`` variable contains a collection of tables. This collection is a ``dict`` with one ``Global`` key, and one other key per impact category: .. code:: ipython3 print(impacts.keys()) .. parsed-literal:: dict_keys(['Global', 'Carbon intensity', 'Human carcinogenic toxicity', 'Fine particulate matter formation', 'Land use']) The ``Global`` table is the *sum across all technologies* for each index, as it is shown for the first few time steps: .. code:: ipython3 display(impacts['Global'].head()) .. raw:: html
Carbon intensity Human carcinogenic toxicity Fine particulate matter formation Land use
2017-02-01 00:00:00 0.460800 0.030586 0.000353 0.007269
2017-02-01 00:15:00 0.460092 0.030610 0.000353 0.007258
2017-02-01 00:30:00 0.460153 0.030682 0.000353 0.007247
2017-02-01 00:45:00 0.457920 0.030642 0.000348 0.007215
2017-02-01 01:00:00 0.458639 0.030747 0.000349 0.007192
The other tables are, for each impact category, the breakdown into all possible sources: .. code:: ipython3 for i in impacts: # Iterate for all impact categories if i=='Global': continue; # Skip the Global, already visualized above. print(f"#############\nimpacts for {i}:") display( impacts[i].head(3).T ) # Transpose table for readability .. parsed-literal:: ############# impacts for Carbon intensity: .. raw:: html
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Carbon intensity_source
Mix_Other 0.006686 0.006624 0.006525
Biomass_AT 0.000330 0.000325 0.000317
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.018306 0.017540 0.017000
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

.. parsed-literal:: ############# impacts for Human carcinogenic toxicity: .. raw:: html
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Human carcinogenic toxicity_source
Mix_Other 0.000446 0.000442 0.000435
Biomass_AT 0.000023 0.000022 0.000022
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000132 0.000126 0.000122
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

.. parsed-literal:: ############# impacts for Fine particulate matter formation: .. raw:: html
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Fine particulate matter formation_source
Mix_Other 0.000010 0.000010 0.000010
Biomass_AT 0.000001 0.000001 0.000001
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000006 0.000006 0.000005
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

.. parsed-literal:: ############# impacts for Land use: .. raw:: html
2017-02-01 00:00:00 2017-02-01 00:15:00 2017-02-01 00:30:00
Land use_source
Mix_Other 0.000192 0.000191 0.000188
Biomass_AT 0.001014 0.000997 0.000974
Fossil_Brown_coal/Lignite_AT 0.000000 0.000000 0.000000
Fossil_Coal-derived_gas_AT 0.000000 0.000000 0.000000
Fossil_Gas_AT 0.000068 0.000065 0.000063
... ... ... ...
Other_renewable_IT 0.000000 0.000000 0.000000
Solar_IT 0.000000 0.000000 0.000000
Waste_IT 0.000000 0.000000 0.000000
Wind_Offshore_IT 0.000000 0.000000 0.000000
Wind_Onshore_IT 0.000000 0.000000 0.000000

101 rows × 3 columns

Group per country ~~~~~~~~~~~~~~~~~ The following piece of code suggests a basic visualization of the Carbon intensity category, grouping the results per country of origin of the tracked electricity. .. code:: ipython3 def compute_per_country(results): """Function to group results per country""" countries = np.unique([c.split("_")[-1] for c in results.columns]) # List of countries per_country = [] for c in countries: cols = [k for k in results.columns if k[-3:]==f"_{c}"] per_country.append(pd.Series(results.loc[:,cols].sum(axis=1), name=c)) return pd.concat(per_country,axis=1) .. code:: ipython3 gwp_per_country = compute_per_country(impacts['Carbon intensity']) # Group Carbon intensity index impacts per country gwp_per_country.plot.area(figsize=(12,4), legend='reverse', color=['r','w','y','b','c','k'], title="Some visualization of the Carbon intensity aggregated per country"); # Build the graph .. image:: images/graph_CC_country.png Group per production type ~~~~~~~~~~~~~~~~~~~~~~~~~ The following piece of code suggests a basic visualization of the Carbon intensity category, grouping the results per technology of origin of the tracked electricity. .. code:: ipython3 def compute_per_type(results): """Function to group datasets per type of unit, regardless of the country of origin""" unit_list = np.unique([k[:-3] if k[-3]=="_" else k for k in results.columns]) # List the different production units per_unit = [] for u in unit_list: cols = [k for k in results.columns if k[:-3]==u] # collect the useful columns per_unit.append(pd.Series(results.loc[:,cols].sum(axis=1), name=u)) # aggregate return pd.concat(per_unit,axis=1) .. code:: ipython3 es13_per_type = compute_per_type(impacts['Carbon intensity']) # Group Carbon intensity index impacts per country es13_per_type.plot.area(figsize=(12,8), legend='reverse', title="Some visualization of the Carbon intensity index aggregated per source"); # Build the graph .. image:: images/graph_CC_source.png