enr_residual_utils

Helper functions to load and preprocess the renewable electricity data from Pronovo and EnergyCharts.

ecodynelec.preprocessing.enr_residual_utils.data_mappings = {'Biogas': [{'end': 'last', 'series': 'Biogas', 'source': 'Pronovo', 'start': '2020-05-01'}], 'Biomass_1_crops': [{'end': '2022-12-31', 'series': 'Biomass_1_crops', 'source': 'Pronovo', 'start': '2020-05-01'}, {'end': 'last', 'series': 'Biomass_1_crops', 'source': 'Pronovo', 'start': '2023-01-01'}], 'Biomass_2_waste': [{'end': '2021-12-31', 'series': 'Biomass_2_waste', 'source': 'Pronovo', 'start': '2020-05-01'}, {'end': '2022-12-31', 'from_end': '2021-12-31', 'from_start': '2021-01-01', 'series': 'Biomass_2_waste', 'source': 'Pronovo', 'start': '2022-01-01'}, {'end': 'last', 'series': 'Biomass_2_waste', 'source': 'Pronovo', 'start': '2023-01-01'}], 'Hydro_Pumpage': [{'end': 'last', 'series': 'Hydro_Pumpage', 'source': 'Pronovo', 'start': '2023-01-01'}], 'Hydro_Pumped_Storage': [{'end': 'last', 'series': 'Hydro_Pumped_Storage', 'source': 'Pronovo', 'start': '2023-01-01'}], 'Sewage_gas': [{'end': 'last', 'series': 'Sewage_gas', 'source': 'Pronovo', 'start': '2020-05-01'}], 'Solar': [{'end': 'last', 'series': 'Solar', 'source': 'Pronovo', 'start': '2020-01-01'}, {'end': 'last', 'from_end': 'last', 'from_start': '2020-01-01', 'series': 'Solar', 'source': 'EC', 'start': '2020-01-01'}], 'Waste': [{'end': '2022-09-30', 'series': 'Waste_1', 'source': 'Pronovo', 'start': '2020-05-01'}, {'end': '2022-11-30', 'from_end': '2021-11-30', 'from_start': '2021-10-01', 'series': 'Waste_1', 'source': 'EC', 'start': '2022-10-01'}, {'end': 'last', 'series': 'Waste_1', 'source': 'Pronovo', 'start': '2023-01-01'}], 'Wind': [{'end': 'last', 'series': 'Wind', 'source': 'Pronovo', 'start': '2020-01-01'}]}

The reorganization rules are only valid for the 2020-2024 period.

A dict giving the mapping between the columns of the final data (result) and the source data from Pronovo and EnergyCharts. Used by the reorganize_enr_data method.

Should follow the structure :

'Category1': [
    {
        'start': 'start date', # start date in the result (and in source data if from_start isn't set)
        'end': 'end date', # end date in the result (and in source data if from_end isn't set) (use 'last' to get the real end of the source data)
        'from_start': '2021-01-01', # optional, start date in the source data (allows to copy data from one date to another)
        'from_end': '2021-12-31', # optional, end data in the source data
        'source': 'Pronovo' or 'EC', # source of the data
        'series': 'Series_Name_In_Source_Data' # name of the column to take in the source data
    },
    ...
],
...

ecodynelec.preprocessing.enr_residual_utils.ec_types_to_types = {'Biogaz': 'Biogas', "Biogaz de station d'épuration": 'Sewage_gas', 'Biomasse': 'Biomass_all', 'Cultures énergétiques': 'Biomass_1_crops', 'Déchets forestiers et agricoles': 'Biomass_2_waste', "Gaz d'égout": 'Sewage_gas', 'Incinération': 'Waste_1', 'Photovoltaïque': 'Solar', 'Éolienne': 'Wind'}: Mapping linking energy charts column names to the actual types of plants used in this project.

ecodynelec.preprocessing.enr_residual_utils.get_enr_data_from_pronovo_ec(path_dir, verbose=False)[source]

Reads the pronovo and energy charts data from the given directory, and returns a dataframe containing the reorganized data. The reorganized data is the best estimation of the real renewable electricity productions (solar, wind, waste…), from what is available.

The reorganization rules are only valid for the 2020-2022 period.

Parameters:

path_dir (str) – The path of the directory containing the pronovo and energy charts data.

The directory should contain two subdirectories:
- pronovo_data: containing the pronovo data (a ‘prod_year’ directory for each input)
- ec_data: containing the energy charts data (annual files)
See the documentation below for more details.
verbose (bool, optional) – Whether to print debug information. The default is False.

Returns:

mapped_data – A dataframe containing the reorganized data, indexed by date.

Return type:

pd.DataFrame

ecodynelec.preprocessing.enr_residual_utils.load_all_pronovo_files(root_dir: str, types: [<class 'str'>], verbose: bool = False) → DataFrame[source]

Loads all pronovo files in the given directories, applying daily scaling with energy charts ecd_enr_model (the hourly variation comes from the pronovo ecd_enr_model, and the daily total from energy charts ecd_enr_model, if available). The scaling is done with csv files starting by “EC”. All other csv files are considered as pronovo files.

Parameters:

root_dir – The root directory containing the pronovo ‘prod_year’ directories
types – The types of plants to extract (in [‘Wind’, ‘Solar’])
verbose – Whether to print debug information

Returns:

A dataframe containing the pronovo ecd_enr_model for all ‘types’, indexed by date

ecodynelec.preprocessing.enr_residual_utils.load_pronovo_file(file: str, types: [<class 'str'>], verbose: bool = False) → DataFrame[source]

Load pronovo ecd_enr_model from a csv file. Supports years from 2020 to 2022 (historically the format of the files changes every semester).

Parameters:

file – the path of the file to load
types – The types of plants to extract (in [‘Wind’, ‘Solar’])
verbose – Whether to print debug information

Returns:

A dataframe containing the pronovo ecd_enr_model for all ‘types’, indexed by date

ecodynelec.preprocessing.enr_residual_utils.pronovo_types_map = {'*': {'Biogas': 'Biogas (kWh)', 'Biomass_1_crops': 'Energiepflanze (kWh)', 'Biomass_2_waste': 'Forst- und Landwirtschaftliche Abfälle (kWh)', 'Sewage_gas': 'Klärgas (kWh)', 'Solar': 'Photovoltaik (kWh)', 'Waste_1': 'Kehrichtverbrennung (erneuerbar) (kWh)', 'Wind': 'Wind (kWh)'}, '2': {'Biogas': '-A.Biogas [kWh]', 'Biomass_1_crops': '-A.Energiepflanze [kWh]', 'Biomass_2_waste': '-A.Forst- und Landwirtschaftliche Abfälle [kWh]', 'Hydro_Pumpage': '+A.Pumpspeicherkraftwerk [kWh]', 'Hydro_Pumped_Storage': '-A.Pumpspeicherkraftwerk [kWh]', 'Sewage_gas': '-A.Klärgas [kWh]', 'Solar': '-A.Photovoltaik [kWh]', 'Waste_1': '-A.Kehrichtverbrennung [kWh]', 'Waste_2.50': '-A.Kehrichtverbrennung (erneuerbar).50 [kWh]', 'Waste_3.100': '-A.Kehrichtverbrennung (erneuerbar).100 [kWh]', 'Waste_4_no_enr': '-A.Kehrichtverbrennung (nicht erneuerbar) [kWh]', 'Wind': '-A.Windenergie [kWh]'}, '5': {'Biomass_all': 'Biomasse', 'Solar': 'Photovoltaic', 'Wind': 'Wind'}, '7': {'Biogas': '-A.Biogas [kWh]', 'Biomass_1_crops': '-A.Energiepflanze [kWh]', 'Biomass_2_waste': '-A.Forst- und Landwirtschaftliche Abfälle [kWh]', 'Hydro_Pumpage': '+A.Pumpspeicherkraft [kWh]', 'Hydro_Pumped_Storage': '-A.Pumpspeicherkraft [kWh]', 'Sewage_gas': '-A.Klärgas [kWh]', 'Solar': '-A.Photovoltaik [kWh]', 'Waste_1': '-A.Kehrichtverbrennung [kWh]', 'Waste_2.50': '-A.Kehrichtverbrennung (erneuerbar).50 [kWh]', 'Waste_3.100': '-A.Kehrichtverbrennung (erneuerbar).100 [kWh]', 'Waste_4_no_enr': '-A.Kehrichtverbrennung (nicht erneuerbar) [kWh]', 'Wind': '-A.Windturbine [kWh]'}}: Mapping linking pronovo column names to the actual types of plants used in this project. There is a different mapping for each pronovo file format (see load_pronovo_file).

ecodynelec.preprocessing.enr_residual_utils.read_enr_data_from_energy_charts(path_dir, verbose=False)[source]

Reads all the energy charts data from the given directory, and returns a dataframe containing the data.

Parameters:

path_dir (str) – The path of the directory containing the energy charts data. The directory should contain a subdirectory named ‘ec_data’. The ‘ec_data’ directory should contain the yearly energy charts data files (.csv files). See the documentation below for more details.
verbose (bool, optional) – Whether to print debug information. The default is False.

Returns:

df_ec_data – A dataframe containing the energy charts data, indexed by date.

Return type:

pd.DataFrame

ecodynelec.preprocessing.enr_residual_utils.read_enr_data_from_pronovo(path_dir, verbose=False)[source]

Reads all the pronovo data from the given directory, and returns a dataframe containing the data.

Parameters:

path_dir (str) – The path of the directory containing the pronovo data. The directory should contain a subdirectory for each year, named ‘prod_year’ (e.g. ‘prod_2020’). Each subdirectory should contain the pronovo data files of this year (.csv files). See the documentation below for more details.
verbose (bool, optional) – Whether to print debug information. The default is False.

Returns:

pronovo_data – A dataframe containing the pronovo data, indexed by date.

Return type:

pd.DataFrame

ecodynelec.preprocessing.enr_residual_utils.reorganize_enr_data(pronovo_data: DataFrame, ec_data: DataFrame) → DataFrame[source]

Reorganizes the pronovo and energy charts data to match the final data format.
The reorganized data is the best estimation of the real renewable electricity productions, from what is available.
The reorganization rules are only valid for the 2020-2024 period.

Parameters:

pronovo_data (pd.DataFrame) – The pronovo data, indexed by date.
ec_data (pd.DataFrame) – The energy charts data, indexed by date.

Returns:

mapped_data – A dataframe containing the reorganized data, indexed by date.

Return type:

pd.DataFrame

Pronovo and EnergyCharts data downloading :

Pronovo data

The Pronovo data can be found on Pronovo, under the “Profils courbe de charge selon technologies” section.

The downloaded .csv files should then be placed in a ‘support_files/pronovo_data/prod_year’ (where year is the electricity production year) folder in the project directory. For example:

As the solar production given by Pronovo doesn’t covers all the solar production in Switzerland, this hourly data is scaled using the daily solar production given by EnergyCharts: you should also download the daily EnergyCharts solar production data (see below) and place it in ‘EC_Solar_year.csv’ files.

EnergyCharts data

The EnergyCharts data is downloaded from EnergyCharts:

EnergyCharts download — *Export settings*

EnergyCharts download categories — *Categories selection*

The downloaded .csv files should then be placed in a ‘support_files/ec_data/’ folder in the project directory.