The Open Grid Emissions Initiative seeks to fill a critical need for high-quality, publicly-accessible, hourly grid emissions data that can be used for GHG accounting, policymaking, academic research, and energy attribute certificate markets. The initiative includes this repository of open-source grid emissions data processing tools that use peer-reviewed, well-documented, and validated methodologies to create the accompanying public dataset of hourly, monthly, and annual U.S. electric grid generation, GHG, and air pollution data.
Please check out our documentation for more details about the Open Grid Emissions methodology.
The Open Grid Emissions Dataset can be downloaded here. An archive of previous versions of the dataset and intermediate data outputs (for research and validation purposes) can be found on Zenodo.
To manage the code environment necessary to run the OGE data pipeline, either pipenv
or conda
may be used. Currently, we utilize pipenv
as our preferred environment manager for running the pipeline that is used for data releases, but conda
will also work if you are more familiar with conda
.
First, navigate to the folder where you want to save the repository and run the following commands:
Note that this option requires to have Python and git installed on your machine.
pip install pipenv
git clone https://github.com/singularity-energy/open-grid-emissions.git
cd open-grid-emissions
pipenv sync
pipenv shell
pip install build
python -m build
pip install .
conda install git
git clone https://github.com/singularity-energy/open-grid-emissions.git
conda update conda
cd open-grid-emissions
conda env create -f environment.yml
conda activate open_grid_emissions
pip install build
python -m build
pip install .
The pipeline can be run as follows:
cd src/oge
python data_pipeline.py --year 2022
independently of the installation method you chose.
A more detailed walkthrough of these steps can be found below in the "Development Setup" section.
The latest release includes data for year 2005-2022 covering the contiguous United States, Alaska, and Hawaii. In future releases, we plan to expand the geographic coverage to additional U.S. territories (dependent on data availability).
Parts of the input data used for the Open Grid Emissions dataset is released by the U.S. Energy Information Administration in the Autumn following the end of each year (2022 data was published in September 2023). Each release will include the most recent year of available data as well as updates of all previous available years based on any updates to the OGE methodology. All previous versions of the data will be archived on Zenodo.
Updated datasets will also be published whenever a new version of the open-grid-emissions repository is released.
There are many ways that you can contribute!
- Tell us how you are using the dataset or python tools
- Request new features or data outputs by submitting a feature request or emailing us at <>
- Tell us how we can make the datasets even easier to use
- Ask a question about the data or methods in our discussion forum
- Submit an issue if you've identified a way the methods or assumptions could be improved
- Contribute your subject matter expertise to the discussion about open issues and questions
- Submit a pull request to help us fix open issues
anomaly_screening
: classes use to flag timeseries for anomalies as proposed in Tyler H. Ruggles et al. Developing reliable hourly electricity demand data through screening and imputation (2020)column_checks
: functions that check that all data outputs have the correct column namesconstants
: specifies conversion factors and constants used across all modulesdata_pipeline
: main script for running the data pipeline from start to finishdownload_data
: functions that download data from the internetdata_cleaning
: functions that clean loaded dataeia930
: functions for cleaning and formatting EIA-930 dataemissions
: functions used for imputing emissions datafilepaths
: used to identify where repository files are located on the user's computergross_to_net_generation
: functions for identifying subplants and gross to net generation conversion factorshelpers
: functions that are used across modulesimpute_hourly_profiles
: functions related to assigning an hourly profile to monthly dataload_data
: functions for loading data from downloaded filesoutput_data
: functions for writing intermediate and final data to csvssubplant_identification
: functions for identifying subplant IDsvalidation
: functions for testing and validating data outputsvisualization
: functions for visualizing data in notebooks
Notebooks are organized into five directories based on their purpose
explore_data
: notebooks used for exploring data outputs and resultsexplore_methods
: notebooks that can be used to explore specific methods step-by-stepmanual_data
: notebooks that are used to create/update certain files indata/manual
validation
: notebooks related to validating resultsvisualization
: notebooks used to visualize datawork_in_progress
: temporary notebooks being used for development purposes on specific branches
All manual reference tables are stored in src/oge/reference_tables
.
All files downloaded/created as part of the pipeline are stored in your HOME directory (e.g. users/user.name/):
$HOME/open_grid_emissions_data/downloads
contains all files that are downloaded by functions inload_data
$HOME/open_grid_emissions_data/outputs
contains intermediate outputs from the data pipeline... any files created by our code that are not final results$HOME/open_grid_emissions_data/results
contains all final output files that will be published
OGE is not yet available on PyPi but can be installed from GitHub. For example, this can be done by adding oge = {git="https://github.com/singularity-energy/open-grid-emissions.git"}
to your Pipfile if you are using pipenv
for your project.
Note that you don't need to run the pipeline to generate the output data as these are available on Amazon Simple Storage Service (S3). Simply, set the OGE_DATA_STORE
environment variable to s3
in the __init__.py file of your project to fetch OGE data from Amazon S3.
To summarize, your __init__.py file would then look like this:
import os
os.environ["OGE_DATA_STORE"] = "s3"
If you would like to run the code on your own computer and/or contribute updates to the code, the following steps can help get you started.
This installation is recommended if you are unfamiliar with git and Python.
We suggest using miniconda or Anaconda to manage the packages needed to run the Open Grid Emissions code. Anaconda and Miniconda install a similar environment, but Anaconda installs more packages by default and Miniconda installs them as needed. These can be downloaded from miniconda or Anaconda
In order to download the repository, you will need to use git. You can either install Git Bash from https://git-scm.com/downloads, or you can install it using conda. To do so, after installing Anaconda or Miniconda, open an Anaconda Command Prompt (Windows) or Terminal.app (Mac) and type the following command:
conda install git
Then you will need set up git following these instructions: https://docs.github.com/en/get-started/quickstart/set-up-git
Using Anaconda command prompt or Git Bash, use the cd
and mkdir
commands to create and/or enter the directory where you would like to download the code (e.g. "Users/myusername/GitHub"). Then run:
git clone https://github.com/singularity-energy/open-grid-emissions.git
Open anaconda prompt, use cd
to navigate to the directory where your local files are stored (e.g. "GitHub/open-grid-emissions"), and then run:
conda update conda
conda env create -f environment.yml
Installation requires that the conda channel-priority be set to "flexible". This is the default behavior, so if you've never manually changed this, you shouldn't have to worry about this. However, if you receive an error message like "Found conflicts!" when trying to install the environment, try setting your channel priority to flexible by running the following command:conda config --set channel_priority flexible
and then re-running the above commands.
The final step is to install the oge
package itself in the conda environment. To do so, run:
conda activate open_grid_emissions
pip install build
python -m build
pip install --editable .
The open_grid_emissions conda environment should now be set up and ready to run.
We recommend that you use Python 3.11. If you don't have Python installed, we recommend that you use pyenv. It lets you easily switch between multiple versions of Python. You will also need to use git to clone the repository. It can be installed from https://git-scm.com/downloads,
This can be done via:
pip install pipenv
As mentioned previously, clone the repository with:
git clone https://github.com/singularity-energy/open-grid-emissions.git
and navigate to the root of the directory:
cd open-grid-emissions
In the root of the directory, create and activate the environment with:
# set up virtual environment (use whichever version of python 3.11 you have installed)
pipenv --python 3.11.4
# if you have updated the pipfile and need to update pipfile.lock, run
pipenv install
# Otherwise, if you just want to install packages from the pipfile.lock, run
pipenv sync
# activate virtual environment
pipenv shell
# install an editable version of the oge package
pip install build
python -m build
pip install –-editable .
If you ever need to remove and reinstall the environment, run pipenv --rm
from the root directory then follow the directions above.
If you would like to run the full data pipeline to generate all intermediate outputs and results files, navigate to open-grid-emissions/src/oge
, and run the following (replacing 2022 with whichever year you want to run):
python data_pipeline.py --year 2022
From time to time, the code will be updated on GitHub. To ensure that you are keeping your local version of the code up to date, open git bash and follow these steps:
# change the directory to where ever your local git repository is saved
# after hitting enter, it should show the name of the git branch (e.g. "(main)")
cd GitHub/open-grid-emissions
# save any changes that you might have made locally to your copy of the code
git add .
# fetch and merge the updated code from github
git pull origin main
If you want to edit the code and do not already have an integrated development environment (IDE) installed, one good option is Visual Studio Code (download: https://code.visualstudio.com/).
If you plan on contributing edits to the codebase that will be merged into the main branch, please follow these best practices:
-
Please do not make edits directly to the main branch. Any new features or edits should be completed in a new branch. To do so, open git bash, navigate to your local repo (e.g.
cd GitHub/open-grid-emissions
), and create a new branch, giving it a descriptive name related to the edit you will be doing:git checkout -b branch_name
-
As you code, it is a good practice to 'save' your work frequently by opening git bash, navigating to your local repo (
cd GitHub/open-grid-emissions
), making sure that your current feature branch is active (you should see the feature name in parentheses next to the command line), and runninggit add .
-
You should commit your work to the branch whenever you have working code or whenever you stop working on it using:
git add .
git commit -m "short message about updates"
-
Once you are done with your edits, save and commit your code using step #3 and then push your changes:
git push
-
Now open the GitHub repo web page. You should see the branch you pushed up in a yellow bar at the top of the page with a button to "Compare & pull request".
- Click "Compare & pull request". This will take you to the "Open a pull request" page.
- From here, you should write a brief description of what you actually changed.
- Click "Create pull request"
- The changes will be reviewed and discussed. Once any edits have been made, the code will be merged into the main branch.
- We generally follow the naming conventions used by the Public Utility Data Liberation Project: https://catalystcoop-pudl.readthedocs.io/en/latest/dev/naming_conventions.html
- Functions should include descriptive docstrings (using the Google style guide https://google.github.io/styleguide/pyguide.html#383-functions-and-methods), inline comments should be used to describe individual steps, and variable names should be made descriptive (e.g.
cems_plants_with_missing_co2_data
notcems_missing
orcpmco2
) - All pandas merge operations should include the
validate
parameter to ensure that unintentional duplicate entries are not created (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html) - All pandas groupby operations should include the
dropna=False
parameter so that data with missing groupby keys are not unintentionally dropped from the data. - All code should be formatted using
ruff
, runningruff format
in the root of the repository will format all files according to the set of configurations enclosed in thepyproject.toml
file. - Clear all outputs from notebooks before committing your work.
- Any manual changes to reported categorical data, conversion factors, or manual data mappings should be loaded from a .csv file
src/oge/reference_tables
rather than stored in a dictionary or variable in the code.