Back out "remove unsupported/unused benchmarks that cause test warnin…

…gs in production diffs"

Summary:
Reason for the undo: The change in the original diff led to the deletion of the entire repo content on GitHub: 1f2db20

This is because ShipIt is configured to sync to GitHub: https://www.internalfb.com/code/configerator/source/opensource/shipit_config/facebookresearch/pplbench.cconf?lines=10

The suggested strategy to resolve this situation is now:
1. Land the present diff.
2. Wait for the GitHub repo to receive the undo.
3. Remove the ShipIt config.
4. Re-do the change in  D48647611.

----
Alternative plan:
1. Undo the deletion commit/PR on GitHub.
2. Remove the ShipIt config.
3. No need to undo and redo the change in the Meta repos.

However, I don't think I have the permissions to perform the actions on GitHub, so the alternative plan would have to be executed by someone else.

----
Original commit changeset: b41d436b2838

Original Phabricator Diff: D48647611

Reviewed By: ToddSmall

Differential Revision: D48819817

fbshipit-source-id: 8189115e9de93c5230fa44289d5a9f968b281892

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,15 @@
+### Issue Description
+A clear and concise description of the issue. If it's a feature request, please add [Feature Request] to the title.
+
+### Steps to Reproduce
+Please provide steps to reproduce the issue attaching any error messages and stack traces.
+
+### Expected Behavior
+What did you expect to happen?
+
+### System Info
+Please provide information about your setup
+- PyTorch Version (run `print(torch.__version__)`
+- Python version
+
+### Additional Context

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,23 @@
+### Motivation
+Please describe your motivation for the changes. Provide link to any related issues.
+
+### Changes proposed
+Outline the proposed changes and alternatives considered.
+
+### Test Plan
+Please provide clear instructions on how the changes were verified. Attach screenshots if applicable.
+
+### Types of changes
+- [ ] Docs change / refactoring / dependency upgrade
+- [ ] Bug fix (non-breaking change which fixes an issue)
+- [ ] New feature (non-breaking change which adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+
+### Checklist
+- [ ] My code follows the code style of this project.
+- [ ] My change requires a change to the documentation.
+- [ ] I have updated the documentation accordingly.
+- [ ] I have read the **[CONTRIBUTING](https://github.com/facebookresearch/pplbench/blob/main/CONTRIBUTING.md)** document.
+- [ ] I have added tests to cover my changes.
+- [ ] All new and existing tests passed.
+- [ ] The title of my pull request is a short description of the requested changes.

.github/workflows/gh-pages.yml

@@ -0,0 +1,36 @@
+name: gh-pages
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    types:
+      - opened
+      - synchronize
+      - reopened
+
+jobs:
+  build_docs_job:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Build the Website
+      id: build
+      run: |
+        cd website
+        yarn
+        yarn build
+
+    - name: Get output time
+      run: echo "The time was ${{ steps.build.outputs.time }}"
+
+    - name: Deploy
+      if: github.event_name == 'push'  # only deploy the website when pushing to main
+      uses: JamesIves/github-pages-deploy-action@releases/v3
+      with:
+          ACCESS_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BRANCH: gh-pages  # The branch the action should deploy to.
+          FOLDER: website/build  # The folder the action should deploy.

.github/workflows/main.yml

@@ -0,0 +1,166 @@
+# this name will be shown on the status badge in README.md in the form "build | passing"
+name: build
+
+on:
+  push:
+    branches-ignore:
+      - gh-pages
+  pull_request:
+    types:
+      - opened
+      - synchronize
+      - reopened
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    strategy:
+      matrix:
+        python-version: [3.7]
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip
+
+    - name: Install linters
+      run: >
+        pip install black==22.3.0 flake8==4.0.1 libcst==0.4.1 ufmt==1.3.2 usort==1.0.2
+        mypy==0.782
+
+    - name: Lint with flake8
+      run: flake8 .
+
+    - name: Lint with ufmt (black + usort)
+      run: ufmt check .
+
+    - name: Install PPL Bench core
+      run: pip install .
+
+    - name: Lint with mypy
+      run: mypy -p pplbench
+
+  find_example_json:
+    runs-on: ubuntu-latest
+    outputs:
+      files: ${{ steps.glob.outputs.files }}
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Recursively search for example JSON files
+      id: glob
+      run: |
+        JSON_FILES=$(find . -type f -wholename '*/examples/*.json' \
+          | jq -Rsc 'split("\n")[:-1]')  # format output of `find` to an JSON array
+        echo "::set-output name=files::$(echo $JSON_FILES)"
+
+  test_example_json:
+    name: Test ${{ matrix.test-file }} (Python ${{ matrix.python-version  }})
+    runs-on: ubuntu-latest
+    needs:
+      - find_example_json
+    strategy:
+      matrix:
+        python-version: [3.7]
+        test-file: ${{ fromJson(needs.find_example_json.outputs.files) }}
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip
+
+    - name: Install PPL Bench with all PPLs
+      run: |
+        pip install .[ppls]
+        # Install Jags-related dependencies
+        sudo apt install pkg-config jags
+        pip install pyjags
+
+    - name: Run PPL Bench with ${{ matrix.test-file }}
+      run: pplbench ${{ matrix.test-file }}
+
+  check_version:
+    runs-on: ubuntu-latest
+    # check the current repository so the pypi release workflow won't be run on forks
+    if: github.event_name == 'push' && github.repository == 'facebookresearch/pplbench'
+    needs:
+      - lint
+      - test_example_json
+    outputs:
+      has_updated: ${{ steps.version.outputs.has_updated }}
+      version_tag: ${{ steps.version.outputs.version_tag }}
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python 3.7
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.7
+
+    - name: See if a tag exists for current version of PPL Bench
+      id: version
+      run: |
+        VERSION_TAG="v$(python setup.py --version)"
+        if  [[ $(git ls-remote --tags origin refs/tags/$VERSION_TAG) ]]; then
+          # there exists a tag with same name as current version
+          echo "::set-output name=has_updated::false"
+        else
+          echo "::set-output name=has_updated::true"
+        fi
+        echo "::set-output name=version_tag::$(echo $VERSION_TAG)"
+        echo "The current version of PPL Bench is $VERSION_TAG"
+
+  release_and_upload_pplbench:
+    name: Release PPL Bench
+    runs-on: ubuntu-latest
+    needs:
+      - check_version
+    if: needs.check_version.outputs.has_updated == 'true'
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python 3.7
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.7
+
+    - name: Upgrade pip
+      run: python -m pip install --upgrade pip
+
+    - name: Install package dependencies
+      run: pip install setuptools wheel
+
+    - name: Build source distribution
+      run: python setup.py sdist
+
+    - name: Build Python 3 wheel
+      run: python setup.py bdist_wheel
+
+    - name: Upload PPL Bench to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        password: ${{ secrets.PYPI_PASSWORD }}
+
+    - name: Create new release on GitHub
+      uses: actions/create-release@v1
+      with:
+        tag_name: ${{ needs.check_version.outputs.version_tag }}
+        release_name: PPL Bench ${{ needs.check_version.outputs.version_tag }}
+      env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -0,0 +1,12 @@
+# we don't want to push changes to example.json by default
+./example.json
+# ignore compilation artifacts
+*.pyc
+*.swp
+beanmachine.log
+
+# Don't track content of these folders
+.vscode/
+*.egg-info/
+outputs/
+**/__pycache__/

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <opensource-conduct@fb.com>. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,55 @@
+# Contributing to PPL Bench
+
+We welcome any contributions - whether it is adding a new model, adding a new PPL  implementation, or modifying an existing PPL implementation.
+
+### How to add a new PPL?
+
+To add a new PPL, first, create a new directory in `pplbench/ppls`. Then, create implementations of `BasePPLInference` specific to the added PPL and chosen inference algorithms available for that system. For example, `pplbench/ppls/stan/inference.py` implements `BaseStanInference` which serves as a basis for two inference algorithms offered by the Stan system: `MCMC` and `VI`.
+
+These inference classes must be able to solve a model implementation object provided to them as an argument. This object must be of a type derived from `BasePPLImplementation`. In the Stan example, the class `BaseStanImplementation` (in `pplbench/ppls/stan/base_stan_impl.py`) provides a Stan-specific interface for  information about Stan models for the inference algorithms to use, and specific model classes are sub-classes of `BaseStanImplementation` (for example, `RobustRegression` in `pplbench/ppls/stan/robust_regression.py`).
+
+### How to add a new model?
+
+To add a new model, you need to write code to generate and test data according to this new model (this data will be shared by all PPLs for evaluation), as well as code for encoding and running inference for this new model for each PPL that will be applied to it.
+
+#### Generating and testing data for a New Model
+
+To write the code to generate and test the data for the model, create a new file in `pplbench/models`. The model will need to inherit from `BaseModel` and implement two methods — `generate_data` and `evaluate_posterior_predictive`.
+
+`generate_data` should return the train and test data by simulating from the generative model.
+
+`evaluate_posterior_predictive` should return the predictive log likelihood of the data given samples from the inferred parameters.
+
+#### Encoding the model in a particular PPL
+
+To encode a new model with a PPL, you need to add a sub-class of `BasePPLImplementation` corresponding to it. One example is the aforementioned `RobustRegression` in `pplbench/ppls/stan/robust_regression.py`, but you can define your own new models by writing classes like that.
+
+Note that, for convenience, a `BaseStanImplementation` class was derived from `BasePPLImplementation` and defines a shared interface for all Stan model implementations. A similar pattern applies to the other PPLs. Therefore you will very likely want to reuse such base classes for convenience when defining new models for these systems, or define such an analogous base class when introducing a completely new PPL.
+
+## Pull Requests
+We welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to pplbench, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.