Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Cloudflare IP range importer #572

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add Cloudflare IP range importer #572

wants to merge 5 commits into from

Conversation

fershad
Copy link
Contributor

@fershad fershad commented Apr 7, 2024

Fixes #569

This PR creates a new Importer for Cloudflare's public IP ranges. It fetches data from two text files which reflect the public IPv4 and IPv6 ranges that Cloudflare maintain.

TXT IPv4 list: https://www.cloudflare.com/ips-v4/#
TXT IPv6 list: https://www.cloudflare.com/ips-v6/#

@gitpod-io gitpod.io
Copy link

gitpod-io bot commented Apr 7, 2024

@fershad fershad requested a review from mrchrisadams May 28, 2024 08:08
@fershad
Copy link
Contributor Author

fershad commented May 28, 2024

@mrchrisadams I think this PR is ready for review. I'm not sure how it can be tested or verified though. Will take your lead.

@mrchrisadams
Copy link
Member

hi @fershad ! If the importer works by fetching data, and then importing it, we could check by writing a test to run the importer and see if the provider, has the new IP ranges added. Are you cool with me adding a test along these lines to this PR?

@fershad
Copy link
Contributor Author

fershad commented May 30, 2024

Thanks @mrchrisadams. Yes that kind of test does make sense. Please do add it, it would be a good reference to have for future importers as well.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024 •
edited
Loading

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 25.8505 464.147 3.84 121
checkout 8.77 3.6079 3.61 1
pip install uv wheel 8.88 3.59008 1.80 2
pip install requirements 70.9933 40.1041 5.73 7
pytest 23.6341 416.845 3.82 109

🌳 CO2 Data:
City: Boydton, Lat: 36.6534, Lon: -78.375
IP: 52.167.166.88
CO₂ from energy is: 0.186587094 g
CO₂ from manufacturing (embodied carbon) is: 0.034522971 g
Carbon Intensity for this location: 402 gCO₂eq/kWh
SCI: 0.221110 gCO₂eq / pipeline run emitted

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024 •
edited
Loading

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 26.0133 465.246 3.75 124
checkout 8.02 3.56159 1.78 2
pip install uv wheel 8.12 3.56308 1.78 2
pip install requirements 67.2167 38.7752 5.54 7
pytest 23.5678 419.346 3.81 110

🌳 CO2 Data:
City: San Jose, Lat: 37.1835, Lon: -121.7714
IP: 52.238.27.78
CO₂ from energy is: 0.077230836 g
CO₂ from manufacturing (embodied carbon) is: 0.035378912 g
Carbon Intensity for this location: 166 gCO₂eq/kWh
SCI: 0.112610 gCO₂eq / pipeline run emitted

@mrchrisadams
Copy link
Member

Hey @fershad - I've added:

  1. tests exercising this code
  2. a management command so we can run this on the command line with update_networks_in_db_cloudflare
  3. some cleanup of the implementation code
  4. updates to the cronjob that would run this. For now the job is commented out though.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024 •
edited
Loading

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 28.2749 438.392 3.95 111
checkout 13.23 3.74373 1.87 2
pip install uv wheel 13.34 3.74414 3.74 1
pip install requirements 68.8867 39.3387 4.92 8
pytest 25.8826 391.565 4.00 98

🌳 CO2 Data:
City: Washington, Lat: 38.7095, Lon: -78.1539
IP: 40.76.238.208
CO₂ from energy is: 0.174041624 g
CO₂ from manufacturing (embodied carbon) is: 0.031669833 g
Carbon Intensity for this location: 397 gCO₂eq/kWh
SCI: 0.205711 gCO₂eq / pipeline run emitted

mrchrisadams
mrchrisadams approved these changes Jul 1, 2024
View reviewed changes
Copy link
Member

@mrchrisadams mrchrisadams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi Fish! Thanks for making this! I can confirm this adds the IP ranges listed at the endpoints you listed.

I think this is good to merge in, but we'll likely need to make a change later to actually switch this on as I think it make sense to make sure we have a named technical contact to speak to first before running it on the regular.

ansible/templates/import_ips_for_large_providers.sh.j2
@@ -14,3 +14,6 @@ source .venv/bin/activate
dotenv run -- ./manage.py update_networks_in_db_amazon
dotenv run -- ./manage.py update_networks_in_db_google
dotenv run -- ./manage.py update_networks_in_db_microsoft

# TODO: activate once we have found our technical contact to talk to
# dotenv run -- ./manage.py update_networks_in_db_cloudflare
Copy link
Member

@mrchrisadams mrchrisadams Jul 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the command we would run on weekly import like we do with AWS, GCP, and Microsoft

apps/greencheck/importers/importer_cloudflare.py
ipv6_data = ipv6_response.text

# destructure the lines of each text file, to make one longer list
return *ipv4_data.splitlines(), *ipv6_data.splitlines()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tweaked this to pull out the entries from file, so rather than returning two strings delimited by newlines like this:

ipv4_data, ipv6_data

We return a single list of strings, with each string representing an ipv4 or ipv6 address

*ipv4_data.splitlines(), *ipv6_data.splitlines()

mrchrisadams
mrchrisadams reviewed Jul 1, 2024
View reviewed changes
apps/greencheck/management/commands/update_networks_in_db_cloudflare.py
from django.core.management.base import BaseCommand


class Command(BaseCommand):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command is what executed when we call update_networks_in_db_cloudflare

mrchrisadams
mrchrisadams reviewed Jul 1, 2024
View reviewed changes
greenweb/settings/common.py
@@ -335,6 +335,9 @@
"GOOGLE_DATASET_ENDPOINT", default="https://www.gstatic.com/ipranges/cloud.json"
)

CLOUDFLARE_PROVIDER_ID = env("CLOUDFLARE_PROVIDER_ID", default=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to set some default values here for the app to start up cleanly

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 22.9241 511.045 3.62 141
checkout 10.8 3.67623 1.84 2
pip install uv wheel 10.87 3.66029 3.66 1
pip install requirements 69.0683 39.6261 5.66 7
pytest 20.8754 464.082 3.63 128

🌳 CO2 Data:
City: Phoenix, Lat: 33.4475, Lon: -112.0866
IP: 20.171.122.191
CO₂ from energy is: 0.258077725 g
CO₂ from manufacturing (embodied carbon) is: 0.040229247 g
Carbon Intensity for this location: 505 gCO₂eq/kWh
SCI: 0.298307 gCO₂eq / pipeline run emitted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrchrisadams mrchrisadams mrchrisadams approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create importer script for Cloudflare public IP ranges
2 participants
@fershad @mrchrisadams