Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[global] Replace use of .format() with f-strings #3505

Merged
merged 1 commit into from
Feb 22, 2024

Conversation

TurboTurtle
Copy link
Member

As part of the project's effort to modernize on f-strings, this commit replaces all usage of .format() with f-string equivalents. The vast majority of these are in-place syntax changes, but a limited number change the order of list items or some formatting tricks in order to appease PEP8.

None of these conversions change the underlying logic of the flows they appear in.

Related: #3472 discussion


Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

  • Is the commit message split over multiple lines and hard-wrapped at 72 characters?
  • Is the subject and message clear and concise?
  • Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
  • Does the commit contain a Signed-off-by: First Lastname email@example.com?
  • Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?

As part of the project's effort to modernize on f-strings, this commit
replaces all usage of .format() with f-string equivalents. The vast
majority of these are in-place syntax changes, but a limited number
change the order of list items or some formatting tricks in order to
appease PEP8.

None of these conversions change the underlying logic of the flows they
appear in.

Related: sosreport#3472 discussion

Signed-off-by: Jake Hunsaker <jacob.r.hunsaker@gmail.com>
Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/sosreport-sos-3505
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@TurboTurtle
Copy link
Member Author

Note: I'll be holding off on merging this until after 4.7 gets cut. While these are simple changes, there's enough of them that I'd prefer we have a full release testing cycle on them as opposed to ~2 weeks.

@arif-ali arif-ali added this to the 4.8.0 milestone Feb 6, 2024

cmd = (
f"PGPASSWORD={self.conf['ENGINE_DB_PASSWORD']} /usr/sbin/sos "
f"report --name=postgresql --batch -o postgresql {sos_opt}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use plugin variable here as well - but it might be ridiculous..?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I can update that.

f"/var/log/{self.apachepkg}*/error_log*",
f"/etc/{self.apachepkg}*/conf/",
f"/etc/{self.apachepkg}*/conf.d/",
f"/var/log/{self.apachepkg}*/katello-reverse-proxy_access_ssl.log*"
Copy link
Contributor

@pmoravec pmoravec Feb 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: it makes sense to have similar copyspecs altogether, so put katello-reverse-proxy_access_ssl.log* just below the other katello-reverse-proxy* one.

Ditto in foreman_proxy plugin.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is due to PEP8 as referenced in the commit message - the only way I found to keep from line-breaking due to going over 79 characters was to be cheeky and stick it at the end without a comma ;-). The other option would involve more use of *, but I'm not familiar with all the log naming here and didn't want to risk messing up some collections.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option is disable PEP8 and pylint warnings on specific lines such as:

LONG LINE over 79 chars here  # noqa, pylint: disable=C0301

FWIW, I absolutely love trailing comma at the end (easy to reorder and/or sort stuff ;-)

@pponnuvel
Copy link
Contributor

This implicitly drop support for Python 3.5 and older.
f-strings were introduced in Python 3.6. Python 3.6 has been EOL'ed for over 2 years now.
However, it's possible several existing machines would still be using Python < 3.6 (possibly even 2.7 in wild cases).
So this implicitly adds a requirement that machines get upgraded to at least Python 3.6 in order to use newer sosreport.
Just wonder if this has been considered and decided as acceptable.

@pmoravec
Copy link
Contributor

.. (possibly even 2.7 in wild cases).

With releasing 4.0, we claimed sos is not further compatible with Python 2.*. Multiple changes introduced in 4.0 dont work on Python 2.7, and I bet more of those were added later on.

Breaking compatibility with Python 3.5 or older is / would be a point. We should at least state it somewhere. In fact, I expect sos already does not work on those Python versions as we already use f-strings on really many places. So the change in this PR won't break anything more. But I agree, we should state somewhere minimal requirements of Python version.

@arif-ali
Copy link
Member

The thing is we have been using fstrings for a while now and older systems would already not work.

We have a situation where 16.04 (xenial) no longer works and we can't backport sos to that distro for this reason. So there we are stuck with 3.9.1. But that is now on security maintenance anyhow, so no urgency on getting new sos.

But the rest for us works, and we don't have this issue. So even though 18.04 (bionic) is now also security maintenance as of last year we are still keen to get that going still. As long as it keeps working there we will be continuing to support it.

Ultimately, I'm keen to go in this way, and after reading a few articles fstrings seem faster and more intuitive, so all up for it.

BTW, I found a python module called flynt which can do changes across the board, if you're interested in going through all the changes it is going to suggest. It does have a dry run option that allows you to see what the changes it would make.

I was also looking at black, which seems to look at coding standards in formatting and consistency, and whether we want use something like that?

@pponnuvel
Copy link
Contributor

pponnuvel commented Feb 17, 2024

In fact, I expect sos already does not work on those Python versions as we already use f-strings on really many places.

That's true.

So the change in this PR won't break anything more.

Not sure if that's correct. I believe only plugins that currently use f-strings would fail and rest of the data would be collected. With this PR, we are introducing more f-strings by converting .format which would work fine otherwise in all Python. So in theory, this could break things that are working today on pre-3.6.

But I agree, we should state somewhere minimal requirements of Python version.

Agree. If that's stated as a minimal requirement, that's fine. It'd be increasingly less of an issue as time goes by anyway. I just wanted to get it clarified and explicitly stated so.

@pponnuvel
Copy link
Contributor

BTW, I found a python module called flynt which can do changes across the board, if you're interested in going through all the changes it is going to suggest. It does have a dry run option that allows you to see what the changes it would make.

I haven't used flynt before - I can look into it.

@TurboTurtle
Copy link
Member Author

Python 3.6 has been EOL'ed for over 2 years now.

At a high level - I'd love to move forward on our minimum python version.

However several of our downstreams still support (or rather, require the use of) python 3.6 which is why we haven't pressed forward with making the minimum version something like 3.9 (still EOL'd I believe?) or more recent 3.11 or 3.12. I know of two specific ones from major contributing companies - RHEL 8 and Ubuntu 18.04, so we'll need to keep 3.6 around until we no longer need to worry about compatibility there.

That doesn't mean we keep 3.6 until these releases are no longer supported at all, though. RHEL 7 still has commercial support, but we don't develop for that target. We have the legacy branch for it if needed, and we'd likely do something similar to that when we drop 3.6 support. We'd need input from @pmoravec and @arif-ali on when we could make a cut like that.

BTW, I found a python module called flynt which can do changes across the board, if you're interested in going through all the changes it is going to suggest. It does have a dry run option that allows you to see what the changes it would make.

Looks interesting. I'll take a look, even if it doesn't work perfectly for us it likely still does a lot of the heavy lifting.

I was also looking at black, which seems to look at coding standards in formatting and consistency, and whether we want use something like that?

We talked about black before, and the short and sweet of it is that while it does some stuff well, the stuff it doesn't do well, or the formatting it does that we don't like, we really don't like.

The off the top of my head example is indention of long strings as parameters to method calls. With the way black does its line breaks it's near impossible to distinguish at a glance what line is a continuation of the previous one, and which is a new parameter.

@pmoravec
Copy link
Contributor

In fact, I expect sos already does not work on those Python versions as we already use f-strings on really many places.

That's true.

So the change in this PR won't break anything more.

Not sure if that's correct. I believe only plugins that currently use f-strings would fail and rest of the data would be collected. With this PR, we are introducing more f-strings by converting .format which would work fine otherwise in all Python. So in theory, this could break things that are working today on pre-3.6.

We use it on more places, like:
https://github.com/sosreport/sos/blob/main/sos/utilities.py#L314
https://github.com/sosreport/sos/blob/main/sos/collector/__init__.py#L625
https://github.com/sosreport/sos/blob/main/sos/collector/clusters/__init__.py#L124
https://github.com/sosreport/sos/blob/main/sos/report/__init__.py#L1733
https://github.com/sosreport/sos/blob/main/sos/policies/__init__.py#L445

But if there is some particular OS with pre-3.6 version that can run the current upstream sos, that would be an argument to pause with this PR, for sure.

I know of two specific ones from major contributing companies - RHEL 8 and Ubuntu 18.04, so we'll need to keep 3.6 around until we no longer need to worry about compatibility there.

Indeed, both OSes use Python 3.6.8 that we will need to support in sos upstream for some/many(?) years. Gladly f-strings were introduced in 3.6, so at least these two distro versions wont be negatively affected by this PR. They will "just" conserve our attempts to move minimal requirements beyond 3.6.

@TurboTurtle
Copy link
Member Author

Indeed, both OSes use Python 3.6.8 that we will need to support in sos upstream for some/many(?) years.

Right, but my main point was that we don't need to necessarily hold upstream for as long as those versions are in use. Again, for RHEL 7 we stopped developing for it even though it is still used/supported today. We made the cut after the final minor release, and left the previous version in the legacy branch for any further remediation that needs to happen.

I think we should do the same here - find a reasonable cut off date for Red Hat and Canonical and mark the current state in a/another legacy branch - otherwise we'll be on 3.6 in 2030 which helps no one.

@dnegreira
Copy link
Contributor

dnegreira commented Feb 19, 2024

I think we should do the same here - find a reasonable cut off date for Red Hat and Canonical and mark the current state in a/another legacy branch - otherwise we'll be on 3.6 in 2030 which helps no one.

I'll circle back on this in the near future.

@dnegreira
Copy link
Contributor

From an Ubuntu standpoint, we are currently working on pushing either sos 4.7 or 4.6.5 to Ubuntu 18.04, which is the last version which has 3.6 support, so from the Ubuntu perspective, we can start deprecating python3.6 already.

Currently Ubuntu 20.04 has python 3.8 and we will need to keep support at least until April 2025, which is when it enters end of standard support, and by then we could probably start deprecating python 3.8.

Hope this makes sense.

@TurboTurtle
Copy link
Member Author

From an Ubuntu standpoint, we are currently working on pushing either sos 4.7 or 4.6.5 to Ubuntu 18.04, which is the last version which has 3.6 support, so from the Ubuntu perspective, we can start deprecating python3.6 already.

Currently Ubuntu 20.04 has python 3.8 and we will need to keep support at least until April 2025, which is when it enters end of standard support, and by then we could probably start deprecating python 3.8.

Hope this makes sense.

Oof. I was really hoping the next Ubuntu release we had to care about after 18.04 had 3.9 at least as that's what EL9 runs with.

According to the python version timeline, 3.8 goes EOL towards the end of this year with 3.9 going EOL the end of 2025.

Tell you what, let's continue this in a github discussion that I'll open later today. For now, the conversation has definitely evolved beyond the original point of this PR.

Speaking of which, if there are no objections, I'll merge this shortly since 4.7.0 was cut yesterday.

@arif-ali arif-ali merged commit 57d2134 into sosreport:main Feb 22, 2024
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants