Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ubuntu/xenial #395

Merged
merged 88 commits into from
May 29, 2020
Merged

Ubuntu/xenial #395

merged 88 commits into from
May 29, 2020

Conversation

blackboxsw
Copy link
Collaborator

@blackboxsw blackboxsw commented May 27, 2020

new-upstream-snapshot for sru into ubuntu/xenial

To create this branch:

cat > drop-deps.patch <<EOF
diff --git a/debian/changelog b/debian/changelog
index 61816397..926a2b0b 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -3,6 +3,8 @@ cloud-init (20.1-50-g7f9f33db-0ubuntu1~16.04.1) UNRELEASED; urgency=medium
   * d/control: drop pyflakes from Build-Depends
     - This python2 pyflakes package is an unneeded dependency now that tests
       use python3-pyflakes.
+  * d/control: drop python3-unittest2 and python3-nose from Build-Depends
+    - upstream now uses pytest
   * refresh patches:
    + debian/patches/ubuntu-advantage-revert-tip.patch
   * refresh patches:
diff --git a/debian/control b/debian/control
index 24328140..6f919475 100644
--- a/debian/control
+++ b/debian/control
@@ -14,7 +14,6 @@ Build-Depends: debhelper (>= 9),
                python3-jinja2,
                python3-jsonpatch,
                python3-mock,
-               python3-nose,
                python3-oauthlib,
                python3-pep8,
                python3-pyflakes,
@@ -23,7 +22,6 @@ Build-Depends: debhelper (>= 9),
                python3-serial,
                python3-setuptools,
                python3-six,
-               python3-unittest2,
                python3-yaml
 XS-Python-Version: all
 Vcs-Browser: https://git.launchpad.net/cloud-init/?h=ubuntu/devel
EOF

cat > manual-changelog.patch <<EOF
diff --git a/debian/changelog b/debian/changelog
index 7e127af6..ae78f242 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -3,6 +3,10 @@ cloud-init (20.2-38-g8377897b-0ubuntu1~16.04.1) UNRELEASED; urgency=medium
   * d/control: drop pyflakes from Build-Depends
     - This python2 pyflakes package is an unneeded dependency now that tests
       use python3-pyflakes.
+  * d/control: add python3-pytest to Build-Depends
+    - This fixes upstream daily builds.  python3-nose is not removed from
+      Build-Depends because, currently, the Ubuntu package builds will still
+      use it for testing.
   * d/control: drop python3-unittest2 and python3-nose from Build-Depends
     - upstream now uses pytest
   * refresh patches:
@@ -11,30 +15,6 @@ cloud-init (20.2-38-g8377897b-0ubuntu1~16.04.1) UNRELEASED; urgency=medium
    + debian/patches/ec2-classic-dont-reapply-networking.patch
    + debian/patches/openstack-no-network-config.patch
    + debian/patches/stable-release-no-jsonschema-dep.patch
-    - HACKING.rst: update CLA link (#199)
-    - Scaleway: Fix DatasourceScaleway to avoid backtrace (#128)
-      [Louis Bouchard]
-    - cloudinit/cmd/devel/net_convert.py: add missing space (#191)
-    - tools/run-container: drop support for python2 (#192) [Paride Legovini]
-    - Print ssh key fingerprints using sha256 hash (#188)
-    - Make the RPM build use Python 3 (#190) [Paride Legovini]
-    - cc_set_password: increase random pwlength from 9 to 20 (#189)
-    - .travis.yml: use correct Python version for xenial tests (#185)
-    - cloudinit: remove ImportError handling for mock imports (#182)
-    - Do not use fallocate in swap file creation on xfs. (#70) [Eduardo Otubo]
-    - .readthedocs.yaml: install cloud-init when building docs (#181)
-    - Introduce an RTD config file, and pin the Sphinx version to the RTD
-      default (#180)
-    - Drop most of the remaining use of six (#179)
-    - Start removing dependency on six (#178)
-    - Add Rootbox & HyperOne to list of cloud in README (#176) [Adam Dobrawy]
-    - docs: add proposed SRU testing procedure (#167)
-    - util: rename get_architecture to get_dpkg_architecture (#173)
-    - Ensure util.get_architecture() runs only once (#172)
-  * d/control: add python3-pytest to Build-Depends
-    - This fixes upstream daily builds.  python3-nose is not removed from
-      Build-Depends because, currently, the Ubuntu package builds will still
-      use it for testing.
   * New upstream snapshot. (LP: #1881018)
     - enable Puppet, Chef mcollective in default config (#385)
       [Mina Galić (deprecated: Igor Galić)]
@@ -203,6 +183,26 @@ cloud-init (20.2-38-g8377897b-0ubuntu1~16.04.1) UNRELEASED; urgency=medium
     - net/cmdline: correctly handle static ip= config (#201)
       [Dimitri John Ledkov]
     - Replace mock library with unittest.mock (#186)
+    - HACKING.rst: update CLA link (#199)
+    - Scaleway: Fix DatasourceScaleway to avoid backtrace (#128)
+      [Louis Bouchard]
+    - cloudinit/cmd/devel/net_convert.py: add missing space (#191)
+    - tools/run-container: drop support for python2 (#192) [Paride Legovini]
+    - Print ssh key fingerprints using sha256 hash (#188)
+    - Make the RPM build use Python 3 (#190) [Paride Legovini]
+    - cc_set_password: increase random pwlength from 9 to 20 (#189)
+    - .travis.yml: use correct Python version for xenial tests (#185)
+    - cloudinit: remove ImportError handling for mock imports (#182)
+    - Do not use fallocate in swap file creation on xfs. (#70) [Eduardo Otubo]
+    - .readthedocs.yaml: install cloud-init when building docs (#181)
+    - Introduce an RTD config file, and pin the Sphinx version to the RTD
+      default (#180)
+    - Drop most of the remaining use of six (#179)
+    - Start removing dependency on six (#178)
+    - Add Rootbox & HyperOne to list of cloud in README (#176) [Adam Dobrawy]
+    - docs: add proposed SRU testing procedure (#167)
+    - util: rename get_architecture to get_dpkg_architecture (#173)
+    - Ensure util.get_architecture() runs only once (#172)
 
  -- Chad Smith <chad.smith@canonical.com>  Thu, 28 May 2020 20:56:53 -0600

EOF

git fetch origin   # or upstream your git preference
git checkout origin/ubuntu/xenial -B ubuntu/xenial
patch -p1 < drop-deps.patch
git commit -m 'd/control: drop python3-nose and python3-unittest2' debian/control 
git commit -am 'changelog: add content for dropping python3-nose and python3-unittest2'

uss-tableflip/scripts/new-upstream-snapshot 8377897b
# manually replace SRU_BUG_NUMBER in the edit view of debian/changelog with 1881018
patch -p1 < manual-changelog.patch
git commit -m 'changelog: manual reorder content'
sed -i -e "1s/UNRELEASED/xenial/" debian/changelog
 git commit -m "releasing cloud-init version 20.2-38-g8377897b-0ubuntu1~16.04.1" debian/changelog

To test:

check diff of your branch vs mine (with out the patches)

run uss-tableflip/scripts/build-package
or make deb to ensure package builds

OddBloke and others added 30 commits March 27, 2020 11:24
We no longer need to support versions which don't have
subprocess.check_output, so we can rely on its presence in the stdlib.
…#287)

We're seeing CI failures currently because a LXD image takes more than
10 minutes to download if the transatlantic link to Canonical's DC is
heavily contested.

This uses a helper provided by Travis[0] to avoid cancellation due to
lack of output for 30 minutes, which should be sufficient to download an
image even at the lowest speeds we've been seeing.

[0] https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
This will be required for the mirror URL sanitisation work,
This also simplifies the implementation to rely on the stdlib, instead
of our own NIH checking.
…anonical#267)

On Focal and later, Ubuntu will prioritize netplan renderer over eni,
even if ifupdown and netplan are both installed.

ENI on Focal and later is considered an unsupported configuration so
cloud-init should generally prefer netplan. On many cloud images,
the /etc/network/interfaces config file does not include the dir
/etc/network/interfaces.d thereby ignoring cloud-init's
/etc/network/interfaces.d/50-cloud-init.cfg file.

LP: #1867029
On some platforms (old heat on OpenTelekomCloud), the user-data mime part
is mislabeled x-shellscript.  cloud-init would not accept this unexpected
mime-type in multipart user-data.  Cloud-init will now run find_ctype()
on the content of the mime-part to check if it matches known include types.

This patch is from Ryan Harper (inspired by my bug report and a
suggestion from Scott Moser) from the discussion on PR canonical#234.


Signed-off-by: Kurt Garloff <kurt@garloff.de>
…nical#291)

This modifies _get_package_mirror_info to convert the hostnames of generated mirror URLs to their IDNA form, and then iterate through them replacing any invalid characters (i.e. anything other than letters, digits or a hyphen) with a hyphen.

This commit introduces the following changes in behaviour:

* generated mirror URLs with Unicode characters in their hostnames will have their hostnames converted to their all-ASCII IDNA form
* generated mirror URLs with invalid-for-hostname characters in their hostname will have those characters converted to hyphens
* generated mirror URLs which cannot be parsed by `urllib.parse.urlsplit` will not be considered for use
  * other configured patterns will still be considered
  * if all configured patterns fail to produce a URL that parses then the fallback mirror URL will be used

LP: #1868232
This shim was required to support Python 2.6, so we no longer need it.
…cal#296)

* distros/tests/test_init: drop needless brackets/indentation

* distros: drop leading/trailing hyphens from mirror URL labels
* Split the "existing contributors" information out to a separate
  section
* Update the "Do these things once" CLA information for the current
  process
functools.lru_cache has been present since Python 3.2, so we no longer need to handle its absence.

(Also sort util's imports while we're modifying them.)
Decouple unlocking passwords when also setting passwords. On OpenBSD skip unlocking
password as `usermode -C no foo` does not  work.
The cc_mounts module does not support NFS mounts in the form of hostname:/ or hostname:/path.

This PR adds support for NFS-style paths in the fs_spec field.

LP: #1870370
Add OpenBSD in the list of supported Operating System.
And document that we can't yield in pytest.fixture functions due to the
age of xenial's pytest.
Swap file size variable was being used before checked if it's set to str
"auto". If set to "auto", it will break with:

  failed to setup swap: unsupported operand type(s) for /: 'str' and 'int'

Signed-off-by: Eduardo Otubo <otubo@redhat.com>
RHBZ: 1772505
We currently have a test system where get_interfaces_by_mac raises an
exception, which is causing these tests to fail as they aren't mocking
get_interfaces_by_mac out.

LP: #1873910
This proposal has been aligned in a human readable.

    There are no changes to this except for spac
    The indentation size is not specifically defined in the YAML RFC
        $ make yaml has passed on my-machine

I set the indentation size to the largest number of articles in the repository (dictionary: 2)

(And also include nishigori as a CLA signer.)
lucasmoura and others added 17 commits May 13, 2020 14:45
Create a schema object for the `apt_configure` module and
validate this schema in the `handle`  function of the module.

There are some considerations regarding this PR:

* The `primary` and `security` keys have the exact same properties. I
  tried to eliminate this redundancy by moving their properties to a
  common place and then just referencing it for both security and
  primary. Similar to what is documented here:
    https://json-schema.org/understanding-json-schema/structuring.html
  under the `Reuse` paragraph.  However, this approach does not work,
  because the `#` pointer goes to the beginning of the file, which is
  a python module instead of a json file, not allowing the pointer to
  find the correct definition. What I did was to create a separate dict
  for the mirror config and reuse it for primary and security, but
  maybe there are better approaches to do that.
* There was no documentation for the config `debconf_selections`. I
  tried to infer what it supposed to do by looking at the code and the
  `debconf-set-selections`  manpage, but my description may not be
  accurate or complete.
* Add a _parse_description function to schema.py to render multi-line
  preformatted content instead of squashing all whitespace

LP: #1858884
This ensures that Travis will not kill our tests if fetching images is
taking a long time.

In implementation terms, this introduces a context manager which will
spin up a multiprocessing.Process in the background and print a dot to
stdout every 10 seconds.  The process is terminated when the context
manager exits.

This also drop the use of travis_wait, which was being used to work
around this issue.
We recently discovered that pylint is failing to report some errors when
invoked across our entire codebase (see
pylint-dev/pylint#3611).  I've run pylint across
every Python file under cloudinit/[0], and this commit fixes the issues
so-discovered.

[0] find cloudinit/ -name "*.py" | xargs -n 1 -t .tox/pylint/bin/python -m pylint
…#368)

We are longer using lxd.readthedocs.io

Signed-off-by: Thomas Parrott thomas.parrott@canonical.com
and slower.
and since we're making it slower, let's cache it, in case boottime gets
called more than once.
Specifically, ensure that given values are either strings, or arrays of strings.
This allows tests to be configured to permit some commands to be run via
util.subp, while still rejecting any unexpected calls.  See the
documentation for further details.
…cal#370)

And add an example of providing a list of assertions.
)

Hopefully this will save some ~pointless round trips on CLA PRs.
These config management things work on BSD, they also claim to work on all distros, so enabling them!

LP: #1880279
Copy link
Collaborator

@raharper raharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I followed your steps and I get the same diff as you.

However, my release string is one different.

cloud-init (20.2-38-g8377897b-0ubuntu1~16.04.1) xenial

and you had these two:

20.2-30-g8bcf1c06-0ubuntu116.04.1
20.2-39-ga99c8bce-0ubuntu1
16.04.1

So, other than that; it seems fine. Hoping this is just mispasted/old values.

@blackboxsw
Copy link
Collaborator Author

blackboxsw commented May 27, 2020

So, other than that; it seems fine. Hoping this is just mispasted/old values.

Yep that was the problem. we also landed another commit in master which threw things off
a new-upstream-snapshot 8377897b fixed everything to make sure we are working on the same commit as ubuntu/devel

Copy link
Collaborator

@raharper raharper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This worked, a couple slightly off steps.

  1. The new upstream snapshot command should use --no-bugs (for whatever reason it did filter bugs even though it was an SRU.
  2. The manual re-ordering commit command needs debian/changelog
    git commit -m 'changelog: manual reorder content' debian/changelog
Commit this change? (Y/n): y
[ubuntu/xenial c83c4489] update changelog (New upstream snapshot 20.2-38-g8377897b).
 1 file changed, 94 insertions(+), 3 deletions(-)
wrote new-upstream-changes.txt for cloud-init version 20.2-38-g8377897b-0ubuntu1~16.04.1.
release with:
sed -i -e "1s/UNRELEASED/xenial/" debian/changelog
git commit -m "releasing cloud-init version 20.2-38-g8377897b-0ubuntu1~16.04.1" debian/changelog
git tag ubuntu/20.2-38-g8377897b-0ubuntu1_16.04.1
# optionally push your tag: git push <remote> ubuntu/20.2-38-g8377897b-0ubuntu1_16.04.1
(crispyboi) cloud-init % patch -p1 < /tmp/manual-changelog.patch 
patching file debian/changelog
Hunk #3 succeeded at 183 with fuzz 2.
(crispyboi) cloud-init % git commit -m 'changelog: manual reorder content' debian/changelog
[ubuntu/xenial c208d8b2] changelog: manual reorder content
 1 file changed, 24 insertions(+), 24 deletions(-)
(crispyboi) cloud-init % sed -i -e "1s/UNRELEASED/xenial/" debian/changelog 
(crispyboi) cloud-init % git commit -m "releasing cloud-init version 20.2-38-g8377897b-0ubuntu1~16.04.1" debian/changelog
[ubuntu/xenial f170f42e] releasing cloud-init version 20.2-38-g8377897b-0ubuntu1~16.04.1
 1 file changed, 1 insertion(+), 1 deletion(-)
(crispyboi) cloud-init % git diff blackboxsw/ubuntu/xenial HEAD 
diff --git a/debian/changelog b/debian/changelog
index b8bea7bb..48b312b6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -204,7 +204,7 @@ cloud-init (20.2-38-g8377897b-0ubuntu1~16.04.1) xenial; urgency=medium
     - util: rename get_architecture to get_dpkg_architecture (#173)
     - Ensure util.get_architecture() runs only once (#172)
 
- -- Chad Smith <chad.smith@canonical.com>  Thu, 28 May 2020 20:56:53 -0600
+ -- Ryan Harper <ryan.harper@canonical.com>  Fri, 29 May 2020 10:58:42 -0500
 
 cloud-init (19.4-33-gbb4131a2-0ubuntu1~16.04.1) xenial; urgency=medium

@blackboxsw blackboxsw merged commit 5985ff4 into canonical:ubuntu/xenial May 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.