Improve text on Packer

[ahrkrak]

In a call with Adobe folks (@mtougeron and @sfotony) it was noted that the docs text[1] on use of Packer was quite scary, so they haven't used it yet (but want to). Given we do create packer images for Cluster API, we should find some way to phrase this that is clearer that this is a valid way to deploy Flatcar, but with some caveats that users should understand.
[1] https://www.flatcar.org/docs/latest/installing/customizing-the-image/customize-the-image/#customization-through-booting-with-packer-vmware-base-vms-or-chrootsystemd-nspawn

[pothos]

With flatcar-reset being available now we can make it a little bit easier for users to not shoot themselves in the foot with OS images that have been booted. But some traps are still there (the insecure build user if /etc/passwd is kept, or a bootstrap Ignition config file on the OEM partition). Also, it will be a inconvenient to use because the call sudo flatcar-reset --keep-paths '/etc/systemd/system/' '/opt/bin/' # and so on… requires an explicit list of files or folders containing these files that were created with packer/ansible and that should be kept. Using /etc/ is not a good idea because that includes the SSH host keys. So it's really up to the user to think about both simple files and wanted state files from actions and come up with regex paths (including paths to removed files under /etc/ so that they don't get recreated during the reset).
Most of these issues users are warned about are not Flatcar-specific and one runs into them with Ubuntu, too. For setups that have all required cleanup logic for, e.g., Ubuntu correctly implemented, the only difference is that Flatcar would also need the first boot flag recreated. The image customization through mounting the image would be possible for Ubuntu as well, and then most issues are avoided - but with Flatcar I think we have even better support for customizing the image through mounting and I think that should be the preferred way. So, in summary yes, it makes sense to improve the text while keeping the preference of customization through mounting, and if Packer/Ansible is used, flatcar-reset is a good tool to help making things right but unfortunately the list of caveats won't be shorter.

[ahrkrak]

We had another call with someone today who was interested in customizing images potentially using Packer.
Perhaps @miao0miao could work with the team to get this text more user friendly.

[ahrkrak]

@miao0miao wdyt?