docs/operator-notes: Note paths that should not be used b/c of switch_root

[dlipovetsky]

I learned this the hard way (see flatcar/Flatcar#1152), and thought it would be useful to have in the documentation.

Feedback and edits welcome!

[travier]

In general, you should not directly write to /proc, /dev, /sys and should use sysctl config files or udev rules.

For /tmp and /run you should use tpmfiles.d configs instead.

Maybe we should have a Butane lint that refuses writing files to those locations?

[dlipovetsky]

Thanks for the feedback!

In general, you should not directly write to /proc, /dev, /sys and should use sysctl config files or udev rules.

That's fair. Want me to add this to the note?

For /tmp and /run you should use tpmfiles.d configs instead.

Ahh, right, /tmp is another one. IIRC, it's mounted after Ignition finishes, not moved by switch_root. I'll update the note.

Maybe we should have a Butane lint that refuses writing files to those locations?

I haven't used Butane myself (I'm using Ignition indirectly, as the result of booting Flatcar Linux machines with Cluster API. Could someone help with this part?

[travier]

The text as written is not correct.

The /proc /sys /dev paths are very much still accessible after switching from the initramfs. Users should not rely on Ignition to write value to those files because the changes are not persisted beyond the first boot.

I think that whether /tmp or /run are a tmpfs mount is up to the distribution, not a requirement of Ignition.

Overall there is nothing invalid with Ignition configs writing to those paths. They only don't do what you likely want.

So I'd phrase this as a recommendation instead. But I'm still not sure that this is the right place for it.

[dlipovetsky]

The /proc /sys /dev paths are very much still accessible after switching from the initramfs.

The text says that the files you write under these paths before switch_root are not accessible after switch_root. It does not say the paths are not accessible. But I can rewrite it to be more clear.

Overall there is nothing invalid with Ignition configs writing to those paths. They only don't do what you likely want.

Any file written under these paths by Ignition is not accessible after switch_root. I can't think of any situation where this is what the user wants. If you can think of one, could you please share it?

So I'd phrase this as a recommendation instead. But I'm still not sure that this is the right place for it.

I want to help other Ignition users avoid a mistake that is easy to make, especially if you are used to cloud-init, which runs after switch_root. Please let me know the right place, in case these docs are not it.

[prestist]

@travier poke

[travier]

Coming back to this and re-reading it again, I think I now understand what you mean. Instead of phrasing it in a negative form, let's write what we recommend instead in a positive form.

[travier]

But I still think that this would be better as a check in Butane as the behavior might be distribution dependent and users are not expected to write Ignition configs directly.

[dlipovetsky]

But I still think that this would be better as a check in Butane as the behavior might be distribution dependent and users are not expected to write Ignition configs directly.

I agree with you.

I won't be able to contribute this check to Butane, though.

[travier]

Small nits but LGTM

[dlipovetsky]

Thanks for reviewing, and for the fixes!

[dlipovetsky]

I think that, earlier in the review, @travier made a good suggestion to give users guidance about managing files under these mount points--namely, to configure the respective systemd services that manage files under tose mount points.

I think it would be good to keep that text, in addition to the guidance on not using Ignition to directly manage files under these mount points. Looks like that guidance is still there, so no need to change anything 😄 .

[dlipovetsky]

Thanks everyone for working together, in good faith, to improve the docs for Ignition users 🙏. It might not be a big change, but the reviews made the text very clear, and I'm sure that will save people from frustration and wasted time.

[jlebon]

I just squashed it all to one commit. Thanks!

[jlebon]

I still find this sentence a bit weird because you wouldn't typically want to create files, directories, or symlinks in there in the first place. But fine as is too.