Add support for operating on byte strings

[danielparks]

This adds Shlex::new_bytes(), quote_bytes(), and join_bytes(). This adds a bytes submodule for operating on byte strings that might contain invalid UTF-8. Where possible I have switched the functions that operate on str to use the _bytes functions internally to avoid duplicating code and eliminate the potential for differing behavior between the two functions.

It includes trivial tests that confirm that the bytes version of functions actually work on invalid UTF-8.

I have not updated the documentation; this is just a first draft to solicit feedback.

Fixes #12

Update: Fixed some problems I mentioned in the comments. Updated the description above; italics are new.

[danielparks]

This doesn’t change the MSRV. cargo msrv says it’s 1.36.0 for both the master branch and this one.

I’m also reasonably confident that this doesn’t change any existing behavior.

[danielparks]

Oh, oops. I didn’t actually deal with Shlex::next() returning a String… and of course as is String::from_utf8_unchecked() isn’t safe in this PR because you can run it on an arbitrary byte string.

I’ll deal with that stuff tomorrow.

[danielparks]

Ok. This should be basically functional now. It passes cargo test --no-default-features and cargo test, and cargo msrv still returns 1.36.0.

Still to do:

• Update documentation and README.
• If it’s not too terrible, implement shlex::Shlex with shlex::bytes::Shlex.
• Check ergonomics with OsString and OsStr, since that’s probably what most people will want to use it with.

[danielparks]

I added a commit to implement Shlex with bytes::Shlex. It used Deref and DerefMut to ensure that accessing the fields (e.g. line_no) still work.

Using OsStr requires std::os::unix::ffi::OsStrExt, but otherwise it doesn’t seem like a problem for users. OsString is basically similar.

[danielparks]

Ok, documentation, README and CHANGELOG updated! Ping @comex, let me know what you think. I generally tried to match your style, but I imagine I diverged a bit, especially in the documentation. Happy to change things.

MSRV remains 1.36.0.

[fenhl]

You can get rid of quote_internal and the Option wrapping by implementing crate::quote like this:

/// Given a single word, return a string suitable to encode it as a shell argument.
pub fn quote(in_str: &str) -> Cow<str> {
    match bytes::quote(in_str.as_bytes()) {
        Cow::Borrowed(out) => {
            // Safety: bytes::quote() was given valid UTF-8, so it will
            // always return valid UTF-8.
            unsafe { core::str::from_utf8_unchecked(out) }.into()
        }
        Cow::Owned(out) => {
            // Safety: bytes::quote() was given valid UTF-8, so it will
            // always return valid UTF-8.
            unsafe { String::from_utf8_unchecked(out) }.into()
        }
    }
}

Or am I missing another reason it's done this way?

[danielparks]

That is nicer. In other projects I don’t use the _unchecked functions, so I have to use Option to avoid doing extra work. Of course, that isn’t necessary here.

[fenhl]

Thank you for your contribution! This has now been released as version 1.2.0.