Unicode input on windows command line does not work

[retep998]

When using stdin with my rust program run from either the windows command line or git bash, any unicode characters are turned into question marks.
So "Алло" becomes [63, 63, 63, 63].
If I change the codepage of the windows command line to utf-8 via chcp 65001, then when I enter in any unicode characters stdin returns EndOfFile.

On the flip side, output using libgreen works fine (because libuv translates utf-8 to utf-16 and calls WriteConsoleW), while output using libnative produces garbage for unicode because it writes utf-8 as if it were ascii.

Using windows 8.1 x64, with 32-bit Rust nightly.

[retep998]

For anyone saying that windows command line does not support unicode, here is some quick code I wrote in C++ which correctly reads and writes unicode from and to the console:

wstring read() {
    auto in = GetStdHandle(STD_INPUT_HANDLE);
    array<wchar_t, 0x1000> buf;
    unsigned long read;
    ReadConsoleW(in, buf.data(), buf.size(), &read, nullptr);
    return{buf.begin(), buf.begin() + read};
}

void write(wstring p_str) {
    auto out = GetStdHandle(STD_OUTPUT_HANDLE);
    unsigned long wrote;
    WriteConsoleW(out, p_str.c_str(), p_str.size(), &wrote, nullptr);
}

int main() {
    write(read());
}

Here is a screenshot of that code working:

[retep998]

A patch for unicode input in libuv is currently going through the process, which will fix libgreen's unicode input.

joyent/libuv#1323

[retep998]

Currently working on writing up a patch for libnative unicode console support. Output works so far:

Still working on input.

[retep998]

Created a PR to address the issue in libnative: #15051

The libuv patch has been merged into libuv master, so that's waiting for libuv to release a new version and then rustuv to update to that new version

[retep998]

Since libnative has been fixed by #15051 and librustuv no longer exists, this issue is nearly done. Code still needs to be written to handle multi-codeunit codepoints broken across reads/writes.

[nagisa]

@retep998 how relevant is this now?

[retep998]

@nagisa For the most part it works. However the current code still cannot cope when characters encoded using multiple codeunits are broken across a read/write boundary. Plus having the API take [u8] is hardly ideal when we should be using the new OsStr thing.

[retep998]

Issue subsumed by #23344