Vendorize latest rb-readline

[deivid-rodriguez]

This project is for Ruby version 2.4 and newer.
For Ruby versions < 2.4 please file an issue here.

What problems are you experiencing?

@MSP-Greg found some problems with rb-readline. I had a look at the code and I think you're vendoring 0.5.4. However, 0.5.5 seems to have been released. Who knows, maybe it improves the experience...

[MSP-Greg]

@larskanis

FYI, 5.5 / master has the same issues. For now, I hope to work on rb-readline with three starting goals:

1. Align methods and attributes with the extension (renaming, moving doc strings).
2. Try to reduce the method count, as many methods exist simply from the 'c translation'.
3. Get it to work with Open3.
4. Get it to pass the tests in ruby.

Any (more) thoughts on seeing if rb-readline can be pulled into oneclick?

Finally, I build ruby-loco with a readline extension, and it does pass tests. Conversely, it doesn't work well with a console window. Hence, I add rb-readline to the builds and have it set as the default.

Thanks, Greg

[deivid-rodriguez]

@MSP-Greg Can you give this branch on my readline-rb fork a try? I've been doing some experiments.

EDIT: Tried it against byebug and it doesn't really work. Not too surprised 😅

[deivid-rodriguez]

I set up appveyor on my rb-readline fork. Passing build here: https://ci.appveyor.com/project/deivid-rodriguez/rb-readline/build/3. I added two tests that use Open3 that make the build hang only on Windows, so at least the bug is now isolated in a test case that doesn't use byebug at all.

[larskanis]

I would really love to get the readline issues fixed. However probably I wouldn't have had included rb-readline into RubyInstaller, if it hasn't been used previously since years. This is mostly due to its poor code quality and bugs. Still it seems to work just enough for most users.

I already addressed several issues with rb-readline in my branch:

• No Unicode support (not even what Microsoft understands under "Unicode"). The whole library is based on ANSI codepage crap.
• No proper encoding support. The library does all multi-byte handling manually, which was necessary in Ruby-1.8, but can be much better handled since 1.9 with builtin string encodings.
• Broken support for wide characters on Windows (characters which need two columns to display).
• Non working keys on Windows.
• The whole library works with zero-terminated strings in Ruby! This is a relict from the C translation.

This all makes the code very hard to maintain. I don't think it makes sense to base further work on https://github.com/ConnorAtherton/rb-readline . Please have a look at https://github.com/larskanis/rb-readline ! It fixes all above issues.

In addition to these issues the library is still hard to maintain. It has a lot of different modes and workarounds for ancient terminal types. And it is based on an very old libreadline version. The purpose of the library is interaction with the user, so that it is hard to test. The given tests are not suitable to verify the correctness 90% of the library.

[deivid-rodriguez]

Thanks @larskanis! So what's your suggestion regarding the maintainance of rb-readline, would you like to move the canonical repository to your fork? Wouldn't it be better to send some patches over there or get contributor access?

[larskanis]

I wan't finished with my fork - that's why I didn't open a pull request so far. Wide character handling wasn't error free on Windows and, as far as I remember, I broke something on Linux. Then I wondered, whether it wouldn't be better to fix the C-based libreadline to properly work on Windows instead. What do you think about this?

Now, that there is some more interest to improve the situation, I could resume my work.

[deivid-rodriguez]

Yes! I thought the same thing, although I think a pure ruby implementation might be valuable too. Getting the C-extension working properly on Windows would be great indeed!

Overall, Windows or Linux, C-based or ruby-based, readline on ruby needs love in my opinion. All major REPL tools made in ruby have long standing issues that seem ultimately caused by bugs in readline.

[deivid-rodriguez]

I'm setting the ball rolling here. The current tests might not be great to ensure correctness but I guess they are a start!

[larskanis]

I was thinking a little bit about how we could improve the situation with readline on Windows. Unfortunately neither libreadline nor libeditline contain any test cases.

From the little work I've done last year on rb-readline, I learned how easy it is to fix one particular issue in rb-readline and brake 5 other things. Knowing this, I'm not even sure, how many things I broke in my branch, because I simply can not verify it, other then manually typing some characters. The current test coverage isn't sufficient by any means.

Therefore I would proceed by adding a kind of terminal emulator (written in Ruby) which receives all outputs of rb-readline for processing/rendering. Using small screens (like 10x5 characters or so) per test case, the resulting screen content could be compared with the expectations of the to be implemented test cases. This way we could verify whether all the character changes and movements on the screen still give the same end result. Having a sufficient test suite, we then could really start to improve the library.

But still I'm not sure if all the work is reasonable. While this improves the situation for ruby, gdb will still not be usable on MINGW (I'm using gdbserver on Windows and connect to it from Linux usually). We also can not benefit from updates to libreadline, but have to maintain our own ruby port.

I therefore will not continue the work on rb-readline for now, but will check if I can port libreadline to Windows. I can use the work on my rb-readline branch as a starting point.

[MSP-Greg]

Lars,

Thanks for the response.

Re libreadline, I've always built trunk with the readline package (7.0.003), and it tests fine in test-all and spec. FYI, the readline.so is included in the ruby-loco builds. The only gem/code I recall trying it with is byebug, and it is unusable. Slow cursor movements; scroll thru history, then 'enter' moves to another string, etc.

So, I'm not that familiar with readline or what it should do, and the vast majority of the code I've written (C#, before ruby) was for US based use, so I'm not at all familiar with encoding issues. And, I'm windows only.

Anyway, I'll start with your work and see what I find, trying the ruby-core tests, etc. io/console does have a method or two that could replace some of the API calls in rb-readline, but all the code for character widths is a big question mark...

The reason that we were looking at it was that some of the byebug tests use open3 for testing (connecting pipes to the app), and the current rb-readline doesn't play well unless it's a real keyboard. That was a relatively simple fix.

Thanks, Greg

[larskanis]

To give you an update: I started to work on libreadline for better Windows support here. I also extracted the ruby binding to libreadline from ruby stdlib to a separate gem. This makes it easier for me to work on it.

One of the first things I'm working on, is to add full Unicode/UTF-8 support. This requires a bunch of changes to both projects. Since I'm not yet familiar with the code base, the progress is fairly slow so far. Still I plan to work on it for the next weeks in my little spare time.

[deivid-rodriguez]

@larskanis That's so awesome and attacks the root of problems! Thanks for doing this! ❤️

[deivid-rodriguez]

@larskanis I see that you've been doing some progress with this 🥇

I just found out about this and thought it could be worth mentioning. I think that means readline willl/might be gemified in the future. Your work fits that direction!

[deivid-rodriguez]

@larskanis Do you intend to do any further work on this? I'm wondering whether I should resurrect ConnorAtherton/rb-readline#146.

[janpio]

(Please do work on this - this cost me too many hours at rubygems/bundler#6902 and https://github.com/bundler/bundler/issues/6907 and only @deivid-rodriguez 's helpful comments helped me to discover ConnorAtherton/rb-readline#147 as the solution for one of these two specific problems. Fixing readline on Windows would be much appreciated)

[larskanis]

I plan to replace rb-readline in RubyInstaller by reline in the future. Also possibly backported to ruby-2.4 to 2-6.

[MSP-Greg]

@larskanis

JFYI, reline is far enough along that with a few minor changes, it can be used in byebug CI, and passes. At present, it isn't compatible with Ruby 2.4.

[larskanis]

@MSP-Greg Yes, CI is working with reline, but interactive use in the Windows console is still somewhat limited (e.g. cursor keys don't work) and buggy (e.g. character sizes are often incorrect for non-ASCII).

[aycabta]

@larskanis

Windows console is still somewhat limited (e.g. cursor keys don't work) and buggy (e.g. character sizes are often incorrect for non-ASCII).

I'm the Reline author. Could you report the bugs to https://github.com/ruby/reline? I'll fix it quickly.

[larskanis]

RubyInstaller-2.7.0-1 is out and ships Reline as replacement to Rb-Readline: https://rubyinstaller.org/2020/01/05/rubyinstaller-2.7.0-1-released.html

@aycabta Thank you so much for implementing the Reline library! Rb-Readline was a dead end but we now finally have a fully working and supported input library for Windows. Sorry for my imprecise comment - I tried a very early state of Reline and but didn't follow the development, so I answered from memory only. I'll add some minor things to the issue tracker.