Fancy remote debugging (aka tab completion and editor controls)

[goodboy]

As part of the journey in #129 I discovered that no-one seems to have solved the problem of getting the fancy features of a modern Python debugger working in remote debugging applications. Though I haven't tested them all, the list in #113 seems to mostly contain systems which rely on telnet servers (or other network IPC magic) but none of them actually solve the issue of how to get the features in the local client that would normally only be possible when the remote process is connected to a tty.

The problem

Standard fancy (read human enhanced) debugger repls (including the stdlib's pdb which uses rlcompleter, and pdb++) rely on libraries such as GNU readline to get things like completion and CLI "editting controls". There seems to be no way to get these features with readline based systems in a remote debugging context since Python's use of readline requires that the process is launched under a tty/pty system. Ideally these features are available in such use cases to make debugging of remote systems sane and efficient for the user.

Evidence

In #129 I was able to verify that launching subprocesses with stdin as a unix pipe indeed causes no readline systems to be loaded. I haven't been able to find a remote debugger that supports this feature either (but of course hopefully someone will prove me wrong!).

Possible solutions

• in the near term: we can not spawn with stdin as a pipe and instead let child processes stay connected to the parent tty (this is actually what trip does and it works):

  • this should allow for nested actor access to stdin since the parent won't have to figure out which pipe to write to.
  • we'll need locking around access to the root actor's tty (which was already part of the plan).
  • this will of course only work for host local actors.

• consider a debugger that doesn't use the stdlib's readline

  • the obvious choice for this is of course python-prompt-toolkit which is specifically a replacement for readline
  • there seems to be test code suggesting ptk can work without being connected to a tty/pty.
  • in theory this might work with ipdb (since ipython uses ptk underneath) if ptk was configured properly (it currently doesn't work any better then the readline options based on my testing in https://github.com/goodboy/tractor/tree/stin_char_relay).

• work with debugger's that want to move to ptk to get this functionality supported in their initial integration such as with pdbpp in Edit debugger line with editor pdbpp/pdbpp#362

Other notes

• pytprocess does get these features in subprocs as we'd expect but ideally we aren't spending time wrapping this with trio since it still won't work for the network remote debugging cases.
  • if we were going to hack it using a pty on *nix systems there is a alternative fork implementation in the code base that looks like a good place to start.
  • there's a great explanation on why the pty process method is used to deal with older tty reliant systems
• further tty/pty resources:
  • examples on setting the local tty to cbreak mode which I've toyed with in the stdin_relay_chars branch.
  • SO answer on how to spawn subprocs in Python connected to a pty (hint: probably better to just look at ptyprocess from above).
  • stty for checking local tty settings
  • the SO newb explanation of a pty on linux
• rlcompleter notes the readline limitation for any doubters.
• epdb's server mode (which seems to have the best integration with pdbpp) has no notion of supporting this afaict.
• tips from trio on how to hook up subprocs for receiving input.

Remote debugging possible hacks or solutions

• python-remote-pdb offers a readline hack using nc or socat which may be usable in the near term
• somehow using some of that ptyprocess has done from tractor spawning machinery and keeping compat with the public trio api

Ideally

• the correct solution to me is getting ptk to support all it's features without requiring a tty whatsoever and then being able to simply talk to a tractor actor running it.

[goodboy]

Relevant links from ptk:

• options for doing this in a tmux-like app: run as background process, connect to active terminal session prompt-toolkit/python-prompt-toolkit#1087
• mention about using IPC to ship ANSI output
• further discussion around capturing the output
• a PR showing how to reroute output to stderr if it's a tty
• there's a utility for avoiding the clobbering of stdout that looks to be exactly what I had envisioned for a minimal control repl
• a very old pdb implementation built on ptpython, ptpdb which is likely where we'll need to start

[goodboy]

Follow up from prompt-toolkit/python-prompt-toolkit#1204:

• @jonathanslenders tried porting ptpdb to ptk 3.0 and it's looking a bit rough.
• we may instead want to look at hacking ipython core to make ipdb work with (as mentioned) new ptk 3.0 features