Generic session manager for Emacs based IDEs

[vspinu]

Sesman is an initiative to create an uniform manager for IDE's REPLs and associated connections. It is designed primarily with CIDER and ESS in mind, but the hope is that it could appeal to other IDEs as well.

A brief description of the system is in the readme. Would really appreciate comments and if you could CC people potentially interested in the project.

Thanks!

@bbatsov, @dgutov, @dpsutton, @jaor, @lionel-, @xiongtx

[dpsutton]

so CIDER would implement the cl-defgeneric sesman-start-session? so the buffer-local sesman-system would be 'cider or something similar? and we crank up an nrepl session and put a name in front of it, returning ("connection-name" . nrepl-connection).

can you walk me through the lifecycle of someone jacking-in to a simple clj project called fizzbuzz?

would cider jack in do something like the following?

(let ((session (sesman-start-session 'cider)))
  (sesman--link-session session 'cider 'project))

I'm not sure i see how to recover the session for us to use it though

[lionel-]

Looks great from a cursory glance!

Could this be used to manage shells as well? Management of eshell sessions is somewhat a pain. Then would it be relevant to think about linking to multiple REPL / shells in a single context? I can think of three big types of relevant processes, REPL, shell, and debugger (e.g. gdb in projects involving compiled code).

[xiongtx]

So this'll supersede the code that's already in clojure-emacs/cider#2069? Next step would be to get CIDER to use this to manage its sessions?

[vspinu]

@dpsutton

so CIDER would implement the cl-defgeneric sesman-start-session?

Yes. A method dispatching on (eql CIDER). Technically this method is not needed for the linking part. It's there to be able to call C-c C-s C-s and start a session irrespective of the IDE. Right now each system does it in its own way.

so the buffer-local sesman-system would be 'cider

Yes.

and we crank up an nrepl session and put a name in front of it, returning ("connection-name" . nrepl-connection)

Yes, but the tail is not just a simple connection. In cljs + clj projects, CIDER session would consist of one Clojure and one Clojurescript connection and maybe a bunch of other stuff; it's entirely up to CIDER what's in the tail.

This is actually the main reason for sessions. They allow dealing with multiple related connections as a unit.

can you walk me through the lifecycle of someone jacking-in to a simple clj project called fizzbuzz?

You jack in as usual with cider-jack-in. Somewhere within cider-jack-in a session (ses-name ...) is created and registered with sesman-register.

From now on every cider's function in need of a connection (eval, completion etc) calls sesman-ensure-session which guaranties to return the most meaningful session (given current context and user's associations).

That's pretty much it. The entire burden of session selection, association, switching etc. is taken over by sesman. Cider need not know where the "most meaningful" session came from.

@lionel-

Could this be used to manage shells as well?

Of course, as long as you have multiple processes which you want to distinguish depending on current context for whichever reason (sending code, completion, associated debugger or compilation process etc) sesman is your friend. You just need to bunch all connected processes into one session and call sesman-register.

Management of eshell sessions is somewhat a pain.

Not just eshell or shell. All emacs's run-xyz suffer from the same problem. They are not designed to work in multi-context, multi-process environments. There is python-x add-on which allows multiple python processes but only dedicated one-per buffer buffer (cc @wavexx).

Then would it be relevant to think about linking to multiple REPL/shells in a single context?

It's possible by design but it's up to the user to configure which context allows 1-to-many and which allow for 1-to-1. By default, project is 1-to-many, but dirs and buffers are 1-to-1. When there are many associations with current context, the question is how to disambiguate between them. You don't want to ask the user for the process each time he or she sends the code to the repl. In ESS (and recently in CIDER) we were using most recently used process to send code to. I think this is a useful (most meaningful?) disambiguation strategy for sesman as well. There is already sesman-greater-p generic in place precisely for this purpose.

I can think of three big types of relevant processes, REPL, shell, and debugger (e.g. gdb in projects involving compiled code).

Indeed. Bundles of heterogeneous processes is why we need sessions. One potentially interesting use case is the multi-modes, when you might have, say, python and R code in one buffer and be willing to send it to separate processes, but those processes should be managed as a bundle etc.

@xiongtx

So this'll supersede the code that's already in clojure-emacs/cider#2069?

Precisely.

Next step would be to get CIDER to use this to manage its sessions?

Well, technically the user manages the sessions. CIDER just calls sesman-ensure-session to retrieve the relevant session. If you wish, sessman-ensure-session is cider-current-connection with super-powers.

[xiongtx]

This looks like good work. It's hard to judge it on its own though. Here's what'd help:

• Copy this package into New connection API and user level connection commands clojure-emacs/cider#2069
• Implement the session management changes w/ this package
• Show that it works and let people test it
• Address feedback
• Put this on MELPA, remove it from CIDER, and make CIDER depend on it

In general CIDER (i.e. @bbatsov) is hesitant to incorporate external deps. This'll demonstrate the value of using sesman, which would justify using the external dep.

[dpsutton]

I'd say dogfooding it inside of CIDER while the api is iterated on could be quite useful.

[bbatsov]

I like this idea as well, as it would allow us to iterate faster on this in the "experimental" period. I do have to take a closer look at the code, though. @xiongtx's plan is pretty much what I would have suggested myself.

In general CIDER (i.e. @bbatsov) is hesitant to incorporate external deps. This'll demonstrate the value of using sesman, which would justify using the external dep.

In particular I don't like deps our team doesn't control, as they often become a liability - like nREPL itself (a lot is riding on the nREPL 0.3 respin that seems stuck right now). Even once this is a proper 3rd party package down the road I'd want a few CIDER devs to have access to the repo, so it's certain we won't get in a situation we have to fork/inline the project.

[vspinu]

Copy this package into clojure-emacs/cider#2069

This certainly sounds like a good plan but it's a lot of work and I don't have many weekly hours to spare at the moment. So you would have to slow down work on CIDER for a month or so starting end february. clojure-emacs/cider#2069 is so overwhelmed with conflicts that it basically means re-writing it from scratch.

In particular I don't like deps our team doesn't control, as they often become a liability

This is a price to pay for modularity. sesman is a small library with pretty clear separation of concerns, so hopefully it wont become a "liability" if things are made right.

One potential show stopper is that sesman requires emacs 25+ (cl-generic, project, seq). Is that ok?

[bbatsov]

This certainly sounds like a good plan but it's a lot of work and I don't have many weekly hours to spare at the moment. So you would have to slow down work on CIDER for a month or so starting end february. clojure-emacs/cider#2069 is so overwhelmed with conflicts that it basically means re-writing it from scratch.

Why would this be more work? Seems to me that whether you hack this within CIDER or outside of it the effort would be comparable.

One potential show stopper is that sesman requires emacs 25+ (cl-generic, project, seq). Is that ok?

That won't be a problem.

[dpsutton]

it sounds like it would be way easier to iterate and keep the CIDER and sesman code changes in lockstep while its nailed down. rather than making sure melpa gets the correct versions at the same time while, presumably, there are changes going through. Either way I'm excited to move the session stuff out of CIDER. Although I'm not sure how it will handle how to choose whether to use a cljs or clj repl

[vspinu]

It does make sense to start with CIDER as a proof of concept because of the complexities with dual REPL environments. If sesman can work in CIDER it will probably will work for everything else.

Although I'm not sure how it will handle how to choose whether to use a cljs or clj repl

That's something to still figure out but it should not be too difficult. I am thinking of a prefix arg which, on universal arg would ask interactively for available possibilities, and would start a session immediately on M-n prefix: M-0 - default, M-1 clojure, M-2 cljs-node, M-3 cljs-figwheel etc. All that configurable by a system specific alist. There might be also a CIDER specific custom variable of how to guess what to start from context on default (unprefixed) startup. It can take values like 'auto, 'ask, 'both, 'clj-only etc. In any case you can keep using jack-in-xyz and use sesman only for switching and re-associations of sessions. Sesman lifecycles should work in parallel to what's already in place.

[vspinu]

@bbatsov

Why would this be more work?

I didn't say more work, just a lot of work which I couldn't do it in one blow as I did last time. Developing as part of CIDER is fine with me as long as sesman is not CIDER specific at the end. Will pop new PR once I have something.

That won't be a problem.

Ok. Great.

[wavexx]

On Tue, Feb 13 2018, Vitalie Spinu wrote: > Management of eshell sessions is somewhat a pain. Not just eshell or shell. All emacs's `run-xyz` suffer from the same problem. They are not designed to work in multi-context, multi-process environments. There is python-x add-on which allows multiple python processes but only dedicated one-per buffer buffer (cc @wavexx).

Generally this is a problem of most modes that support an underlying REPL. It's even worse than that: in python.el, once a "global" interpreter is created, creating a new dedicate process is even more painful. Either behavior can be severely limiting depending on what you're trying to do and I'm definitely not happy with it. I come late to the party here, but I'm happy vspinu is working on something to tackle this. But I feel the main README in sesman isn't giving me (as an elisp writer) the big picture. Let's say I'm working on python-x, and I want to implement an underlying session management to handle background processes. The interface that I want to expose to the user is: - let the user have one or more processes - run stuff! If there is no background process, make a new one - run stuff in a different process (let me choose existing ones!) - show buffer<>process associations How would I use sesman to implement this?

[vspinu]

Hi @wavexx, you are definitely not late. The project is in the inception state.

But I feel the main README in sesman isn't giving me (as an elisp writer) the big picture.

The inner details are not fully figured out as there is no canonical implementation yet. Once CIDER has a working implementation I will create a detailed manual.

let the user have one or more processes

This is automatic and taken care by the sesman. You just implement sesman-start-session for your system (python-x). When the user calls sesman-start (C-c C-s C-s) new python session is automatically started and associated with current context (project or directory). To make this work with run-python one would need to wrap it in an advice and invoke sesman-register in order to make sesman aware of the new python session.

• run stuff in a different process (let me choose existing ones!)

Users can start as many sessions as they need and associate, disassociate or re-associate sessions with projects, directories or buffers at any given time. As an IDE developer you never worry about those details.

• run stuff! If there is no background process, make a new one

Each time you want the current process (to eval the code from a buffer) simply call sesman-ensure-session which will return the most meaningful session (depending on current sessions, current associations, current context and user customization). If no session exists, sesman asks the user for new session or cranks it automatically (again, depending on user customization). If there are sessions already running (in other projects for example) user can chose to associate current context with one of those sessions.

For non-interactive use (eldoc, company, flymake etc) you don't want to ask the user for the session. For such cases specialized session retrieving commands are available. For instance sesman-linked-session would return the most relevant linked to current context session or nil if there is none.

• show buffer<>process associations

This part is not there yet. Once the UI is properly figured out I might add a tabular display where user can see (and manipulate) all running sessions across all IDEs with their current associations. But the added value of such display is questionable. The association and life-cycle management commands in sesman-map display current system's sessions during the completion anyway. For instance, each time you (re)associate a process with a new context you will be provided with currently running sessions (sorted in the order of relevance).

[wavexx]

On Sat, Feb 17 2018, Vitalie Spinu wrote: > - run stuff in a different process (let me choose existing ones!) Users can start as many sessions as they need and associate, disassociate or re-associate sessions with projects, directories or buffers at any given time. As an IDE developer you never worry about those details.

<snip>

> - show buffer<>process associations

I need to worry though. I might need to keep state in sync between the current buffer and the associated process. Spawning a new process or dissociating from an existing one might require some extra work. This is the tricky stuff I was always too lazy to implement. And different modes might do different things. The basic example that comes to mind here is process execution tracking. In python-x a "session" would equal to one inferior running, because all the tracking revolves around the currently associated inferior buffer, and not vice-versa. From the inferior buffer I need to know which buffers are associated in order to trigger the relevant events. If I associate a python-x buffer to a *new* inferior, I'm actually moving the buffer to a different session. I think the relationship is a bit more intimate when it boils down to running processes.

[vspinu]

In python-x a "session" would equal to one inferior running,

Right, and it's probably the most common use case for sesman.

From the inferior buffer I need to know which buffers are associated in order to trigger the relevant events.

This is a very good point and should definitely be part of the sesman's system API - given a session return all associated buffers. It essentially boils down to mapping through the buffers and filtering the ones which are associated with the session.

It also makes sense to introduce permanent associations which users cannot change. For instance inferior buffer is always associated with the session to which it belongs. Something like sesman-permanent-session-name buffer local variable. A process variable would make sense as well (or even more sense, in order to avoid with-current-buffer indirections). The downside is that system developers will have to set those variables themselves. Sesman doesn't assume anything about the structure of the session object besides that it's a list starting with a session name.

If I associate a python-x buffer to a new inferior, I'm actually moving the buffer to a different session.

I think there is no "best" rule of how to automatically associate the context on session creation. Currently new sessions are associated with the coarsest context (project, or directory if no project). This means that if current buffer is already explicitly associated with another session the association is not overwritten. I think explicit association with buffers should rarely be used by the users. Such micro-management is often annoying and very often leads to confusion.

There are some sharp corners regarding automatic associations and I am still thinking about it, but one thing I am certain about, it should be up to the user to decide how their associations work. In the future these should be configurable to all likings.

[wavexx]

On Sat, Feb 17 2018, Vitalie Spinu wrote: I think there is no "best" rule of how to automatically associate the context on session creation. Currently new sessions are associated with the coarsest context (project, or directory if no project, or buffer if no directory). This means that if current buffer is already explicitly associated with another session the association is not overwritten. I think explicit association with buffers should rarely be used by the users. Such micro-management is often annoying and very often leads to confusion.

It might make sense to not over-generalize here. Managing the inferior and the associated buffers is the central issue that I had with most issues with. This is quite specific, and has some common usage patterns. For example, like we said before, having the buffers<>inferior map at hand is necessary. The way I do it in python-x is, like you said, iterate though buffers and look at all buffers that have a known buffer-local variable pointing back to the inferior. Quite ugly, but handles some corner cases nicely. As an user, I'd also like some specific behaviors. For example, when I need to select [new session]/[attach to existing], I don't want to see all sessions/IDE/projects. I just want to see buffers<>inferior associations which *also* have the same major mode as I'm currently on. That is, when attaching to a new inferior, I need a compatible one. I don't want a list of generic sessions.

There are some sharp corners regarding automatic associations and I am still thinking about it, but one thing I am certain about, it should be up to the user to decide how their associations work. In the future these should be configurable to all likings.

When setting up the inferior you definitely need to know if it's a new one or an existing one and the list of associated buffers. To be more useful, sesman should probably mediate the common things you need to do with comint-mode as well. For example, setup already the various comint-functions to point to the right inferior and setup any advice if necessary so that things work "automagically". Knowing if the inferior is still working or has completed is something that I'd love to have everywhere, not only in python-x. Like I said, this is the juicy part in setting up inferior management when dealing with multiple buffers. I think I've seen several "session management" libraries, which aim to tack basic "group of buffers and associated files", such as saving and restoring buffer state. I feel that's pretty covered.

[vspinu]

I don't want to see all sessions/IDE/projects.

You see only the ones from the current system (IDE) and they are sorted meaningfully - all sessions already associated with current context, then all friendly sessions (implementation dependent), then all other sessions.

To be more useful, sesman should probably mediate the common things you need to do with comint-mode as well.

This is out of scope I am afraid. Sesman session need not even be composed of inferiors, or processes.
But I do relate to the issue though.

If I understand correctly you mean the interaction part - a set of commands that would send portions of the buffer to the inferior. For that a separate library seems appropriate.

Knowing if the inferior is still working or has completed is something that I'd love to have everywhere, not only in python-x.

This one is inferior specific and for comint buffers boils down to waiting for prompts and/or pinging for the process, not sure if this could be or should be abstracted.

I think I've seen several "session management" libraries,

I am not aware of anything comparable to what sesman aims at.

[wavexx]

On Sun, Feb 18 2018, Vitalie Spinu wrote: > I think I've seen several "session management" libraries, I am not aware of anything comparable to what sesman aims at.

Just for discussion, did you try any package or mode that does something similar or that I should have a look at for ideas?

[vspinu]

I don't think there is any such mode. Every IDE does its own process management. From the ones that I have tried ESS proc management is the most user friendly, but still very basic (buffer centric).

[tarsius]

Now that the recipe has been merged into Melpa, are you going to remove seman.el from the cider recipe?

[vspinu]

Yes. Just haven't had time for this so far.

[bbatsov]

I guess now we can close this ticket.