Make sure promnesia works under Windows

[Pfedj]

Windows 10 with Python 3.7.6.

When I run promnesia demo --port 16789 https://github.com/karlicoss/exobrain
I get error:

c:\anaconda\lib\site-packages\promnesia\kython\klogging2.py:27: UserWarning: You might want to install 'logzero' for nice colored logs!
  warnings.warn("You might want to install 'logzero' for nice colored logs!")
[INFO    2020-05-18 20:49:57,102 promnesia common.py:329] extracting via promnesia.sources.guess:index ('https://github.com/karlicoss/exobrain',) {} ... ...
[INFO    2020-05-18 20:49:57,102 promnesia common.py:329] extracting via promnesia.sources.guess:index ('https://github.com/karlicoss/exobrain',) {} ... ...
Traceback (most recent call last):
  File "c:\anaconda\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\anaconda\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\anaconda\Scripts\promnesia.exe\__main__.py", line 7, in <module>
  File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 173, in main
    do_demo(index_as=getattr(args, 'as'), params=args.params, port=args.port, config_file=args.config)
  File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 110, in do_demo
    errors = _do_index()
  File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 49, in _do_index
    hist, errors = previsits_to_history(ex, src=ex.src)
  File "c:\anaconda\lib\site-packages\promnesia\common.py", line 333, in previsits_to_history
    previsits = list(extr()) # TODO DEFENSIVE HERE!!!
  File "c:\anaconda\lib\site-packages\promnesia\sources\guess.py", line 33, in index
    yield from index_(path, *args, **kwargs)
  File "c:\anaconda\lib\site-packages\promnesia\sources\vcs.py", line 15, in index
    check_call(['git', 'clone', repo, tp])
  File "c:\anaconda\lib\subprocess.py", line 358, in check_call
    retcode = call(*popenargs, **kwargs)
  File "c:\anaconda\lib\subprocess.py", line 339, in call
    with Popen(*popenargs, **kwargs) as p:
  File "c:\anaconda\lib\subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "c:\anaconda\lib\subprocess.py", line 1148, in _execute_child
    args = list2cmdline(args)
  File "c:\anaconda\lib\subprocess.py", line 555, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'WindowsPath' is not iterable

So how can I run this program on Windows?

[karlicoss]

Hey.. Uhoh, it would need some work for that first, sorry! I'm trying to keep things as portable as possible, but had to keep Windows in mind, I haven't used it for years.
I think github actions got windows available for CI, so maybe at some point I'll have time for this.

[karlicoss]

Some related comments here (apparently can be run via WSL?) #114 (comment)

[karlicoss]

Right, I played a bit with CircleCI Windows runners and SSH debugging -- not too pleasant experience, but did tackle some issues.
#143

This particular problem is a bug in Python 3.7! https://bugs.python.org/issue33617
I patched this particular place, but it's possible that there are more. If anyone notices more of these, happy to receive a PR (or at least post a comment).

Most tests are passing now, however, hug (the HTTP framework I'm using) itself doesn't seem to start at all, which is of course a major issue.
I wonder if it's related to the firewall (e.g. similar to hugapi/hug#470), but don't know about Windows, so will have to research it.

Not sure when I'll have time for another round of investigation, but hopefully this motivates someone else to carry on (especially if you have a proper Windows system around!).

[mdroidian]

Alright. I don't have much python knowledge, but here's where I'm at

c:\>promnesia index

[INFO    2020-09-30 07:25:36,953 promnesia common.py:417] extracting via promnesia.sources.auto:index () {} ... ...
[ERROR   2020-09-30 07:25:36,954 promnesia common.py:424] index() missing 1 required positional argument: 'path'
    Traceback (most recent call last):
      File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\common.py", line 422, in previsits_to_history
        previsits = list(extr())
      File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\common.py", line 410, in <lambda>
        extr = lambda: ex.ff(*ex.args, **ex.kwargs)
    TypeError: index() missing 1 required positional argument: 'path'
[INFO    2020-09-30 07:25:37,015 promnesia dump.py:51] saved database to C:\Users\Uncle Awesome\AppData\Local\promnesia\promnesia\promnesia.sqlite

c:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia>promnesia index

Traceback (most recent call last):
  File "c:\users\uncle awesome\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\uncle awesome\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\Scripts\promnesia.exe\__main__.py", line 7, in <module>
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\__main__.py", line 254, in main
    do_index(config_file=args.config)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\__main__.py", line 59, in do_index
    config.load_from(config_file)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 87, in load_from
    instance = import_config(config_file)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 103, in import_config
    spec.loader.exec_module(mod) # type: ignore
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 9, in <module>
    from .common import PathIsh, get_tmpdir, appdirs, default_output_dir
ImportError: attempted relative import with no known parent package

c:\>promnesia serve
Immediate responds with

[INFO    2020-09-30 07:32:49,837 promnesia server.py:305] Running server: ['python', '-m', 'hug', '-p', '13131', '-f', 'C:\\Users\\Uncle Awesome\\AppData\\Roaming\\Python\\Python38\\site-packages\\promnesia\\server.py']
c:\>

But a second later gives

Traceback (most recent call last):
  File "C:\Users\Uncle Awesome\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Uncle Awesome\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\__main__.py", line 3, in <module>
    hug.development_runner.hug.interface.cli()
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 650, in __call__
    raise exception
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 646, in __call__
    result = self.output(self.interface(**pass_to_function), context)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 129, in __call__
    return __hug_internal_self._function(*args, **kwargs)
  File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\development_runner.py", line 65, in hug
    api_module = importlib.machinery.SourceFileLoader(file.split(".")[0], file).load_module()
  File "<frozen importlib._bootstrap_external>", line 462, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 962, in load_module
  File "<frozen importlib._bootstrap_external>", line 787, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 779, in exec_module
  File "<frozen importlib._bootstrap_external>", line 915, in get_code
  File "<frozen importlib._bootstrap_external>", line 972, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Uncle'

[karlicoss]

Just promnesia index won't do much if you don't have a config, I recommend following the setup/troubleshooting guide https://github.com/karlicoss/promnesia#setup

As for promnesia serve, at the first glance, this looks like a bug in hug with the whitespace handling in paths it splits Uncle Awesome for some reason..

[mdroidian]

RE: promnesia index ah of course.🤦‍♂️ I added a proper path and looks good now 👍

RE: promnesia serve I added an issue at hug hugapi/hug/issues/872

[karlicoss]

I looked a bit and I think I was wrong about hug -- more likely an issue with handling whitespace in os.execvpe call.

• https://bugs.python.org/issue19066
• https://stackoverflow.com/questions/53218984/how-can-i-avoid-the-windows-exec-and-spawn-functions-splitting-arguments-at

I think you can try enclosing __file__ here in extra quotes:

promnesia/src/promnesia/server.py

Line 303 in 15088ba

'-f', __file__,

, e.g. '-f', '"' + __file__ + '"',.

If that works, I guess I'll look up what should be used instead so it's portable.. or perhaps hack it just for Windows.

[mdroidian]

can confirm '-f', '"' + __file__ + '"', works 👍

[karlicoss]

User reported similar error when checking promnesia doctor config (fails during compileall check). I guess need to setup some mypy rule to check that all paths are passed as strings

[karlicoss]

Also sqlitebrowser doesn't (or might not?) register itself in PATH on windows, so maybe need a better way of launching it

[karlicoss]

Perhaps need to document: paths on Windows need to be prepended with r because of backslashes https://stackoverflow.com/questions/18084554/why-do-i-get-a-syntaxerror-for-a-unicode-escape-in-my-file-path/18084594#18084594

Also find command on Windows is... maybe need to fallback to os.walk to start with

[karlicoss]

all right, fixed a bunch of things here... #197

• the argument of type 'WindowsPath' is not iterable thing -- worked around via a whapper for subprocess functions 8ec3a24
• traversing the files/grepping should work now 42e2604
• not sure how to configure libmagic on Windows so just worked around it defensively 313fd25
• 6/75 tests are still failing, most have to do with that Path.is_dir doesn't like when path contains an asterisk, which globbing in HPI relies upon.. will need to swap these lines, will do later https://github.com/karlicoss/HPI/blob/6d9bc2964b24cfe6187945f4634940673dfe9c27/my/core/common.py#L160-L165

still haven't added static analysis for these, so without running tests on Windows some of it might go stale.. but hopefully I'll remember this for a while

[karlicoss]

karlicoss/HPI#124 .. and after this update all Promnesia tests pass (except test_example_config, which has some linux/macos specific paths -- not sure what's the best to do with it)

[Cobertos]

Have you considered just saying Linux support works only under WSL, @karlicoss ? With the reliance on fd, find, and other *nix utilities, it would probably be easier to just direct users to use WSL instead of writing new implementations. It works especially well with a library like this because at the end of the day, it's a CLI program + filesystem ingest + webserver, all of which are able to traverse the WSL boundary fairly easily.

It would be more overhead on the part of the users to install and configure WSL, but it's something that's documented in a lot of places and IMO was easy the last time I did it (I recently switched to Linux after about 10+ years of mainly Windows dev-ing)

[karlicoss]

Yeah, I guess I don't know much about it, so figured I'd give it a go to fix (last time I did anything on Windows was ~10 years ago). I guess worth mentioning in the docs that via WSL it would be much more likely to work properly!

[AB1908]

So, is the WSL version working properly?

Update: It does! Strangely, there's some path issues so have to use python3 -m promnesia vs promnesia.

[karlicoss]

Oh that's good to know. Yeah issues like these can always happen, and on linux as well...
Generally running everything as python3.x -m is the safest thing to do, it's just a bit verbose

[seanbreckenridge]

Recently setup a old thinkpad on windows -- will probably be testing HPI/promnesia on there eventually, will report/PR some docs if I have any issues

Edit: Had no issues setting it up in WSL

[mdroidian]

I have a (fairly) fresh Windows install and had no issues with Promnesia this go around!
Right in Windows, not via WSL.

• Installed Python 3.10.2 (Windows installer (64-bit)) via python.org
• pip3 install --user promnesia
• added a folder to PATH
• promnesia demo https://github.com/karlicoss/exobrain seemed to run fine
• promnesia config create, added a local folder with .md files, promnesia index
• promnesia serve

Seems good 👍

[AB1908]

Very cool! I've shifted back to Arch and haven't had the time to try it out yet.

[karlicoss]

I did a bunch of fixes, so pretty sure it should work both under WSL or regular shell (or whatever it's called in Windows world). It also runs on CI now so hopefully won't regress. Feel free to reopen/comment if you still have issues with Windows

[Anhunghezo]

• added a folder to PATH

What do you mean in "add a folder to PATH", sir?

I installed Python 3.10.7.
I run "pip3 install --user promnesia" and successfully.

but when I run "promnesia demo https://github.com/karlicoss/exobrain" as your suggestion, I received an error like this:

C:\Python3107>promnesia demo https://github.com/karlicoss/exobrain
'promnesia' is not recognized as an internal or external command,
operable program or batch file.

Do I make mistake in something, sir?

[AB1908]

python promnesia since you installed it as a user package and not a global package.