automl-gs assumes the Python that's running is the global python

[Julian]

(I need to fill this in with a full reproducer) but as a placeholder:

I just ran automl-gs for the first time, and it errored out with:

/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping

  0%|          | 0/100 [00:00<?, ?trial/s]

  0%|          | 0/20 [00:00<?, ?epoch/s]�[ATraceback (most recent call last):
  File "model.py", line 2, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'


  0%|          | 0/20 [00:00<?, ?epoch/s]�[A/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py:858: ResourceWarning: subprocess 12646 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Solving a classification problem, maximizing accuracy using tensorflow.

Modeling with field specifications:
message.received: numeric
message.sender: categorical
message.subject: text
message.size: numeric
message.recipients: categorical
response.sent: numeric
response.sender: categorical
response.subject: text
response.size: numeric
response.recipients: categorical
lag_readable: text
Traceback (most recent call last):
  File "/Users/jberman/.local/bin/automl_gs", line 10, in <module>
    sys.exit(cmd())
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/automl_gs/automl_gs.py", line 175, in cmd
    tpu_address=args.tpu_address)
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/automl_gs/automl_gs.py", line 87, in automl_grid_search
    "metadata", "results.csv"))
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/pandas/io/parsers.py", line 702, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/pandas/io/parsers.py", line 429, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/pandas/io/parsers.py", line 1122, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Users/jberman/.local/share/virtualenvs/automl/lib/python3.7/site-packages/pandas/io/parsers.py", line 1853, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 387, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'automl_train/metadata/results.csv' does not exist: b'automl_train/metadata/results.csv'
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='automl_results.csv' mode='w' encoding='UTF-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback

Where at least that first error there is because the auto-generated .py file doesn't contain a shebang:

⊙  head automl_train/model.py                                                                                                jberman@USNYHJBERMANMB2 ●
import argparse
import pandas as pd
from pipeline import *

and seems to be being executed by just calling python, whereas it needs to use sys.executable from the original Python that was used to run automl-gs (and which is where pandas would be installed to).

(The blow up to be clear is because it's installed into a virtualenv, and in fact whatever other Python it's running model.py with in fact does not have pandas installed to it).

[Julian]

https://github.com/minimaxir/automl-gs/blob/master/automl_gs/utils_automl.py#L293-L297 look like the relevant lines -- automl-gs shouldn't go off trying to find a Python, since the Python that it's installed to might not be on the path at all -- it's just sys.executable.

[Julian]

... which I can confirm if I change those lines to just use sys.executable, I at least move on to the next error (one telling me I need to install an ML framework).