Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

macOS: code works in jupyter/ipython but not in terminal #3872

Closed
fohria opened this issue Apr 6, 2020 · 1 comment
Closed

macOS: code works in jupyter/ipython but not in terminal #3872

fohria opened this issue Apr 6, 2020 · 1 comment

Comments

@fohria
Copy link

fohria commented Apr 6, 2020

bayesian newbie here, thanks for this nice library!

Description of your problem

  • running below code from macos command line crashes with compilation error seen in exhibit A

  • issue remains in new conda environment (tested with python=3.7 and python=3.8)

  • the same code works fine in jupyter notebook/jupyter kernel on the same machine with the same conda environment (this part blew my mind 🙃 )

  • the same code works fine from command line on ubuntu with python37/38

  • as suggested in theano max works but not min #3831, adding this theano config line solves the issue theano.config.gcc.cxxflags = "-Wno-c++11-narrowing"

  • however, we then instead reach a new issue with multithreading and the execution just stops with the message seen in exhibit B

  • again, the same code works in jupyter!

  • but here there's a difference between python versions, conda env with python=3.7 works now, but python=3.8 we need the next step as well

  • so as suggested in another issue, Parallel sampling hangs on macOS with Python 3.8 with pickling error #3844, we can solve this new issue with:

import multiprocessing as mp
mp.set_start_method("fork")

I've no idea what's going on here, but since there are several related issues, i thought it might be something weird going on that's macOS specific?

potentially related to:
#3831
#3695
#3844

Please provide a minimal, self-contained, and reproducible example.
sorry about the not-very-minimal example. main model definition is below, it uses theano scan and full code in this gist

actions = theano.shared(np.array([0,1,0,1,1,0,0], dtype='int16'))
rewards = theano.shared(np.array([4,2,6,3,2,5,4], dtype='int16'))

with pm.Model() as qlearn3:

    alpha = pm.Beta('alpha', alpha=1, beta=1)
    tau = pm.HalfNormal('tau', 10)
    gamma = pm.Beta('gamma', alpha=1, beta=1)

    probs = categorical_actionsQL(actions, rewards, alpha, tau, gamma)

    like = pm.Categorical('like', p=probs, observed=actions)

    trace = pm.sample()

Please provide the full traceback.

exhibit A

You can find the C code in this temporary file: /var/folders/kj/ztrcmdjs58ncsdg37pg5pynw0000gn/T/theano_compilation_error_7sn1f20a
Traceback (most recent call last):
  File "model.py", line 41, in <module>
    probs = categorical_actionsQL(actions, rewards, alpha, tau, gamma)
  File "model.py", line 23, in categorical_actionsQL
    output, updates = theano.scan(fn=update_qvalsQL,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/scan_module/scan.py", line 1077, in scan
    scan_outs = local_op(*scan_inputs)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/op.py", line 669, in __call__
    thunk = node.op.make_thunk(node, storage_map, compute_map,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/scan_module/scan_op.py", line 861, in make_thunk
    self.fn = function(wrapped_inputs,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/compile/function.py", line 306, in function
    fn = pfunc(params=inputs,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/compile/pfunc.py", line 483, in pfunc
    return orig_function(inputs, cloned_outputs, mode,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/compile/function_module.py", line 1841, in orig_function
    fn = m.create(defaults)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/compile/function_module.py", line 1714, in create
    _fn, _i, _o = self.linker.make_thunk(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/link.py", line 697, in make_thunk
    return self.make_all(input_storage=input_storage,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/vm.py", line 1087, in make_all
    thunks.append(node.op.make_thunk(node,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/op.py", line 954, in make_thunk
    return self.make_c_thunk(node, storage_map, compute_map,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/op.py", line 857, in make_c_thunk
    outputs = cl.make_thunk(input_storage=node_input_storage,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cc.py", line 1215, in make_thunk
    cthunk, module, in_storage, out_storage, error_storage = self.__compile__(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cc.py", line 1153, in __compile__
    thunk, module = self.cthunk_factory(error_storage,
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cc.py", line 1623, in cthunk_factory
    module = get_module_cache().module_from_key(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cmodule.py", line 1189, in module_from_key
    module = lnk.compile_cmodule(location)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cc.py", line 1520, in compile_cmodule
    module = c_compiler.compile_str(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/theano/gof/cmodule.py", line 2410, in compile_str
    raise Exception('Compilation failed (return status=%s): %s' %
Exception: ('The following error happened while compiling the node', Elemwise{mul,no_inplace}(InplaceDimShuffle{x,0}.0, InplaceDimShuffle{x,x}.0), '\n', "Compilation failed (return status=1): /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:506:27: error: non-constant-expression cannot be narrowed from type 'npy_intp' (aka 'long') to 'int' in initializer list [-Wc++11-narrowing].     int init_totals[2] = {V1_n0, V3_n1};.                           ^~~~~. /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:506:27: note: insert an explicit cast to silence this issue.     int init_totals[2] = {V1_n0, V3_n1};.                           ^~~~~.                           static_cast<int>( ). /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:506:34: error: non-constant-expression cannot be narrowed from type 'npy_intp' (aka 'long') to 'int' in initializer list [-Wc++11-narrowing].     int init_totals[2] = {V1_n0, V3_n1};.                                  ^~~~~. /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:506:34: note: insert an explicit cast to silence this issue.     int init_totals[2] = {V1_n0, V3_n1};.                                  ^~~~~.                                  static_cast<int>( ). /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:518:12: error: non-constant-expression cannot be narrowed from type 'ssize_t' (aka 'long') to 'int' in initializer list [-Wc++11-narrowing].         0, V3_stride1, .            ^~~~~~~~~~. /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:518:12: note: insert an explicit cast to silence this issue.         0, V3_stride1, .            ^~~~~~~~~~.            static_cast<int>( ). /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:520:1: error: non-constant-expression cannot be narrowed from type 'ssize_t' (aka 'long') to 'int' in initializer list [-Wc++11-narrowing]. V1_stride0, V1_stride1. ^~~~~~~~~~. /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:520:1: note: insert an explicit cast to silence this issue. V1_stride0, V1_stride1. ^~~~~~~~~~. static_cast<int>( ). /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:520:13: error: non-constant-expression cannot be narrowed from type 'ssize_t' (aka 'long') to 'int' in initializer list [-Wc++11-narrowing]. V1_stride0, V1_stride1.             ^~~~~~~~~~. /Users/foh/.theano/compiledir_macOS-10.14.6-x86_64-i386-64bit-i386-3.8.2-64/tmpzsu2aa1s/mod.cpp:520:13: note: insert an explicit cast to silence this issue. V1_stride0, V1_stride1.             ^~~~~~~~~~.             static_cast<int>( ). 5 errors generated.. ", '[Elemwise{mul,no_inplace}(<TensorType(float64, row)>, <TensorType(float64, (True, True))>)]')

exhibit B

model.py", line 47, in <module>
    trace = pm.sample()
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/pymc3/sampling.py", line 469, in sample
    trace = _mp_sample(**sample_args)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/pymc3/sampling.py", line 1053, in _mp_sample
    sampler = ps.ParallelSampler(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/pymc3/parallel_sampling.py", line 355, in __init__
    self._samplers = [
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/pymc3/parallel_sampling.py", line 356, in <listcomp>
    ProcessAdapter(
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/site-packages/pymc3/parallel_sampling.py", line 242, in __init__
    self._process.start()
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/Users/foh/anaconda3/envs/pymc/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Please provide any additional information below.

oh i also updated gcc via homebrew and rebooted macos to no avail

Versions and main components

  • PyMC3 Version: 3.8
  • Theano Version: 1.0.4
  • Python Version: 3.8
  • Operating system: macOS 10.14.6
  • How did you install PyMC3: conda
@AlexAndorra
Copy link
Contributor

Hi Henrik,
Thanks for reporting! I'm closing since I think this is fixed by #3919, but feel free to reopen if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fohria @AlexAndorra