Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using gpu and from multiprocessing import Pool, mxnet==1.2 #41

Open
chen849157649 opened this issue Jan 11, 2019 · 4 comments
Open

Using gpu and from multiprocessing import Pool, mxnet==1.2 #41

chen849157649 opened this issue Jan 11, 2019 · 4 comments

Comments

@chen849157649
Copy link

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mxnet/symbol/symbol.py", line 1522, in simple_bind
ctypes.byref(exe_handle)))
File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 251, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:18:11] src/storage/storage.cc:65: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x39008a) [0x7efdbb00208a]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x3906c1) [0x7efdbb0026c1]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31aba9d) [0x7efdbde1da9d]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31b3e58) [0x7efdbde25e58]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2a808f6) [0x7efdbd6f28f6]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b31baa) [0x7efdbd7a3baa]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32235) [0x7efdbd7a4235]
[bt] (7) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32528) [0x7efdbd7a4528]
[bt] (8) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b41d46) [0x7efdbd7b3d46]
[bt] (9) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b42264) [0x7efdbd7b4264]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/chengt1/workspace/ArcFaceVideo/mxnet_align/helper.py", line 168, in detect_first_stage_warpper
return detect_first_stage(*args)
File "/home/chengt1/workspace/ArcFaceVideo/mxnet_align/helper.py", line 156, in detect_first_stage
output = net.predict(input_buf)
File "/usr/local/lib/python3.5/dist-packages/mxnet/model.py", line 700, in predict
self._init_predictor(data_shapes, type_dict)
File "/usr/local/lib/python3.5/dist-packages/mxnet/model.py", line 623, in _init_predictor
self.ctx[0], grad_req='null', type_dict=type_dict, **shapes)
File "/usr/local/lib/python3.5/dist-packages/mxnet/symbol/symbol.py", line 1528, in simple_bind
raise RuntimeError(error_msg)
RuntimeError: simple_bind error. Arguments:
conv4_1_weight: (2, 32, 1, 1)
conv1_weight: (10, 3, 3, 3)
conv4_2_bias: (4,)
conv4_2_weight: (4, 32, 1, 1)
prelu1_gamma: (10,)
conv4_1_bias: (2,)
conv2_bias: (16,)
data: (1, 3, 104, 184)
conv2_weight: (16, 10, 3, 3)
conv3_weight: (32, 16, 3, 3)
prelu3_gamma: (32,)
conv1_bias: (10,)
conv3_bias: (32,)
prelu2_gamma: (16,)
[19:18:11] src/storage/storage.cc:65: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x39008a) [0x7efdbb00208a]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x3906c1) [0x7efdbb0026c1]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31aba9d) [0x7efdbde1da9d]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31b3e58) [0x7efdbde25e58]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2a808f6) [0x7efdbd6f28f6]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b31baa) [0x7efdbd7a3baa]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32235) [0x7efdbd7a4235]
[bt] (7) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32528) [0x7efdbd7a4528]
[bt] (8) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b41d46) [0x7efdbd7b3d46]
[bt] (9) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b42264) [0x7efdbd7b4264]

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "demo.py", line 11, in
warped = utils_api.detect_centor_face(detector, full_path,chips_method=False)
File "/home/chengt1/workspace/ArcFaceVideo/utils_api.py", line 39, in detect_centor_face
results = detector.detect_face(img)
File "/home/chengt1/workspace/ArcFaceVideo/mxnet_align/mtcnn_detector.py", line 241, in detect_face
zip(repeat(img), self.PNets[:len(batch)], [scales[i] for i in batch], repeat(self.threshold[0])) )
File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: simple_bind error. Arguments:
conv4_1_weight: (2, 32, 1, 1)
conv1_weight: (10, 3, 3, 3)
conv4_2_bias: (4,)
conv4_2_weight: (4, 32, 1, 1)
prelu1_gamma: (10,)
conv4_1_bias: (2,)
conv2_bias: (16,)
data: (1, 3, 104, 184)
conv2_weight: (16, 10, 3, 3)
conv3_weight: (32, 16, 3, 3)
prelu3_gamma: (32,)
conv1_bias: (10,)
conv3_bias: (32,)
prelu2_gamma: (16,)
[19:18:11] src/storage/storage.cc:65: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x39008a) [0x7efdbb00208a]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x3906c1) [0x7efdbb0026c1]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31aba9d) [0x7efdbde1da9d]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x31b3e58) [0x7efdbde25e58]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2a808f6) [0x7efdbd6f28f6]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b31baa) [0x7efdbd7a3baa]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32235) [0x7efdbd7a4235]
[bt] (7) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b32528) [0x7efdbd7a4528]
[bt] (8) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b41d46) [0x7efdbd7b3d46]
[bt] (9) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2b42264) [0x7efdbd7b4264]

@DuckJ
Copy link

DuckJ commented Apr 23, 2019

Have you solved this problem? I also met this question

@mingzhang96
Copy link

also meet the same question

@guangyuan502
Copy link

the same to me,please save me

@leviethung2103
Copy link

killall python wil solve the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants