Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't export dataset #1219

Closed
l3xis opened this issue Mar 1, 2020 · 9 comments · Fixed by #1567
Closed

Can't export dataset #1219

l3xis opened this issue Mar 1, 2020 · 9 comments · Fixed by #1567
Assignees
Labels
bug Something isn't working

Comments

@l3xis
Copy link

l3xis commented Mar 1, 2020

I can't export dataset. Possible formats not shown.
изображение

In developer console i see that /api/v1/server/dataset/formats returns http 500 error. Error without details.
изображение

Inside container at same moment:
2020-02-28 11:18:33,801 DEBG 'runserver' stderr output:
[Fri Feb 28 11:18:33.801370 2020] [core:notice] [pid 10:tid 139667012917120] AH00051: child pid 199 exit signal Illegal instruction (4), possible coredump in /tmp/mod_wsgi-localhost:8080:1000

gdb information about coredump:
Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done.
[New LWP 586]
[New LWP 594]
[New LWP 599]
[New LWP 600]
[New LWP 597]
[New LWP 598]
[New LWP 596]
[New LWP 595]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `(wsgi:localhost:8080:1000) -f /tmp/mod_wsgi-localhost:8080:1000/httpd.conf -DMO'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x00007f06c1d4774d in poll () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (Thread 0x7f06c2b11780 (LWP 586))]
(gdb) where
#0 0x00007f06c1d4774d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f06c22553f7 in apr_poll () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f06bf778ff8 in wsgi_daemon_main (daemon=0x7f06c2aa3e10, p=0x7f06c2b19028) at src/server/mod_wsgi.c:9667
#3 wsgi_start_process (p=0x7f06c2b19028, daemon=0x7f06c2aa3e10) at src/server/mod_wsgi.c:10395
#4 0x00007f06bf77affb in wsgi_manage_process (reason=, data=0x7f06c2aa3e10, status=)
at src/server/mod_wsgi.c:8169
#5 0x00007f06c225048e in apr_proc_other_child_alert () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#6 0x00007f06c11dbe14 in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
#7 0x000055647616988e in ap_run_mpm ()
#8 0x0000556476162740 in main ()
(gdb) bt full
#0 0x00007f06c1d4774d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1 0x00007f06c22553f7 in apr_poll () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
No symbol table info available.
#2 0x00007f06bf778ff8 in wsgi_daemon_main (daemon=0x7f06c2aa3e10, p=0x7f06c2b19028) at src/server/mod_wsgi.c:9667
buf = ""
nbytes = 1
i =
rv =
thread_rv = 1
thread_attr = 0x7f06c2ad8fc0
reaper = 0x7f06c2ab4078
poll_fd = {p = 0x7f06c2abc008, desc_type = APR_POLL_FILE, reqevents = 1, rtnevents = 0, desc = {f = 0x7f06c2ad8c28,
s = 0x7f06c2ad8c28}, client_data = 0xc6227c26fdd88100}
poll_count = 0
#3 wsgi_start_process (p=0x7f06c2b19028, daemon=0x7f06c2aa3e10) at src/server/mod_wsgi.c:10395
status =
lr =
i =
entries =
entry =
daemon = 0x7f06c2aa3e10
p = 0x7f06c2b19028
status =
#4 0x00007f06bf77affb in wsgi_manage_process (reason=, data=0x7f06c2aa3e10, status=)
at src/server/mod_wsgi.c:8169
mpm_state = 1
stopping =
daemon = 0x7f06c2aa3e10
#5 0x00007f06c225048e in apr_proc_other_child_alert () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
No symbol table info available.
#6 0x00007f06c11dbe14 in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
No symbol table info available.
#7 0x000055647616988e in ap_run_mpm ()
No symbol table info available.
#8 0x0000556476162740 in main ()
No symbol table info available.
(gdb) backtrace
#0 0x00007f06c1d4774d in poll () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f06c22553f7 in apr_poll () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f06bf778ff8 in wsgi_daemon_main (daemon=0x7f06c2aa3e10, p=0x7f06c2b19028) at src/server/mod_wsgi.c:9667
#3 wsgi_start_process (p=0x7f06c2b19028, daemon=0x7f06c2aa3e10) at src/server/mod_wsgi.c:10395
#4 0x00007f06bf77affb in wsgi_manage_process (reason=, data=0x7f06c2aa3e10, status=)
at src/server/mod_wsgi.c:8169
#5 0x00007f06c225048e in apr_proc_other_child_alert () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
#6 0x00007f06c11dbe14 in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
#7 0x000055647616988e in ap_run_mpm ()
#8 0x0000556476162740 in main ()
(gdb) quit

Anyone has any idea on why this is happening?
Probably similar problem is described at stackoverflow: https://stackoverflow.com/questions/60316150/cvat-computer-vision-annotation-tool-export-impossible

@azhavoro
Copy link
Contributor

azhavoro commented Mar 2, 2020

@l3xis Hi, thanks for the report, what version of cvat do you use (git hash would be nice)?

@l3xis
Copy link
Author

l3xis commented Mar 2, 2020

Hi, current commit id is 9850094

@zhiltsov-max
Copy link
Contributor

@l3xis, the problem you described looks similar to the issue here: https://unix.stackexchange.com/questions/253465/php-app-crashes-child-pid-exits-with-illegal-instruction-4, so it may be some conflict with your hardware and Apache library libapr, which might be worth reporting after some investigation. Can you reproduce the issue?

@efcy
Copy link
Contributor

efcy commented Mar 6, 2020

I have the same problem with 24130cda . I hosted the application successfully on a ubuntu 18.04 lts system with up to date packages. I updated everything a couple of days ago and have this problem since then.

@nmanovic nmanovic added the bug Something isn't working label Mar 6, 2020
@nmanovic nmanovic added this to the 0.6.0 - Release milestone Mar 6, 2020
@l3xis
Copy link
Author

l3xis commented Mar 11, 2020

Hi. I found that crash causes plugin "tf_detection_api_format". I disable it (delete from plugins directory). Now i can export dataset.

@zhiltsov-max
Copy link
Contributor

zhiltsov-max commented Mar 11, 2020

@l3xis, thanks for finding the possible cause! Could you please check if you're able to do python -c 'import tensorflow' inside your environment?

@l3xis
Copy link
Author

l3xis commented Mar 11, 2020

Of course
django@4969d0312ecd:~$ python3 -c 'import tensorflow'
Illegal instruction (core dumped)

Crash under gdb:

django@4969d0312ecd:~$ gdb --args python3 -c 'import tensorflow'
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/python3 -c import\ tensorflow
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7f5439bfe700 (LWP 783)]

Thread 1 "python3" received signal SIGILL, Illegal instruction.
0x00007f5432b4b8d0 in nsync::nsync_mu_init(nsync::nsync_mu_s_*) ()
from /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/_pywrap_tensorflow_internal.so
(gdb)

@alalek
Copy link

alalek commented Mar 11, 2020

  • run this gdb command after the crash: (gdb) x/i $rip
  • search for "avx" in lscpu or cat /proc/cpuinfo output

@l3xis
Copy link
Author

l3xis commented Mar 11, 2020

(gdb) x/i $rip
=> 0x7fbc8b09b8d0 <_ZN5nsync13nsync_mu_initEPNS_11nsync_mu_s_E>: vpxor %xmm0,%xmm0,%xmm0

I understood. My CPU too old and does not support AVX instruction set.

I found this issue tensorflow/tensorflow#24548

Can you add information about tensorflow pre-built package hardware requirements to readme?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants