Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault (core dumped) #9

Closed
alexbogus opened this issue May 21, 2020 · 25 comments
Closed

Segmentation fault (core dumped) #9

alexbogus opened this issue May 21, 2020 · 25 comments
Assignees

Comments

@alexbogus
Copy link

Hi Mike,

Me again, sorry for the inconvenience.
After a few days of working the docker, I appreciate in the logs a segmentation fault.

These are the logs

,[mlat-client] Delaying mlat-client startup until rbfeeder receives station sn...
,[rbfeeder] [2020-05-21 16:15:08]  Starting RBFeeder Version 0.2.6 (build 20180313082038)
,[rbfeeder] [2020-05-21 16:15:08]  Using configuration file: /etc/rbfeeder.ini
,[rbfeeder] [2020-05-21 16:15:08]  Network-mode enabled.
,[rbfeeder] [2020-05-21 16:15:08]  		Remote host to fetch data: xx.xx.xx.xx
,[rbfeeder] [2020-05-21 16:15:08]  		Remote port: 30005
,[rbfeeder] [2020-05-21 16:15:08]  		Remote protocol: BEAST
,[rbfeeder] [2020-05-21 16:15:08]  System: raspberry
,[rbfeeder] [2020-05-21 16:15:08]  Start date/time: 2020-05-21 16:15:07
,[rbfeeder] [2020-05-21 16:15:08]  Socket for ANRB created. Waiting for connections on port 32088
,[rbfeeder] [2020-05-21 16:15:09]  Connection established.
,[rbfeeder] [2020-05-21 16:15:10]  Connection with RadarBox24 server OK! Key accepted by server.
,[rbfeeder] [2020-05-21 16:15:38]  ******** Statistics updated every 30 seconds ********
,[rbfeeder] [2020-05-21 16:15:38]  Packets sent in the last 30 seconds: 40, Total packets sent since startup: 40
,[mlat-client] Thu May 21 16:15:38 2020 mlat-client 0.2.11 starting up
,[mlat-client] Thu May 21 16:15:38 2020 Listening for Beast-format results connection on port 30105
,[mlat-client] Thu May 21 16:15:38 2020 Connected to multilateration server at mlat1.rb24.com:40900, handshaking
,Segmentation fault (core dumped)

My apologies for the inconvenience
Thanks again four your support

Kind regards
Alex

@mikenye mikenye self-assigned this May 22, 2020
@mikenye
Copy link
Member

mikenye commented May 22, 2020

Hi @alexbogus, no inconvenience mate, good to hear from you again.

Firstly, is your underlying host still this?:

Linux myhostname 5.3.0-46-generic #38-Ubuntu SMP Fri Mar 27 17:37:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Secondly, was this a once off or does it happen over and over continually?

Lastly, can you please post the output of docker inspect rbfeeder (you might need to change rbfeeder to your container name)?

Please sanitise this output by censoring your ALT, LAT, LONG and SHARING_KEY.

Thanks.

-Mike

@alexbogus
Copy link
Author

Hi Mike,
Thanks for your support.

Output of my uname -a:

Linux myhostname 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Output of my rbfeeder container:

[
    {
        "Id": "78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad",
        "Created": "2020-05-22T06:08:16.692389629Z",
        "Path": "/init",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2049801,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-05-22T06:08:17.564829414Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:d8fffc8c09285f9758e5646cdd6b90efd28730586eaca100598576cc1534b0b3",
        "ResolvConfPath": "/var/lib/docker/containers/78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad/resolv.conf",       
        "HostnamePath": "/var/lib/docker/containers/78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad/hostname",
        "HostsPath": "/var/lib/docker/containers/78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad/hosts",
        "LogPath": "/var/lib/docker/containers/78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad/78fd9a5168584cdd8cc475bead93c8318e10306e196d60c7777b0e5ce0a84aad-json.log",
        "Name": "/radarbox",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/docker/adsb/radarbox/rbfeeder.ini:/etc/rbfeeder.ini:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "5",
                    "max-size": "10m"
                }
            },
            "NetworkMode": "adsb",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/39fbdd00d600b5b1914609af75d48e5080eaed832a72b737b295294ea9621607-init/diff:/var/lib/docker/overlay2/4db903e0d7e3d16ca01ac369cba5abb76de7a061dc06cefb3d5f4bf4f5067d8d/diff:/var/lib/docker/overlay2/a3719a00c0c85c58acbfe26f6c9096632ffba3743b719b68a4d3ffbc7f0414b3/diff:/var/lib/docker/overlay2/d2f72dfae6b2c35bcb132932e08f4bdba05c7a87de7b7065b413e3cb461b05ee/diff:/var/lib/docker/overlay2/441606064ddcf55c4b5a9048156722bf9d347b61b4c898e69c6d9656d984eed7/diff:/var/lib/docker/overlay2/db365569bfdb408bd028f00d2d542b029cec8ef066a9c37aa21a3c1ba16b65e4/diff:/var/lib/docker/overlay2/f1f6f6bffee3f6c1b13f4eaa39bfbfdb9c35c1469a3bfb30ef8f2139bdca4cfd/diff:/var/lib/docker/overlay2/cf5c7d5fb37883b266f18903c48526784e1a14cc84576c47f278ab14ef2fa79d/diff:/var/lib/docker/overlay2/42d14487348d68ebef163368b471c8ddf3471af1b428d1892eb0e31448e5b98c/diff:/var/lib/docker/overlay2/a59aaedc75d400fe67878006650cc1aab4067120b44f47f4d15ba49b963f0a9f/diff",
                "MergedDir": "/var/lib/docker/overlay2/39fbdd00d600b5b1914609af75d48e5080eaed832a72b737b295294ea9621607/merged",
                "UpperDir": "/var/lib/docker/overlay2/39fbdd00d600b5b1914609af75d48e5080eaed832a72b737b295294ea9621607/diff",
                "WorkDir": "/var/lib/docker/overlay2/39fbdd00d600b5b1914609af75d48e5080eaed832a72b737b295294ea9621607/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/docker/adsb/radarbox/rbfeeder.ini",
                "Destination": "/etc/rbfeeder.ini",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "xxxxxxxxxxxxx",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "30105/tcp": {},
                "32088/tcp": {}
            },
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "TZ=xxxxxxxxxxxxx",
                "LAT=xx.xx",
                "LONG=-xx.xx",
                "BEASTHOST=xx.xx.xx.xx",
                "ALT=xx",
                "SHARING_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "BEASTPORT=30005",
                "MLAT_SERVER=mlat1.rb24.com:40900"
            ],
            "Cmd": null,
            "Image": "mikenye/radarbox:latest",
            "Volumes": {
                "/etc/rbfeeder.ini": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/init"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "bf2491c980766e73ee1d71f6cd417f17dc68499bcae922c679c2563264e09c08",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "adsb",
                "com.docker.compose.service": "rbfeeder",
                "com.docker.compose.version": "1.20.0-rc2"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b7f70b5eb2c59dd50d66fa65bb0c9c7ef666fc7446f4b20ec0d74c9e58d125a5",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "30105/tcp": null,
                "32088/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/b7f70b5eb2c5",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "adsb": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "78fd9a516858",
                        "rbfeeder"
                    ],
                    "NetworkID": "712df2cbee948ea19eb1bb486f5f8dd34951d8d702f8f034d50119bcc74ccef2",
                    "EndpointID": "84794e62e039991b5503d69b947f50be87b82b54272cdcfd0b1504014366c5c7",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.7",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "xx:xx:xx:xx:xx:xx",
                    "DriverOpts": null
                }
            }
        }
    }
]

Thank you for taking the time to help me
Kind regards
Alex

@mikenye
Copy link
Member

mikenye commented May 22, 2020

Is the segfault a once off or does it happen over and over continually?

@alexbogus
Copy link
Author

alexbogus commented May 22, 2020

It happen over and over continually

@mikenye
Copy link
Member

mikenye commented May 22, 2020

Are you able to post the output of tail -100 /var/log/kern.log ?

You may need to censor some of the information in this file... I'm looking for more information on what is causing the segfault.

Thanks.

@alexbogus
Copy link
Author

This is the output of tail -100 /var/log/kern.log

br-cf6a80473db0: port 15(vethbbc9800) entered disabled state
eth0: renamed from veth9b9ac73
IPv6: ADDRCONF(NETDEV_CHANGE): vethbbc9800: link becomes ready
br-cf6a80473db0: port 15(vethbbc9800) entered blocking state
br-cf6a80473db0: port 15(vethbbc9800) entered forwarding state
br-cf6a80473db0: port 15(vethbbc9800) entered disabled state
veth9b9ac73: renamed from eth0
br-cf6a80473db0: port 15(vethbbc9800) entered disabled state
device vethbbc9800 left promiscuous mode
br-cf6a80473db0: port 15(vethbbc9800) entered disabled state
br-cf6a80473db0: port 15(veth5f8aef7) entered blocking state
br-cf6a80473db0: port 15(veth5f8aef7) entered disabled state
device veth5f8aef7 entered promiscuous mode
br-cf6a80473db0: port 15(veth5f8aef7) entered blocking state
br-cf6a80473db0: port 15(veth5f8aef7) entered forwarding state
br-cf6a80473db0: port 15(veth5f8aef7) entered disabled state
eth0: renamed from veth854e18f
IPv6: ADDRCONF(NETDEV_CHANGE): veth5f8aef7: link becomes ready
br-cf6a80473db0: port 15(veth5f8aef7) entered blocking state
br-cf6a80473db0: port 15(veth5f8aef7) entered forwarding state
br-cf6a80473db0: port 15(veth5f8aef7) entered disabled state
veth854e18f: renamed from eth0
br-cf6a80473db0: port 15(veth5f8aef7) entered disabled state
device veth5f8aef7 left promiscuous mode
br-cf6a80473db0: port 15(veth5f8aef7) entered disabled state
br-cf6a80473db0: port 15(veth61d9f88) entered blocking state
br-cf6a80473db0: port 15(veth61d9f88) entered disabled state
device veth61d9f88 entered promiscuous mode
br-cf6a80473db0: port 15(veth61d9f88) entered blocking state
br-cf6a80473db0: port 15(veth61d9f88) entered forwarding state
br-cf6a80473db0: port 15(veth61d9f88) entered disabled state
eth0: renamed from vethf24b0ce
IPv6: ADDRCONF(NETDEV_CHANGE): veth61d9f88: link becomes ready
br-cf6a80473db0: port 15(veth61d9f88) entered blocking state
br-cf6a80473db0: port 15(veth61d9f88) entered forwarding state
br-cf6a80473db0: port 15(veth61d9f88) entered disabled state
vethf24b0ce: renamed from eth0
br-cf6a80473db0: port 15(veth61d9f88) entered disabled state
device veth61d9f88 left promiscuous mode
br-cf6a80473db0: port 15(veth61d9f88) entered disabled state
br-cf6a80473db0: port 15(veth8e823d7) entered blocking state
br-cf6a80473db0: port 15(veth8e823d7) entered disabled state
device veth8e823d7 entered promiscuous mode
br-cf6a80473db0: port 15(veth8e823d7) entered blocking state
br-cf6a80473db0: port 15(veth8e823d7) entered forwarding state
br-cf6a80473db0: port 15(veth8e823d7) entered disabled state
eth0: renamed from veth8cf97b9
IPv6: ADDRCONF(NETDEV_CHANGE): veth8e823d7: link becomes ready
br-cf6a80473db0: port 15(veth8e823d7) entered blocking state
br-cf6a80473db0: port 15(veth8e823d7) entered forwarding state
br-cf6a80473db0: port 15(veth8e823d7) entered disabled state
veth8cf97b9: renamed from eth0
br-cf6a80473db0: port 15(veth8e823d7) entered disabled state
device veth8e823d7 left promiscuous mode
br-cf6a80473db0: port 15(veth8e823d7) entered disabled state
br-cf6a80473db0: port 15(veth87c69a7) entered blocking state
br-cf6a80473db0: port 15(veth87c69a7) entered disabled state
device veth87c69a7 entered promiscuous mode
br-cf6a80473db0: port 15(veth87c69a7) entered blocking state
br-cf6a80473db0: port 15(veth87c69a7) entered forwarding state
br-cf6a80473db0: port 15(veth87c69a7) entered disabled state
eth0: renamed from veth9c66b03
IPv6: ADDRCONF(NETDEV_CHANGE): veth87c69a7: link becomes ready
br-cf6a80473db0: port 15(veth87c69a7) entered blocking state
br-cf6a80473db0: port 15(veth87c69a7) entered forwarding state
veth9c66b03: renamed from eth0
br-cf6a80473db0: port 15(veth87c69a7) entered disabled state
br-cf6a80473db0: port 15(veth87c69a7) entered disabled state
device veth87c69a7 left promiscuous mode
br-cf6a80473db0: port 15(veth87c69a7) entered disabled state
br-cf6a80473db0: port 15(vethb37a64a) entered blocking state
br-cf6a80473db0: port 15(vethb37a64a) entered disabled state
device vethb37a64a entered promiscuous mode
br-cf6a80473db0: port 15(vethb37a64a) entered blocking state
br-cf6a80473db0: port 15(vethb37a64a) entered forwarding state
br-cf6a80473db0: port 15(vethb37a64a) entered disabled state
eth0: renamed from veth4552c77
IPv6: ADDRCONF(NETDEV_CHANGE): vethb37a64a: link becomes ready
br-cf6a80473db0: port 15(vethb37a64a) entered blocking state
br-cf6a80473db0: port 15(vethb37a64a) entered forwarding state
br-cf6a80473db0: port 15(vethb37a64a) entered disabled state
veth4552c77: renamed from eth0
br-cf6a80473db0: port 15(vethb37a64a) entered disabled state
device vethb37a64a left promiscuous mode
br-cf6a80473db0: port 15(vethb37a64a) entered disabled state
br-cf6a80473db0: port 15(veth64794f1) entered blocking state
br-cf6a80473db0: port 15(veth64794f1) entered disabled state
device veth64794f1 entered promiscuous mode
br-cf6a80473db0: port 15(veth64794f1) entered blocking state
br-cf6a80473db0: port 15(veth64794f1) entered forwarding state
br-cf6a80473db0: port 15(veth64794f1) entered disabled state
eth0: renamed from veth66ad37d
IPv6: ADDRCONF(NETDEV_CHANGE): veth64794f1: link becomes ready
br-cf6a80473db0: port 15(veth64794f1) entered blocking state
br-cf6a80473db0: port 15(veth64794f1) entered forwarding state
br-cf6a80473db0: port 15(veth64794f1) entered disabled state
veth66ad37d: renamed from eth0
br-cf6a80473db0: port 15(veth64794f1) entered disabled state
device veth64794f1 left promiscuous mode
br-cf6a80473db0: port 15(veth64794f1) entered disabled state

@mikenye
Copy link
Member

mikenye commented May 22, 2020

Hmm. I can’t see any helpful information in there. The next time you see a segfault could you capture the last 100 lines of /var/log/syslog, /var/log/kern.log and dmesg?

@alexbogus
Copy link
Author

alexbogus commented May 22, 2020

I'm currently reviewing the logs, but I cannot not see any useful information
But if I move to /var/lib/docker/overlay2/<overlayID>/merged/run/s6/services/rbfeeder I can see some qemu core dumps. Maybe those files could be useful

This is the output:

86M     qemu_rbfeeder_20200522-104123_2946.core
86M     qemu_rbfeeder_20200522-104022_2930.core
86M     qemu_rbfeeder_20200522-103922_2914.core
86M     qemu_rbfeeder_20200522-103821_2898.core
86M     qemu_rbfeeder_20200522-103720_2882.core
86M     qemu_rbfeeder_20200522-103620_2866.core
86M     qemu_rbfeeder_20200522-103519_2850.core
86M     qemu_rbfeeder_20200522-103418_2834.core
86M     qemu_rbfeeder_20200522-103317_2818.core
86M     qemu_rbfeeder_20200522-103217_2802.core

Maybe those file can be useful to debug?

@mikenye
Copy link
Member

mikenye commented May 22, 2020

Yes please, I’d like to have a look at them. Any way I can get hold of them? Do you need me to set up a drop box? You probably don’t want to post them publicly as they may contain sensitive info.

@alexbogus
Copy link
Author

I'll send you an email with the details to download the coredumps

Regards

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Core analysis:

root@a0efe1f26c08:/# file /tmp/rbfeeder-coredump/qemu_rbfeeder_20200522-104930_3074.core
/tmp/rbfeeder-coredump/qemu_rbfeeder_20200522-104930_3074.core: ELF 32-bit LSB core file ARM, version 1 (SYSV), SVR4-style, from 'rbfeeder', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0
root@a0efe1f26c08:/# gdb /usr/bin/rbfeeder /tmp/rbfeeder-coredump/qemu_rbfeeder_20200522-104930_3074.core
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/rbfeeder...(no debugging symbols found)...done.
[New LWP 3082]
[New LWP 3074]
[New LWP 3077]
[New LWP 3078]
[New LWP 3079]
[New LWP 3080]
[New LWP 3081]
[New LWP 3083]
[New LWP 3084]
[New LWP 3085]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `���'.
#0  __isoc99_fscanf (stream=0x0, format=0x390d4 "%f") at isoc99_fscanf.c:30
30      isoc99_fscanf.c: No such file or directory.
[Current thread is 1 (Thread 0xfbbfc440 (LWP 3082))]
(gdb) bt
#0  __isoc99_fscanf (stream=0x0, format=0x390d4 "%f") at isoc99_fscanf.c:30
#1  0x0002a6b8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) l
25      in isoc99_fscanf.c

It looks like rbfeeder is segfaulting when trying to open a file?

I can see from your docker inspect that you have a bind mount for /etc/rbfeeder.ini:

"Mounts": [
            {
                "Type": "bind",
                "Source": "/home/docker/adsb/radarbox/rbfeeder.ini",
                "Destination": "/etc/rbfeeder.ini",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

Could you please post the contents of this file (censor lat, lon, alt, sn, key, external_host), and also the output of the command stat /home/docker/adsb/radarbox/rbfeeder.ini.

Thanks.

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Could you also try running the container without the bind mount for /etc/rbfeeder.ini?

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Could I also please have the output of cat /sys/class/thermal/thermal_zone0/temp? Thanks.

@alexbogus
Copy link
Author

Could you also try running the container without the bind mount for /etc/rbfeeder.ini?

Volume lines commented in docker-compose

  ##volumes:
      ##- ./radarbox/rbfeeder.ini:/etc/rbfeeder.ini

And this is the output, another Segmentation fault (core dumped)

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
,[s6-init] ensuring user provided files have correct perms...exited 0.
,[fix-attrs.d] applying ownership & permissions fixes...
,[fix-attrs.d] done.
,[cont-init.d] executing container initialization scripts...
,[cont-init.d] 01-rbfeeder: executing... 
,[cont-init.d] 01-rbfeeder: exited 0.
,[cont-init.d] done.
,[services.d] starting services
,[services.d] done.
,[mlat-client] Delaying mlat-client startup until rbfeeder receives station sn...
,[rbfeeder] [2020-05-23 14:02:44]  Starting RBFeeder Version 0.2.6 (build 20180313082038)
,[rbfeeder] [2020-05-23 14:02:44]  Using configuration file: /etc/rbfeeder.ini
,[rbfeeder] [2020-05-23 14:02:44]  Network-mode enabled.
,[rbfeeder] [2020-05-23 14:02:44]  		Remote host to fetch data: xx.xx.xx.xx
,[rbfeeder] [2020-05-23 14:02:44]  		Remote port: 30005
,[rbfeeder] [2020-05-23 14:02:44]  		Remote protocol: BEAST
,[rbfeeder] [2020-05-23 14:02:44]  System: raspberry
,[rbfeeder] [2020-05-23 14:02:44]  Start date/time: 2020-05-23 14:02:44
,[rbfeeder] [2020-05-23 14:02:44]  Socket for ANRB created. Waiting for connections on port 32088
,[rbfeeder] [2020-05-23 14:02:46]  Connection established.
,[rbfeeder] [2020-05-23 14:02:46]  Connection with RadarBox24 server OK! Key accepted by server.
,[rbfeeder] [2020-05-23 14:03:15]  ******** Statistics updated every 30 seconds ********
,[rbfeeder] [2020-05-23 14:03:15]  Packets sent in the last 30 seconds: 0, Total packets sent since startup: 0
,[mlat-client] Sat May 23 14:03:15 2020 mlat-client 0.2.11 starting up
,[mlat-client] Sat May 23 14:03:16 2020 Listening for Beast-format results connection on port 30105
,[mlat-client] Sat May 23 14:03:16 2020 Connected to multilateration server at mlat1.rb24.com:40900, handshaking
,[mlat-client] Sat May 23 14:03:23 2020 Server says: 
,[mlat-client] 
,[mlat-client]         AirNAv Server
,[mlat-client] 
,[mlat-client]         The multilateration server source code is available under
,[mlat-client]         the terms of the Affero GPL (v3 or later). You may obtain
,[mlat-client]         a copy of this server's source code at the following
,[mlat-client]         location: https://github.com/mutability/mlat-server
,[mlat-client]         
,[mlat-client] Sat May 23 14:03:23 2020 Handshake complete.
,[mlat-client] Sat May 23 14:03:23 2020   Compression:       zlib2
,[mlat-client] Sat May 23 14:03:23 2020   UDP transport:     disabled
,[mlat-client] Sat May 23 14:03:23 2020   Split sync:        disabled
,[mlat-client] Sat May 23 14:03:23 2020 Input connected to xx.xx.xx.xx:30005
,[mlat-client] Sat May 23 14:03:23 2020 Input format changed to BEAST, 12MHz clock
,Segmentation fault (core dumped)
,[rbfeeder] [2020-05-23 14:03:45]  Starting RBFeeder Version 0.2.6 (build 20180313082038)
,[rbfeeder] [2020-05-23 14:03:45]  Using configuration file: /etc/rbfeeder.ini
,[rbfeeder] [2020-05-23 14:03:45]  Network-mode enabled.
,[rbfeeder] [2020-05-23 14:03:45]  		Remote host to fetch data: xx.xx.xx.xx
,[rbfeeder] [2020-05-23 14:03:45]  		Remote port: 30005
,[rbfeeder] [2020-05-23 14:03:45]  		Remote protocol: BEAST
,[rbfeeder] [2020-05-23 14:03:46]  System: raspberry
,[rbfeeder] [2020-05-23 14:03:46]  Start date/time: 2020-05-23 14:03:45
,[rbfeeder] [2020-05-23 14:03:46]  Socket for ANRB created. Waiting for connections on port 32088
,[rbfeeder] [2020-05-23 14:03:47]  Connection established.

@alexbogus
Copy link
Author

Could I also please have the output of cat /sys/class/thermal/thermal_zone0/temp? Thanks.

My system It's working dockerized in a VM inside a ESXi Server (v6.5)

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Could I also please have the output of cat /sys/class/thermal/thermal_zone0/temp? Thanks.

My system It's working dockerized in a VM inside a ESXi Server (v6.5)

Could you SSH into the VM and run the command cat /sys/class/thermal/thermal_zone0/temp, and post the output?

@alexbogus
Copy link
Author

Here is the output of stat /home/docker/adsb/radarbox/rbfeeder.ini

Fichero: /home/docker/adsb/radarbox/rbfeeder.ini
Tamaño: 283           Bloques: 8          Bloque E/S: 4096   fichero regular
Dispositivo: 811h/2065d Nodo-i: 26214432    Enlaces: 1
Acceso: (0664/-rw-rw-r--)  Uid: ( 1000/  myusername)   Gid: ( 1000/  myusername)
Acceso: 2020-05-23 16:01:52.470883362 +0200
Modificación: 2020-05-23 16:01:33.882753643 +0200
Cambio: 2020-05-23 16:01:33.882753643 +0200
Creación: -

SSh into the VM, and I can't see /sys/class/thermal/thermal_zone0/temp
I only can see In the /sys/class/thermal directory:

drwxr-xr-x  2 root root 0 may 23 16:01 ./
drwxr-xr-x 66 root root 0 may 23 12:23 ../
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device0 -> ../../devices/virtual/thermal/cooling_device0/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device1 -> ../../devices/virtual/thermal/cooling_device1/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device2 -> ../../devices/virtual/thermal/cooling_device2/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device3 -> ../../devices/virtual/thermal/cooling_device3/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device4 -> ../../devices/virtual/thermal/cooling_device4/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device5 -> ../../devices/virtual/thermal/cooling_device5/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device6 -> ../../devices/virtual/thermal/cooling_device6/
lrwxrwxrwx  1 root root 0 may 23 16:01 cooling_device7 -> ../../devices/virtual/thermal/cooling_device7/

But doing a recursive search, I cannot find any temp files

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Hi @alexbogus, I think the problem is with the missing /sys/class/thermal/thermal_zone0/temp.

Looking at the rbfeeder source code, searching the files for fscanf (shown as the source of the segfault by gdb above), we can see the following:

...
    thermal = fopen("/sys/class/thermal/thermal_zone0/temp", "r");
    n = fscanf(thermal, "%f", &millideg);
...

Reference: https://github.com/mutability/rbfeeder/blob/8f002217ffe5d5dffced9845bed117ffe2a95a0e/airnav.c#L2529

I think what's happenning is because this code is designed to be run on an RPi (RadarBox don't make binaries for other platforms), I'm using qemu-arm-static to run their binaries on an x86_64 platform. rbfeeder is looking for the file /sys/class/thermal/thermal_zone0/temp which doesn't exist on your VM (likely because it's virtualised), and causing a segmentation fault.

On my Linux host that I run this on (physical), I have a kernel module loaded called x86_pkg_temp_thermal:

$ lsmod | grep therm
x86_pkg_temp_thermal    16384  0

Could you run the lsmod | grep therm command in your VM and see if this module is loaded?

If not, you could try loading it with modprobe x86_pkg_temp_thermal and see if that creates the /sys/class/thermal/thermal_zone0/temp file.

@mikenye
Copy link
Member

mikenye commented May 23, 2020

Alternatively, you could try running the commands:

mkdir -p /sys/class/thermal/thermal_zone0
echo 24000 > /sys/class/thermal/thermal_zone0/temp

...which will "spoof" a temperature of 24 degrees C and hopefully fool rbfeeder. :)

@alexbogus
Copy link
Author

alexbogus commented May 23, 2020

mkdir -p /sys/class/thermal/thermal_zone0 as root
#mkdir: cannot create the directory «/sys/class/thermal/thermal_zone0»: Operation not permitted

Ups, I'm trying to pass from host to virtual machine information about temperature.

@mikenye
Copy link
Member

mikenye commented May 23, 2020

OK try this.

echo 24000 > /home/docker/adsb/radarbox/faketemp

Then add a bind mount to your docker-compose.yml:

under - /home/docker/adsb/radarbox/rbfeeder.ini:/etc/rbfeeder.ini, add:

  - /home/docker/adsb/radarbox/faketemp:/sys/class/thermal/thermal_zone0/temp:ro

Then recreate the container with docker-compose up -d.

@alexbogus
Copy link
Author

Trying the workaround:

ERROR: for rbfeeder Cannot start service rbfeeder: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/docker/adsb/faketemp\\\" to rootfs \\\"/var/lib/docker/overlay2/5d6ed3cf951ef291336be45fc4dbc406f162d07309aa5f75f998c7a390b4a41c/merged\\\" at \\\"/var/lib/docker/overlay2/5d6ed3cf951ef291336be45fc4dbc406f162d07309aa5f75f998c7a390b4a41c/merged/sys/class/thermal/thermal_zone0/temp\\\" caused \\\"mkdir /var/lib/docker/overlay2/5d6ed3cf951ef291336be45fc4dbc406f162d07309aa5f75f998c7a390b4a41c

@mikenye
Copy link
Member

mikenye commented May 26, 2020

Can you share your updated docker-compose.yml file? The workaround worked in my environment, although I already had the /sys/class/thermal/thermal_zone0/temp file present on my system...

Outside of the suggested workarounds, I'm not sure how to solve this problem without either:

  • Modifying the rbfeeder source code (which I'm not able to do); or
  • Somehow get CPU/system temperature into your VM

Sorry Alex. :(

@alexbogus
Copy link
Author

Thanks Mike, I really appreciate your effort to help me with the issue. No problem if we cannot solve this issue.

This is my docker-compose.yml

  rbfeeder:
    image: mikenye/radarbox:latest
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: 10m
    tty: true
    container_name: radarbox
    restart: always
    environment:
      - TZ=${TZ}
      - LAT=${LATITUDE}
      - LONG=${LONGITUDE}
      - BEASTHOST=${READSBHOST}
      - ALT=${ALTITUDE}
      - SHARING_KEY=${RADARBOXKEY}
    volumes:
      - ./radarbox/rbfeeder.ini:/etc/rbfeeder.ini
      - ./radarbox/faketemp:/sys/class/thermal/thermal_zone0/temp:ro
    networks:
      - adsb

@alexbogus
Copy link
Author

Consider this issue closed as no further resolution is possible until Radarbox releases new code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants