[Config Support]: How to reconnect to camera after connection fails? #12769

SY-EOS · 2024-08-05T16:35:56Z

SY-EOS
Aug 5, 2024

Describe the problem you are having

My cameras are constantly freezing after a few hours, all at random times. Logs show a 20 second timeout and then ffmpeg exiting. Is there a way to reconnect to a rtsp stream if it loses connection (reconnect until it established a connection again)? Currently using Lilin cameras.

If I refresh the browser, the cameras all come back up immediately.

Any feedback will be greatly appreciated!

Version

0.14.0-da913d8

Frigate config file

mqtt:
  enabled: false

  # Include all cameras by default in Birdseye view
birdseye:
  enabled: true
  mode: continuous
  width: 2560
  height: 1440
  restream: true

detectors:
  coral:
    type: edgetpu
    device: usb

live:
  # Optional: Set the height of the live stream. (default: 720)
  # This must be less than or equal to the height of the detect stream. Lower resolutions
  # reduce bandwidth required for viewing the live stream. Width is computed to match known aspect ratio.
  height: 720
  # Optional: Set the encode quality of the live stream (default: shown below)
  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 1

go2rtc:
  streams:
    bulwark_port:
      - rtsp://admin:pass@192.168.170.25:554/stream1
    bulwark_stb:
      - rtsp://admin:pass@192.168.170.13:554/stream1
    cockpit_port:
      - rtsp://admin:pass@192.168.170.20:554/stream1
    cockpit_stb:
      - rtsp://admin:zcpm3ETO@192.168.170.140:554/stream1
    main_engine_port:
      - rtsp://admin:pass@192.168.170.19:554/stream1
    main_engine_stb:
      - rtsp://admin:pass@192.168.170.12:554/stream1
    fwd_mast_port:
      - rtsp://admin:pass@192.168.170.29:554/stream1
    fwd_mast_stb:
      - rtsp://admin:pass@192.168.170.28:554/stream1
    er_upper_port_fwd:
      - rtsp://admin:pass@192.168.170.17:554/stream1
    er_upper_stbd_inboard:
      - rtsp://admin:pass@192.168.170.150:554/stream1
    er_lower_stb_aft:
      - rtsp://admin:pass@192.168.170.24:554/stream1
    mizzen_mast_port:
      - rtsp://admin:pass@192.168.170.26:554/stream0

cameras:
  bulwark_stb:
    enabled: true
    birdseye:
      order: 1
    ffmpeg:
     # hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/bulwark_stb
          roles:
            - detect
    detect:
      enabled: false
      fps: 30
  cockpit_stb:
    enabled: true
    birdseye:
      order: 2
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/cockpit_stb
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  er_upper_stbd_inboard:
    enabled: true
    birdseye:
      order: 3
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/er_upper_stbd_inboard
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  fwd_mast_stb:
    enabled: true

    birdseye:
      order: 4
    ffmpeg:
 #     hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/fwd_mast_stb
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  er_upper_port_fwd:
    enabled: true
    birdseye:
      order: 5
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/er_upper_port_fwd
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  main_engine_stb:
    enabled: true
    birdseye:
      order: 6
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/main_engine_stb
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  main_engine_port:
    enabled: true
    birdseye:
      order: 7
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/main_engine_port
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  cockpit_port:
    enabled: true
    birdseye:
      order: 8
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/cockpit_port
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  er_lower_stb_aft:
    enabled: true
    birdseye:
      order: 9
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/er_lower_stb_aft
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  bulwark_port:
    enabled: true
    birdseye:
      order: 10
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/bulwark_port
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  fwd_mast_port:
    enabled: true
    birdseye:
      order: 11
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/fwd_mast_port
          roles:
            - detect
    detect:
      enabled: false
      fps: 15
  mizzen_mast_port:
    enabled: true
    birdseye:
      order: 12
    ffmpeg:
#      hwaccel_args: preset-vaapi
      retry_interval: 10
      inputs:
        - path: rtsp://localhost:8554/mizzen_mast_port
          roles:
            - detect
    detect:
      enabled: false
      fps: 30
version: 0.14

Relevant log output

Frigate Logs:

error
2024-08-05 10:09:29
ffmpeg.bulwark_port.detect
[rtsp @ 0x561a81fb3e00] method DESCRIBE failed: 404 Not Found
error
2024-08-05 10:09:29
ffmpeg.bulwark_port.detect
rtsp://localhost:8554/bulwark_port: Server returned 404 Not Found
error
2024-08-05 10:09:32
frigate.video
fwd_mast_stb: Unable to read frames from ffmpeg process.
error
2024-08-05 10:09:32
frigate.video
fwd_mast_stb: ffmpeg process is not running. exiting capture thread...
error
2024-08-05 10:09:32
frigate.video
bulwark_port: Unable to read frames from ffmpeg process.
error
2024-08-05 10:09:32
frigate.video
bulwark_port: ffmpeg process is not running. exiting capture thread...
error
2024-08-05 10:09:39
watchdog.fwd_mast_stb
Ffmpeg process crashed unexpectedly for fwd_mast_stb.
error
2024-08-05 10:09:39
watchdog.fwd_mast_stb
The following ffmpeg logs include the last 100 lines prior to exit.
error
2024-08-05 10:09:39
ffmpeg.fwd_mast_stb.detect
[rtsp @ 0x5626a677ae00] method DESCRIBE failed: 404 Not Found
error
2024-08-05 10:09:39
ffmpeg.fwd_mast_stb.detect
rtsp://localhost:8554/fwd_mast_stb: Server returned 404 Not Found
error
2024-08-05 10:09:39
watchdog.bulwark_port
Ffmpeg process crashed unexpectedly for bulwark_port.
error
2024-08-05 10:09:39
watchdog.bulwark_port
The following ffmpeg logs include the last 100 lines prior to exit.
error
2024-08-05 10:09:39
ffmpeg.bulwark_port.detect
[rtsp @ 0x560a4fb31e00] method DESCRIBE failed: 404 Not Found
error
2024-08-05 10:09:39
ffmpeg.bulwark_port.detect
rtsp://localhost:8554/bulwark_port: Server returned 404 Not Found
error
2024-08-05 10:09:42
frigate.video
fwd_mast_stb: Unable to read frames from ffmpeg process.
error
2024-08-05 10:09:42
frigate.video
fwd_mast_stb: ffmpeg process is not running. exiting capture thread...
error
2024-08-05 10:09:42
frigate.video
fwd_mast_port: Unable to read frames from ffmpeg process.
error
2024-08-05 10:09:42
frigate.video
fwd_mast_port: ffmpeg process is not running. exiting capture thread...
error
2024-08-05 10:09:49
watchdog.fwd_mast_stb
Ffmpeg process crashed unexpectedly for fwd_mast_stb.
error
2024-08-05 10:09:49
watchdog.fwd_mast_stb
The following ffmpeg logs include the last 100 lines prior to exit.
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_stb.detect
[rtsp @ 0x5616e336be00] method DESCRIBE failed: 404 Not Found
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_stb.detect
rtsp://localhost:8554/fwd_mast_stb: Server returned 404 Not Found
error
2024-08-05 10:09:49
watchdog.fwd_mast_port
Ffmpeg process crashed unexpectedly for fwd_mast_port.
error
2024-08-05 10:09:49
watchdog.fwd_mast_port
The following ffmpeg logs include the last 100 lines prior to exit.
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af97a780] [swscaler @ 0x5651af987740] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af97a780] [swscaler @ 0x5651afc81f80] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651afc7e680] [swscaler @ 0x5651afc8b640] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651afc7e680] [swscaler @ 0x5651af990b40] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af9a12c0] [swscaler @ 0x5651afc7e680] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af9a12c0] [swscaler @ 0x5651af97ec00] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af9a12c0] [swscaler @ 0x5651afc7e680] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
[swscaler @ 0x5651af9a12c0] [swscaler @ 0x5651af97a780] deprecated pixel format used, make sure you did set range correctly
error
2024-08-05 10:09:49
ffmpeg.fwd_mast_port.detect
rtsp://localhost:8554/fwd_mast_port: Connection timed out`

Go2rtc logs:

warning
2024-08-05 10:06:32
rtsp
error="streams: dial tcp 192.168.170.12:554: i/o timeout" stream=main_engine_stb
warning
2024-08-05 10:06:32
rtsp
error="streams: dial tcp 192.168.170.26:554: i/o timeout" stream=mizzen_mast_port
warning
2024-08-05 10:06:36
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:45718->192.168.170.28:554: i/o timeout" url=rtsp://admin:pass@192.168.170.28:554/stream1
warning
2024-08-05 10:06:39
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:36356->192.168.170.25:554: i/o timeout" url=rtsp://admin:pass@192.168.170.25:554/stream1
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.13:554: i/o timeout" stream=bulwark_stb
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.150:554: i/o timeout" stream=er_upper_stbd_inboard
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.17:554: i/o timeout" stream=er_upper_port_fwd
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.20:554: i/o timeout" stream=cockpit_port
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.24:554: i/o timeout" stream=er_lower_stb_aft
warning
2024-08-05 10:06:42
rtsp
error="streams: dial tcp 192.168.170.26:554: i/o timeout" stream=mizzen_mast_port
warning
2024-08-05 10:06:52
rtsp
error="streams: dial tcp 192.168.170.13:554: i/o timeout" stream=bulwark_stb
warning
2024-08-05 10:06:52
rtsp
error="streams: dial tcp 192.168.170.150:554: i/o timeout" stream=er_upper_stbd_inboard
warning
2024-08-05 10:06:52
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:06:52
rtsp
error="streams: dial tcp 192.168.170.17:554: i/o timeout" stream=er_upper_port_fwd
warning
2024-08-05 10:06:52
rtsp
error="streams: dial tcp 192.168.170.29:554: i/o timeout" stream=fwd_mast_port
warning
2024-08-05 10:06:53
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:34140->192.168.170.140:554: i/o timeout" url=rtsp://admin:zcpm3ETO@192.168.170.140:554/stream1
warning
2024-08-05 10:07:02
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:07:02
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:07:02
rtsp
error="streams: dial tcp 192.168.170.29:554: i/o timeout" stream=fwd_mast_port
warning
2024-08-05 10:07:12
rtsp
error="streams: dial tcp 192.168.170.140:554: i/o timeout" stream=cockpit_stb
warning
2024-08-05 10:07:22
rtsp
error="streams: dial tcp 192.168.170.140:554: i/o timeout" stream=cockpit_stb
warning
2024-08-05 10:07:32
rtsp
error="streams: dial tcp 192.168.170.140:554: i/o timeout" stream=cockpit_stb
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:33682->192.168.170.26:554: i/o timeout" url=rtsp://admin:pass@192.168.170.26:554/stream0
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:49166->192.168.170.25:554: i/o timeout" url=rtsp://admin:pass@192.168.170.25:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:43018->192.168.170.24:554: i/o timeout" url=rtsp://admin:pass@192.168.170.24:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:37628->192.168.170.19:554: i/o timeout" url=rtsp://admin:pass@192.168.170.19:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:43222->192.168.170.29:554: i/o timeout" url=rtsp://admin:pass@192.168.170.29:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:41288->192.168.170.150:554: i/o timeout" url=rtsp://admin:pass@192.168.170.150:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:48960->192.168.170.13:554: i/o timeout" url=rtsp://admin:pass@192.168.170.13:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:44096->192.168.170.12:554: i/o timeout" url=rtsp://admin:pass@192.168.170.12:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:43156->192.168.170.140:554: i/o timeout" url=rtsp://admin:zcpm3ETO@192.168.170.140:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:37532->192.168.170.28:554: i/o timeout" url=rtsp://admin:pass@192.168.170.28:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:37150->192.168.170.20:554: i/o timeout" url=rtsp://admin:pass@192.168.170.20:554/stream1
warning
2024-08-05 10:08:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:46328->192.168.170.17:554: i/o timeout" url=rtsp://admin:pass@192.168.170.17:554/stream1
warning
2024-08-05 10:08:52
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:08:52
rtsp
error="streams: dial tcp 192.168.170.19:554: i/o timeout" stream=main_engine_port
warning
2024-08-05 10:08:52
rtsp
error="streams: dial tcp 192.168.170.12:554: i/o timeout" stream=main_engine_stb
warning
2024-08-05 10:08:52
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:08:52
rtsp
error="streams: dial tcp 192.168.170.29:554: i/o timeout" stream=fwd_mast_port
warning
2024-08-05 10:09:02
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:09:02
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:09:02
rtsp
error="streams: dial tcp 192.168.170.29:554: i/o timeout" stream=fwd_mast_port
warning
2024-08-05 10:09:12
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:09:12
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:09:22
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:09:22
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:09:32
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb
warning
2024-08-05 10:09:32
rtsp
error="streams: dial tcp 192.168.170.25:554: i/o timeout" stream=bulwark_port
warning
2024-08-05 10:09:32
startup
github.com/AlexxIT/go2rtc/internal/streams/producer.go:171 > error="read tcp 172.18.0.2:57418->192.168.170.29:554: i/o timeout" url=rtsp://admin:pass@192.168.170.29:554/stream1
warning
2024-08-05 10:09:42
rtsp
error="streams: dial tcp 192.168.170.28:554: i/o timeout" stream=fwd_mast_stb

Frigate stats

Coral inference: 10ms
CPU usage: < 20%
RAM usage: < 20%

Operating system

Debian

Install method

Docker Compose

Object Detector

Coral

Any other information that may be helpful

No response

NickM-27 · 2024-08-05T16:37:24Z

NickM-27
Aug 5, 2024
Collaborator Sponsor

frigate automatically reconnects to cameras when ffmpeg exits. This can be seen how there are multiple errors for the same camera

5 replies

SY-EOS Aug 5, 2024
Author

Can you perhaps see any clues as to why it might be failing/freezing if it is reconnecting on connection lost?

NickM-27 Aug 5, 2024
Collaborator Sponsor

one problem I see is that your detect fps is very high, this does not need to match the cameras fps and it should really just be using the default of 5 in most cases

SY-EOS Aug 5, 2024
Author

Would that be an issue even though I have detect currently set to false? I haven't had it enabled since installing Frigate

NickM-27 Aug 5, 2024
Collaborator Sponsor

yes, the stream is always decoded no matter what

SY-EOS Aug 5, 2024
Author

Got it. Thank you, that's very useful to know. I will adjust it and see if it makes a difference

hawkeye217 · 2024-08-05T16:58:07Z

hawkeye217
Aug 5, 2024
Collaborator

Seems like it's network related since they all go down at the same time. Are they wifi cameras?

1 reply

SY-EOS Aug 5, 2024
Author

They are connected via LAN. Topology is:

Lilin cameras -> Cisco Switch -> Cisco Switch -> Frigate

Switches are connected via 10G uplink. Frigate and switches all belong to the same VLAN.

The strange thing is that they will freeze at random times. Sometimes one will freeze, then another, while the others carry on going. But it's not the same cameras or consistent.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Config Support]: How to reconnect to camera after connection fails? #12769

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Config Support]: How to reconnect to camera after connection fails? #12769

SY-EOS Aug 5, 2024

Describe the problem you are having

Version

Frigate config file

Relevant log output

Frigate stats

Operating system

Install method

Object Detector

Any other information that may be helpful

Replies: 2 comments · 6 replies

NickM-27 Aug 5, 2024 Collaborator Sponsor

SY-EOS Aug 5, 2024 Author

NickM-27 Aug 5, 2024 Collaborator Sponsor

SY-EOS Aug 5, 2024 Author

NickM-27 Aug 5, 2024 Collaborator Sponsor

SY-EOS Aug 5, 2024 Author

hawkeye217 Aug 5, 2024 Collaborator

SY-EOS Aug 5, 2024 Author

SY-EOS
Aug 5, 2024

Replies: 2 comments 6 replies

NickM-27
Aug 5, 2024
Collaborator Sponsor

SY-EOS Aug 5, 2024
Author

NickM-27 Aug 5, 2024
Collaborator Sponsor

SY-EOS Aug 5, 2024
Author

NickM-27 Aug 5, 2024
Collaborator Sponsor

SY-EOS Aug 5, 2024
Author

hawkeye217
Aug 5, 2024
Collaborator

SY-EOS Aug 5, 2024
Author