Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[boot performance] long swss/syncd/teamd start due to interfaces-config.service #12948

Closed
stepanblyschak opened this issue Dec 5, 2022 · 1 comment
Assignees
Labels
NVIDIA Triaged this issue has been triaged

Comments

@stepanblyschak
Copy link
Collaborator

Description

interfaces-config.service performs /etc/network/interfaces generation where configuration for the management interface resides and restarts networking service.
Swss, teamd, syncd, bgp services are configured to run After= interfaces-config.service.
This makes data plane services block for management interfaces to be configured.

interfaces-config.service is one of the longest starting services:

2min 42.490s warmboot-finalizer.service
     30.436s hw-management.service
     13.623s interfaces-config.service
      9.318s syncd.service
      9.145s rasdaemon.service
      7.994s database.service
      6.574s docker.service
      5.436s networking.service
      4.762s snmp.service
      4.364s dev-sda3.device

We could have saved 13 sec if we had no dependency on interfaces-config.service.

Steps to reproduce the issue:

  1. Perform warm-reboot
  2. Do: "sudo zless /var/log/teamd.log | grep 'SIGUSR1|carrier changed to UP'
  3. Measure time between SIGUSR1 and port channel carrier UP log message

Describe the results you received:

We experience >80 sec in port channel restoration time which is under a risk of port channel beeing flapped.

Describe the results you expected:

If we had interfaces-config.service indepdenently started from data plane services we could have saved some time and be under 80 sec.

Output of show version:

SONiC Software Version: SONiC.202205_1_rc.4-7aa8502d0_Internal
Distribution: Debian 11.5
Kernel: 5.10.0-12-2-amd64
Build commit: 7aa8502d0
Build date: Thu Nov 24 12:21:41 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci03-244

Platform: x86_64-mlnx_msn2700-r0
HwSKU: ACS-MSN2700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1550X12437
Model Number: MSN2700-CS2F
Hardware Revision: Not Specified
Uptime: 00:42:25 up 4 min,  1 user,  load average: 1.58, 1.65, 0.82
Date: Sun 27 Nov 2022 00:42:25

Docker images:
REPOSITORY                                         TAG                                IMAGE ID       SIZE
docker-orchagent                                   202205_1_rc.4-7aa8502d0_Internal   4934c5e9a09c   478MB
docker-orchagent                                   latest                             4934c5e9a09c   478MB
docker-fpm-frr                                     202205_1_rc.4-7aa8502d0_Internal   df24b8dda4f0   489MB
docker-fpm-frr                                     latest                             df24b8dda4f0   489MB
docker-teamd                                       202205_1_rc.4-7aa8502d0_Internal   2b65c7a94402   459MB
docker-teamd                                       latest                             2b65c7a94402   459MB
docker-platform-monitor                            202205_1_rc.4-7aa8502d0_Internal   66bc72ac3bea   867MB
docker-platform-monitor                            latest                             66bc72ac3bea   867MB
docker-macsec                                      latest                             7eb2050765e6   461MB
docker-syncd-mlnx                                  202205_1_rc.4-7aa8502d0_Internal   c7ce148bb5aa   862MB
docker-syncd-mlnx                                  latest                             c7ce148bb5aa   862MB
docker-snmp                                        202205_1_rc.4-7aa8502d0_Internal   40c5b00be439   488MB
docker-snmp                                        latest                             40c5b00be439   488MB
docker-dhcp-relay                                  latest                             41cb40231b85   453MB
docker-sonic-telemetry                             202205_1_rc.4-7aa8502d0_Internal   1af21b53dbbf   524MB
docker-sonic-telemetry                             latest                             1af21b53dbbf   524MB
docker-lldp                                        202205_1_rc.4-7aa8502d0_Internal   ea3af087e7ce   486MB
docker-lldp                                        latest                             ea3af087e7ce   486MB
docker-database                                    202205_1_rc.4-7aa8502d0_Internal   b66c8dcf5852   443MB
docker-database                                    latest                             b66c8dcf5852   443MB
docker-mux                                         202205_1_rc.4-7aa8502d0_Internal   cb30d70c7d4a   492MB
docker-mux                                         latest                             cb30d70c7d4a   492MB
docker-router-advertiser                           202205_1_rc.4-7aa8502d0_Internal   36d2d71c98b3   443MB
docker-router-advertiser                           latest                             36d2d71c98b3   443MB
docker-sonic-mgmt-framework                        202205_1_rc.4-7aa8502d0_Internal   1df66ddde189   557MB
docker-sonic-mgmt-framework                        latest                             1df66ddde189   557MB
docker-nat                                         202205_1_rc.4-7aa8502d0_Internal   5b47aa79540c   430MB
docker-nat                                         latest                             5b47aa79540c   430MB
docker-sflow                                       202205_1_rc.4-7aa8502d0_Internal   5441306adc85   428MB
docker-sflow                                       latest                             5441306adc85   428MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/doroce      1.0.0-master-internal-8            5c9f7110ace8   611MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.3.1-202205-internal-7            369c313f8355   648MB

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NVIDIA Triaged this issue has been triaged
Projects
None yet
Development

No branches or pull requests

2 participants