Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bulk route API printing not implemented errors when routes are added and removed #5758

Closed
daall opened this issue Oct 30, 2020 · 7 comments
Closed

bulk route API printing not implemented errors when routes are added and removed #5758

daall opened this issue Oct 30, 2020 · 7 comments
Assignees
@kcudnik
Labels
Master Branch Quality P0 Priority of the issue

Comments

@daall
Copy link
Contributor

daall commented Oct 30, 2020

Description
When we load a recent master image onto a device, it constantly prints errors from syncd reporting that the bulk route API is not implemented.

Steps to reproduce the issue:

  1. reboot a device that is running the latest master image

Describe the results you received:

Oct 30 16:21:49.014739 str-dx010-acs-4 ERR syncd#syncd: message repeated 2 times: [ :- bulkCreate: create_route_entries is not supported]
Oct 30 16:21:49.015033 str-dx010-acs-4 ERR syncd#syncd: :- bulkSet: set_route_entries_attribute is not supported
Oct 30 16:21:49.051703 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported
Oct 30 16:21:49.101384 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002
Oct 30 16:21:49.117268 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.117443 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.129999 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported
Oct 30 16:21:49.231620 str-dx010-acs-4 ERR syncd#syncd: :- bulkSet: set_route_entries_attribute is not supported
Oct 30 16:21:49.251872 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002,10.0.0.61@PortChannel0003,10.0.0.63@PortChannel0004
Oct 30 16:21:49.258997 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002,10.0.0.63@PortChannel0004
Oct 30 16:21:49.296391 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.296495 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.462893 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.462893 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.462957 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.61@PortChannel0003,10.0.0.63@PortChannel0004
Oct 30 16:21:49.468695 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.63@PortChannel0004
Oct 30 16:21:49.504041 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported
Oct 30 16:21:49.504538 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::72@PortChannel0001,fc00::76@PortChannel0002,fc00::7e@PortChannel0004
Oct 30 16:21:49.526490 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::72@PortChannel0001,fc00::7e@PortChannel0004
Oct 30 16:21:49.598758 str-dx010-acs-4 ERR syncd#syncd: :- bulkSet: set_route_entries_attribute is not supported
Oct 30 16:21:49.636098 str-dx010-acs-4 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002,10.0.0.63@PortChannel0004
Oct 30 16:21:49.636645 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.636971 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.657133 str-dx010-acs-4 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002
Oct 30 16:21:49.834021 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.834397 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.883522 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.883876 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.908087 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002,10.0.0.61@PortChannel0003
Oct 30 16:21:49.912149 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002
Oct 30 16:21:49.960503 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.59@PortChannel0002,10.0.0.63@PortChannel0004
Oct 30 16:21:49.963339 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:49.963554 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:49.987830 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::72@PortChannel0001,fc00::76@PortChannel0002,fc00::7a@PortChannel0003,fc00::7e@PortChannel0004
Oct 30 16:21:49.988176 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported
Oct 30 16:21:50.009971 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::72@PortChannel0001,fc00::76@PortChannel0002
Oct 30 16:21:50.029345 str-dx010-acs-4 ERR syncd#syncd: :- bulkSet: set_route_entries_attribute is not supported
Oct 30 16:21:50.029797 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group fc00::72@PortChannel0001,fc00::7a@PortChannel0003
Oct 30 16:21:50.159141 str-dx010-acs-4 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group 10.0.0.57@PortChannel0001,10.0.0.61@PortChannel0003,10.0.0.63@PortChannel0004
Oct 30 16:21:50.163696 str-dx010-acs-4 ERR syncd#syncd: :- bulkRemove: not implemented, FIXME
Oct 30 16:21:50.163802 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidRemove: bulkRemove api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.167122 str-dx010-acs-4 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group 10.0.0.57@PortChannel0001,10.0.0.63@PortChannel0004
Oct 30 16:21:50.204085 str-dx010-acs-4 NOTICE swss#orchagent: :- removeNextHopGroup: Delete next hop group fc00::72@PortChannel0001,fc00::7e@PortChannel0004
Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- meta_generic_validation_remove: object 0x5000000000686 reference count is 2, can't remove
Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- removeNextHopGroup: Failed to remove next hop group 5000000000686, rv:-17
Oct 30 16:21:50.217595 str-dx010-acs-4 ERR syncd#syncd: :- bulkRemove: not implemented, FIXME
Oct 30 16:21:50.217709 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidRemove: bulkRemove api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.513498 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.513598 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.518887 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.518992 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.534637 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.534776 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.553897 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.553999 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.575291 str-dx010-acs-4 NOTICE swss#orchagent: :- addNextHopGroup: Create next hop group 10.0.0.57@PortChannel0001,10.0.0.61@PortChannel0003
Oct 30 16:21:50.575783 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.576057 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.578636 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME
Oct 30 16:21:50.578928 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45
Oct 30 16:21:50.628600 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported

Describe the results you expected:
We shouldn't see any errors from syncd.

Additional information you deem important (e.g. issue happens only occasionally):

Output of show version:

admin@sonic:~$ show ver

SONiC Software Version: SONiC.master.461-6088bd59
Distribution: Debian 10.6
Kernel: 4.19.0-9-2-amd64
Build commit: 6088bd59
Build date: Thu Oct 29 19:06:46 UTC 2020
Built by: sonic@jenkins-worker-8

Uptime: 16:49:47 up 28 min,  1 user,  load average: 2.13, 1.96, 1.75

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-snmp                   latest                da467be983be        426MB
docker-snmp                   master.461-6088bd59   da467be983be        426MB
docker-teamd                  latest                29df3c93ee45        425MB
docker-teamd                  master.461-6088bd59   29df3c93ee45        425MB
docker-sonic-mgmt-framework   latest                5c067f4b99f2        549MB
docker-sonic-mgmt-framework   master.461-6088bd59   5c067f4b99f2        549MB
docker-router-advertiser      latest                071d37bcd803        390MB
docker-router-advertiser      master.461-6088bd59   071d37bcd803        390MB
docker-platform-monitor       latest                a74214f9bd0f        503MB
docker-platform-monitor       master.461-6088bd59   a74214f9bd0f        503MB
docker-lldp                   latest                03cb08ffa877        456MB
docker-lldp                   master.461-6088bd59   03cb08ffa877        456MB
docker-dhcp-relay             latest                57b818251555        397MB
docker-dhcp-relay             master.461-6088bd59   57b818251555        397MB
docker-database               latest                92fb2157bf9d        390MB
docker-database               master.461-6088bd59   92fb2157bf9d        390MB
docker-orchagent              latest                462cc6634084        437MB
docker-orchagent              master.461-6088bd59   462cc6634084        437MB
docker-nat                    latest                524f117bfe7c        428MB
docker-nat                    master.461-6088bd59   524f117bfe7c        428MB
docker-sonic-telemetry        latest                7235ee54ccbd        459MB
docker-sonic-telemetry        master.461-6088bd59   7235ee54ccbd        459MB
docker-fpm-frr                latest                736695efe884        441MB
docker-fpm-frr                master.461-6088bd59   736695efe884        441MB
docker-sflow                  latest                8e9c073d8380        428MB
docker-sflow                  master.461-6088bd59   8e9c073d8380        428MB
docker-syncd-brcm             latest                e9af209ac842        527MB
docker-syncd-brcm             master.461-6088bd59   e9af209ac842        527MB

syslog.txt
sairedis.rec.txt
@daall daall added P0 Priority of the issue Master Branch Quality labels Oct 30, 2020
@daall
Copy link
Contributor Author

daall commented Oct 30, 2020

Seems like it could be related to: sonic-net/sonic-sairedis#656

@kcudnik
Copy link
Contributor

kcudnik commented Oct 30, 2020

yes, related to sonic-net/sonic-sairedis#656
@dmytroxshevchuk can you check this out ? processBulkOidRemove/Create functions are only in your PR

@kcudnik
Copy link
Contributor

kcudnik commented Oct 30, 2020 •
edited
Loading

there is secondary issue here:
Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- meta_generic_validation_remove: object 0x5000000000686 reference count is 2, can't remove
Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- removeNextHopGroup: Failed to remove next hop group 5000000000686, rv:-17

@kcudnik kcudnik closed this as completed Oct 30, 2020
@kcudnik kcudnik reopened this Oct 30, 2020
@kcudnik
Copy link
Contributor

kcudnik commented Oct 30, 2020

Ok, couple issues here:

this is OA issues, is trying to remove next hop group, but reference is still in use, there is bug in OA for counting references

Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- meta_generic_validation_remove: object 0x5000000000686 reference count is 2, can't remove
Oct 30 16:21:50.210939 str-dx010-acs-4 ERR swss#orchagent: :- removeNextHopGroup: Failed to remove next hop group 5000000000686, rv:-17

next issue:

Oct 30 16:21:49.963554 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidCreate: bulkCreate api is not implemented or not supported, object_type = 45

this is just error LOG message since bulk API is not supported, it logs the messaged, but api is executed 1by1 in loop, so this message level should be loweredsimilar for this:

Oct 30 16:21:50.163802 str-dx010-acs-4 ERR syncd#syncd: :- processBulkOidRemove: bulkRemove api is not implemented or not supported, object_type = 45

next issue:

Oct 30 16:21:50.217595 str-dx010-acs-4 ERR syncd#syncd: :- bulkRemove: not implemented, FIXME
Oct 30 16:21:50.513498 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: not implemented, FIXME

those messages are from sonic VendorSai class in syncd, it means that some some of the bulk api is not implemented, but actual api then is executed in 1by1 in loop, loglevel of this message should stay as is, there is easy fix for this, and i just fount that is a bug: i added PR to fix this: sonic-net/sonic-sairedis#685

next issue:

Oct 30 16:21:49.129999 str-dx010-acs-4 ERR syncd#syncd: :- bulkCreate: create_route_entries is not supported

this means that vendor sai have no pointer for bulk API, but api is executed 1by1, loglevel of this could be lowered

last one:

Oct 30 16:21:49.598758 str-dx010-acs-4 ERR syncd#syncd: :- bulkSet: set_route_entries_attribute is not supported

vendor SAI also dont have bulk API pointer for this, but api is executed 1by1 anyway

so actually except that bug i found, this is just bunch of warnings, and instead of bulk, 1by1 is executed
if this is async mode, on any actual error syncd would crash

@kcudnik
Copy link
Contributor

kcudnik commented Oct 30, 2020

here is loglevel update: sonic-net/sonic-sairedis#687

@daall daall closed this as completed Nov 4, 2020
@kcudnik
Copy link
Contributor

kcudnik commented Nov 4, 2020

you just closed this? are there any comments whether this was resolved on OA level ?

@daall
Copy link
Contributor Author

daall commented Nov 4, 2020

we're opening a new issue for the OA problems. the syncd issue is resolved.

@daall daall mentioned this issue Nov 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kcudnik kcudnik

Labels
Master Branch Quality P0 Priority of the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kcudnik @daall