Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

acquire lock before accessing shared redis context #489

Merged
merged 12 commits into from
Jul 24, 2019

Conversation

anilkpandey
Copy link
Contributor

Multiple threads in syncd access ASIC_DB through the same redis context. g_mutex is used to synchronize the access as hiredis is not thread safe.
Function try_translate_vid_to_rid(), which accesses ASIC_DB was being called unprotected. This caused one of the threads to stuck on redisBufferRead().

(gdb) bt
#0 0x00007f15301cd22d in read () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f1537ebf6a2 in read (__nbytes=16384, __buf=0x7f152d6b0910,
__fd=) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2 redisBufferRead (c=0x55c450ffdde0) at hiredis.c:802
#3 0x00007f1537ebf9a0 in redisGetReply (c=0x55c450ffdde0,
reply=reply@entry=0x7f152d6b4a00) at hiredis.c:888
#4 0x00007f1537c612d0 in swss::RedisReply::RedisReply (this=0x7f152d6b4a00,
db=0x55c451001e80, command=...) at redisreply.cpp:33
#5 0x00007f1537c614c2 in swss::RedisReply::RedisReply (this=0x7f152d6b4a00,
db=, command=..., expectedType=3) at redisreply.cpp:45
#6 0x00007f1537c7efba in swss::RedisClient::hset (this=0x55c450ffd510,
key="ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x26000000000e17","mac":"3C:2C:99:8B:35:BC","switch_id":"oid:0x21", '0' <repeats 12 times>, ""}", field="SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID",
value="oid:0x3a00000000104d") at redisclient.cpp:56

…d access from different threads

syncd main thread was doing g_redisClient->hget without acquiring g_mutex, thereby causing either syncd main thread or notification thread to block indefinitely on redisBufferRead().
@prsunny prsunny requested a review from kcudnik July 18, 2019 23:22
Copy link
Collaborator

@kcudnik kcudnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, this is due to move processFlexCounterEvent to be executed to separate theread
which is result of making workaround for postpone counters subscribe which should be synchronous on first place

@lguohan
Copy link
Contributor

lguohan commented Jul 19, 2019

@anilkpandey, can you resolve the conflict?

@anilkpandey
Copy link
Contributor Author

@anilkpandey, can you resolve the conflict?

I am not able to, may be because I don't have write access.

@prsunny
Copy link
Contributor

prsunny commented Jul 19, 2019

You can merge/rebase from master and push to this PR. You don't have to use the resolve button above.

@prsunny
Copy link
Contributor

prsunny commented Jul 20, 2019

Looks like the editor added a lot of tab spaces. Kindly revert the space changes and recommit.

@anilkpandey
Copy link
Contributor Author

Looks like the editor added a lot of tab spaces. Kindly revert the space changes and recommit.

Done

@kcudnik kcudnik merged commit 3a8fc6f into sonic-net:master Jul 24, 2019
pettershao-ragilenetworks pushed a commit to pettershao-ragilenetworks/sonic-sairedis that referenced this pull request Nov 18, 2022
* fixed issue of permenent block on redisBufferRead() due to unprotected access from different threads

syncd main thread was doing g_redisClient->hget without acquiring g_mutex, thereby causing either syncd main thread or notification thread to block indefinitely on redisBufferRead().

* Delete .DS_Store

.

* resolved merge conflict

* Revert "resolved merge conflict"

This reverts commit f009491.

* Revert "Merge remote-tracking branch 'upstream/master'"

This reverts commit 550aca9, reversing
changes made to 676c0b5.

* Revert "Revert "Merge remote-tracking branch 'upstream/master'""

This reverts commit 03fc996.

* Revert "Revert "resolved merge conflict""

This reverts commit e840a6b.

* Update syncd.cpp

* Update syncd.cpp

* Update syncd.cpp

* Update syncd.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants