orchangent crash during crm acl_resouce_list polling in armhf environment #801
Comments
|
@taahme, any idea why the regression is caused the boost 1.71 upgrade? |
…or call. Default value set to value_t::array in constructor. Issue: sonic-net/sonic-sairedis#801 Signed-off-by: Rajkumar Pennadam Ramamoorthy <rpennadamram@marvell.com>
…or call. Default value set to value_t::array in constructor. Issue: sonic-net/sonic-sairedis#801 Signed-off-by: Rajkumar Pennadam Ramamoorthy <rpennadamram@marvell.com>
|
When setting a hardware watchpoint to the stack memory address, the value is not changing from 2 (value_t::array) to 0 (value_t::null), instead value 0 is directly written to memory. (gdb) x/256wa $sp Thread 1 "orchagent" hit Hardware watchpoint 2: *0xbe9b5368 Old value = -1225460960 Attached full log of gdb. |
|
kernel changing memory ? rather user space |
No other thread in orchagent process context is writing in this memory. |
|
hmm, this code is executed inside sairedis lib, and all apis are protected by mutex, not sure how other thread could modify this, and why kernel ? that seems really messed up, unless there is some bad allocation in json.hpp |
Issue is not related to json.hpp. I wrote simple main.cpp (with specific code causing problem), no crash observed. |
|
if it's not json.hpp, do you have suspect for a root cause? and if it's not json.hpp what's about then #802?? and then moved this to swss-common to fix json.hpp internally? this suggest this is something wrong with json.hpp, or compiler issue, or perhaps emulation of armrh |
|
@rajkumar38, how frequently do you see this issue on arm? |
Issue is consistently reproducible. |
|
will you be able to narrow this down by removing more and more components from OA since you said that simple main.cpp worked fine (did you also compiled this with exact same flags that sairdis is compiled? and with the same input for serialize method?). Or is there a possibility to access your device ? i can then try do do it myself. Also, is this the only place this happens? i mentioned before that array() json is used in many places in serialieze, just want to narrow this if this is strictly related to json or acl_resource_list serialiaze |
Can you be more specific on statement "will you be able to narrow this down by removing more and more components from OA" ? compiled with flags I'm checking on the possibility of device access internally. will get back to you on this.
We are consistently see the issue only in array() json in acl_resource_list_serialize. |
|
thanks for sharing example, but you are missing a lot of flags when sairedis/meta is compiled for example saiserialize.cpp where acl serialize method exists is compiled like this: and those options can have different effect on compilation (also this is cmd for x64 not for arm), warnings can be probably skipped but those options: -ansi -fPIC -std=c++11 potentially can generate some other binary code for test program i was rather thinking about something like this: #include "meta/sai_serialize.h"
int main()
{
sai_acl_resource_list_t res = { }; // populate with some values
sai_serialize_acl_resource_list(res, false); // or use sai_serialize_attr_value instead
}and compile this like this g++ main.cpp -I sairedis/meta -Lsairedis/meta/.libs -lsaimeta -lsaimetadata and saimeta and saimetadata should be compiled by buildimage, to rule out some other factor, and confirm whether issue still exists, and this will give some more insight, if it exists still, then problem is in json.hpp or compilaton of saimeta/saimetadata, and if error don't exists, then it will seem like it's strictly related to orchagent only and not actual json array issue. you can clone sairedis repo, do ./autogen.sh; ./configure; cd meta; make if you need some help, maybe it will be faster to setup some meeting and live solve problem, this will be also definitely faster if this investigation will still show no error, then it would suggest that error is related strictly to OA, and by "narrow this down" i meant removing code from OA, compile it and test if it still hit the issue, function that takes those resources is located in |
Thanks for the inputs. I followed the procedure and compiled the main.cpp linking shared-libs from the compiled debians. |
|
Ok, so now we have a different exception or maybe it's the same ? push_back seems to come from json.hpp, and place is probably similar as in case when OA calls the code. For future root causing this, we can take out function code from sai_serialize_acl_resource_list, and put that function directly into main.cpp, so no libs would be required to compile this. If error will still persist, then you can try to remove compile flags one by one to see if there is any effect on that. Also you code is missing "-O2" argument, which can be crucial when compiling code inside main.cpp. I'm still betting this some compiler code optimization bug on armhf |
It is the same exception similar to OA call.
Issue is seen with or without O2 flag. FYI, this was working before in armhf env. So I tried to isolate the commit that was causing this error. |
|
Code in sonic-net/sonic-swss-common#452 is not related to serialization in any way (It can possibly generate some extra code from json.hpp since everything there is template). So not sure why would error go away if you remove it. You are mentioning also that issue is seen with O2 or without, but for what scenario ? when you compile main with libs ? O2 does not affect already compiled libs, you would need to recompile libs without O2 to potentially see the difference. ps. if error goes away if you remove code in 452, then this maybe pointing to some issues with compiler. What compiler do you use to compile armhf gcc abi?? what version? ps. i still suggest to have a quick call if possible, it would definitely speed up finding out the issue. |
I didn't recompile libs without O2. Can you suggest on the list of submodules that O2 has to be disabled ? We can have a call to discuss further. Pls share your email and time zone, will send zoom invite. |
|
by default O2 is enabled on all modules, that serialization methods only uses saimeta from sairedis and swsscommon which has it's own dependencies, but since only json.hpp is used, actually no swsscommon should be required at all. and probably to reproduce this, only saiserialize.cpp would be required and json.hpp im in CET/GMT+1 time zone, you can send invite to my alias at microsoft.com "kcudnik", my available hours are 10am-10pm |
|
we spend some time on zoom call, and we find out that when sonic is build using docker the issue is present @rajkumar38 one question here, will you be able to test this on different hardware device ? just in case |
|
i had in mind test this on second armhf device if you have possible access to 2nd device |
We have seen same crash on different armhf HW. |
|
ok, thanks for confirmation |
|
I was able to reproduce this issue locally on our lab device: but im' not sure why we don't see any issues with that on our side, if we run orchagent, maybe those available resources are not queried properly, or tests are wrong, no clue. also difference i noticed, that there is a lot of less libraries linked to binary and that's it, and on your platform this list was like 2x size, but i guess that's not matter now since we have a repro, |
|
Last night i spent couple hours after midnight to figure this out, and i still have no root cause, but issue is definitely related to armhf g++/linker and linux elf dynamic linker and loader on our lab device: libs looks like this: and are copied directly from swss docker as in previous post Makefile all: main
main: main.cpp Makefile
g++ main.cpp -o main -I /home/admin/sonic-sairedis/SAI/inc/ \
libswsscommon.so.0 libsaimeta.so.0 libsaimetadata.so.0
.PHONY: clean
clean:
rm -f mainhere is main.cpp used for test: #include "saitypes.h"
#include <string>
extern std::string sai_serialize_acl_resource_list(
_In_ const sai_acl_resource_list_t& aclresource,
_In_ bool countOnly);
int main()
{
sai_acl_resource_t res;
sai_acl_resource_list_t list = { .count = 1, .list = &res };
sai_serialize_acl_resource_list(list, false);
}compilation and execution that was basic scenario and it's reproducible 100% of the time issue is observed on push_back() method in json.hpp file playing with only libswsscommon.so.0.0.0I build setup gcc, libnl (libnl-dev packages will be needed) and other libraries on device itself to test everything on device. configuration on sonic-swsscommon: (on commit f01fede8500a499ce94f3573f0bb6c3c5991efe7) this will produce common/.libs/libswsscommon.so.0.0.0, if we copy that to our test dir and recompile main, the error will still be present, this commit contains line in dbconnector.cpp: (btw this code is not used in serialization methods) void DBConnector::hmset(const std::unordered_map<std::string, std::vector<std::pair<std::string, std::string>>>& multiHash)
{
SWSS_LOG_ENTER();
// make sure this will be object (not null) when multi hash is empty
json j = json::object();if we change that that to: void DBConnector::hmset(const std::unordered_map<std::string, std::vector<std::pair<std::string, std::string>>>& multiHash)
{
SWSS_LOG_ENTER();
// make sure this will be object (not null) when multi hash is empty
json j ; // = json::object();
then when we recompile swss-common and copy again common/.libs/libswsscommon.so.0.0.0 to our test directory then error is magically gone!, this suggests that there could be some issues with linking/dynamic loading of libraries and some symbols issues so now we have 2 swsscommon libraries, which one is causing error, and second not. libsaimetadata.so is untouched so what's the difference? i tried to do a "nm -D" and "radelf -a" to compare between good and bad lib so if we do ** diff -U0 good bad** for command "nm -DC" --- good 2021-04-02 11:37:11.154726016 +0000
+++ bad 2021-04-02 11:37:15.954742416 +0000
@@ -600,0 +601 @@
+W nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::basic_json(std::initializer_list<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> >, bool, nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::value_t)
@@ -601,0 +603 @@
+W nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::basic_json(std::initializer_list<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> >, bool, nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::value_t)in bad, there are 2 extra symbols - which are actually the same. "readelf -a" contaisn like 6k differnces, and addresses are differnt, so there is no good way to quickly compare this. i will try different approach: i stripped entire code from swss-common library that is not used by libsaimetadata.so, (only Logger and tokenizer are used) i also removed all references to json.hpp, and created own swss.cpp which contains minimal interface needed for libsaimetadata.so: swss.cpp #include <string>
#include <vector>
namespace swss {
class Logger
{
public:
enum Priority { SWSS_EMERG, };
void write(Priority prio, const char *fmt, ...);
void wthrow(Priority prio, const char *fmt, ...);
static Logger &getInstance();
class ScopeLogger
{
public:
ScopeLogger(int line, const char *fun);
~ScopeLogger();
};
class ScopeTimer
{
public:
ScopeTimer(int line, const char* fun, const char *msg, ...);
~ScopeTimer();
};
};
Logger &Logger::getInstance() { static Logger log; return log; }
Logger::ScopeLogger::ScopeLogger(int line, const char *fun){}
Logger::ScopeLogger::~ScopeLogger(){}
Logger::ScopeTimer::ScopeTimer(int line, const char* fun, const char *msg, ...){}
Logger::ScopeTimer::~ScopeTimer(){}
void Logger::write(Priority prio, const char *fmt, ...){}
void Logger::wthrow(Priority prio, const char *fmt, ...){}
std::vector<std::string> tokenize(const std::string &str, const char token) { throw; }
}Makefile all: main
main: main.cpp Makefile libswsscommon.so.0.0.0
g++ main.cpp -o main -I /home/admin/sonic-sairedis/SAI/inc/ \
libswsscommon.so.0 libsaimeta.so.0 libsaimetadata.so.0
libswsscommon.so.0.0.0: swss.cpp
g++ -shared -fPIC swss.cpp -o libswsscommon.so.0.0.0
.PHONY: clean
clean:
rm -f main libswsscommon.so.0.0.0ok, at this point we have original libsaimeta.so libsaimetadata.so and minimalistic libswsscommon.so, which compiles, with main.cpp and everything works, serialization works in main at this point. at this point we are introducing new library "foo.so" from foo.cpp foo.cpp #include "json.hpp"
using json = nlohmann::json;
void bar() { json j = json::object(); }
void foo() { json j = json::array(); }and compile like this: all: main
main: main.cpp Makefile libswsscommon.so.0.0.0 foo.so
g++ main.cpp -o main -I /home/admin/sonic-sairedis/SAI/inc/ \
foo.so libswsscommon.so.0 libsaimeta.so.0 libsaimetadata.so.0
libswsscommon.so.0.0.0: swss.cpp
g++ -shared -fPIC swss.cpp -o libswsscommon.so.0.0.0
foo.so: foo.cpp Makefile
g++ -shared -O2 -ansi -fPIC -std=c++11 -Wno-psabi foo.cpp -o foo.so
.PHONY: clean
clean:
rm -f main libswsscommon.so.0.0.0 foo.soand error strikes again when we add totally unrelated library foo.so, which is just linked, error comes back ! and here is fun part
i think we could treat this json::parse("{}") as temporary/workaround solution to replace code in sonic-net/sonic-swss-common#452 but this is not fixing root cause, i think still problem is somewhere on compiler/linker/loader level and may or may not be mitigated with some specific compiler flags, but no guarantee for that. playing with libsaimetadata.soif i compile manually libsaimetadata.so or even if i use this: and get library from there directly, and copy libsaimetadata.so.0.0.0 and keep original libsaimeta and libswsscommon, error goes away, and i don't know why :/ i also noticed that when during fake root build, some extra parameters are pushed to compiler like: especially that stack protector is causing to produce extra code, but even with this, error is gone when compiled locally so far i could not make scenario that i locally compile libsaimetadata.so and i will still have error present, i will try build using sonic-buildimage, and see if this help to bring back error, and then try to narrow down code from build docker, i tried to get artifacts from azure pipeline like from here: https://dev.azure.com/mssonic/build/_build/results?buildId=7329&view=artifacts&pathAsName=false&type=publishedArtifacts and if i take those debs from there (just libsaimeta, libsaimetadata and libswsscommon) error still exists, and this shows that there are still some differences when building docker and building locally final thoughtsi also noticed that when i compare libsaimetadata.so compiled locally (no error) vs compiled by azure pipeline (error exists) some symbols are named differently some have [c++11::abi] inside them, but most parts is the same, still i think there is some compatibility issue but adding that extra foo.so library clearly shows that this is some linker/loader related and narrowing this down would require to have locally build libsaimetadata that also causes this error to show up, and then cutting json.hpp until we get smallest example what's going on, it also could be hard to achieve json.hpp is about 8k lines, and even if we would be able to make 10 lines class that could reproduce that, it still could be cross-compiler issue unfortunately im not familiar with armhf assembly to figure out the differences between good and bad code and figure out wha't happening there :/ notice also that json.hpp as only file, produces "note" info from compiler (that's why added -Wno-psabi), there were errors in gcc5 that introduces some api interface errors, like functions could have different parameter size because of this, but this is fixed already since then, |
|
Thanks @lguohan and @yxieca to jump in internal discussion. Hear some more clues: Guohan asked:
Good observation, I was natively compiled on device, but docker is cross-compiling and maybe that’s the key. This is next on my list to build that in docker and do similar code stripping to find if that could be the case.
In our case, even if that would be the case, all symbols should be exactly the same, since entire sonic is build on the same docker with the same version of gcc cross compiler for armhf. @ying Xie, thanks for analysis, I'm aware of that symbol strip, and there is even a special flag for linker that uses local symbols if symbol is multiple defined in several libraries “-Bsymbolic”, but I does not make effect if I compile foo.cpp with it. As for changing order of the libraries, I just tried that, and if i put foo.so with (foo and bar functions) the error goes way (yey) even if I remove -O2 from foo compilation. And original issue goes away if we swap order of libswsscommon and libsaimetadata, so great you are right here 😊 But I think this I still not root causing the problem, and changing order not always could be possible due to other dependencies that can be required. I will ask marvell to try this order change on OA, but libswsscommon and libsaimetadata are also in many different places in siredislib at this order like saiplayer. Attempt to workaround: sonic-net/sonic-swss#1689 As for json::parse("{}") it can still generate json::object inside parse function when expanded but then it could be marked by compiler as internal object, and removed/hidded when -O2 is present also it’s about 3x slower (just measured) than json::object(). What I also observed that even foo.cpp is compiled with foo and bar methods, and -O2, there are still some weak json symbols exported even thou json.hpp is self-contained library and everything in foo.cpp is used internally, if I mark foo and bar as static method, no other json symbols are exported and error goes away, but when skip O2 param, error goes back, since json symbols will be showed up (probably error caused by symbols). My conclusion here is that there is some issue with symbols that are generated via cross compilation. Question here is, whether we want to dig more into that if lib swap will be good workaround for that. @rajkumar38 is possible, please cherry pick sonic-net/sonic-swss#1689 and test on your side if this will be possible workaround fix, and if it will move forward a little bit :P |
|
One more workaround i found is to make all json.hpp namespace hidden namespace nlohmann __attribute__ ((visibility ("hidden")))this will also work with/without -O2 param Potential fix: sonic-net/sonic-swss-common#471 @rajkumar38 you could also try to cherrypick this one too maybe independently even Edit: visibility attribute causes compile issue on swss, since GearParserBase exposes json to public |
|
@rajkumar38 , can you share the root cause analysis, we need to make sure we fully understand the root cause. |
|
There is possible another workaround to add gcc flag where json.hpp is used, so swss-common, swss and sairedis |
|
I tried cross compile but on some older gcc version with swss-common, sairedis and swss, with everything compiles fine, and when i copy compiled libs and binaries to target device, there is no error, i will need to build entire sonic image locally and keep build docker, and then try to narrow this down |
|
@kcudnik sonic-net/sonic-swss#1689 (recompiled and installed swss_1.0.0_armhf.deb) I also tried with sample main.cpp with above PRs and no exception seen. |
|
Great!, so each of the PR's worked separetly ? |
Yes. I tried separately. It works. |
|
Thanks for confirmation! That's good, 1689 is already merged, so we could proceed with that and update sonic-buildimage, just to make things going and possibly hit otter issues on armhf XD. We still probably need to root cause this exactly, i will start playing with docker tomorrow if i will post my progress. Please also note that this is not the only issue I'm working on :) |
|
Getting back to "foo.cpp" example with foo and bar functions, when we compile that example with -O2 flag, issue still exists, but there is only 11 exported symbols related to json, and it was easy to check which symbol is causing problem, and if we rename this one in compiled foo.so:
then issue goes away, same symbol also exists in libsaimetadata.so, but since it can't be found in foo.so it uses local version of it and "everything" works, i mean there is no crash with this particular example, in full code there still could be other symbols that cause similar issues playing with single symbols like that would be bad idea, so it's better to go with -fvisibility-inlines-hidden flag |
|
ok, i finally found root cause for this issue, i build locally sonic image and debs, and i noticed that by default there are 4 docker images created, 2 for streach and 2 for buster: sonic-buildimage makefiles are using those to build deb packages, and you can select whether those are build on stretch or buster (ex using BLDENV or NOSTRECH=1) I used those 2 images, to locally build libswsscommon and libsaimetadata, but in both cases i didn't see the issue, and i started to think that, this probably is not compiler bug, but rather something else. What i also noticed that gcc version is different for them: and if we combine this with gcc NOTE message when compiling (on stretch) swsscommon which includes json.hpp (or libsaimetadata saiserialize.cpp): and note here: will change in GCC 7.1 - notice WILL and when compiling on buster: and note here: changed in GCC 7.1 - notice CHANGED And then it hit me, the problem is that somehow libswsscommon is compiled on stretch, and libsaimetdata on buster, even thou buster was selected! gcc was informing us all along, that there will be changes in passing parameter, and if that's true then gcc v6 is NOT compattible with v8 for this particular function, which happens to be json constructor. Now for confirmation, i took artifacts from here: https://dev.azure.com/mssonic/build/_build/results?buildId=7329&view=artifacts&pathAsName=false&type=publishedArtifacts from sonic-buildimage.marvell-armhf/target/debs/buster since release packages were stripped from all debug info, we need to to use debug symbols to find out compiler: Bingo!, 2 different versions for gcc, so json constructor symbols are incompatible between libswsscommon and libsaimetadatta, and hence we are getting exception on serialization, which is actually lucky that we get exception instead of crash. now for the good part: why libswsscommon is compiled with different version of gcc? i have no idea :D I will need help from someone who designed makefiles for sonic-buildimage to find why stretch/gccv6 is used, could be azure pipeline artifact selection bad? - rather no, since locally build also have this issue |
|
some more insights from azure pipeline artifacts from libsaimetadata-dbg_1.0.0_armhf.deb: this is wired, since first one contains both gcc6 and gcc8 versions, but in that docker which builds libsaimetadata and libsaimeta there is no gccv6 installed those are steps which i did locally on latest ubuntu 20.04.1-Ubuntu (+ installing dockers etc from apt) "SONIC_IMAGE_VERSION" : "master.0-0b16ca4a" clone commit: sonic-net/sonic-buildimage@0b16ca4 after this, everything is compiled with gcc version v8.3.0, and there is no issue present when testing with main.cpp |
|
Raised below PR which moves syncd compilation from stretch to buster. With this libswsscommon,libsaimetadata and libsairedis debs will be compiled only for buster with gccv8x. This PR fixes the orchagent json issue. |
|
what if someone would like to build it still on stretch ? |
@kamil, as I understand community is moving towards buster. Is there any reason why do we need to support stretch? |
|
not all vendors moved to buster, and they still use stretch, we will be encouraging all vendors to move to buster |
We had already moved to Buster on armhf platform, so no need to support stretch anymore. |
Below is the gdb stack trace.
Looks like libboost1.71 pkg causing the crash. Issue is not seen after reverting PR, sonic-net/sonic-buildimage#6532
Looks like basic_json constructor is invoked with value_t::null instead of value_t::array.
gdb) bt
#0 0xb68cb746 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
#1 0xb68d90ae in raise () from /lib/arm-linux-gnueabihf/libc.so.6
#2 0xb68cb1f2 in abort () from /lib/arm-linux-gnueabihf/libc.so.6
#3 0xb6ac4888 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#4 0xb6ac31ec in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#5 0xb6ac3242 in std::terminate() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#6 0xb6ac347c in __cxa_throw ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
#7 0xb6c20a70 in nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::push_back (this=0xbecee3a8,
val=...) at /usr/include/swss/json.hpp:4710
#8 0xb6c16458 in sai_serialize_acl_resource_list[abi:cxx11](_sai_acl_resource_list_t const&, bool) (aclresource=..., countOnly=countOnly@entry=false)
at saiserialize.cpp:1277
#9 0xb6c1688e in sai_serialize_attr_value[abi:cxx11](_sai_attr_metadata_t const&, _sai_attribute_t const&, bool) (meta=..., attr=...,
countOnly=countOnly@entry=false) at saiserialize.cpp:1613
#10 0xb6c0bf1e in saimeta::SaiAttributeList::serialize_attr_list[abi:cxx11](_sai_object_type_t, unsigned int, _sai_attribute_t const*, bool) (
objectType=objectType@entry=SAI_OBJECT_TYPE_SWITCH,
--Type for more, q to quit, c to continue without paging--
attr_count=attr_count@entry=1, attr_list=attr_list@entry=0xbecee808,
countOnly=countOnly@entry=false) at SaiAttributeList.cpp:120
#11 0xb6e72862 in sairedis::RedisRemoteSaiInterface::get (
this=this@entry=0x8a1bd8,
objectType=objectType@entry=SAI_OBJECT_TYPE_SWITCH,
serializedObjectId="oid:0x21", '0' <repeats 12 times>,
attr_count=attr_count@entry=1, attr_list=attr_list@entry=0xbecee808)
at RedisRemoteSaiInterface.cpp:804
#12 0xb6e72aac in sairedis::RedisRemoteSaiInterface::get (this=0x8a1bd8,
objectType=SAI_OBJECT_TYPE_SWITCH, objectId=, attr_count=1,
attr_list=0xbecee808) at RedisRemoteSaiInterface.cpp:566
#13 0xb6c30426 in saimeta::Meta::get (this=0x8a3f0c,
object_type=SAI_OBJECT_TYPE_SWITCH, object_id=,
attr_count=1, attr_list=0xbecee808) at Meta.cpp:1528
#14 0xb6e5ce34 in sairedis::Sai::get (this=0x889c84,
objectType=SAI_OBJECT_TYPE_SWITCH, objectId=, attr_count=1,
attr_list=0xbecee808) at Sai.cpp:240
#15 0xb6e5877c in redis_get_switch_attribute (object_id=,
attr_count=1, attr_list=0xbecee808) at sai_redis_switch.cpp:27
#16 0x00580016 in ?? ()
#17 0x0058032a in ?? ()
#18 0x004e0818 in ?? ()
#19 0x004c3f44 in ?? ()
--Type for more, q to quit, c to continue without paging--
#20 0xb68cb524 in __libc_start_main () from /lib/arm-linux-gnueabihf/libc.so.6
#21 0x004d9d38 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) fr 7
#7 0xb6c20a70 in nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::push_back (this=0xbecee3a8,
val=...) at /usr/include/swss/json.hpp:4710
4710 /usr/include/swss/json.hpp: No such file or directory.
(gdb) p* this
$1 = {
m_type = nlohmann::basic_json<std::map, std::vector, std::_cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::object, m_value = {
object = 0x9d2c70, array = 0x9d2c70, string = 0x9d2c70, boolean = 112,
number_integer = 10300528, number_unsigned = 10300528,
number_float = 5.0891370188258436e-317}}
(gdb) fr 8
#8 0xb6c16458 in sai_serialize_acl_resource_list[abi:cxx11](sai_acl_resource_list_t const&, bool) (aclresource=..., countOnly=countOnly@entry=false)
at saiserialize.cpp:1277
1277 saiserialize.cpp: No such file or directory.
(gdb) info local
item = {
m_type = nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::object, m_value = {
object = 0x9cd930, array = 0x9cd930, string = 0x9cd930, boolean = 48,
number_integer = 10279216, number_unsigned = 10279216,
number_float = 5.0786074917816749e-317}}
i = 0
logger__LINE = {m_line = 1258,
m_fun = 0xb6c39414 <sai_serialize_acl_resource_list[abi:cxx11](_sai_acl_resource_list_t const&, bool)::FUNCTION> "sai_serialize_acl_resource_list"}
FUNCTION = "sai_serialize_acl_resource_list"
j = {
m_type = nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::object, m_value = {
object = 0x9cb870, array = 0x9cb870, string = 0x9cb870, boolean = 112,
number_integer = 10270832, number_unsigned = 10270832,
number_float = 5.0744652454069419e-317}}
arr = {
m_type = nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::object, m_value = {
--Type for more, q to quit, c to continue without paging--
object = 0x9d2c70, array = 0x9d2c70, string = 0x9d2c70, boolean = 112,
number_integer = 10300528, number_unsigned = 10300528,
number_float = 5.0891370188258436e-317}}
With break-points:
Thread 1 "orchagent" hit Breakpoint 2, nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::basic_json (
this=0xbef1a3a8, init=..., type_deduction=false,
manual_type=nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::null) at json.hpp:1520
1520 json.hpp: No such file or directory.
(gdb) bt
#0 nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::basic_json (this=0xbef1a3a8, init=...,
type_deduction=false,
manual_type=nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::null) at json.hpp:1520
#1 0xb6c2443e in nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::array (init=...)
at /usr/include/swss/json.hpp:1615
#2 sai_serialize_acl_resource_list[abi:cxx11](_sai_acl_resource_list_t const&, bool) (aclresource=..., countOnly=countOnly@entry=false)
at saiserialize.cpp:1271
#3 0xb6c2488e in sai_serialize_attr_value[abi:cxx11](_sai_attr_metadata_t const&, _sai_attribute_t const&, bool) (meta=..., attr=...,
countOnly=countOnly@entry=false) at saiserialize.cpp:1613
#4 0xb6c19f1e in saimeta::SaiAttributeList::serialize_attr_list[abi:cxx11](_sai_object_type_t, unsigned int, _sai_attribute_t const*, bool) (
objectType=objectType@entry=SAI_OBJECT_TYPE_SWITCH,
attr_count=attr_count@entry=1, attr_list=attr_list@entry=0xbef1a808,
countOnly=countOnly@entry=false) at SaiAttributeList.cpp:120
#5 0xb6e80862 in sairedis::RedisRemoteSaiInterface::get (
--Type for more, q to quit, c to continue without paging--
this=this@entry=0x1493c80,
objectType=objectType@entry=SAI_OBJECT_TYPE_SWITCH,
serializedObjectId="oid:0x21", '0' <repeats 12 times>,
attr_count=attr_count@entry=1, attr_list=attr_list@entry=0xbef1a808)
at RedisRemoteSaiInterface.cpp:804
#6 0xb6e80aac in sairedis::RedisRemoteSaiInterface::get (this=0x1493c80,
objectType=SAI_OBJECT_TYPE_SWITCH, objectId=, attr_count=1,
attr_list=0xbef1a808) at RedisRemoteSaiInterface.cpp:566
#7 0xb6c3e426 in saimeta::Meta::get (this=0x1496084,
object_type=SAI_OBJECT_TYPE_SWITCH, object_id=,
attr_count=1, attr_list=0xbef1a808) at Meta.cpp:1528
#8 0xb6e6ae34 in sairedis::Sai::get (this=0x147bc84,
objectType=SAI_OBJECT_TYPE_SWITCH, objectId=, attr_count=1,
attr_list=0xbef1a808) at Sai.cpp:240
#9 0xb6e6677c in redis_get_switch_attribute (object_id=,
attr_count=1, attr_list=0xbef1a808) at sai_redis_switch.cpp:27
#10 0x004f0016 in ?? ()
#11 0x004f032a in ?? ()
#12 0x00450818 in ?? ()
#13 0x00433f44 in ?? ()
#14 0xb68d9524 in __libc_start_main () from /lib/arm-linux-gnueabihf/libc.so.6
#15 0x00449d38 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p manual_type
$1 = nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, bool, long long, unsigned long long, double, std::allocator>::value_t::null
(gdb)
The text was updated successfully, but these errors were encountered: