[DocDB] Correctly return error when attempting to serialize a too large message rather than crashing

[mdbridge]

Jira Link: DB-11216

Description

Today, we have the following code:

~/code/yugabyte-db/src/yb/rpc/serialization.cc:67:
Status SerializeMessage(
    AnyMessageConstPtr msg, size_t body_size, const RefCntBuffer& param_buf,
    size_t additional_size, size_t offset) {
  DCHECK_EQ(msg.SerializedSize(), body_size);
  auto size = SerializedMessageSize(body_size, additional_size);


  auto total_size = size + additional_size;
  if (total_size > FLAGS_rpc_max_message_size) {
    return STATUS_FORMAT(InvalidArgument, "Sending too long RPC message ($0 bytes)", total_size);
  }

where

~/code/yugabyte-db/src/yb/rpc/lightweight_message.cc:30:
DEFINE_UNKNOWN_uint64(rpc_max_message_size, 255_MB,
    "The maximum size of a message of any RPC that the server will accept. The sum of "
    "consensus_max_batch_size_bytes and 1KB should be less than rpc_max_message_size");

However, SerializedMessageSize has:

~/code/yugabyte-db/src/yb/rpc/serialization.cc:62:
size_t SerializedMessageSize(size_t body_size, size_t additional_size) {
  auto full_size = body_size + additional_size;
  return body_size + CodedOutputStream::VarintSize32(narrow_cast<uint32_t>(full_size));

This code will crash the TServer if full_size is greater than the maximum integer value for uint32_t.  (narrow_cast crashes the process if it is unable to complete the cast.)

This task is to prevent the TServer from crashing due to narrow_cast's in serialization.cc.

At a minimum, add a test that serializing a message greater than 4 GiB returns a InvalidArgument Status rather than crashing the process.

Tthe log snippet of the crash is:

6 06:30:43 ulcpcxygdb004 yb-tserver: F0506 06:30:43.382462 13246 <http://casts.cc:21]|casts.cc:21]> Bad narrow cast: 5867568307 &gt; 4294967295
May  6 06:30:43 ulcpcxygdb004 yb-tserver: Fatal failure details written to /yuga01/yb-data/tserver/logs/yb-tserver.FATAL.details.2024-05-06T06_30_43.pid26522.txt
May  6 06:30:43 ulcpcxygdb004 yb-tserver: F20240506 06:30:43 ../../src/yb/gutil/casts.cc:21] Bad narrow cast: 5867568307 &gt; 4294967295
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d1354bf47  google::LogMessage::SendToLog()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d1354ce8d  google::LogMessage::Flush()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d1354d509  google::LogMessageFatal::~LogMessageFatal()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d13d53609  yb::BadNarrowCast()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d1454b0bd  yb::rpc::YBInboundCall::Respond()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d144acb2e  yb::rpc::RpcContext::RespondSuccess()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d147f309c  yb::tserver::(anonymous namespace)::ReadQuery::Complete()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d147ec2cc  yb::tserver::PerformRead()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d14844433  yb::tserver::TabletServiceImpl::Read()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d149805a4  std::__1::__function::__func&lt;&gt;::operator()()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d1498204f  yb::tserver::TabletServerServiceIf::Handle()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d14537f8a  yb::rpc::ServicePoolImpl::Handle()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d14477d7f  yb::rpc::InboundCall::InboundCallTask::Run()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d14546b23  yb::rpc::(anonymous namespace)::Worker::Execute()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x562d14c18b82  yb::Thread::SuperviseThread()
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x7fa58a181694  start_thread
May  6 06:30:43 ulcpcxygdb004 yb-tserver: @     0x7fa58a68341d  __clone

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[es1024]

To reproduce:

CREATE TABLE test(pk INT PRIMARY KEY, text00 TEXT, text01 TEXT, text02 TEXT, text03 TEXT, text04 TEXT, text05 TEXT, text06 TEXT, text07 TEXT, text08 TEXT, text09 TEXT, text10 TEXT, text11 TEXT, text12 TEXT, text13 TEXT, text14 TEXT, text15 TEXT, text16 TEXT);
INSERT INTO test(pk) VALUES(0);
UPDATE test SET text00 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text01 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text02 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text03 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text04 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text05 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text06 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text07 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text08 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text09 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text10 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text11 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text12 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text13 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text14 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text15 = repeat('0', 254000000) WHERE pk = 0;
UPDATE test SET text16 = repeat('0', 254000000) WHERE pk = 0;
SELECT * FROM test LIMIT 1;

[rthallamko3]

One of the customers is running into this and they need the fix in 2.20, so adding corresponding backport labels.