Skip to content

Commit

Permalink
[Bug](column) fix append_data_by_selector_impl reserve too mush usele…
Browse files Browse the repository at this point in the history
…ss memory (#39581)

## Proposed changes

```
for (auto* place : local_state._value_places) {
                            SCOPED_TIMER(local_state._selector_block_timer);
                            RETURN_IF_ERROR(place->append_block_by_selector(input_block, eos));
}
    for (int i = 0; i < mutable_columns.size(); ++i) {
        columns[i]->append_data_by_selector(mutable_columns[i], _selector);
    }
```
the columns[I] size is 4096, and _selector size is 1;
so the (4096 - 1) rows is useless;
in a usercase, the block have 4096 rows and 34 columns.
so need bytes is:
1 block * 4096 rows * 4096 reserve * 16 bytes * 34 column * 48 instance
= 438086664192 bytes;

it's will cause the query canceled as not have enough memory.
  • Loading branch information
zhangstar333 authored and dataroaring committed Aug 26, 2024
1 parent 053cdca commit 61db2e2
Showing 1 changed file with 5 additions and 2 deletions.
7 changes: 5 additions & 2 deletions be/src/vec/columns/column_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,11 @@ void IColumn::append_data_by_selector_impl(MutablePtr& res, const Selector& sele
"Size of selector: {} is larger than size of column: {}",
selector.size(), num_rows);
}

res->reserve(num_rows);
DCHECK_GE(end, begin);
// here wants insert some value from this column, and the nums is (end - begin)
// and many be this column num_rows is 4096, but only need insert num is (1 - 0) = 1
// so can't call res->reserve(num_rows), it's will be too mush waste memory
res->reserve(res->size() + (end - begin));

for (size_t i = begin; i < end; ++i) {
static_cast<Derived&>(*res).insert_from(*this, selector[i]);
Expand Down

0 comments on commit 61db2e2

Please sign in to comment.