DDL statements hang when the compiled statements cache is enabled

[ilya071294]

FB5. How to reproduce:

1. Prepare data:

connect '127.0.0.1:employee' user SYSDBA password 'masterkey';

create table TEST_TABLE(ID integer, STR1 varchar(50));

set term ^;

create or alter procedure GEN_DATA (
    i integer)
as
begin
  while (i > 0) do
  begin
    insert into TEST_TABLE values (:i, '01234567890123456789012345678901234567890123456789');
    i = i - 1;
  end
end^

set term ;^

execute procedure GEN_DATA(5000000);
commit;
create index TEST_TABLE_IDX1 on TEST_TABLE (ID);
commit;

Close this connection.

2. Execute statements in the first ISQL and leave it open:

connect '127.0.0.1:employee' user SYSDBA password 'masterkey';
select first 10 * from TEST_TABLE where ID > 0;
commit;

3. Execute statements in the second ISQL, and it will hang on the CREATE TABLE statement:

connect '127.0.0.1:employee' user SYSDBA password 'masterkey';
create table T1 (ID integer);

Seems like the issue is related to the compiled statements cache and occurs when DsqlStatementCache::shrink() is called.
MaxStatementCacheSize = 0 solves the issue.
Also when I try these steps on the debug build with MaxStatementCacheSize = 32K, I get a violated fb_assert in DsqlStatementCache::purgeAllAttachments().

[asfernandes]

This is regression introduced with fix for #7385

[hvlad]

I run DEBUG build with default firebird.conf.
I can't reproduce hang.

Do you run SS ?
Is it necessary to insert 5M recs into table ?
Looks like commit is missing after execute procedure, correct ?

But assert with MaxStatementCacheSize = 32K is reproduced.
Could you try following patch against it ?
gh7598.diff.txt

[ilya071294]

I run DEBUG build with default firebird.conf.

Same for me.

Do you run SS ?

Yes.

Is it necessary to insert 5M recs into table ?

Yes, and CREATE INDEX is also important here. I believe there is a more simple way to reproduce but I haven't figured it out yet.

Looks like commit is missing after execute procedure, correct ?

Yes, you are right. I edited the comment.

[hvlad]

Finally, I've reproduced it. Previous attempts failed because I insert 500K recs into table.
It makes me look why 2MB is not enough to cache such simple statement as

select first 10 * from TEST_TABLE where ID > 0;

Right after prepare, in DsqlStatementCache::putStatement the statement size is 9376 bytes.
After execution, in DsqlStatementCache::statementGoingInactive, the statement size become 2882880 bytes.

The such big difference is because of BitmapTableScan::Impure::irsb_bitmap that contains 5M record numbers and uses a lot of memory (allocated from statement pool). This is why both CREATE INDEX and 5M records is important in test case.

Also, it raise a question - should we explicitly clear (or reset) this bitmap (and may be some other resources) when RecordStream is closing ? I.e. not postpone it to the deletion of statement's pool.

[ilya071294]

But assert with MaxStatementCacheSize = 32K is reproduced.
Could you try following patch against it ?

I tried it, and it seems to fix both problems.

[hvlad]

Fixed, thanks.