ArrayIndexOutOfBound Exception when using binary protocol (Custom ID 39)

[henryzhao81]

Switch to using testing 2.0 source code, I found a issue when I using Console which goes to binary protocol. Following are the cases:

create class tuser;
create class tgroup;
create property tuser.name STRING;
create property tuser.groups LINKSET tgroup;
create property tgroup.name STRING;
create property tgroup.memberUsers LINKSET tuser;
insert into tuser set name = 'test1'; //Inserted record 'tuser#105:0{name:test1} v1'
insert into tuser set name = 'test2'; //Inserted record 'tuser#105:1{name:test1} v1'
insert into tuser set name = 'test3'; //Inserted record 'tuser#105:2{name:test1} v1'
insert into tuser set name = 'test4'; //Inserted record 'tuser#105:3{name:test1} v1'
insert into tuser set name = 'test5'; //Inserted record 'tuser#105:4{name:test1} v1'
insert into tuser set name = 'test6'; //Inserted record 'tuser#105:5{name:test1} v1'
insert into tuser set name = 'test7'; //Inserted record 'tuser#105:6{name:test1} v1'
insert into tuser set name = 'test8'; //Inserted record 'tuser#105:7{name:test1} v1'
insert into tuser set name = 'test9'; //Inserted record 'tuser#105:8{name:test1} v1'
insert into tgroup set name = 'testgroup1'; //Inserted record 'tgroup#107:0{name:testgroup1} v1'
update #105:0 set groups = [#107:0];
update #107:0 set memberUsers = [#105:1,#105:2,#105:3,#105:4,#105:5,#105:6,#105:7,#105:8];

When I execute : select from #105:0, on server side it keep throw ArrayIndexOutOfBoundsException, it works fine when I execute on studio.
Error on iterating record collection
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #105:2
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:945)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:79)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:264)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:43)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:302)
at com.orientechnologies.orient.core.db.record.OLazyRecordIterator.next(OLazyRecordIterator.java:75)
at com.orientechnologies.orient.core.db.record.OLazyRecordIterator.next(OLazyRecordIterator.java:38)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeLinkCollection(ORecordSerializerBinaryV0.java:586)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeSingleValue(ORecordSerializerBinaryV0.java:467)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serialize(ORecordSerializerBinaryV0.java:180)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinary.toStream(ORecordSerializerBinary.java:81)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:1884)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:654)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.getRecordBytes(OBinaryNetworkProtocolAbstract.java:418)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.writeRecord(OBinaryNetworkProtocolAbstract.java:353)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.writeIdentifiable(OBinaryNetworkProtocolAbstract.java:129)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1163)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:343)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:180)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:65)
Caused by: java.lang.ArrayIndexOutOfBoundsException
Error on iterating record collection
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #105:3
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:945)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:79)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:264)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:43)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:302)
at com.orientechnologies.orient.core.db.record.OLazyRecordIterator.next(OLazyRecordIterator.java:75)
at com.orientechnologies.orient.core.db.record.OLazyRecordIterator.next(OLazyRecordIterator.java:38)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeLinkCollection(ORecordSerializerBinaryV0.java:586)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeSingleValue(ORecordSerializerBinaryV0.java:467)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serialize(ORecordSerializerBinaryV0.java:180)
at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinary.toStream(ORecordSerializerBinary.java:81)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:1884)
at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:654)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.getRecordBytes(OBinaryNetworkProtocolAbstract.java:418)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.writeRecord(OBinaryNetworkProtocolAbstract.java:353)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.writeIdentifiable(OBinaryNetworkProtocolAbstract.java:129)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1163)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:343)
at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:180)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:65)
Caused by: java.lang.ArrayIndexOutOfBoundsException
… …

After debug the source code, find out due to 2.0 we did lots of changes on protocol.

When iterator memberUsers which is a linkset, somehow recordSerializer was set in ONetworkThreadLocalSerializer.(Fiirst time read serializer was set to null in thread local). when call fromStream, it was expect first bytes as serilizerVersion followed by document binary, but somehow it was missing. Then throw ArrayIndexOutofBound exception,

ORecordSerializerBinary.java
@OverRide
public ORecord fromStream(final byte[] iSource, ORecord iRecord, final String[] iFields) {
if (iSource.length == 0)
return iRecord;
if (iRecord == null)
iRecord = new ODocument();
else
checkTypeODocument(iRecord);

BytesContainer container = new BytesContainer(iSource);
container.skip(1);
serializerByVersion[iSource[0]].deserialize((ODocument) iRecord, container);//ArrayIndexOutOfBoundsException
ORecordInternal.clearSource(iRecord);
return iRecord;

}

[henryzhao81]

If I create brand new DB, in console, when i open database, String serializeName = getStorage().getConfiguration().getRecordSerializer(); //serializeName is ORecordSerializerBinary, I do not see any arrayIndexOutofBound exception in server log

But if I use existing DB and create those classes, serializeName will be ORecordDocument2csv, the issue will happen. Do we support backward compatible?

[tglman]

hi @henryzhao81,

following the stack trace i think i identified the problem and fixed it, working to write a test case.
I'll close this when i've the test case.

bye

[henryzhao81]

hi Emanuel, i think the issue have not been complete fixed.
Following are my test case, execute on console not studio:

create class tuser extends OTriggered;
create class tcontact extends OTriggered;
create property tuser.name STRING;
create property tuser.contact LINK tcontact;
create property tcontact.name STRING;
create property tcontact.owner LINK tuser;
insert into tuser set name = 'test1'; //#119:0
insert into tcontact set name = ‘contact1’; //#120:0

alter class tuser custom onAfterRead = null;
alter class tuser custom onBeforeUpdate = null;
create property tcontact.updateTime DATETIME;
update #119:0 set contact = #120:0;
update #120:0 set owner = #119:0;
alter class tcontact custom onAfterRead = com.hook.UserHandler.afterReadContact;

alter class tcontact custom onBeforeUpdate = com.hook.UserHandler.beforeUpdateContact;

Following are the implementation of 2 functions, will be triggered by execute: select from #120:0

//on afterReadContact, which do not change document in database. It temporarily set updateTime field as current timestamp and show on the query result

public String afterReadContact(ODocument doc) {
    doc.field("updateTime", System.currentTimeMillis());
    return RESULT.RECORD_NOT_CHANGED.name();
}

//Looks like beforeUpdate hook will be triggered even i declare RECORD_NOT_CHANGED in read hook. In beforeUpdateContact, it loads itself from database as origin document, try to compare with updated document. But at this time, server throw ArrayIndexOutofBoundException.

public String beforeUpdateContact(ODocument doc) {
    ODocument loaded = ODatabaseRecordThreadLocal.INSTANCE.get().load(doc.getIdentity(), "*:0", true);
    return RESULT.RECORD_CHANGED.name();
}

[henryzhao81]

Change afterReadContact, add one line to unsetDirty, then beforeUpdateContact will not be triggered, and exception not popup, please let me know if it is the right approach.

public String afterReadContact(ODocument doc) {
    doc.field("updateTime", System.currentTimeMillis());
    ORecordInternal.unsetDirty(doc);
    return RESULT.RECORD_NOT_CHANGED.name();
}

[tglman]

hi @henryzhao81,
if you set a field in the document for the database that document is changed and it need to be saved, but i understand your case, so is correct to call unsetDirty for your case.

if this workaround solve your case i'll leave this open with lower priority, for improve the Hook managment, the exception happen because you have in the hook context the network serializer instead of the database serializer.

[tglman]

This is fixed for the main use case, in case of document change inside an hook on the server side the problem is still there, but with clear workaround, so i'm going to move this forward with less priority

[tglman]

after last changes with the dirty manager, and the remove of the ONetworkThreadLocalSerializer this should not happen anymore, closing