Exception thrown when importing backup from 3.0 to 3.1

[madmac2501]

OrientDB Version: 3.1.1

Java Version: 1.8

OS: Linux

Expected behavior

Database import process finish

Actual behavior

An exception is thrown in the import process from 3.0 to 3.1 and the import stops

Steps to reproduce

Run these commands in the console

CONNECT plocal:../databases/xxx admin admin
EXPORT DATABASE /tmp/xxx

CREATE DATABASE plocal:../databases/xxx
IMPORT DATABASE /tmp/xxx.json.gz

This is the log from the import database

Importing indexes ...

• Index 'InvestigatedUserTo_out_in'...OK
• Index 'accountDomain'...OK
• Index 'AlertedSystemTo_out_in'...OK
• Index 'System.hostname'...OK
• Index 'Alert.alertId'...OK
• Index 'AccessIn_out_in'...Error on database import happened just before line 0, column -1290445527
  com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record GraphDB implementation [moved] #60:307: found duplicated key 'OCompositeKey{keys=[ORDER BY in queries [moved] #47:6504, Add direct conversion of ODocument in JSON [moved] #16:38589]}' in index 'AccessIn_out_in' previously assigned to the record Enhance the base-build.xml to create all required directories [moved] #58:489940
  DB name="xxx" INDEX=AccessIn_out_in RID=Enhance the base-build.xml to create all required directories [moved] #58:489940
  at com.orientechnologies.orient.core.index.OIndexUnique.lambda$new$0(OIndexUnique.java:51)
  at com.orientechnologies.orient.core.storage.index.hashindex.local.v3.OLocalHashTableV3.doPut(OLocalHashTableV3.java:1403)
  at com.orientechnologies.orient.core.storage.index.hashindex.local.v3.OLocalHashTableV3.lambda$put$3(OLocalHashTableV3.java:1323)
  at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:267)
  at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideComponentOperation(OAtomicOperationsManager.java:259)
  at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.calculateInsideComponentOperation(ODurableComponent.java:96)
  at com.orientechnologies.orient.core.storage.index.hashindex.local.v3.OLocalHashTableV3.put(OLocalHashTableV3.java:1301)
  at com.orientechnologies.orient.core.storage.index.hashindex.local.v3.OLocalHashTableV3.validatedPut(OLocalHashTableV3.java:281)
  at com.orientechnologies.orient.core.storage.index.engine.OHashTableIndexEngine.validatedPut(OHashTableIndexEngine.java:264)
  at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:3630)
  at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.lambda$validatedPutIndexValue$19(OAbstractPaginatedStorage.java:3599)
  at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.calculateInsideAtomicOperation(OAtomicOperationsManager.java:170)
  at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:3596)
  at com.orientechnologies.orient.core.index.OIndexUnique.doPut(OIndexUnique.java:113)
  at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:93)
  at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:36)
  at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:947)
  at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1052)
  at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:586)
  at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:260)
  at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:120)
  at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:46)
  at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:200)
  at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importIndexes(ODatabaseImport.java:1604)
  at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:214)
  at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2555)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:445)
  at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:283)
  at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:144)
  at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:160)

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'xxx' from file: /tmp/xxx.json.gz

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #60:307: found duplicated key 'OCompositeKey{keys=[#47:6504, #16:38589]}' in index 'AccessIn_out_in' previously assigned to the record #58:489940
DB name="xxx" INDEX=AccessIn_out_in RID=#58:489940

[andrii0lomakin]

Hi @madmac2501 , I suppose I would need the initial JSON to reproduce the issue, could you send it to me?

[madmac2501]

I'm sorry but I cannot provide access to this specific backup. Let me know if this is enough, if not I can also try to create a fake dataset.

After this error and the new database started in OrientDB 3.1.1 what I found was that the problem was that there were duplicated records as the message exception show, these duplicated records broke an index constraint. I manually deleted the duplicated records and then I was able to create the indexes again.

The index is created in an edge class to enforce uniqueness in an edge class. This is the index definition

CREATE INDEX AccessIn_out_in IF NOT EXISTS ON AccessIn (out, in) UNIQUE_HASH_INDEX;

[andrii0lomakin]

Hi @madmac2501 , the problem with duplicated records, did exist, but it should be absent in latest 3.0.x and 3.1.x versions.
I do not think that is the problem of import/export. It was bug in tx handling which caused given behaviour.

[madmac2501]

Great thanks!

could it be possible to handle this exception in the backup import? this way the backup import will not stop and the restore of other indexes will continue even if one index fails.

[andrii0lomakin]

But how then will you know which data to keep and which to delete, in such case that will be a quite random procedure? Do not think that it could be a solution for everyone.

[madmac2501]

Sorry I didn't explain well.

The current import behavior is that after the exception is thrown the import process stops and the rest of indexes are not imported even if they are valid. I think that the import behavior can be improved a little if the import process ignores the indexes that don't follow the constrains (like key uniqueness) by handling the exception and then the import continues with the rest of indexes.

[andrii0lomakin]

Ah I see, let me think about it.