tableInfoStore should be scoped to transaction

[tie]

Currently database.Database holds a reference to tableInfoStore that is shared between all transactions which, among other things, may create/rename/alter/drop tables. This violates the transaction isolation.

Make sure INSERT is indeed isolated.

package main

import (
	"github.com/dgraph-io/badger/v2"
	"github.com/genjidb/genji"
	"github.com/genjidb/genji/engine/badgerengine"
)

func main() {
	ng, err := badgerengine.NewEngine(badger.DefaultOptions("").WithInMemory(true))
	if err != nil { panic(err) }
	db, err := genji.New(ng)
	if err != nil { panic(err) }

	err = db.Exec("CREATE TABLE tb (id INTEGER PRIMARY KEY)")
	if err != nil { panic(err) }

	// Does panic with "duplicate document" error unless committed.
	for i := 0; i < 2; i++ {
		tx, err := db.Begin(true)
		if err != nil { panic(err) }
		defer tx.Rollback()
		err = tx.Exec("INSERT INTO tb (id) VALUES (?)", 42)
		if err != nil { panic(err) }
	}
}

Make sure that CREATE INDEX is indeed isolated.

package main

import (
	"github.com/dgraph-io/badger/v2"
	"github.com/genjidb/genji"
	"github.com/genjidb/genji/engine/badgerengine"
)

func main() {
	ng, err := badgerengine.NewEngine(badger.DefaultOptions("").WithInMemory(true))
	if err != nil { panic(err) }
	db, err := genji.New(ng)
	if err != nil { panic(err) }

	err = db.Exec("CREATE TABLE tb")
	if err != nil { panic(err) }

	// Does panic with "index already exists" error unless committed.
	for i := 0; i < 2; i++ {
		tx, err := db.Begin(true)
		if err != nil { panic(err) }
		err = tx.Exec("CREATE UNIQUE INDEX idx ON tb(id)")
		if err != nil { panic(err) }
	}
}

Reproduce the bug. Panics with "table already exists" error.

package main

import (
	"github.com/dgraph-io/badger/v2"
	"github.com/genjidb/genji"
	"github.com/genjidb/genji/engine/badgerengine"
)

func main() {
	ng, err := badgerengine.NewEngine(badger.DefaultOptions("").WithInMemory(true))
	if err != nil { panic(err) }
	db, err := genji.New(ng)
	if err != nil { panic(err) }

	// We never commit the transaction, so these changes should be isolated
	// from other concurrent (but not necessarily parallel) transactions.
	for i := 0; i < 2; i++ {
		tx, err := db.Begin(true)
		if err != nil { panic(err) }
		err = tx.Exec("CREATE TABLE tb")
		if err != nil { panic(err) }
	}
}

Tangentially related to #210 since it needs concurrent transactions.

[tie]

Moving tableInfoStore to transaction also removes the need for transaction IDs.

[asdine]

Actually, tableInfoStore has been shared between transactions on purpose because it's the first step towards implementing different levels of transaction isolation.

To my understanding, transaction isolation and concurrent access management are slightly different.

When two transactions are trying to do the following actions:

• Creating / modifying the same table (the table itself, not its content)
• Creating a document with the same primary key
• Updating the same document, or reading a document that is currently updated by another transaction.

the database should lock that resource for the first transaction and block the other transaction until the first one is either rolled back or commited (ideally with a timeout).
We currently don't have a locking mechanism, which makes the behavior odd, while not incorrect.

It means that yes, transactions can be aware of other transactions doings on locked resources.
To understand that, run two mysql or postgres clients on two separate shells and do the following:

- shell1> BEGIN;
- shell2> BEGIN;
- shell1> CREATE TABLE foo(a INT PRIMARY KEY);
- shell2> CREATE TABLE foo(a INT PRIMARY KEY);
 this will block, until you either commit or rollback the first transaction

The same goes for an update for example, when one transaction does an update on a given document and the other tries to read that document.
That's why they have things like SELECT ... FOR UPDATE and other explicit locking mechanisms.

What is currently missing with Genji is that locking mechanism. This is obviously not important for Bolt and the inmemory engine, which, like SQLite, only allow one concurrent write transaction.
But for Badger, locking instead of returning an error would be best.

[tie]

My understanding is that ACID/MVCC/locking/etc properties in Genji are expected to be implemented by K/V engines, and not at the SQL layer, so I think we need a LockStores and/or LockKeys methods along with some TxOptions in engine then.

[tie]

this will block, until you either commit or rollback the first transaction

Since BadgerDB uses snapshots isolation, I wouldn’t expect Genji to block in this case.

[tie]

Another issue with current implementation of tableInfoStore is that it’s in-memory only, so we can’t keep it up-to-date if e.g. some other process modifies table info.

[asdine]

My understanding is that ACID/MVCC/locking/etc properties in Genji are expected to be implemented by K/V engines, and not at the SQL layer

I guess it depends on the direction the project is taking, which can be summed up with the following question:

Do we want Genji to provide a unified experience but with specialized engines or do we want to delegate 100% of transaction management and complexity to the engines?

First option Pros:

• Add unified implicit and explicit lock management (i.e. row locks, SELECT ... FOR UPDATE, SKIP LOCKED etc.)
• Add different transaction isolation levels (i.e. Read committed, read uncommitted, serializable, etc.)
• Support lots of different KV stores. Genji would provide the MVCC layer. Engines wouldn't need to support transactions but only dealing with multiple updates atomically (like Pebble does, but Bolt and Badger already naturally do it. Basically this Store#Update(...kvPair) error).

First option Cons:

• Implementing MVCC correctly is difficult
• Database files can't be shared between processes (I don't really think this is a con, there are protocols for that)
• Genji may not use 100% of engine implementation capabilities

Second option Pros:

• Some engines already have MVCC baked-in and it's working well
• Genji is just a SQL layer + Document layer and only focuses on that

Second option Cons:

• Experience depends highly on the engines
• Implementing an engine with all the properties required by Genji is complex
• No transaction management options, no locking capabilities

Genji started as the second option but with the ambition of becoming the first one. I don't think it should only be a layer otherwise there will never be a good reason to be using Genji instead of SQLite for example (besides the fact that Genji doesn't use CGO).

Regarding the issue at hand, and to explain why I'm talking about this, it was done this way as the first step towards the first option.

Another issue with current implementation of tableInfoStore is that it’s in-memory only, so we can’t keep it up-to-date if e.g. some other process modifies table info.

I don't think this is an issue because I don't think this is a good idea to let other processes (local or remote) modify the same data without some kind of control.

[tie]

I don't think it should only be a layer otherwise there will never be a good reason to be using Genji instead of SQLite for example

@asdine which features do you think we need to implement to become a viable SQLite alternative?

Regarding the second option cons, I’m not sure the last two points are exactly true.

• Implementing an engine with all the properties required by Genji is complex

  The issue comes down to distributing the complexity, and since SQL and document layers are already complex enough, I think it’s fair to delegate MVCC and locking to engine implementations.

• No transaction management options, no locking capabilities

  Is there something we can’t achieve by adding LockKeys method and allowing using engine.Store outside of transaction?

[asdine]

which features do you think we need to implement to become a viable SQLite alternative?

There are lots of missing SQL features that would make Genji a true SQLite alternative, but I'll focus on what I think are the most important ones:

• JOINs with good query planning: Implementing JOINs is not that complex per se, but choosing the right query plan for a given JOIN is very difficult. If we manage to implement a decent cost-based optimizer I'd be more than happy.
• Performance: This is not a nice to have but an important feature. Until now, I've tried my best not to optimize anything (well except for some low hanging fruits, as long as they don't get in the way), but rather design with scaling in mind. Once we'll lock the database file format, we'll be able to invest in performance measurement and improvement.
• Locks and transaction modes: SQLite only supports the "serializable" mode, i.e. one concurrent writer / multiple concurrent readers, pretty much the same as what the Bolt and Memory engines propose. However, Postgres and others provide more transaction modes as well as very nice APIs for explicit and implicit locking. Having that would allow Genji to truly shine when concurrency is involved. Not only in terms of performance with concurrent writers/reads but also in terms of useful capabilities, which would make it a true SQLite alternative.
• Checks: Being a document database doesn't mean we can't have control over the data we are inserting. I'd like to add more power to the schema definitions by implementing CHECKs.
• CTEs / Triggers, etc.
• Multi-platform: I'd like Genji to work for Python, Ruby, etc. but also on the browser (see Genji.js).

The issue comes down to distributing the complexity, and since SQL and document layers are already complex enough, I think it’s fair to delegate MVCC and locking to engine implementations.

There is a huge problem with that, which can basically be summed up by this code:

package main

import (
	"fmt"
	"log"
	"net/http"
	"sync/atomic"

	"github.com/dgraph-io/badger/v2"
	"github.com/genjidb/genji"
	"github.com/genjidb/genji/document"
	"github.com/genjidb/genji/engine/badgerengine"
)

func main() {
	ng, err := badgerengine.NewEngine(badger.DefaultOptions("foo/"))
	if err != nil {
		panic(err)
	}
	db, err := genji.New(ng)
	if err != nil {
		panic(err)
	}
	defer db.Close()

	err = db.Exec("CREATE TABLE foo; INSERT INTO foo (a) VALUES (0)")
	if err != nil {
		panic(err)
	}

	var count int64

	http.HandleFunc("/", func(rw http.ResponseWriter, r *http.Request) {
		reqID := atomic.AddInt64(&count, 1)

		log.Printf("Handling request #%d\n", reqID)
		defer fmt.Printf("Finished handling request #%d\n", reqID)

		err = db.Update(func(tx *genji.Tx) error {
			d, err := tx.QueryDocument("SELECT a FROM foo")
			if err != nil {
				return err
			}
			var a int
			err = document.Scan(d, &a)
			if err != nil {
				return err
			}
			return tx.Exec("UPDATE foo SET a = ?", a+1)
		})
		if err != nil {
			log.Println(err)
			rw.WriteHeader(http.StatusInternalServerError)
		}
	})

	http.ListenAndServe(":8080", nil)
}

If you run concurrent queries against this server, you have this:

badger 2020/11/02 23:21:33 INFO: All 1 tables opened in 0s
badger 2020/11/02 23:21:33 INFO: Replaying file id: 0 at offset: 17270
badger 2020/11/02 23:21:33 INFO: Replay took: 731.8µs
badger 2020/11/02 23:21:33 DEBUG: Value log discard stats empty
2020/11/02 23:21:37 Handling request #1
2020/11/02 23:21:37 Handling request #2
2020/11/02 23:21:37 Handling request #4
2020/11/02 23:21:37 Handling request #3
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #2
2020/11/02 23:21:37 Handling request #5
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #4
2020/11/02 23:21:37 Handling request #6
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #1
2020/11/02 23:21:37 Handling request #7
2020/11/02 23:21:37 Handling request #8
2020/11/02 23:21:37 Handling request #9
2020/11/02 23:21:37 Handling request #10
2020/11/02 23:21:37 Handling request #11
2020/11/02 23:21:37 Handling request #12
2020/11/02 23:21:37 Handling request #13
2020/11/02 23:21:37 Handling request #14
2020/11/02 23:21:37 Handling request #15
2020/11/02 23:21:37 Handling request #16
2020/11/02 23:21:37 Handling request #17
2020/11/02 23:21:37 Handling request #18
2020/11/02 23:21:37 Handling request #19
2020/11/02 23:21:37 Handling request #20
2020/11/02 23:21:37 Handling request #21
2020/11/02 23:21:37 Handling request #22
2020/11/02 23:21:37 Handling request #23
2020/11/02 23:21:37 Handling request #24
2020/11/02 23:21:37 Handling request #26
2020/11/02 23:21:37 Handling request #25
2020/11/02 23:21:37 Handling request #27
2020/11/02 23:21:37 Handling request #28
2020/11/02 23:21:37 Handling request #29
2020/11/02 23:21:37 Handling request #30
2020/11/02 23:21:37 Handling request #31
2020/11/02 23:21:37 Handling request #32
2020/11/02 23:21:37 Handling request #33
2020/11/02 23:21:37 Handling request #34
2020/11/02 23:21:37 Handling request #35
2020/11/02 23:21:37 Handling request #37
2020/11/02 23:21:37 Handling request #36
2020/11/02 23:21:37 Handling request #38
2020/11/02 23:21:37 Handling request #40
2020/11/02 23:21:37 Handling request #39
2020/11/02 23:21:37 Handling request #41
2020/11/02 23:21:37 Handling request #42
2020/11/02 23:21:37 Handling request #43
2020/11/02 23:21:37 Handling request #44
2020/11/02 23:21:37 Handling request #45
2020/11/02 23:21:37 Handling request #46
2020/11/02 23:21:37 Handling request #47
2020/11/02 23:21:37 Handling request #48
2020/11/02 23:21:37 Handling request #49
2020/11/02 23:21:37 Handling request #50
2020/11/02 23:21:37 Handling request #51
2020/11/02 23:21:37 Handling request #52
2020/11/02 23:21:37 Handling request #53
Finished handling request #3
2020/11/02 23:21:37 Handling request #54
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #5
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #7
2020/11/02 23:21:37 Handling request #55
2020/11/02 23:21:37 Handling request #56
Finished handling request #6
2020/11/02 23:21:37 Handling request #57
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #57
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #9
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #10
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #11
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #12
2020/11/02 23:21:37 Handling request #59
2020/11/02 23:21:37 Handling request #58
2020/11/02 23:21:37 Handling request #60
2020/11/02 23:21:37 Handling request #61
2020/11/02 23:21:37 Handling request #62
Finished handling request #8
2020/11/02 23:21:37 Handling request #63
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #14
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #63
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #15
2020/11/02 23:21:37 Handling request #64
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #16
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #17
2020/11/02 23:21:37 Handling request #65
2020/11/02 23:21:37 Handling request #67
2020/11/02 23:21:37 Handling request #66
2020/11/02 23:21:37 Handling request #68
Finished handling request #13
2020/11/02 23:21:37 Handling request #69
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #20
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #19
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #69
2020/11/02 23:21:37 Handling request #70
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #21
2020/11/02 23:21:37 Handling request #71
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #22
2020/11/02 23:21:37 Handling request #72
2020/11/02 23:21:37 Handling request #73
2020/11/02 23:21:37 Handling request #74
Finished handling request #18
2020/11/02 23:21:37 Handling request #75
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #24
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #26
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #25
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #27
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #28
2020/11/02 23:21:37 Handling request #76
2020/11/02 23:21:37 Handling request #77
2020/11/02 23:21:37 Handling request #78
2020/11/02 23:21:37 Handling request #79
2020/11/02 23:21:37 Handling request #80
Finished handling request #23
2020/11/02 23:21:37 Handling request #81
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #30
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #31
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #32
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #81
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #33
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #34
2020/11/02 23:21:37 Handling request #83
2020/11/02 23:21:37 Handling request #84
2020/11/02 23:21:37 Handling request #82
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #35
2020/11/02 23:21:37 Handling request #85
2020/11/02 23:21:37 Handling request #86
2020/11/02 23:21:37 Handling request #87
2020/11/02 23:21:37 Handling request #88
Finished handling request #29
2020/11/02 23:21:37 Handling request #89
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #89
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #38
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #39
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #40
2020/11/02 23:21:37 Handling request #90
2020/11/02 23:21:37 Handling request #91
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #41
2020/11/02 23:21:37 Handling request #92
2020/11/02 23:21:37 Handling request #93
2020/11/02 23:21:37 Handling request #94
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #37
2020/11/02 23:21:37 Handling request #95
Finished handling request #36
2020/11/02 23:21:37 Handling request #96
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #42
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #45
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #44
2020/11/02 23:21:37 Handling request #97
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #96
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #46
2020/11/02 23:21:37 Handling request #98
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #47
2020/11/02 23:21:37 Handling request #99
2020/11/02 23:21:37 Handling request #100
2020/11/02 23:21:37 Handling request #101
2020/11/02 23:21:37 Handling request #102
Finished handling request #43
2020/11/02 23:21:37 Handling request #103
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #103
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #50
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #49
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #51
2020/11/02 23:21:37 Handling request #104
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #52
2020/11/02 23:21:37 Handling request #105
2020/11/02 23:21:37 Handling request #106
2020/11/02 23:21:37 Handling request #107
2020/11/02 23:21:37 Handling request #108
Finished handling request #48
2020/11/02 23:21:37 Handling request #109
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #109
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #54
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #53
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #56
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #59
2020/11/02 23:21:37 Handling request #110
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #60
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #58
2020/11/02 23:21:37 Handling request #111
2020/11/02 23:21:37 Handling request #112
2020/11/02 23:21:37 Handling request #113
2020/11/02 23:21:37 Handling request #114
2020/11/02 23:21:37 Handling request #115
2020/11/02 23:21:37 Handling request #116
Finished handling request #55
2020/11/02 23:21:37 Handling request #117
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #62
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #117
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #64
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #65
2020/11/02 23:21:37 Handling request #118
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #67
2020/11/02 23:21:37 Handling request #119
2020/11/02 23:21:37 Handling request #120
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #66
2020/11/02 23:21:37 Handling request #121
2020/11/02 23:21:37 Handling request #122
2020/11/02 23:21:37 Handling request #123
Finished handling request #61
2020/11/02 23:21:37 Handling request #124
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #70
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #124
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #71
2020/11/02 23:21:37 Transaction Conflict. Please retry
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #73
Finished handling request #72
2020/11/02 23:21:37 Handling request #126
2020/11/02 23:21:37 Handling request #125
2020/11/02 23:21:37 Handling request #127
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #74
2020/11/02 23:21:37 Handling request #129
2020/11/02 23:21:37 Handling request #128
2020/11/02 23:21:37 Handling request #130
Finished handling request #68
2020/11/02 23:21:37 Handling request #131
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #76
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #77
2020/11/02 23:21:37 Handling request #132
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #78
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #79
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #80
2020/11/02 23:21:37 Handling request #133
2020/11/02 23:21:37 Handling request #134
2020/11/02 23:21:37 Handling request #135
2020/11/02 23:21:37 Handling request #136
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #83
2020/11/02 23:21:37 Handling request #137
Finished handling request #75
2020/11/02 23:21:37 Handling request #138
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #82
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #85
2020/11/02 23:21:37 Transaction Conflict. Please retry
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #86
Finished handling request #138
2020/11/02 23:21:37 Handling request #139
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #87
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #88
2020/11/02 23:21:37 Handling request #141
2020/11/02 23:21:37 Handling request #140
2020/11/02 23:21:37 Handling request #142
2020/11/02 23:21:37 Handling request #143
2020/11/02 23:21:37 Handling request #144
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #90
Finished handling request #84
2020/11/02 23:21:37 Handling request #145
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #92
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #93
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #145
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #95
2020/11/02 23:21:37 Handling request #146
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #94
2020/11/02 23:21:37 Handling request #147
2020/11/02 23:21:37 Handling request #148
2020/11/02 23:21:37 Handling request #149
2020/11/02 23:21:37 Handling request #150
Finished handling request #91
2020/11/02 23:21:37 Handling request #151
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #98
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #99
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #100
2020/11/02 23:21:37 Handling request #152
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #151
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #102
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #104
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #101
2020/11/02 23:21:37 Handling request #153
2020/11/02 23:21:37 Handling request #155
2020/11/02 23:21:37 Handling request #154
2020/11/02 23:21:37 Handling request #156
2020/11/02 23:21:37 Handling request #157
2020/11/02 23:21:37 Handling request #158
Finished handling request #97
2020/11/02 23:21:37 Handling request #159
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #107
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #106
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #108
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #110
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #111
2020/11/02 23:21:37 Handling request #160
2020/11/02 23:21:37 Handling request #161
2020/11/02 23:21:37 Handling request #162
2020/11/02 23:21:37 Handling request #163
2020/11/02 23:21:37 Handling request #164
Finished handling request #105
2020/11/02 23:21:37 Handling request #165
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #165
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #113
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #115
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #114
2020/11/02 23:21:37 Handling request #166
2020/11/02 23:21:37 Handling request #167
2020/11/02 23:21:37 Handling request #169
2020/11/02 23:21:37 Handling request #168
Finished handling request #112
2020/11/02 23:21:37 Handling request #170
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #118
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #120
2020/11/02 23:21:37 Handling request #171
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #121
2020/11/02 23:21:37 Handling request #172
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #119
Finished handling request #116
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #122
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #123
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #127
2020/11/02 23:21:37 Handling request #174
2020/11/02 23:21:37 Handling request #173
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #129
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #125
2020/11/02 23:21:37 Handling request #175
2020/11/02 23:21:37 Handling request #176
Finished handling request #126
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #130
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #131
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #132
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #133
2020/11/02 23:21:37 Handling request #178
2020/11/02 23:21:37 Transaction Conflict. Please retry
Finished handling request #135

Delegating this kind of issue to the engine might create bad surprises for users and a mixed experience.
Also, forcing engines to implement something like LockKeys would duplicate something that could be done once by Genji.

Etcd implements their own MVCC layer on top of Bolt, I wouldn't be surprised if Dgraph had their own locking mechanism over Badger. Cockroach only uses serializable atomic operations in Pebble and deal with MVCC itself.
I really think we can do something in Genji as well without sacrificing what makes Badger and Bolt great.

[asdine]

Closing this as the issue no longer exists with #322. Once the project will be more mature I'll reopen a separate discussion on the subject.
Please reopen if you have any question