Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin the version of cockroach for tests to a specific version #2971

Merged
merged 1 commit into from
Apr 18, 2023

Conversation

mhutchinson
Copy link
Contributor

./storage/crdb tests have started failing in GCP recently. Investigation shows that this is pulling the latest tagged version, which breaks hermetic builds. This PR is a test to see if using the previous version fixes the issue, and if so, I propose merging this to unblock things and then discussing with the community to look for better solutions.

./storage/crdb tests have started failing in GCP recently. Investigation shows that this is pulling the latest tagged version, which breaks hermetic builds. This PR is a test to see if using the previous version fixes the issue, and if so, I propose merging this to unblock things and then discussing with the community to look for better solutions.
@mhutchinson mhutchinson requested a review from a team as a code owner April 18, 2023 15:16
@mhutchinson
Copy link
Contributor Author

@JAORMX FYI

@mhutchinson
Copy link
Contributor Author

Failing snippet for those that want the gory details:

...
2023/04/17 22:02:29 waiting for download of /tmp/cockroach-22-2-8
2023/04/17 22:02:29 waiting for download of /tmp/cockroach-22-2-8
2023/04/17 22:02:29 Using automatically-downloaded binary: /tmp/cockroach-22-2-8
2023/04/17 22:02:30 executing: /tmp/cockroach-22-2-8 start-single-node --logtostderr --insecure --host=localhost --port=0 --http-port=0 --store=type=mem,size=0.20 --listening-url-file=/tmp/cockroach-testserver898281996/0/listen-url --external-io-dir=disabled
2023/04/17 22:02:30 process 10042 started. env=[COCKROACH_MAX_OFFSET=1ns COCKROACH_TRUST_CLIENT_PROVIDED_SQL_REMOTE_ADDR=true GID=0 HOME=/root PATH=/usr/local/go/bin:/go/bin:/testbase/protoc/bin:/usr/local/bin:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin USER=root UID=0]; cmd: /tmp/cockroach-22-2-8 start-single-node --logtostderr --insecure --host=localhost --port=0 --http-port=0 --store=type=mem,size=0.20 --listening-url-file=/tmp/cockroach-testserver898281996/0/listen-url --external-io-dir=disabled
W0417 22:02:31.310299    9780 tree_storage.go:86] failed verifying database connection: missing "=" after "&bogus*:::?" in connection info string"
W0417 22:02:44.230324    9780 tree_storage.go:363] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/732/2/1): "sql txn" meta={id=222e8833 key=/Table/616/1/7040283176384859160/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.00107903 epo=0 ts=1681768961.179821613,2 min=1681768956.877110741,0 seq=3000} lock=true stat=PENDING rts=1681768956.877110741,0 wto=false gul=1681768957.377110741,0, stack:
goroutine 84 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x72
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000185a20, {0x1184a10, 0xc00003e068})
	/workspace/storage/crdb/tree_storage.go:363 +0x305
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0x11861e8?, {0x1184a10, 0xc00003e068}, 0x0?, {0xc000834800, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/workspace/storage/crdb/log_storage.go:281 +0x1a8
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc00048f860)
	/workspace/storage/crdb/log_storage_test.go:246 +0x418
testing.tRunner(0xc00048f860, 0x106f730)
	/usr/local/go/src/testing/testing.go:1446 +0x217
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1493 +0x75e
--- FAIL: TestQueueLeavesDuplicateBigBatch (12.92s)
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/732/2/1): "sql txn" meta={id=222e8833 key=/Table/616/1/7040283176384859160/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.00107903 epo=0 ts=1681768961.179821613,2 min=1681768956.877110741,0 seq=3000} lock=true stat=PENDING rts=1681768956.877110741,0 wto=false gul=1681768957.377110741,0
FAIL
I0417 22:02:48.610299    9780 common_test.go:70] Cleaning up database for test TestCRDBAdminStorage
I0417 22:02:48.942298    9780 common_test.go:70] Cleaning up database for test TestNodeRoundTrip/store-4-read-4
...

@pavelkalinnikov in case this is interesting to you. No expectation or pressure :-)

@pav-kv
Copy link
Contributor

pav-kv commented Apr 20, 2023

@pavelkalinnikov in case this is interesting to you. No expectation or pressure :-)

@mhutchinson There are a couple of CRDB issues with similar errors:

See if your symptoms match, or transaction does anything similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants