Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot adopt more than 65535 repositories with PostgreSQL #19137

Closed
Mcdostone opened this issue Mar 19, 2022 · 0 comments · Fixed by #19140
Closed

Cannot adopt more than 65535 repositories with PostgreSQL #19137

Mcdostone opened this issue Mar 19, 2022 · 0 comments · Fixed by #19140
Labels
Milestone

Comments

@Mcdostone
Copy link

Mcdostone commented Mar 19, 2022

Gitea Version

1.16.4

Git Version

2.25.1

Operating System

Linux

How are you running Gitea?

Nothing fancy

curl -L "https://github.com/go-gitea/gitea/releases/download/v1.16.4/gitea-1.16.4-linux-amd64" --output /tmp/gitea
/tmp/gitea

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

Mar 18 21:54:55 linux- gitea[584975]: 2022/03/18 21:54:55 ...pi/v1/admin/adopt.go:51:ListUnadoptedRepositories() [E] InternalServerError: Count: pq: got 236972 parameters but PostgreSQL only supports 65535 parameters
Mar 18 21:54:55 linux gitea[584975]: 2022/03/18 21:54:55 router: completed GET /api/v1/admin/unadopted?page=1&limit=1000&pattern=&access_token=7feec787664a21a74aec942a71343b46d2d52360&sudo=&token= for 129.6.76.113:0, 500 Internal Server Error in 3618.9ms @ admin/adopt.go:20(admin.ListUnadoptedRepositories)

Description

The bug

I have more than 65535 unadopted repositories in gitea-repositories.
When I want to list them, I have a 500 HTTP status:

curl -X 'GET' 'http://localhost:3000/api/v1/admin/unadopted?page=1&limit=1000' -H 'authorization: Basic XXXXX'`

The Cause

After some voodoo debugging, I figured out the error happens in /services/repository/adopt.go:249, after the function call at services/repository/adopt.go:347

// Caller is services/repository/adopt.go:L347
// services/repository/adopt.go:249
       repos, _, err := models.GetUserRepositories(&models.SearchRepoOptions{
		Actor:   ctxUser,
		Private: true,
		ListOptions: db.ListOptions{
			Page:     1,
			PageSize: len(repoNamesToCheck),     // len(repoNamesToCheck) is greater than 65535
		}, LowerNames: repoNamesToCheck,             // 💀 the array contains more than 65535 items 
	})
	if err != nil {
		return err                                   // 🔥 Count: pq: got 236972 parameters but PostgreSQL only supports 65535 parameters"
	}
	```
@lunny lunny added the type/bug label Mar 19, 2022
zeripath added a commit to zeripath/gitea that referenced this issue Mar 19, 2022
The adoption page checks directories to see if they are repositories by querying the
db on a per user basis. This can lead to problems if a user has a large number of
repositories or putative repositories.

This PR changes the buffering to check the db in IterataeBufferSize batches instead.

Fix go-gitea#19137

Signed-off-by: Andrew Thornton <art27@cantab.net>
6543 pushed a commit that referenced this issue Mar 21, 2022
…eck (#19140)

The adoption page checks directories to see if they are repositories by querying the
db on a per user basis. This can lead to problems if a user has a large number of
repositories or putative repositories.

This PR changes the buffering to check the db in IterataeBufferSize batches instead.

Fix #19137

Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added a commit to zeripath/gitea that referenced this issue Mar 21, 2022
…eck (go-gitea#19140)

Backport go-gitea#19140

The adoption page checks directories to see if they are repositories by querying the
db on a per user basis. This can lead to problems if a user has a large number of
repositories or putative repositories.

This PR changes the buffering to check the db in IterataeBufferSize batches instead.

Fix go-gitea#19137

Signed-off-by: Andrew Thornton <art27@cantab.net>
lunny pushed a commit that referenced this issue Mar 22, 2022
…eck (#19140) (#19160)

Backport #19140

The adoption page checks directories to see if they are repositories by querying the
db on a per user basis. This can lead to problems if a user has a large number of
repositories or putative repositories.

This PR changes the buffering to check the db in IterataeBufferSize batches instead.

Fix #19137

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath added this to the 1.16.5 milestone Mar 23, 2022
Chianina pushed a commit to Chianina/gitea that referenced this issue Mar 28, 2022
…eck (go-gitea#19140)

The adoption page checks directories to see if they are repositories by querying the
db on a per user basis. This can lead to problems if a user has a large number of
repositories or putative repositories.

This PR changes the buffering to check the db in IterataeBufferSize batches instead.

Fix go-gitea#19137

Signed-off-by: Andrew Thornton <art27@cantab.net>
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants