504 error when browsing directories in large repo

[bagasme]

• Gitea version (or commit ref): main (commit 8fa3bbc, built with Go 1.16.4)
• Git version: 2.31.1
• Operating system: Debian 10
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No (not relevant)
• Log gist: log

Description

I have a copy of GCC repo on my Gitea instance on LXD container.
I'm trying to browse gcc/ subdirectory, but it always returned 504 error, presumably because the subdirectory contained 946 entries (directories and files). I used to be able to access the subdirectory sometimes in the past.

Inspecting using Developer Tools on Firefox, I noticed that the HTTP request during the attempt was initiated by BrowserTabChild.jsm script, but slow response time from server (1 minute) might cause the error.

I'm running Gitea behind NGINX as reverse proxy, and the relevant NGINX config is:

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

Screenshots

[not relevant]

[zeripath]

This is a timeout.

Please test #16059 to see if this solves your problem - if not try #16063 or try building with gogit enabled instead.

[bagasme]

@zeripath I don't see any difference when building with TAGS="bindata gogit"; still returned 504.

And I can't test #16063 because it means downgrading database, and I only have one Gitea instance (as production), and I don't want to spin up another instance because of memory constraints on my computer (I only have 4 GB RAM).

[6543]

@zeripath on first load of subfolder gcc it took 169161ms to load! (dbms: mysql; version 0966349; no-gogit)

[6543]

@bagasme for now you just have to raise git timeout limits via config

[richmahn]

I have this same problem, at least getting a 504 when working with a big repo. Our prod server, running v1.14.6 of Gitea is NOT having this problem, but every time I populate our develop server running Gitea v1.15.0 with the same DB and filebase, if I go to https://develop.door43.org/unfoldingWord/en_tn it will sometimes timeout. It doesn't do it every time, sometimes it does show the page, sometimes it doesn't. I have debugged and I found where it is stopping is when it tries to do the Discard(1) when b.n is 0 for the README file (which in my problematic repo has a size of more than 4096 bytes). It is failing here:

https://github.com/go-gitea/gitea/blob/main/modules/git/blob_nogogit.go#L126

I don't know if this is the same as the gcc repo being used, as it has a smaller README file. @zeripath I am not sure why the Discard(1) never returns, not even if I set my TIMEOUT really high. This page shows that it should return an error when there are no bytes in the buffer, which I assume is the case when the code reaches the line linked above? See https://pkg.go.dev/bufio#Reader.Discard

[richmahn]

Oh wait, maybe @6543 fixed this with #16899? Does that sound like the issue that was fixed @6543 ?

[richmahn]

Hmm, I am using the latest code from release/v1.15 and does have the DataAsync fix. Yet do wonder about that Discard(1) call when there are no bytes to read...

[richmahn]

Ok, looks like the above fix fixes the Close() call that seemed to previously be hanging.

[richmahn]

Not sure if this is/was the same problem that the OP repo was having though. @bagasme you may want to try the latest of of the development branch (main) or the release/v1.15 branch.

[zeripath]

@richmahn is the hold up definitely occurring at blob.Close()? Two options for what is going on here :

1. A miscount with the buffering code:

echo $SHA | git cat-file --batch 

will return a header line which read by ReadBatchLine followed by the blob content + a trailing '\n'.

But I can't see a miscount there.

2. Some buffering or slow flushing?

This may be similar to the reasons why windows users seem to have trouble with the native backend. On #16773 I've had to use an os.Pipe and explicitly Sync the stdinWriter - maybe we need something like that for the stdoutReader?

Looking at the code for go's exec it appears that using io.Pipe doesn't avoid an os.Pipe from being created so we can get get rid of the io.Pipe.

---

I'm sorry that there appears to be a bug here - I'm really kinda stumped as to what's going on though. Perhaps the fixes from @6543 will have fixed the issue. What I'll do it is clone the repo and see if I can make it happen locally.

---

Some questions:

1. What OS are you running on?
2. What version of git?

---

Not sure if this is/was the same problem that the OP repo was having though. @bagasme you may want to try the latest of of the development branch (main) or the release/v1.15 branch.

Yeah I think this issue is regarding something different and is to do with the old algorithm. Those problems should be a lot less frequent now.

[richmahn]

@zeripath Running on an AWS server:

$ cat /proc/version
Linux version 4.14.241-184.433.amzn2.x86_64 (mockbuild@ip-10-0-1-230) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-13) (GCC)) #1 SMP Wed Aug 4 14:35:15 UTC 2021
$ git version
git version 2.32.0

So no, not Windows. It was occurring right there at the .Close() call I put Printf()s everywhere. It was the Close() for when the repo home page reads the README file and was trying to close it in the defer. I increased my timeout in the config to 600 and it still never returned for the page. However, if I refreshed the page, it would sometimes work.

[zeripath]

Ok, looks like the above fix fixes the Close() call that seemed to previously be hanging.

Does this imply that the problem is fixed now?

[zeripath]

Should be fixed by #16467

[richmahn]

I just now today found what was going on with my repos and my 504. I am working from a fork of Gitea for my company which adds features, one being to look at a manifest.yaml file in the root directory of a repo and read it like Gitea does the README.md file. I found since Gitea change to using a CatFileReader, my YAML file reader was closing it when it did a ReadAll(). I just today fixed it to use a MultiReader and leave the CatFileReader open for other batch calls. So it was an error in my own code.