Experiment: replacing modernc.org/sqlite with github.com/ncruces/go-sqlite3 in gotosocial #78
Replies: 7 comments 29 replies
-
Not sure this is the best forum for this discussion (rather than something under Any specific questions/concerns you might have for me, please make them more explicit, 'cause at first glance I don't think I can contribute much to the discussion. I'll still open a few threads for things that come up. |
Beta Was this translation helpful? Give feedback.
-
Busy handler not being context awareThe philosophy of this driver is that the busy handler is the maximum you'll ever want to wait. Queries should still bail out early if a context is cancelled. If this doesn't happen, I consider it a bug. Acting on context done isn't immediate (it's hard to interrupt file IO, and I don't even try). Also, the default timeout handler sleeps increments of 100ms, so you can always miss cancellation by at least that much. But I don't think you should ever miss context cancellation by more than (e.g.) 250ms. |
Beta Was this translation helpful? Give feedback.
-
ExecContext is weird
Everything else (including This is my attempt at enforcing some correctness properties. |
Beta Was this translation helpful? Give feedback.
-
Error translationIn general I'd advising going with idiomatic Go and using this pattern for error translation: Obviously, that's just a matter of style and you may prefer to keep it how you did it. This is just to say that this driver tries to, as much as possible, respect Go idioms in error handling. |
Beta Was this translation helpful? Give feedback.
-
Update on the situation: memory leak resolved. It was me adding All (bar two things) seems to be working as expected now, with less code than we had previously for the modernc.org/sqlite implementation, since the busy timeout handler can now successfully be relied on! The only 2 things remaining are a particularly slow running query (again this could be an issue of our own making, no fingers being pointed here), which we don't experience under modernc.org/sqlite, and our tests seem to fail using |
Beta Was this translation helpful? Give feedback.
-
So, question: do you see this getting some traction? Any blockers remaining? |
Beta Was this translation helpful? Give feedback.
-
So, this got merged in superseriousbusiness/gotosocial#2863! If you're ever serious about releasing a FreeBSD binary with this enabled, something needs to be done about #85. Otherwise, what you have should be good for both Linux and macOS. |
Beta Was this translation helpful? Give feedback.
-
Just starting an open discussion for this so I can document things as I experience them, and serves as a useful space for any related discussion to this Go sqlite implementation. I'm not 100% sure this is something we'd look at PR'ing to main, but there's definitely a very spirited discussion between our core devs on matrix :D.
So far there's a working build replacing modernc.org/sqlite with github.com/ncruces/go-sqlite3 here: https://github.com/NyaaaWhatsUpDoc/gotosocial/tree/experiment/use-wasm-sqlite3
I might look at upstreaming the funky little driver wrapper I made (see
./internal/db/sqlite
) as it makes wrapping the driver yourself easier if you need to, and i've kept our standardretryOnBusy()
handler in place over using the built-in busy-timeout handler both due to it not being context aware (though change should be easy enough, i have a change for y'all ready if you want it upstream) and as i'm not sure if they 100% replace one-another.This working build runs gotosocial as expected, with decent performance, minus a memory leak (which doesn't occur on current main) which causes the application to get oom killed every few hours. With the fact we're using
github.com/uptrace/bun
(which in turn wrapsdatabase/sql
, which then uses our busy / error handling wrapper...) it's hard to narrow down exactly where it's being introduced, but i shall keep looking!Anyways, i hope at least someone here in this discussion board finds this interesting! Either way i'll probably continue to ramble with my progress as i make it :)
Beta Was this translation helpful? Give feedback.
All reactions