chore!: increase pg connection defaults #21572
Conversation
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
spikecurtis
left a comment
There was a problem hiding this comment.
I'm pretty nervous about making this breaking change.
First of all, us testing in dogfood is just one team/use case; I'm not sure we should assume it generalizes.
Second of all, while increasing these values decrease measured wait time, do we have any evidence that actual humans are having an actually better experience since we made these changes to our deployment?
This is a particularly sharp edge to ship to customers because they'll install the new version on staging, where loads are probably low and everthing will be fine. Then they could ship to prod and run out of connections and have requests start failing.
|
Turns out we had a misunderstanding internally. I'm going to close this, and we're instead going to call out https://coder.com/docs/tutorials/best-practices/scale-coder#connection-pool-tuning in the release notes of the upcoming v2.30 to advise operators to adjust their own settings accordingly. |
Context:
We noticed by observing
go_sql_in_use_connectionsandgo_sql_idle_connectionsthat we were saturating the max connections even in our modestly-sized dogfood installation.After landing #21403 we conducted some experiments to see where the sweet-spot would be, and we landed on 30 max open connections and 15 idle connections per coder replica.
This will be the new default to give operators the best chance of a smooth experience when using the product.
However, operators who have existing installations need to pay close attention when upgrading to v2.30, which this change will be included in. Postgres'
max_connectionsdefaults to100, and with a 10-replica installation they might've gotten away with not tuning this value with each replica only using 10 connections.Release manager: we need to call this change out prominently in the changelog.
docs/tutorials/best-practices/scale-coder.mdcontains some scaling guidance for these values.