Over 5 error rate once right after deploy then after 1 hour
The theory is that when we deploy, we clear CF cache - so our throughput goes up. That causes us to run more db queries, creating more db connections. There were too many idle connections, but we were creating more.
We use:
ActiveRecord::Base.connection.execute(query)
Instead the plan is to use
ActiveRecord::Base.connection_pool.with_connection
MR: https://gitlab.com/powr/powr/-/merge_requests/7038
This will ensure we don't constant try to create new connections.
For more details: https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html#method-i-with_connection
Connections can be obtained and used from a connection pool in several ways:
Simply use ActiveRecord::Base.connection as with Active Record 2.1 and earlier (pre-connection-pooling). Eventually, when you're done with the connection(s) and wish it to be returned to the pool, you call ActiveRecord::Base.clear_active_connections!. This will be the default behavior for Active Record when used in conjunction with Action Pack's request handling cycle.
Manually check out a connection from the pool with ActiveRecord::Base.connection_pool.checkout. You are responsible for returning this connection to the pool when finished by calling ActiveRecord::Base.connection_pool.checkin(connection).
Use ActiveRecord::Base.connection_pool.with_connection(&block), which obtains a connection, yields it as the sole argument to the block, and returns it to the pool after the block completes.