Skip to main content

Admin Alert Failure

Issue: Admin alert failure

What happened?

The admin alert blocking MR has ENV['ADMIN_EMAILS_LIMIT_PER_HOUR'] which sets the upper limit of admin alerts per hour for all apps before blocking. If this ENV is not defined then we use the default value of 10. QA was testing the task with default settings and wasn't aware of the existence of this ENV. After getting approval the MR was pushed to prod and everything was good. On the next day, the Support Team requested to increase the limit up to 25 emails per hour. After increasing the config's value admin alerts failed. Failure was due to a mistake, ENV's value wasn't converted to an integer before comparing with another integer.

How can we escape this kind of issue in the future?

  • The engineer should provide information about dynamically changing configuration parameters inside the MR. QA should be aware of these configs to cover additional test cases with/without ENV.
  • When we create ENV for the first time on production we should request QA to verify that this config doesn't break the functionality.