There have been a few bugs and regressions that have surfaced lately and I believe a number of these could have been captured with better testing. We should have E2E tests, unit testing and proper evals configured for each model.

We should be able to:

• Have predictable output for known problems / input
• Check if scheduled playbooks (out of the box and custom) are able to run as expected
• Check if notifications are being configured and sent as expected
• Check if chats can be created, deleted and interacted with
• Check if database connectivity is persisted and refreshed appropriately
• Check if custom playbooks can be added, updated or deleted

I'm sure there are more, but this should help get the discussion started.