TL;DR On Nov 20 2023 from 1pm CET to 3pm CET our ENS resolver returned a static name.

Summary

Customers discovered the issue and operator initiated a swift rollback after the issue was discovered.

Root Cause A bug was deployed to production.

5 Whys

Why was a bug deployed to production?

We occasionally make mistakes. We know this and hence and try to catch these in lower stages.

Why was the bug not discovered in lower stages?

The name service feature was not covered in integration tests.

Why was the feature not covered in integration tests?

We never prioritized.

Why was this never prioritized?

The mix of effort and fun to write of the tests didn’t immediately check out.

Why did it not check out?

The integration test suite is difficult to maintain.

What could we have done better?

• The feature was not covered by integration tests
• Other integration tests were failing for a while not alerting operator to dig into what’s going on
• The integration test suite was not fun/fast to work with
• The rollback took a long time due to the images not being tagged
• Discovery could have been quicker

Action items

1. Write a more fun to work with integration test suite @Derek Rein ✅
2. Cover the Name Service in integration tests @Derek Rein ✅
3. Cover transaction history in integration tests @Max Kalashnikov ✅
4. Make sure integration tests pass again @Max Kalashnikov ✅
5. Deploy only tagged images @Max Kalashnikov ✅