I think this is the first article that truly gave me “slop nausea”. So many “It’s not X. It’s Y.” Do people not realize how awful this reads? It’s not a novel either, just a few thousand words, just fucking write it and edit it yourself.
interesting that they have an agent that is triggered on flaky CI failures. but it seems far too specific -- you can have pull request on many other triggers.
there doesn't seem to be any upside on having it only for flaky tests because the workflow is really agnostic to the context.
a test pass rate of 100% is a fairy tale. maybe achievable on toy or dormant projects, but real world applications that make money are a bit more messy than that.
I think this is the first article that truly gave me “slop nausea”. So many “It’s not X. It’s Y.” Do people not realize how awful this reads? It’s not a novel either, just a few thousand words, just fucking write it and edit it yourself.
interesting that they have an agent that is triggered on flaky CI failures. but it seems far too specific -- you can have pull request on many other triggers.
there doesn't seem to be any upside on having it only for flaky tests because the workflow is really agnostic to the context.
> Every commit to main triggers an average of 221 parallel jobs
Jesus, this is why Bazel was invented.
Two problematic statements in this article:
1. Test pass rate is 99.98% is not good - the only acceptable rate is 100%.
2. Tests should not be quarantined or disabled. Every flaky test deserves attention.
a test pass rate of 100% is a fairy tale. maybe achievable on toy or dormant projects, but real world applications that make money are a bit more messy than that.
On top of 2., new tests should be stress-tested to make sure they aren't flaky so that the odds of merging them go down.
I can run flaky tests on my machine a thousand times without failure, whereas they fail in CI sometimes.