While I can’t really comment on how good this specific implementation is simple diff based testing frameworks like this dont get enough press. My first job involved working on an internal programming language. The test suite was just a ton of statements that were executed top down and their output was compared against a single file. Yes that’s a bit absurd but it worked remarkably well. If they’d bothered to add a bit more structure around it I think it would have been perfect.
It's not absurd at all (in my view). A test checks that some obtained result matches the expected result - and if that obtained result is something that got printed out and redirected to a file, and that expected result is something that was produced the same way from a known good run (that was determined to be good by somebody looking at it with their eyes), and the match is performed by comparing the two output files... then there you go.
This is how basically all of the useful tests I've written have ended up working. (Including, yes, tests for an internal programming language.) The language is irrelevant, and the target system is irrelevant. All you need to be able to do is run something and capture its output somehow.
(You're not wrong to note that the first draft basic approach can still be improved. I've had a lot of mileage from adding stuff: producing additional useful output files (image diffs in particular are very helpful), copying input and output files around so they're conveniently accessible when sizing up failures, poking at test runner setup so it scales well with core count, more of the same so that it's easy to re-run a specific problem test in the debugger - and so on. But the basic principle is always the same: does actual output match expected output, yes (success)/no (fail).)
Afaik this is usually know as “snapshot”, “golden master”, or “characterisation” testing, and it is quite common and popular. And effective. Although it is quite sensitive to non-determinisms as well as less helpful at tracking down issues.
I wrote https://github.com/mmastrac/clitest because I needed a more complex testing harness for CLI tests that does something similar. It's not exactly the same, but it's definitely in the same universe.
One-file-per testcase like `tc` does works, but it tends to fall apart a bit at large scale in my experience.
While I can’t really comment on how good this specific implementation is simple diff based testing frameworks like this dont get enough press. My first job involved working on an internal programming language. The test suite was just a ton of statements that were executed top down and their output was compared against a single file. Yes that’s a bit absurd but it worked remarkably well. If they’d bothered to add a bit more structure around it I think it would have been perfect.
It's not absurd at all (in my view). A test checks that some obtained result matches the expected result - and if that obtained result is something that got printed out and redirected to a file, and that expected result is something that was produced the same way from a known good run (that was determined to be good by somebody looking at it with their eyes), and the match is performed by comparing the two output files... then there you go.
This is how basically all of the useful tests I've written have ended up working. (Including, yes, tests for an internal programming language.) The language is irrelevant, and the target system is irrelevant. All you need to be able to do is run something and capture its output somehow.
(You're not wrong to note that the first draft basic approach can still be improved. I've had a lot of mileage from adding stuff: producing additional useful output files (image diffs in particular are very helpful), copying input and output files around so they're conveniently accessible when sizing up failures, poking at test runner setup so it scales well with core count, more of the same so that it's easy to re-run a specific problem test in the debugger - and so on. But the basic principle is always the same: does actual output match expected output, yes (success)/no (fail).)
Afaik this is usually know as “snapshot”, “golden master”, or “characterisation” testing, and it is quite common and popular. And effective. Although it is quite sensitive to non-determinisms as well as less helpful at tracking down issues.
Agreed, this is the default testing methodology I reach for. Other methodologies are useful in some situations, but those are the minority.
I wrote https://github.com/mmastrac/clitest because I needed a more complex testing harness for CLI tests that does something similar. It's not exactly the same, but it's definitely in the same universe.
One-file-per testcase like `tc` does works, but it tends to fall apart a bit at large scale in my experience.
Integration tests are language agnostic too: you drive a browser through your app and all it matters is the output.
In this tc framework all it matters is the output of the command. The only part that must be customized to adapt to different languages is
It's at https://github.com/ahoward/tc/blob/main/specs/002-we-need-a/...