Perhaps I’ll get to model-based testing someday. Meanwhile, there’s plenty to do testing simple functions. One issue that comes up with property-based testing is test coverage. It’s very easy for...
Perhaps I’ll get to model-based testing someday. Meanwhile, there’s plenty to do testing simple functions.
One issue that comes up with property-based testing is test coverage. It’s very easy for randomly generated test data to have a very low probability of finding interesting behavior.
A simple example is generating random strings. If you pick Unicode code points uniformly at random, you will see mostly Chinese characters and rarely ASCII. If you’re testing data where a particular character like a comma is treated specially, you need to generate that character fairly often. As a hack, I changed the probabilities in my random string generator to pick ASCII 50% of the time, but in principle, any character could have a special meaning somewhere.
You can use a code coverage tool to get an idea how far your tests are getting. The trouble is, it’s not enough; I was able to get to 100% code coverage in repeatTest and found some bugs along the way, but I still find logic bugs.
A sometimes assertion goes beyond code coverage by making sure your testing covers interesting logical conditions. That is, the assertion should sometimes be true and sometimes false, or your testing isn’t good enough yet. (Or maybe one side of the condition is unreachable.)
Yep. Fast-check doesn’t do that and neither does my library (maybe someday), but you can still run code coverage and adjust your tests manually. Fuzzers look pretty interesting, but it seems like...
Yep. Fast-check doesn’t do that and neither does my library (maybe someday), but you can still run code coverage and adjust your tests manually.
Fuzzers look pretty interesting, but it seems like they are more about testing whole programs? Meanwhile, property-test libraries are more about unit testing an API.
Really enjoyable read, thanks for posting! I will say though that using anything financial as your example is almost cheating given that it is the quintessential case for tests. I also liked that...
Really enjoyable read, thanks for posting! I will say though that using anything financial as your example is almost cheating given that it is the quintessential case for tests.
I also liked that they touched on proofs, and that the more you constrain the code you write, the easier it is to validate. In most cases it doesn't make sense, but if you really really wanted to, you can write your code such that individual components can be proven correct through induction proofs.
In practice, it just isn't feasible outside of unique situations. I've only needed to do so a few times, when writing scientific simulations. That was a challenge because I needed to both heavily optimize, and prove correctness.
Perhaps I’ll get to model-based testing someday. Meanwhile, there’s plenty to do testing simple functions.
One issue that comes up with property-based testing is test coverage. It’s very easy for randomly generated test data to have a very low probability of finding interesting behavior.
A simple example is generating random strings. If you pick Unicode code points uniformly at random, you will see mostly Chinese characters and rarely ASCII. If you’re testing data where a particular character like a comma is treated specially, you need to generate that character fairly often. As a hack, I changed the probabilities in my random string generator to pick ASCII 50% of the time, but in principle, any character could have a special meaning somewhere.
You can use a code coverage tool to get an idea how far your tests are getting. The trouble is, it’s not enough; I was able to get to 100% code coverage in repeatTest and found some bugs along the way, but I still find logic bugs.
A sometimes assertion goes beyond code coverage by making sure your testing covers interesting logical conditions. That is, the assertion should sometimes be true and sometimes false, or your testing isn’t good enough yet. (Or maybe one side of the condition is unreachable.)
Often coverage-guided fuzzers (e.g. https://en.m.wikipedia.org/wiki/American_Fuzzy_Lop_(software)) work pretty well here.
Yep. Fast-check doesn’t do that and neither does my library (maybe someday), but you can still run code coverage and adjust your tests manually.
Fuzzers look pretty interesting, but it seems like they are more about testing whole programs? Meanwhile, property-test libraries are more about unit testing an API.
Really enjoyable read, thanks for posting! I will say though that using anything financial as your example is almost cheating given that it is the quintessential case for tests.
I also liked that they touched on proofs, and that the more you constrain the code you write, the easier it is to validate. In most cases it doesn't make sense, but if you really really wanted to, you can write your code such that individual components can be proven correct through induction proofs.
In practice, it just isn't feasible outside of unique situations. I've only needed to do so a few times, when writing scientific simulations. That was a challenge because I needed to both heavily optimize, and prove correctness.