This was illuminating and has challenged some of my beliefs and assumptions about programming. I related to his descriptions of adult ADHD - and the discussions after the talk - more than I...
This was illuminating and has challenged some of my beliefs and assumptions about programming.
I related to his descriptions of adult ADHD - and the discussions after the talk - more than I anticipated. I almost find those discussions more valuable than the particular "coping mechanisms".
Up till this point it never even occurred to me to think of these things as accessibility tools, but that lens gives some very insightful context on the kinds of tasks I struggle with in software development. I hope that awareness puts me in a better place to look for other tools to make things easier on myself.
As far as the technical details; I find value in static types for the same reasons discussed. I'm not familiar with property based testing as a term, I guess it is a more nuanced sort of fuzz testing? I feel I would fall into the trap of over-complicating the constraints.
There is overlap with fuzz testing in that it uses randomly generated data, but there is a different emphasis. The earliest property-testing library I know about is the QuickCheck library for...
There is overlap with fuzz testing in that it uses randomly generated data, but there is a different emphasis.
The earliest property-testing library I know about is the QuickCheck library for Haskell. The idea is that you have some function where the return value is supposed to have a mathematical property (like, for a sort() function, the result should always be sorted). You give it the function to test and an assertion to run on the result, and it automatically looks for a way to falsify the assertion.
It was called QuickCheck because it's not rigorous like a proof and it's lightweight enough to run routinely as part of a test suite. It only runs 100 randomly-generated tests by default. It's more like unit testing in that you test one function at a time.
Since then, property-testing libraries have been written for various languages. I'm working on a property-testing library for Deno.
I'm not all that familiar with fuzzers, but I believe they are more often used to find new security bugs in full-fledged programs, and they use a variety of tricks to look for ways to (for example) crash a program written in C. It's my understanding that people set them up and let them run for days to search for bugs. So, more like whole-program testing than unit testing?
Here's my takeaway on fuzzing and property testing; anyone more familiar please correct me on any nuance here. In fuzzing, you litter code with assertions to check pre- and post-conditions; if the...
Here's my takeaway on fuzzing and property testing; anyone more familiar please correct me on any nuance here.
In fuzzing, you litter code with assertions to check pre- and post-conditions; if the program is ever in an invalid state, it fails fast. Then the fuzzer runs with as much input as possible; if it never fails, then you have confidence that every property implied by every assertion holds. Run for longer on more inputs, and that confidence (in principle) goes higher. So if that's right, that's why I see some parallel to property testing. Every assert is a property that's being tested.
Property testing is more local; you're suggesting similar granularity to unit tests. So you can reasonably enforce constraints on the shape of the input data to avoid wasting time checking invalid inputs. The number of properties checked in any particular unit is less, so you don't need to run as long to gain confidence you've checked everything. I guess you also get some sensible "coverage" metric to be sure the properties you want to check actually do get checked.
My sense is these strategies aren't exclusive. You could litter code with pre- and post-condition assertions; constrain property test inputs normally, the test checks all those assertions you're interested in for the unit. All the same assertions are in play when you run the fuzzer. Property tests then are units that check the happy path, and fuzzing is to be sure things don't fail even in arbitrary potentially-invalid inputs.
Yeah, that seems about right. Having assertions outside the production code (in the test suite) makes it more like other unit test. But I think the biggest difference might be that I run property...
Yeah, that seems about right. Having assertions outside the production code (in the test suite) makes it more like other unit test. But I think the biggest difference might be that I run property tests as part of a test suite, interactively and as part of a continuous build if I set one up. To run quickly, they can't be all that rigorous. It's for catching dumb errors.
But you could run them a lot longer sometimes and I think some people do?
Some fuzzers also use more sophisticated algorithms to get better coverage - otherwise they could run a long time without getting anywhere. There's an older one called "american fuzzy lop" that apparently uses lightweight code coverage as feedback for a genetic algorithm.
When writing unit tests, you get better coverage by looking at the coverage report and writing new tests; there's a person in the loop. I got to 100% code coverage and now it's not that useful a signal for finding more bugs.
This absolutely clicks with me. I've always thought that my interest in type theory ultimately stemmed from my tendency to make dumb mistakes, and my interest in abstractions and generalizations...
This absolutely clicks with me. I've always thought that my interest in type theory ultimately stemmed from my tendency to make dumb mistakes, and my interest in abstractions and generalizations stemmed from my inability to keep track of details.
This was illuminating and has challenged some of my beliefs and assumptions about programming.
I related to his descriptions of adult ADHD - and the discussions after the talk - more than I anticipated. I almost find those discussions more valuable than the particular "coping mechanisms".
Up till this point it never even occurred to me to think of these things as accessibility tools, but that lens gives some very insightful context on the kinds of tasks I struggle with in software development. I hope that awareness puts me in a better place to look for other tools to make things easier on myself.
As far as the technical details; I find value in static types for the same reasons discussed. I'm not familiar with property based testing as a term, I guess it is a more nuanced sort of fuzz testing? I feel I would fall into the trap of over-complicating the constraints.
Thanks to whoever updated the tags.
There is overlap with fuzz testing in that it uses randomly generated data, but there is a different emphasis.
The earliest property-testing library I know about is the QuickCheck library for Haskell. The idea is that you have some function where the return value is supposed to have a mathematical property (like, for a sort() function, the result should always be sorted). You give it the function to test and an assertion to run on the result, and it automatically looks for a way to falsify the assertion.
It was called QuickCheck because it's not rigorous like a proof and it's lightweight enough to run routinely as part of a test suite. It only runs 100 randomly-generated tests by default. It's more like unit testing in that you test one function at a time.
Since then, property-testing libraries have been written for various languages. I'm working on a property-testing library for Deno.
I'm not all that familiar with fuzzers, but I believe they are more often used to find new security bugs in full-fledged programs, and they use a variety of tricks to look for ways to (for example) crash a program written in C. It's my understanding that people set them up and let them run for days to search for bugs. So, more like whole-program testing than unit testing?
Here's my takeaway on fuzzing and property testing; anyone more familiar please correct me on any nuance here.
In fuzzing, you litter code with assertions to check pre- and post-conditions; if the program is ever in an invalid state, it fails fast. Then the fuzzer runs with as much input as possible; if it never fails, then you have confidence that every property implied by every assertion holds. Run for longer on more inputs, and that confidence (in principle) goes higher. So if that's right, that's why I see some parallel to property testing. Every assert is a property that's being tested.
Property testing is more local; you're suggesting similar granularity to unit tests. So you can reasonably enforce constraints on the shape of the input data to avoid wasting time checking invalid inputs. The number of properties checked in any particular unit is less, so you don't need to run as long to gain confidence you've checked everything. I guess you also get some sensible "coverage" metric to be sure the properties you want to check actually do get checked.
My sense is these strategies aren't exclusive. You could litter code with pre- and post-condition assertions; constrain property test inputs normally, the test checks all those assertions you're interested in for the unit. All the same assertions are in play when you run the fuzzer. Property tests then are units that check the happy path, and fuzzing is to be sure things don't fail even in arbitrary potentially-invalid inputs.
Yeah, that seems about right. Having assertions outside the production code (in the test suite) makes it more like other unit test. But I think the biggest difference might be that I run property tests as part of a test suite, interactively and as part of a continuous build if I set one up. To run quickly, they can't be all that rigorous. It's for catching dumb errors.
But you could run them a lot longer sometimes and I think some people do?
Some fuzzers also use more sophisticated algorithms to get better coverage - otherwise they could run a long time without getting anywhere. There's an older one called "american fuzzy lop" that apparently uses lightweight code coverage as feedback for a genetic algorithm.
When writing unit tests, you get better coverage by looking at the coverage report and writing new tests; there's a person in the loop. I got to 100% code coverage and now it's not that useful a signal for finding more bugs.
This absolutely clicks with me. I've always thought that my interest in type theory ultimately stemmed from my tendency to make dumb mistakes, and my interest in abstractions and generalizations stemmed from my inability to keep track of details.