In unit testing, a “unit” does not have to be the smallest possible section of code. It can be a while class or module or even set of related classes/modules. Testing individual functions in isolation leads to brittle tests that easily break during refactoring. Testing overall system behaviour results in more robust tests that require fewer changes during refactoring which gives you more confidence then you have not introduced a regression.
IMO, you should usually test only stable interfaces.
If you have no stable interface all the way into the UI, then you shouldn’t test anything all the way into the UI, and focus your tests there. Odds are that your code isn’t very good, because it is rare that you don’t need anything stable all the way through, but well, “rare” is not the same as “impossible”.
Martin Fowler called them sociable tests. The only way to properly test your units’ behavior is to pull in their dependencies. Isolated tests are useless, brittle and slow to write.
Honestly, I don’t think unit tests are a useful name. Everyone has a different idea of what a unit is and the line between unit tests and integrations tests is IMO not very useful. As long as your tests are
isolated from external factors (ie they completely control the test environment),
fast to run
repeatable, aka not flaky
can identify problems easily
Then where you draw the line between unit and integration is meaningless. It was meant to be that ingratiation tests were slow, so you wanted to shrink them down to make them faster to run. But I have not had a problem with the speed of more integration style tests in a long time.
I also don’t think interacting with the OS is such a bad idea. For instance the filesystem (what everyone always points to as an example) IMO is fine if done right. The big issue with interacting with the FS is keeping your tests isolated - too many people end up reading/writing the same file locations and thus breaking isolation. But you can always create a unique tmp dir for each test and do what ever you want inside that. Interacting with the filesystem on modern system is fast, and reliable - especially given that tmp locations are generally in ram these days.
I think the better term you are looking for is mocks and mocking. IMO these should be kept to a minimum. Like the above - you dont need to mock out the filesystem API when you can just use the filesystem in an isolated way. Same with network services - I really like gos httptest module, it lets you easily spin up a webserver that you can respond with whatever you need to. No need to create a mockable API when you can spin up a fast and reliable http endpoint to respond how you need it to.
Which leads to fakes (ie fake, simple implementations of a real external API). IMO these are far more useful than mocks and should be your first resort with mocks being your last resort. Such as things like gofakes3 an in memory s3 implementation in go that you can use any s3 client to talk to. Things like this let you create tests that you spin up the server (a unique one for each test), put objects into it to set things up how you need them, run your function and assert the contents are what you expect. Makes your tests more complete (and that you are not just testing your mock implementation rather than your actual logic) while keeping them isolated and fast - all the benefits of a small unit test combined with the wider scope of an integration test.
Couldn’t agree more with this comment and the thread in general, it’s a relief to see. I get so frustrated as so many of my colleagues seem to cling to this very old concept of the testing pyramid and associated definitions. It’s completely meaningless in a modern setting. We should mock as little and as far back as possible, yet others seem to delight in locking huge chunks of functionally out of the test base just ‘because’.
I believe we should have a new word that differentiates between ultra-basic tiny unit tests, and bigger unit tests that are still not integration tests.
E.g. rust and some other newer languages have a way to write basically an inline test for a function — that would constitute my former category. These make sense during development as a reality check. “Yes, this ad hoc stack I need inside this class should have two elements if I push two elems” sort of thing. That implementation may not even be accessible from the outside in case of an OOP language so you can’t even properly test it. Also, these are the ones that should change with the code and removing them is no big deal.
The other kind should work against the public APIs of libs/classes and they should not be rewritten on internal changes at all.
I actually love that they’re so brittle because it quickly catches problems that need fixing.
Tests are not brittle when they catch actual problems. Tests are brittle when they break for no good reason. And really when you are refactoring something tests should not break - you are not changing behaviour, just reorganising things. If you need to do big changes, rewrite or even delete tests when you refactor something then your tests IMO are brittle and much less likely to catch regressions in behaviour. Yeah you will need to tweak them some times, but these should be kept to a minimum and not be happening every time you refactor anything.
When writing new code small tests can feel good, they are easy to write and you can test things quickly in isolation. But after that point how much value do they give you? Tests do have a cost - beyond the original time you spent writing them, you have to maintain them, and keep them uptodate, they take time to run etc… If they cannot properly catch a regression when it happens then IMO they are not worth the cost of keeping them around. Larger tests tend to be better at catching regressions and are less prone to need to be tweaked when you refactor the internals of something.
So, generally speaking I tend to prefer testing public APIs of something and ignore internal helper functions (unless some helper is sufficiently large/complex enough that it warrants extra dedicated tests, which is not common). Note that this does not mean public to downstream users of your library/script/binary, but also larger internal modules API that the rest of the application can use. Though I do find quite a few smaller applications you only need to test the actual public API from an end users perspective.
A lot of that has to do with type checking and a lot of the methods would have huge consequences if they were off.
Uhg, This is a big reason I don’t like loosely typed languages. Yeah that might be one case where smaller tests are needed but IMO testing input types should not be required at all - that is something a compiler can do for you for free. At least assuming you have a strongly typed language. People always say loosely typed languages are faster to code in - but this benefit is completely lost when you spend ages writing tests for different inputs types that a compiler in a stronger typed language would just reject.
In unit testing, a “unit” does not have to be the smallest possible section of code. It can be a while class or module or even set of related classes/modules. Testing individual functions in isolation leads to brittle tests that easily break during refactoring. Testing overall system behaviour results in more robust tests that require fewer changes during refactoring which gives you more confidence then you have not introduced a regression.
IMO, you should usually test only stable interfaces.
If you have no stable interface all the way into the UI, then you shouldn’t test anything all the way into the UI, and focus your tests there. Odds are that your code isn’t very good, because it is rare that you don’t need anything stable all the way through, but well, “rare” is not the same as “impossible”.
This is the correct comment.
Martin Fowler called them sociable tests. The only way to properly test your units’ behavior is to pull in their dependencies. Isolated tests are useless, brittle and slow to write.
Yeah I’m of the opinion that unit tests are usually a waste of time and people should only write integration tests.
The only time I think unit tests are valuable is for checking edge cases when e.g. interacting with the operating system.
Honestly, I don’t think unit tests are a useful name. Everyone has a different idea of what a unit is and the line between unit tests and integrations tests is IMO not very useful. As long as your tests are
Then where you draw the line between unit and integration is meaningless. It was meant to be that ingratiation tests were slow, so you wanted to shrink them down to make them faster to run. But I have not had a problem with the speed of more integration style tests in a long time.
I also don’t think interacting with the OS is such a bad idea. For instance the filesystem (what everyone always points to as an example) IMO is fine if done right. The big issue with interacting with the FS is keeping your tests isolated - too many people end up reading/writing the same file locations and thus breaking isolation. But you can always create a unique tmp dir for each test and do what ever you want inside that. Interacting with the filesystem on modern system is fast, and reliable - especially given that tmp locations are generally in ram these days.
I think the better term you are looking for is mocks and mocking. IMO these should be kept to a minimum. Like the above - you dont need to mock out the filesystem API when you can just use the filesystem in an isolated way. Same with network services - I really like gos httptest module, it lets you easily spin up a webserver that you can respond with whatever you need to. No need to create a mockable API when you can spin up a fast and reliable http endpoint to respond how you need it to.
Which leads to fakes (ie fake, simple implementations of a real external API). IMO these are far more useful than mocks and should be your first resort with mocks being your last resort. Such as things like gofakes3 an in memory s3 implementation in go that you can use any s3 client to talk to. Things like this let you create tests that you spin up the server (a unique one for each test), put objects into it to set things up how you need them, run your function and assert the contents are what you expect. Makes your tests more complete (and that you are not just testing your mock implementation rather than your actual logic) while keeping them isolated and fast - all the benefits of a small unit test combined with the wider scope of an integration test.
Couldn’t agree more with this comment and the thread in general, it’s a relief to see. I get so frustrated as so many of my colleagues seem to cling to this very old concept of the testing pyramid and associated definitions. It’s completely meaningless in a modern setting. We should mock as little and as far back as possible, yet others seem to delight in locking huge chunks of functionally out of the test base just ‘because’.
I believe we should have a new word that differentiates between ultra-basic tiny unit tests, and bigger unit tests that are still not integration tests.
E.g. rust and some other newer languages have a way to write basically an inline test for a function — that would constitute my former category. These make sense during development as a reality check. “Yes, this ad hoc stack I need inside this class should have two elements if I push two elems” sort of thing. That implementation may not even be accessible from the outside in case of an OOP language so you can’t even properly test it. Also, these are the ones that should change with the code and removing them is no big deal.
The other kind should work against the public APIs of libs/classes and they should not be rewritten on internal changes at all.
deleted by creator
Tests are not brittle when they catch actual problems. Tests are brittle when they break for no good reason. And really when you are refactoring something tests should not break - you are not changing behaviour, just reorganising things. If you need to do big changes, rewrite or even delete tests when you refactor something then your tests IMO are brittle and much less likely to catch regressions in behaviour. Yeah you will need to tweak them some times, but these should be kept to a minimum and not be happening every time you refactor anything.
When writing new code small tests can feel good, they are easy to write and you can test things quickly in isolation. But after that point how much value do they give you? Tests do have a cost - beyond the original time you spent writing them, you have to maintain them, and keep them uptodate, they take time to run etc… If they cannot properly catch a regression when it happens then IMO they are not worth the cost of keeping them around. Larger tests tend to be better at catching regressions and are less prone to need to be tweaked when you refactor the internals of something.
So, generally speaking I tend to prefer testing public APIs of something and ignore internal helper functions (unless some helper is sufficiently large/complex enough that it warrants extra dedicated tests, which is not common). Note that this does not mean public to downstream users of your library/script/binary, but also larger internal modules API that the rest of the application can use. Though I do find quite a few smaller applications you only need to test the actual public API from an end users perspective.
Uhg, This is a big reason I don’t like loosely typed languages. Yeah that might be one case where smaller tests are needed but IMO testing input types should not be required at all - that is something a compiler can do for you for free. At least assuming you have a strongly typed language. People always say loosely typed languages are faster to code in - but this benefit is completely lost when you spend ages writing tests for different inputs types that a compiler in a stronger typed language would just reject.