cross-posted from: https://programming.dev/post/864349
I have spent some time trying to simplify the release process. For a variety of reasons, we can only release on Thursdays. The code is “frozen” on Tuesday before it can be released on Thursday. But we sometimes squeeze in a quick fix on Wednesday as well.
The question, is when should QA test the code?
Here is what I have seen happen:
- Dev writes code and sends it to QA.
- QA finds problems, sends it back to the Dev.
- Dev fixes and sends it back to QA.
I have seen a Dev fix their code on Tuesday, and then QA comes back on Wednesday with problems, when the code should have been frozen anyway.
I am looking, what should be the best solution here.
We have several problems going on at once:
- Developers test on the same server as QA tests. I am working to switch developers to a separate Dev server, but it is a long work in progress.
- We don’t have an easy way to revert code back from the QA server. It is easier to build revisions than revert changes. We can try to revert code more, but it will require a culture change.
- QA don’t really have a schedule when they are supposed to do functional testing vs regression testing.
I don’t know what is the best way to proceed forward. Thus far, I haven’t thought too much about the QA because I was focused more on getting releases out. Now that releasing is more simplified, that we can potentially do weekly releases, I am trying to see how we should proceed with the testing process.
It sounds like your process isn’t working because you’re treating Tuesday’s deadline as just a feature freeze rather than a full freeze.
If you want to go ahead with the full freeze then if it fails QA the feature should simply be rejected. Revert the change make the fixes and wait for the next Tuesday deadline to resubmit it.
If you’re keen to continue the feature freeze then you need to move forward the freeze deadline and agree very specific timings with QA. For example: Feature freeze at 9 am Tuesday morning - so devs must submit all features before end of the day Monday. QA have all Tuesday to review for a deadline 9 am Wednesday.
That gives the dev at a minimum some time on Wednesday to address any issues, but more likely QA can come back quicker so they’ll have some time on Tuesday as well.
Dev must submit fixes before a 2 pm deadline on Wednesday. QA do a second review and have all feedback by 9 am Thursday at which point it’s simple commit/revert.
Your company seem to have a significant technical debt in environments, tooling, test automation, CI/CD, that is slowing down releases. Is that a problem acknowledged in your org? Will you get support for continuing changes and is there an understanding of what is a good enough level?
Would a fixed weekly or bi-weekly schedule work? Both devs and QA can plan their work accordingly. For example: devs and QA agree on what to work on next on a Thursday and how to test (acceptance criterias). Dev start development, QA prepare tests (functional and regression). Dev work for a week, tests and feature freezes code next Thursday. QA starts acceptance test on Friday, they ping-pong Fri-Mon-Tue, regression test and full code freeze on Wednesday, release on Thursday. If quality of the code and tooling makes the process smooth enough, then a similar weekly cycle could also be possible.
Gradually increasing test automation could speed up the whole process. Devs and QA can write and maintain automated tests together.
Feature flags could help isolating bugs that are discovered late in the process.
If there are several rounds between devs and QA, then a root cause analysis can help there as well. Are devs and QA aligned on how to test? Are devs testing enough? Is QA giving enough info for devs on the bugs found? Etc.
It seems to me that your problem is that your definition of “freeze” seems to allow fixes for QA issues. So, not a freeze at all if the idea is to give QA a chance to have a clean testing framework on Wednesday.
I see two alternatives:
-
Make Tuesday a true freeze. Any defects found by QA drop that feature out of the Thursday release.
-
Stop “throwing features over the wall” to QA. Make the QA testers/process part of the development team/process. Features are considered “done” and ready for submission to production only once tested. Freeze on Thursday morning with only integration testing to be done before release.
In truth, both approaches yield the same results. If programmers have to get it right by Tuesday, then they’ll need to work more closely with QA during development. Eventually, the Wednesday testing becomes little more than a rubber stamp and they’ll push to move the freeze back to Wednesday.
Most importantly it seems that in this situation the “definition of done”, has to be more than just “coding completed”.
I like this response. Indeed, if we don’t have strong expectations from the devs to finish by a certain time, then we shouldn’t have such expectations on QA.
I wanted to suggest something like this. Code-freeze wise, you can have a “minor” and “major” problems, major problems block the feature, minor ones let it go (but you now have a tech debt, and make sure that THIS process to fixing up found issues is higher-prio then new features). Of course, you decide what is minor and what major. E.g. maybe a typo in the UI is acceptable, maybe not.
As for throwing features over the wall - I would actually suggest just changing the perspective - make QA involved earlier. The feature is not ready and not frozen unless it’s been looked at by QA. Then when a thing is frozen, it’s really ready. (Of course you’ll still have regressions etc but that’s another topic.)
I think that defining “done” as QA tested is way better than “the code compiles”, which is essentially what most teams seem to use.
Developers need to get into the habit of not writing bugs. That’s technically the answer to all of these problems. “We have issues dealing with bugs found in the QA phase”. So stop writing bugs for QA to find, then the problem goes away.
If the attitude is “Bugs found in QA kick the feature out of the release”, then the programmers are going to find that they work all week and end up contributing nothing to the release. Maybe the release is completely empty. Hold them responsible for it. Attitudes will change.
-
Ephemeral test environments are a great tool for this kind of stuff. We do work on a feature branch and spin up a test environment for that feature. QA happens there, then it gets merged to master once they approve it.
In our workflow, it deploys immediately, but you could just as easily adapt that to cut a release every Thursday. Then each release can just contain whatever makes its way into master by Thursday. You might need to add more process if there’s release notes etc that need to be coordinated. My suggestion on that would be to cut the release on Tuesday or Wednesday, then whoever does that documentation can look at what made the cut and spend a day documenting it, then just press the deploy button on Thursday once they’re done.
It depends… The myriad of reasons to have a dedicated release day have often to do with synchronizing marketing, support and the other departments.
My question is what does QA mean for your org? Does it mean defect detection? Testing? Acceptance? Those are all different things. The teams i see that are able to release every day have a strict separation of Quality Control and Functional Acceptance. QC used to detect defects and regression and is handled by highly automated processes accounted for by engineering. Then acceptance is done by a dedicated product/quality team that figure out if the new functionality actually is built to spec and solves the customer problems. This also involves blogs, documentation, customer contact, release notes, tutorials and workshop for the support team etc… This second part is handled by feature flagging, so that the product teams can bèta test, run a limited release and track adoption.
It really depends on what kind of software youre running and what your relationship is towards the end user and the rest of the org. Something that is the same in all cases is that your requirements and acceptance criteria need to be very clear from the start and regression resting needs to be fully automated.
What’s on the test env? In-dev? Approved changes that will go through QA for release? Release-ready changes?
What does freeze consequently mean? Freeze development? Freeze what you consider for a release? Freeze the release?
If you’re developing on the test env with QA testing changes for approval to a release you can’t “freeze” what you still have to test. You need a inclusion-freeze to have time for QA testing and approval, and then a cleanup and factual freeze of what a release includes.
Once a release is factually frozen you either do the release or you don’t. If an issue is found big enough to not make that release, you make a new release or wait a week. Smaller issues can live on prod for a week - or be fixed earlier through patch releases if you do those.
Your QA team should be building automated regression tests and automating their tests so that they can be run locally as part of the development cycle or as a job in your CI system.
That would decouple this incredibly tight release cycle that you currently have
All of these things are less serious problems if things move faster. QA review of the master branch is the only thing you’re doing that has to be long running and synchronous.
For your acknowledged problems:
- This is huge, and should be a focus for you.
- This isn’t actually a big deal. Outside of fixing bad prod deploys, reversion causes more problems than it solves. If you get the devs out of QA, the need for this will decrease.
- If you freeze the code on tuesday and release on thursday, that sounds like a schedule!
It sounds like the QA department is overwhelmed. You need more test automation in order get the turnaround for regression testing down. Many end to end test tools have recording functions - a senior QA or engineer in test could quickly get simple but brittle automation in place.
Devs need to understand that QA stopping the line is a last resort, not a personal safety net. More unit and integration testing will keep more of the work on the devs until it actually is ready.
What does your source control and CICD situation look like?
