The Original Post

The QA team starts testing a software/product and there are “way too many” defects…Every other scenario is failing, new flows are explored & clarifications sought. What would be the strategy now?

It’s a complex application with tight schedule. Too many defects add cherry on the cake. The blame game starts. But at the end, development & QA are expected to work as a team & deliver the product/software. What are the options?

• Reject the build. But that would mean delay in delivery.
• Get into a war room, daily. It helps to triage the defects & increase the seriousness.
• Prioritize. Work as a team & ensure priority flows are working as soon as possible.
• Identify super-devs & super-QAs to take up some additional tasks & ensure minimum turn-around.
• Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise.
• Since timelines are fixed, make sure turnaround time is as minimum as possible – be it defect fix, retesting, business clarifications, anything. Everybody has to be on their toes.

Note: Retrospective w.r.t. estimations, build quality, management screw-up, etc. is altogether a different discussion.

The Reactions/Responses | How to handle Defects

<Lead QE/ SDET>

QE instead of QA: Based on my experience in Expedia and now in Walmart, there are small teams and QE (note I am not using QA here) who can fix or help in fixing issues. The emphasis is not on logging the defects but reporting it effectively and getting it resolved as soon as possible based on priority.

…………………………

Unit Tests: This is a cat-mouse game. You can only win when you will deliver the product with best quality. Reporting defects, fixes etc. These are part of this game. You have to decide how you can achieve it. As per me this is one team game against your challenge (Challenge here is to deliver product in time with best quality).

First we have to control on any new defects, for that you have do proper unit testing before making any changes. Do the checking with proper code review, so it will help all to ensure that new code is not breaking any existing feature. We have to set the priority for defects, estimate the fix time of new deployment to start testing. We can also increase the deployment frequency but in control way.

<Quality Assurance Engineer>

Reject the build, never compromise on quality – even under pressure of product release.

<QA Analyst>

Prioritize & then Retrospect: I am a peace lover but frankly when such a chaos arise, before getting into a solution my mind battles for below questions. Surely probing these questions will not handle the situation then and there but I can’t help it but think…

• Isn’t the developer is testing before delivering the code to testers?
• Aren’t the testers doing smoke testing to before jumping right into the testing?
• What is the process and approach they have…”Big bang model”?
• Wasn’t there a strategy/plan defined..?
• And was this not a foresighted risk … what did the Analyst do then?
• And what did they learn in their previous retrospective?

We all know that 80% of problems in software arise due to 20% of bugs, I think same applies to the process and people.

Honestly all of us when face a similar situation would definitely prioritize our tasks to control the scene of crime, but for the long run one needs to think on above questions as well and RETROSPECT them.

<Me>

Thank You for the valuable inputs ? Agree that ‘precaution’ is always better than ‘cure’. But that’s the way of life – ideal vs. practical.

• There are projects which collaborate & there is continuous communication.
• Then there are some projects which end up in such situations. Help it or not.

I agree these kind of projects are ‘learning in disguise’ ? Best candidates for a retrospective. “You got to know the worst in order to identify the best”.

Thank You again for highlighting an important point of ‘avoiding’ these kind of situations from the very start ? to focus on planning-team-and-communication!

For the record, just some random cases…

• Aren’t the developers unit testing? — in many cases, No!
• Aren’t Testers doing Smoke Tests? — There is no time. The build was delayed.
• What is the process & approach? — Psuedo-Agile ?
• Wasn’t there a strategy/plan? — Yes. The ‘Documents’.
• Foresighted risk — Risk of too many issues popping up is always there.
• What did they learn in their previous retrospective? — All retrospectives are not always implemented ?

I have heard people saying that we are “highly agile”. They release twice a day but do not have automation to make testing a valued effort. I find that funny. ?

<Agile Testing Evangelist – Helping teams test better!>

Risk-based Testing: My questions are

• What was the strategy for identifying the test cases especially Regression?
• Where any of the tests prioritized up front? If yes how?
• What was done to identify the risks involved in project and the product? What were the parties involved in risk identification? Was only the “risk aware higher management” involved or were others like product owners, developers and most importantly testers involved too?

Here are my 2 cents…

Instead of thinking about a strategy after being in this situation, Why not think about a strategy up front? Do a thorough risk analysis and use Risk Based Testing as the primary test strategy and build on that.

Risk based testing that counts both risk mitigation and contingency plans.

<Test Lead>

QA Report: Good Suggestions. For the QA to be on safe side, preparing a quality report having list of reproducing defects and also the QA recommendations for release is vital. This report can be shared with all the required stakeholders a day or two before release.

“QA Report” with recommendations!

<Agile Enthusiast, Learner>

Find the real Solution: Before answering this question, my question would be –

• Where were these folks who’re testing application now; when ‘someone’ was writing such a crappy code?
• Why these two ‘departments’ working in silos?
• Who ceased these departments not to collaborate right from the ‘very beginning’?

<Me>

Agree that ‘precaution’ is always better than ‘cure’. But that’s the way of life – ideal vs. practical.

• There are projects which collaborate & there is continuous communication.
• Then there are some projects which end up in such situations. Help it or not.

The discussion is how to tackle the latter one. After all, delivery is important – no matter what.

I agree these kind of projects are ‘learning in disguise’ ? Best candidates for a retrospective. “You got to know the worst in order to identify the best”.

Thank You again for highlighting an important point of ‘avoiding’ these kind of situations from the very start ? to focus on planning-team-and-communication!

<Agile Enthusiast, Learner>

The current situation is more like a ‘Firefighting’ scene which is not a Normal Way of Life and it would never be. At the edge of ‘Chaos’ whatever seems good do it because in ‘chaotic’ domain the first thing you should do is to act. Things got crazy here, stuffs are off the rails. Your first ‘firefighting’ steps should be correct problem and contain the issue. Your initial steps may not be best but it’ll help you stop bleeding and somewhat be soothing. But don’t stop here find out the ‘Real Solution’ of problem.

Contain the issue >> may not be best >> don’t stop >> find the real solution ?

<Test Manager (PMP, CSTE)>

Pareto Analysis: This is where we can apply Pareto. 80% of problems / defects / blockages in test is due to 20% of causes. Start by doing a practical review of all your observations and group/sort to find your top most issues. 2nd- address defects with high impact & retest in quick turnaround. 3rd As problems are starting to get addressed create a practical and reliable plan. 4th most important key point is to work collaboratively through all this instead of blame and shrugging responsibility / ownership. Collectively all together create the mess and all clean it together. Based on real life experience.

Oh Yes, it brings the memories back. However, I do wonder if the first option is chosen (“Reject the build. But that would mean delay in delivery”) would there be any true delays. And by true delays, I mean the delays over and above the delays that you are going to face if you are in this situation. However if you do have to fix and go in QA (not recommended), data is your best friend. You will need a super team to go identify, prioritize and resolve issue.

…………………………

Team Work: On this situation firstly we need to team work then ensure priority flow, and the most important thing if you want deliver the project to your client, Make sure he is not trouble with basic thing of your software, because your software must walk at least normal and good condition.

Basically Team work + Prioritized Tests ?

<Software Quality Assurance Analyst>

Requirement Freeze: First of all is there any signed and locked requirement document? If yes then first stop development and prepare test case based on the requirement document. Verify the test cases with the client. After that start testing case by case. If require divide the one functionality in multiple cases. Then based on the test cases pass or fail management can decide what else development is require or not. Also must use any testing bug tracking software.

In current situation war room discussion, blame games etc. are also started when requirements are not lock and no one knows clearly what the requirement. Because when product is in QA developer shouldn’t involve directly until any broken things stop the QA.

<Counter>

How do you propose to handle ever changing requirements? Consider the case of Facebook or WhatsApp. Or for that matter the Microsoft website. The website contents change dynamically throughout the day. How do you handle such situations in testing when requirements are never locked?

…………………………

Release Notes: These things to be reported in the release notes of the product before delivers it to the client side if the things are not fully covered

<Sr. QA Lead>

Unit Tests + Testing: I would suggest to take preventive actions along with corrective actions for fixing issues. Preventive action could be get proper Unit testing practices in place and let developers also own quality for their own part. QA should not end up doing unit testing (meaning just verifying basic functions and CRUD operations). QA must get good Quality builds so their bandwidth is utilized in effective testing and verifying real cases. All the Best!

For Dev to focus on unit tests & QA to deep-five ?

<Senior Software Engineer | QA>

“And the blame game begins”.

Communication: A typical case, which I believe everyone has come across. But people often forget that QA and DEV are OPPOSITE sides of SAME coin. For a coin to be of any value, both side have to be perfect.

In my opinion, communication with an open mind is most important in this scenario. Keep each other informed of every development.

Finding who, how and why we missed is something to discuss later. In this chaos focus on NOW, work as a team.

<Prince2 ® Registered Practitioner>

Left Shift Testing: It’s a genuine current phenomena in all the companies. We need to deliver the product as cohesive Engineering efforts. Try to convince and get confidence from Dev team to adopt left shift testing and follow bottom up approach in building test suites for Unit testing (70%), API testing (20%) and Functionalities UI testing (10%).  Which gives us better test coverage with low maintenance.

<Software Test engineer>

Environment Management: What we do – There are there environments for filtering.

1. ST – Dev
2. SIT – Test
3. UAT – Business

First point is timelines should be fixed generously for the User Stories.

1. Before the deployment we need to identify the already present defects in Test environment, which will reduce the confusion whether it’s a regression or newly added defects and raise them.
2. Ask the dev team to prioritize the regression defects and fix them before the new deployment
3. Start testing in the dev environment (System Testing) after the dev team completes their own unit testing (2 days timeline depending on the complexity of the user story)- Try to find as many bugs as possible to filter out more
4. After these defects are fixed deployment is given and 2 weeks of testing will be done in SIT environment along with regression- More filtering will be done
5. UAT testing will be done and make sure no major defects may come out during this phase which is our responsibility

Happy Testing.

…………………………

Prioritize. Work as a team & ensure priority flows are working as soon as possible.

Finger pointing is a frequent occurrence on failed projects. So it’s good to involve the higher management (risk aware) just in case something goes wrong.

< Senior QA Specialist >

Critical Fixes: Focus should be in fixing crucial bugs first and asking the test team to again run build verification tests and Ad-hoc tests. Once it is stable then focus should be on regression tests, etc.

< Senior Quality Analyst >

Report: First step is to report this situation to management and then start prioritizing the higher risks areas and works as a team. By doing this management got aware early and cannot blame that they came to know when it is hard to control.

Management should be informed to take advice/support, to keep all stakeholders updated so that it can be re-planned/adjusted OR to avoid the blame game? I agree with the action but not the motive ?

< QA Lead >

1. Work as a team & ensure priority flows are working as soon as possible.
2. Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise.

< QA Lead >

Prioritize: I have tried all the points mentioned by you but the only thing that worked was — Prioritize. Work as a team & ensure priority flows are working as soon as possible. Rest everything didn’t work in favor. I believe the last thing to do in the project is blame game which unfortunately happens the most.

Haha! ‘Blame game’ is an implicit part of human nature ? Thanks for pointing out the most practical reaction ?missed it..!!

…………………………

Defect Report: First focus on creating the defect report and send it to development leads and then basis the priority and severity first those bugs should be fixed which are directly impacting the business and prepare the build cycle for rest of the bugs for next release

Fixing priority bugs in the current release and postpone the rest ??

< Senior Tester >

So, proposed solutions are;

• Reject the build. But that would mean delay in delivery. – Instead you can try to inspect quality into a known low quality build? – Please lord no!
• Get into a war room, daily. It helps to triage the defects & increase the seriousness. – Instead of discovering the root cause and fixing. Waste time reviewing and re-reviewing defects. Fixing the stable door after the horse has bolted.
• Work as a team & ensure priority flows are working as soon as possible. – Neglect the root cause and look busy doing something that gives an illusion of control.
• Identify super-devs & super-QAs to take up some additional tasks & ensure minimum turn-around. – Ensure your ‘stars’ are demotivated and want to leave.
• Involve the higher management (risk aware) just in case something goes wrong. It shouldn’t come as a surprise. – CYA
• Since timelines are fixed, make sure turnaround time is as minimum as possible – be it defect fix, retesting, business clarifications, anything. Everybody has to be on their toes. – Persecute the innocent.

< Me >

? Thank You for the counter ? Let me tell you a story –

Lead: We need to improve. Let’s brainstorm.

A1: I have an idea-1.

Lead: Ridiculous. That won’t work because…

A2: I have an idea-2.

Lead: Nah! Stupid. Next…

A3: I have an idea-3.

Lead: Do you think that will work? I don’t think so.

Team (thinking): Leave it. He/She has a counter for everything. He/She won’t listen. Just that want to ‘Brainstorm’.

A4: So, what do you suggest Lead?

Lead: Ooops!!

“If you don’t listen to people, why would they talk” ??

< Senior Tester >

I’m sure that the shrewd can determine the difference between the points of my comments and a leader that can’t listen and build on ideas, when a donkey is dead and needs to be dropped, and determining the actual problem to solve sounds so easy, but in reality is the trickiest bit.

< Me >

“Determining the actual problem to solve sounds so easy, but in reality is the trickiest bit” ? couldn’t agree more…!!

< Senior Tester >

• You can’t make a crappy release good, only less crappy, potentially.
• You can’t solve a problem unless you know what the actual problem is.
• You can waste time, a lot of time, patching symptoms of the problem.
• Everyone is responsible for quality, and should know they are.
• This is what you get when you test at the end.

Stop the Release: So the only proposal I have is stop the release and do it again, but right this time. When done right it will be a lot quicker than doing it wrong and then trying to make it right.

< Me >

Now that sounds like a solution ? “Stop the release and do it again, but right this time” ?

— If the stakeholders agree!

< Senior Tester >

It’s the stakeholders’ money and therefore their choice – waste money trying to make a silk purse out of a sow’s ear or spend money trying to build what is wanted. Seems obvious to me!

Of course there is another option – leave and find a professional outfit to work for! ?

…………………………

< Test Consultant >

Stop & Stabilize: Stop delivering new functionality and stabilize the product. Track bugs and regression test well. New functionality has to be prioritized in terms of impact once it does start being checked in again.

Bugs should also be grouped together by area so they can be fixed together and possibly allow section to be rewritten and refactored. Obviously prioritizing using severity and impact and risk analysis.

< QA Automation Engineer >

I liked this approach. I also had this problem with one of my products I worked for and I followed the same strategy. Just test that whole sections and provide correct scenarios with acceptance criteria and it really worked.

< Product Management | KMP 1 | CSPO | SAFe 4.5 Agilist | Lean Six Sigma GB Trained & Tested >

Focus on Defects, not Delivery: I am sure in this kind of environment there might be a big backlog of defects, so team will be firefighting between defect backlog and product backlog. It means that the system is not stable. First Identify the list of all defects and take the list to client, here transparency is important, discuss with them on stabilizing the system rather than focusing on delivery. Doing this we need not separately discuss on extending the timelines, they will be aligned with your plan. Without fixing the defects and focusing on delivery will make the situation worse.

Requirement Gathering: If the situation is not as stated above. Defects raised are because of missed scenarios and requirements are not clear. Change the approach in requirement gathering .To start with, gather requirements collectively along with BA, QA and Dev (one or more from each team), document the gathered requirements and get them validated from client. Post validation inform client that requirements are freeze for the sprint and there will not be any changes in between. We need not be stubborn or strict to do this with client, if we follow a best practice transparently, clients understands and respects the discipline.

< Solutions Architect >

Fire the CTO ?

< Solution Architect >

Wow. Trust the SDET, trust your first instinct.

< Senior Technical Project Manager >

Pair up testers and developers.  You are all engineers working to achieve the same result.

< QA Team Lead >

No Sign-off: I think you should log all the defects starting from critical one. Don’t give your testing sign off. Say them clearly we need all issues fixed. So it will not come to QA head at end. Go by process. If sign off process is not there list out issues and send to all leads /managers to just show your participation and concern that build is not stable.

< Experienced mobile/web QA engineer >

Exit Criteria: Ideally, long before the product reaches this point there should be a well-defined set of exit criteria agreed upon by the product owners, the developers and QA that define the bar that the product is expected to meet, and if the build is as bad as this scenario makes it out to be then there’s no way it’s meeting the bar.  Unfortunately, there’s only so much the QA team can do (especially in a black box scenario) but they’re just the messengers in this scenario.  If all of this is in writing beforehand it shifts a lot of the pressure away from QA.

Of course, QA needs to be doing their job too and making sure they prioritize correctly.  Ideally the test suites should do this already, but the focus needs to be on meeting the exit criteria, and spending a bunch of time filling low severity bugs doesn’t help any here.

< Senior Software Developer >

Prioritize: As per agile methodology convert application functionalities into smaller business deliverable epics or features and get it prioritized from client. Get client into confidence and deliver product as per priority. Going into war room and pressurizing developers and testers may solve your problem for now but it may create a poor maintainable product for future.

< Vice President QA >

MVP: There are obviously multiple problems here but let’s address the pressing issue of the release. No QA professional ever said “lets release this software which doesn’t meet the client requirements and hope they don’t notice”  If the major use cases aren’t working then NO ONE will be happy, regardless of whether you meet the delivery deadline.  It’s far better to prioritize the bug fixing based on the clients priorities and get them something that meets their basic needs even if it means missing your deadline.  Communication is also key, be honest about backlog but have a plan to remedy this in the future with maintenance releases.   AFTER the initial release there are obviously issues that need addressing in the way QA interacts with development and the point in the development cycle at which bugs are identified and fixed.

< Software Test Engineer >

Test Design: First of all try to make test cases covering each and every joints that would help the application getting failed at the start up only and then on every release run those cases and fail the build if anything gets failed. This will save your work load again and again and make sure developers are properly guided with the UNIT TESTING. Which they did was not up to the mark and if required you can guide them.

< Test Manager >

Unit Test + Pair Up: Identify the true cause of the issues so they can be addressed otherwise the problems will continue. What about Unit Testing? Is Unit Testing being undertaken and passing successfully? If yes, then you need to look at whether the tests are robust enough, if no, this might demonstrates the importance of ensuring all activities of the SDLC are executed correctly. On this occasion my suggestion would be to pair the developers and testers up, before code delivery (even of fixes), to identify if the correct scenarios have been considered by the developers and requirements have been understood correctly. In similar situation I introduced developer peer review of test scripts. The result was increased quality out of development by improved understanding of the requirements and unit testing. The added benefit was stronger working relationships with improved moral. It’s not a blame game it’s about coming together as ONE team.

< Sr. Team Lead – QA >

Round-table: This is the story of every fixed time project. The thing higher management have to create a mitigation plan. Now coming to the fact that this way testing defect rate will increase why because in hurry developer miss many thing like unit testing, code quality , business flow implementation gap. Sit with developer round the table along with product manager and start doing thing then and there.

1. Try to cover and plan out testing activity within QA team and share it with Dev team to get timelines for each module
2. Work with developer to raise, get fix and verify it to fasten the dev and bug fix cycle.
3. Try to share one bug status report in morning and one in the evening including management people to showcase what is the fix rate and how much effort QA are putting to get these done with developers.
4. Always ready with proper QA plan in-case of any kind of delay in delivery or any major gap.
5. Always make sure that in these type of timelines low and medium bug are acceptable but no critical and blocker.
6. Work as a team even if someone is not supporting QA because ultimate culprit is QA for any kind of quality issue.

< IT Quality Consultant >

There are different responses possible based on your role – for a PM:

1. Maintain motivation of the team
2. Secure critical path
3. Communicate risk
4. Prioritize functionalities
5. Control scope
6. Repeat

Thank You for that ‘management’ perspective ?

< SAP PS Consultant >

First identify the nature and type of defects, Go live critical, critical, or rare scenario, this will help to focus.

< Quality Professional, Lean Sigma, Agile Evangelist -SAFe Agilist, ICC-ACC, CSM, CSPO, CSP, KMP >

Early Testing: In a complex application like this, why first of all testing coming too late in the cycle? Was there any convergence between testing and development team.  This is where agility proves.  Stop delivery. Fix the issues and streamline things.   Get customer into confidence first.

‘Stop Delivery’ ? and then streamline/fix ? thank you for the inputs.

< Lead consultant, Scrum Master >

1. Send an email to higher ups to let them aware of the situation with facts & figure.
2. Do risk based testing and let the team collaborate without any fear of being pointed out for flaws. Let them share/ bear the responsibilities. If you really want to deliver the project else restructure now to avoid failure at later stage.
3. Nothing much can be done for such situation unless people interact and collaborate.

< UAT Test Analyst >

Risk Analysis and re-prioritization

…………………………

Whoa! With this we come to an end of this wonderful discussion I had on LinkedIn. Unit Test, Prioritize, Stop the release, Find the real Solution, Test Design, Environment, Team Work, Pair Up, Defect Report, Communication, and what not. This was one healthy & very informative discussion. So the next time you face such a situation (I hope not :P), do refer this article for some guidance. And if you already faced one – don’t forget to provide your valuable comments. Anyways, there is no harm in sharing the article with greater audience ? Thank You and Happy Testing!

BECOME IOS SOFTWARE TEST ENGINEER WITH CODEFITNESS!

• We are not typical bootcamp. We teach through day-to-day work tasks using Agile approach. From day one you will start automating real use cases of a real startup-like iOS app. Thus you will gain real experience.
• You will learn programming via hands-on task completion — focusing you on topics that are necessary to complete your task
• You will work in pair just like at real-world job. Pair programming is proven to be very effective and used heavily industry wide.
• You will experience best software development practices:
  • Each test case code will be committed to common GitHub repository via PullRequest. We will teach everything about Git and GitHub from scratch
  • Upon successful code review and merge, your test code will be executed on Jenkins — your work will be run along with existing tests thus will be checked if your newly automated test is not breaking existing tests. This practice is called pre-merged testing and in recent years has become industry standard
• We will start each session with methodical approach on how to solve most asked Interview Programming problems thus will prepare you for coding interview screening