The Importance of Error Logs

Introduction

Has someone reviewed your systems’ error logs today? What about yesterday? Within the past week? Before, I would have assumed a singular answer from everyone-- “Yes”. Of course people are reviewing their error logs, because why else would they create them to begin with? Of course error log reviews are part of the QA process. Of course someone is managing and reviewing the errors in these logs. Now, however, after years of experience across multiple companies, I can tell you the answer is most likely “No” or maybe even “I don’t know”.

The Truth About Error Logs

The truth is that error logs tend to fall by the wayside of other projects and requests. Developers often times don’t have the resources to review the error logs in depth and business/marketing resources tend to not understand the errors enough to be of help diagnosing them. Most of the time QA resources don’t even have access to these logs. So who looks at them? In my experience you can set up all the email chains or permissions in the world for users to see error logs but unless the responsibility is distinctly defined, no one will review them. It is seen as an additional “chore” that can be intimidating to tackle, especially when the logs previously remained untouched or reviewed. Simply achieving a goal like having a useful error log can be difficult. So where do you begin?

Taking Charge

Take charge of the situation. As a Business Analyst, part of your job is to determine where the company can improve their current systems, and that includes utilizing the error log to identify both serious issues that need immediate attention and low priority items that can be addressed long-term to improve system processes or user accessibility. If the error log currently being utilized is too cluttered with meaningless warnings or errors to be of use, work with the development team to alleviate that. The goal is to have a functional error log that can indicate when something has gone wrong or when new releases introduce an issue. After ensuring the error log is readable and useable, the next step is to become familiar with what a “normal” error log looks like for the system. How many total errors per day is normal? This number will likely change per day depending on typical web traffic. On days where your websites receive more traffic, you can expect more organic error messages than on a day in which traffic is lighter. Which errors are more often at the top of the error log? Essentially the goal in this step is to understand your error logs (or to word it another way, become the “knowledge expert” on your error logs) so you can identify new issues in the error log at a glance. If you are used to seeing more of one error and instead see another at the top of the log for that day, investigate what may have caused that change. Anytime an error log changes in any tangible way, you should be asking “why”. If you encounter an error often enough in the error logs, you should also take the time to understand what that error means. Whether that means googling the terms or asking a development resource for a moment of their time, it’s important to understand where an error log line item belongs in the spectrum of importance.

The Outcome

Some of the most severe bugs my clients have fixed have ended up being identified as a result of error log reviews. At one of my prior companies, I noticed a small error log line item I hadn’t seen previously a few months in regarding our search functionality and I reached out to one of our lead developers via email. Before the end of business that day we had released a hotfix to resolve the issue, as that small new error identified a glaring security hole in our product. At my current company I have helped identify issues introduced in releases as well as identify issues introduced by our third party partners that interact with our systems. I’m not even on the official mailing list for our error logs; instead, a coworker has to forward me the email daily; but I’m still the one setting eyes on it every day and reviewing the results for action items. The result wherever I have taken charge of the error log is this: a higher level of quality assurance at a low level of effort and the bonus of being far more knowledgeable about technical issues and error severity. So-- has someone reviewed your systems’ error logs today? If the answer is “no”, why haven’t you?

Process Implementation and Improvement: A High-Level How-To

Every company has processes. They may be variable based on a number of factors, but they exist; that’s how a business achieves replicable success-- by finding a process and sticking with it. Unfortunately, what works for one project within a company may not work well with other projects. It may encounter bottlenecks that reduce performance or profits, which is something no one wants. So, what happens when a company realizes their approach may not be ideal and looks for new ways to improve? Business analysts are often tasked with reviewing current systems and recommending improvements based off of that analysis. There are 5 basic stages to identifying and implementing successful and cost-efficient business processes: Document current processes, identify bottlenecks current processes, Brainstorm and document the proposed updated processes, implement the process within the iteration of a project, and finally, perform a retrospective analysis of the process performance to improve the process as necessary.

Often the most daunting task may seem to be thoroughly documenting any current processes executed by a company. If nothing is documented prior to your intervention, it is likely that one of two scenarios has occurred: one individual holds all of the knowledge you need and want to document or many people hold “piece-meal” versions of the knowledge based on the area of the systems and processes that they deal with on a day-to-day basis. Both of these scenarios put the company at a disadvantage. One person holding all the knowledge causes a bottleneck anytime another team member needs information because that person becomes dependent on the knowledge resource’s availability, and that resource may become stretched thin. It also puts the company in a position where they are very dependent on a specific resource; if that resource is out sick or decides to switch companies, the company is at a severe disadvantage. When multiple people hold a “piece-meal” version of the systems and processes used, different problems arise. If many people hold small views of the systems and processes, they may not be as efficient as they could be with the appropriate knowledge and no one holds an overall top-level view of the systems and processes, which would minimize confusion. Often, when a resource with all the information leaves a company, the knowledge base switches to the “piece-meal” scenario. Documenting current system dependencies, charting current process flows, defining a glossary of terms used by the company, clearly mapping data, and documenting credentials for testing purposes are just some of the documents that should be produced during this stage of the process.

Once processes and overall knowledge of the company have been documented and distributed to the team, it is time to begin analyzing these processes to identify bottlenecks within the processes. Where does progress in the project lag the most and what can be done to prevent that? Is it a matter of outdated or difficult processes that can be updated in some way? Do additional resources need to be added to the process in some way? These potential issues and solutions need to be identified and proposed to the team. Once everyone is in agreement, documentation of the new processes can begin.

Documentation of all knowledge regarding processes is important, whether it’s new or old. Any processes that have been identified to have bottlenecks should be updated and documented to indicate these updates. If a new resource or position is added to the company process, their role within that process must be defined and all existing process charts must be updated to reflect this responsibility. This will also allow business and marketing employees to better understand the proposed changes and updates. All updates should have a clearly defined goal and a way to measure if that goal has been met. Is the goal to reduce time required per iteration of the project? Is it to increase quality or sales lead numbers? Before proceeding with process updates, ensure that the progress related to these updates can be clearly measured or evaluated in some way.

Once the documentation and visual representations of the new proposed processes have been completed, they can be presented to the team for approval. In order to implement a new system or process, everyone on the team must be on board. If someone has concerns, discuss them with the team and address them to the best of your abilities. As soon as the team is in agreement about the process updates, define a date to have the process fully implemented into the project cycle and focus to have all dependencies in place prior to this date. Once that date is hit, run through a project iteration using these newly defined processes. Ensure that any unexpected issues or problems are documented along the way.

No system is perfect, and there is always room for improvement. At the end of each project iteration, meet with the team to reflect on the last run-through. Discuss any issues encountered and what can be done to resolve them. Processes are a constantly evolving project of their own that need constant tweaking based on company growth and presented projects. Be sure to update any documents related to decisions that come from this retrospection.

Implementing and updating processes for companies that are disorganized or inefficient can be quite an intimidating task. However, the process becomes a lot less daunting when broken down into 5 stages: Document current processes, identify bottlenecks current processes, Brainstorm and document the proposed updated processes, implement the process within the iteration of a project, and finally, perform a retrospective analysis of the process performance to improve the process as necessary. If a BSA follows these stages when implementing a new process, they are sure to assist the company with profitability-- that’s something everyone can get behind!

The D.E.B.S.C.I. Bug Reporting Format

As QA, we've all run across it at one time or another - a user enters a bug that is incredibly vague and leaves us clueless as to what the problem is. It's a sentence, or a fragment of a sentence, or a blank bug with the title "Page Got Broke" and an attached screenshot of a seemingly normal page (to you, at least). As a QA Resource who has interacted with outdated bug tracking systems and little guided structure to how to enter a bug, this issue can be persistent and troublesome. The time spent on back and forth between the client and resource, trying to squeeze all the information that is needed out of them could better be spent on resolving or discussing the issue with the development team as necessary.

It's clear that people need direction on how to enter a bug and what information to include. If you suggest providing "as much information as possible", you're likely to be flooded with unnecessary detail about the smallest issues. Suggesting that someone "use their best judgement" often leaves them feeling uneasy and unsure of themselves. The best solution, then, is short instruction that can be remembered by the vast majority of people. Why not take that idea and create a short acronym that people can easily remember? The D.E.B.S.C.I. bug reporting format is a way to minimize the amount of poorly reported bugs within a loosely structured bug tracking system. Let's break down the acronym:

Description - Provide a small paragraph explaining the issue. For example, "I navigated to the homepage in a new browser, went to the contact us page and saw an error"

Environment - Provide the environment this issue was identified in. Alpha, Beta, Production, etc.

Browser - Provide the browser you saw the issue in. This works for web applications and websites. If you are a software QA tester, you could replace this with "Version" for OS or program version

Steps to Replicate - Provide step by step instructions on what you did when you saw the issue. For example:

1.) Navigate to [URL of homepage]

2.) Locate and click on "Contact us" link in the footer

3.) Determine if user sees an error instead of the contact us page

Current Outcome - Provide the current outcome, or current bug. For example, "When a user clicks on the "Contact Us" link, they are taken to an error page

Intended Outcome - Provide the intended or expected outcome. For example, "When a user clicks on the "Contact Us" link, they are taken to the Contact Us page without error

Although not perfect, this simple acronym can assist in the QA process by allowing team members and clients to more easily remember the information that should be provided in a bug report. Clients no longer feel that they have no instruction on how to enter an issue; they can be reminded of the DEBSCI format via a quick e-mailed explanation or document as needed. While it has not fully eliminated vague bug reports, it has certainly diminished the number of bugs I get that make me scratch my head.

How do you guide clients and coworkers on entering bug reports?