Troubleshooting Secrets: How Top DevOps Engineers Find and Resolve Production Issues Quickly
Troubleshooting is an important skill for DevOps Engineers. Learn the step experience DevOps Engineers take to find and resolve Software Issues quickly
Errors are Common Occurrences, Don’t Get Intimidated
After a software product has been developed and deployed for its users, it is expected that the application will be subjected to many changes throughout its lifecycle. Within a DevOps team, the lifecycle of a software application is a recursive flow of planning, building, testing, deploying, observing, and planning. These changes are necessary due to changing customer or user requirements, optimization, or fixing of errors and problems in the software.
Errors and bugs in software cannot be 100% unavoidable so it is important to have plans and systems for handling errors and restoring the system to a state of functionality as quickly as possible. Going a step further to prevent errors from reoccurring is also very important, especially when working in a critical production environment. When errors, bugs or any other event in a system disrupts its availability or reliability, such a situation is known as an incident.
Important Skills for Troubleshooting During Incidents
In a software engineering team, when incidents occur on an application that has been deployed and is in use, it is common for all eyes to turn toward the DevOps team, after all, they are the group of friends that are in charge of deploying the application.
As a DevOps Engineer, it is important to be skilled in troubleshooting, to find and resolve incidents when they arise. To do so, you need to understand the infrastructure that has been configured for running the applications and the applications that have been deployed on your infrastructure.
Having this knowledge would help you to be more effective in your troubleshooting.
Knowing the software products are probably going to encounter errors and incidents is fine. However, when incidents occur, the reputation of the product is negatively affected, the business and its users can lose money and critical data can be compromised. So as much as it cannot be completely prevented, the team needs to be able to effectively respond to issues and resolve them quickly.
Speed and Accuracy are the two most important results in troubleshooting. During troubleshooting, a team needs Engineers who can quickly find the exact root cause of an incident and resolve it, thereby keeping the MTTR (Mean Time to Recovery) at a minimum. For this reason, effective DevOps Engineers are most often the go-to people when serious incidents occur within an organization. They have the knowledge of the systems, and the experience to quickly discover what may be causing an error in the system. But what are the steps that can be followed in a team to resolve issues and incidents within a low MTTR?
Staying Ahead of Errors
We have often heard that prevention is better than cure. But what we may not have heard is that detection is the best of them all. Indeed, we may not be able to completely prevent issues from arising in a software product, but it is important to be able to detect when there’s an abnormal behavior occurring in your systems or infrastructure
The first and most critical aspect of handling an issue is knowing when an issue is occurring. This is achieved through proper monitoring and alerting. Monitoring systems help to keep a constant view of the state of a system and alerting systems send out notifications to relevant teams and individuals when a system is in an abnormal state. This is how teams can detect issues and start working on a resolution before it begins to affect users.
Another important aspect of staying ahead of issues is how you learn from mistakes that have occurred before. A “god-state” in system reliability is to ensure that errors that have occurred before don’t occur again. This includes using post-mortems to document issues from detection to resolutions, developing playbooks for resolutions, setting better monitoring, and optimizing systems to prevent a particular known error.
So now, you have got your monitoring system properly set up, and you have your alerting setup to catch abnormalities and notify the appropriate parties. But you have a problem ahead of you that you have to solve. Let’s call it, Incident Chaos.
Having an Incident Management Plan
When an incident occurs anywhere, even outside a software team, the next thing that follows is chaos. Think of a house fire for instance, people around immediately have two primary goals; save yourself and/or stop the fire every other form of logic is thrown into the fire.
But a firefighting team comes with a more systematic approach;
first, quick analysis: the size of the building, type of roof, are there people in the building, then rescue, then fire containment and suppression, after which probably a post-fire analysis. Please note: this might not be the actual firefighting process.
However, the point is that to avoid chaos when handling an incident, there has to be a well-outlined plan of how to approach a resolution.
In software engineering, we call this an Incident Management Plan.
Incident Management Plan
Incident Management is the process by which a software development or IT operations team responds to emergency events and service disruptions to restore stability and functionality to the system.
An incident Management Plan would contain various policies around handling incidents including; communication procedures, escalation procedures, roles and responsibilities of team members during incidents, response procedures, documentation procedures, and many more
Some Incident Management steps include:
Detection and Reporting: When an issue is discovered by a monitoring system, a team member, or a customer, it would need to be logged for further investigation. This is a common use case of tools like JIRA, ServiceNow, etc.
Categorization and Prioritization: The issue is then given a level of severity based on the magnitude of its effect on the system. And it is prioritized based on that.
Response: This involves the steps taken to resolve the issue. Assigning it to the right team to investigate and resolve, escalating based on severity, communication with customers and the rest of the organization, and resolution are all done in this stage.
Learning and Improvement: These steps are also known as postmortems. They involve creating detailed documentation of what went wrong, how it was resolved, and how it has been prevented to avoid a future occurrence.
It is important to keep incident management blameless. Handling an incident is not the time to point fingers and punish those who caused an issue. All this does is erode the sense of ownership and autonomy of the engineers in your team. Rather focus on resolving the issue and learning from the incident.
Steps in Troubleshooting
So you’ve got your monitoring and alerting setup and are now able to detect issues before they begin to affect your customers. You have your Incident Management strategy to avoid chaos and to have a well-outlined approach to handling issues. But when you are looking to resolve an issue, what steps do you take to find the root cause and provide a solution?
In troubleshooting, the steps you follow may vary depending on the kind of issue you are fixing, however, there are fundamental steps that you can follow to make the process straightforward.
Identify the Indicators of Issues: The first thing during troubleshooting is to recognize when the performance of a system is abnormal. Your monitoring system is set up to show you the behavior of your infrastructure and applications, it takes knowledge and experience to understand when the behavior of a system is not optimal.
Analyzing Signals and Information: To understand the error better, it is then important to analyze the data available. This could include logs, traces, and metrics on the system. Logs give you a deeper view of what is going on within the application and traces would give you information on the flow of data across the various components of the system. This would help you pinpoint the area of impact and give you an idea of the cause.
Based on Analysis, List likely causes of the Problem: This is where knowledge of the system and experience becomes very important. The information gathered during the analysis of the logs, traces, and metrics would be used to infer possible causes of the problem. Which would then be further investigated to confirm the most probable cause of the problem
Propose a Solution: After the problem has been found, the next and most important step is to fix the problem. Solutions can come in the form of refactoring code, updating software, scaling the infrastructure, creating new features to address problems, etc.
Test the Solution: Evaluate the fix and confirm that the issue is resolved
Conclusion
Troubleshooting is an important skill for DevOps Engineers. It is important to understand the components of your infrastructure to be able to find and resolve issues that arise and get the system back up in a functional state as quickly as possible.
Support Our Journey
Our newsletter has the mission of creating valuable learning materials for DevOps and Infrastructure Engineers of all levels. We do the research and simplify complex concepts to make learning and development easier.
If you found this post useful, subscribe to our newsletter and share it with other Engineers in your network.
Subscribe to our paid newsletter for Exclusive Content, Deep Dives, and Behind the Scenes.