Inforonics Blog

Mistake 2 when Fixing Major System Failures: Assuming…

ciohooper — Wed, 16 Feb 2011 16:43:12 +0000

Handling major system failure mistake number 2: Broad based assumptions without facts. “It’s probably the network” or “the server” or “the last code release”… or any other scapegoat. A systemic system failure that has been plaguing an organization is rarely impacted by these discrete failures in and of themselves. Assuming these areas to be the cause pre-disposes the team to certain opinions. You’ve heard the expression about what happens when you assume. I’ll spare you the spelling lesson and get to the point. Don’t excite your team to chase red herrings. When you point fingers and play the blame game you waste time and generally end up in the same place you started. It’s not productive it creates animosity and most of all it distracts from the issue at hand. Every possible cause of the problem across the entire infrastructure must be considered for fault isolation. That’s a waste of time and impossible, you say! Not at all. In fact, once you have mastered this troubleshooting technique, you will find isolating the “where” of a problem will almost immediately tell you the “what”. Here is how you do it: Start at the user.

Either through remote tools, or even better if you are able stand over the shoulder of an affected user and see the issue in action, that will really help. It helps the user community see presence from IT, which is good politically, and it helps quantify the business impact. Most importantly it gives you the ability to rapidly execute the process of elimination. (Need a tool? See below. )

By starting at the user interface you are able to very quickly determine the availability and response of many of the underlying components. For instance, if the user is able to get past the login screen, you have validated PC, LAN, WAN, DC, and server to server connectivity in one shot. DNS, authentication and some business logic is also being validated as the user works through the system to the point of impact. Note: If you are having sporadic performance or availability failures, you may have to utilize and end-user experience tool to capture over longer periods of time or across geographic locations.

Collect your data and go to the whiteboard. Starting at the user-interface, start tracing the transaction to the underlying systems that support that specific problematic business process. Here is when a well configured Configuration Management System pays for itself 10-fold. Now you can deploy very targeted diagnostic tools at these specific locations to determine response times that are affecting the overall business impact. Only then can you accuse or excuse a particular component.

Obviously having a well-executed end-to-end monitoring system accelerates this process. None the less, it should not take more than 1 week to get to the diagnostic phase and start isolating bottlenecks within the infrastructure.

Email me at Matthew.Hooper@inforonics.com and I will send you a utility we used to trace the end-user experience and inserts markers into the network stream for easier diagnostics. I have many other blog links of my own and others who have written about tools that I am happy to share. Of course I’m happy to have the Inforonics team speak to you about getting tools as a service.

Classic mistakes when handling critical incidents

ciohooper — Thu, 03 Feb 2011 00:44:22 +0000

‘It’s obvious that when a critical production system goes offline restoring the system back to operation is a mission worthy of focused resources. How many times have you sat in a meeting where ridiculous ideas are thrown around in a vain attempt to restore the business service? Probably many! A great deal of time and energy is spent “brainstorming” in an attempt to restore the system often with unproductive results.

Here are a couple of classic mistakes that organizations make when a system crashes:

1) You start fixing the issue without knowing what the problem is.

I’m not talking about Problem Management, so you can put your ITIL bible down for a second. What I mean is, what is the impact, what’s not happening from a business perspective because of this condition.

Because IT Operations folks are traditionally reactive, they all too often start fixing system issues without knowing what is truly broken. Engineers will start applying solutions to situations without fully understanding what happened in the first place. It is imperative that you take a moment to define what “resolved” will mean in each scenario. The concern must be clearly identified before solutions can be thrown out. Putting a system back on-line may not solve the business impact.

HINT: Identify the Critical Success Factor (CSF) and then identify 3 Key Performance Indicators (KPIs) that will confirm that the issues are resolved. I’m not talking about writing a book here. Simply state what success will look like and how we will measure it. For instance, 40 users are able to input 200 policies in a work day period. That’s the CSF. Now how will you measure this? We will record how many policies are entered into the system per hour. We will measure the number of users logged on to the system per hour. We will measure the number of failed attempts. Thus we know have Key Performance Indicators that we state: 1) x numbers of policies can be done per hour. 2) x number of policies fail out of y 3) x number of agents will experience a y% fail rate.

CSF and KPIs should be measureable so that an evaluation can be made as to whether the issue has been resolved. In this example if we hit KPI1 -30 policies per hour, KPI2 – Less than 1 out of 6 fail KPI3 – 5 agents or less experience an 15% fail rate, than we know that we are achieving the required business condition, and we have solved the problem.

Do you notice the mindset difference here? Have we solved all the problems? Well if you ask the engineering team the answer is obviously no. We are still experiencing failures. However, from a business perspective, we have solved the problem.

Stuff breaks, crap happens, that’s life. IT can and never will be able to provide a perfect operating environment. That is why we need SLA’s that are realistic and service driven, not metric driven. So remember, before you dive deep and call in the SWAT team (please don’t do that, we’ll get to that in an upcoming blog) make sure you define what the problem and the success look like. This little bit of effort and time will get the business up and running faster and save you lots of frustration.

Next blog, let’s break down another classic mistake in handling incidents: Assuming anything.

Benefits Change Management Brings to your Organization: Faster time to value

ciohooper — Mon, 20 Dec 2010 20:31:25 +0000

Innovation occurs with precision and understanding. Protection of critical business processes can be secured by reducing the risks and exposures to changes of the technology that automate and manage these business processes. Using composite technologies like a Configuration Management System (CMS) allow disparate teams to utilize a common tools space to visualize and analyze impacts to business services from the assets undergoing change. Thus hands-on resources can make better business decisions and provide better guidance to management of risk by correlating request for changes (RFCs) to configuration items (CIs). This brings manay benefits to the operations, here are just a few:

Guarding against unapproved changes that may open unintended security holes.
Improvement of service and business continuity by identifying single points of failure.
Improved monitoring by seeing the dependencies which can enable engineers to quickly identify a new incident that occurs and know instantly why or how it happened.
Ensure vital business functions were properly tested to protect the operation of the business.
Provide visibility into each request for a changes affect on CI’s & ensure new CI’s are provisioned with the appropriate levels of managemt (i.e. Monitoring, Documentation, Anti-Virus, etc…)

Putting Change Management in place and getting an organization to readily adopt a consistent and measurable process is difficult. It takes executive sponsorship and the right level of training and tools to make it work. Below are some guidlines we fee will help a Change Management implementation strategy bring success:

1) Create, Document, and Communicate Policy

The first step in implementing Change Management is changing your people. Individuals need to learn the reward or penalty for making changes outside of the approved procedures. Start with solid Change and Release policies backed by an enterprise architecture standard that is supported by strong configuration management. Unless you build traceablility into your systems, you will never be able to govern this process. Thus stating a violation in your policy without the ability to govern it will be seen as nothing more than an idle threat. Undocumented and unapproved changes to your environment are a risk to your business and potential the jobs of your employees. Treat it this way!

2) Create Process Integration Points

You must define and determine the inputs into the RFC process. Leaving change management out as a silo’d process is setting up the team and charter for failure. ITIL preaches the 7 R’s of change, which are good to know, but it’s more than asking questions. Change Management is a governer, not a designers, creator or supporter. Thus if your process does not have the necessary give and take influences and touchpoints to these other outcome aspects, it will fail. The “must-haves” touchpoints must include:

Incident or problem fixes
New feature functions
Architectural improvements for availability capacity or security
Continuous Service Improvement
Finding or Business Alignment adjustment

3) Create effective change categories and build approval procedures accordingly.

All to often I have seen change advisory boards of 20 or more employees spending an entire afternoon reviewing hundreds of alterations. Seriously, this is a waste of valuable time! Knock it off, there is real work to do. Remember the benefit is accelerated “managed changes” not useless beauracracy that holds meetings as a facade of risk protection. A group of select people who are responsible, informed, and empowered are more than capable of approving changes if they have been categorized correctly and approved functionally. Keep your CABs (Change Advisory Boards) small and focused. Your RFCs (Request for Changes) will have better analysis against them, more evaluation of impact and given better attention.

Before the next major issue arises, seize the opportunity to make a change. Put controls in place that build visibility into your organizational efforts and you will see that projects and tasks do not have to turn into chaos even at today’s fast and furious business pace. Dramatically increase the efficiency of change requests in your organization by utilizing tools and processes that streamline and track your efforts. Focus resources on innovation – not fighting fires.’ ‘The Real Value of Change Management’

Now the big question: How do you manage changes with Cloud Computing?

We’ll tackle that in our next posting.

Is there value to Change Management?

ciohooper — Mon, 18 Oct 2010 21:05:54 +0000

Change is usually something people generally avoid. It typically causes uneasiness for people, when they hear things are going to change. However, for various reasons change is a reality of businesses. New applications, hardware, outsourcing, or cloud adoption are actions that will require changes to enterprise IT organizations. Even smaller scale changes like integrating systems or increasing capacity to accommodate growth requires changing something, usually something that is working. So why the fear? “If it ain’t broke, don’t fix it” is the old adage that helps illustrate the fact that things typically break because someone changes something. In IT, we know all too well that changes can have a negative impact on business services when proper procedures are not followed. What separates a low risk transition from the utter chaos that can consume a flawed release? Strong Change Management. As IT becomes the business enabler, it must embrace Change Management within its operating procedures. Change Management is a core process discipline in IT Service Management. The objective of ensuring that standardized methods and procedures are used for efficient handling of changes within the IT infrastructure protecting the value of services being consumed. It is well documented that implementing Change Management in your IT organization will minimize the number and severity of incidents related to adjustments that will occur. Change Management helps businesses adapt by implementing controlled, predictable, and measurable procedures. It can also cause an IT organization to crawl to a halt, introducing an overwhelming amount of red-tape, bureaucracy. When companies begin thinking about implementing a Change Management process they are often faced with striking a balance between tight and rigid controls that mitigate risk and the expedience and business demand for the needed modification. So how do we get there?

Change Management – “Seems like a lot of paperwork and red-tape to me.”

No argument here. However, this fact does not mean we should forgo the controls and documentation. For example, think about changing the ownership of a house from one person to another. It’s bureaucracy when the transaction goes smoothly. It’s protection when it doesn’t. Why do we say that? Say your house has some damage that was not reported, thus the seller over-stated the value of the home. The failure to get the right level of inspection (testing), risk mitigation (title and lien clearance), and insurance (roll-back plans) in place, could leave you in a lot of trouble. Whereas, if everything goes fine you say “why did we need all that stuff”.

When IT operations are running smoothly it doesn’t seem necessary to put sign-off scheduling and controls into place. However, when an implementation goes wrong or conflicts with another set of work-efforts people get frustrated with the chaos and demand that order be restored. Too often, it’s only after a business takes a hit from a change gone wrong that people scramble to identify the problem. In the mean time revenue decreases, customer service deteriorates, client satisfaction slumps – in short business fails. Time and money are wasted on resolving issues that could otherwise have been prevented with better coordination of configuration, build, and deployment efforts. Once the fire has been put out the issue seems to fade along with the urgency to control the chaos. That is until another problem arises.

Next article: Benefits Change Management Brings to your Organization.

Inforonics SupportCast Weekly – Episode 8: Managing Operations

Inforonics — Fri, 15 Oct 2010 00:26:18 +0000

Managing operations requires a clear picture of what you are doing and what you need to be doing. For any IT Organization SLA’s are a way to ensure you have the right resource doing the right thing. For Inforonics SLA’s are a competitive advantage, we view them as an opportunity to help manage our client’s expectations. Listen in as Inforonics VP of Client Services, Jeff Roberts, and CIO, Matt Hooper, discuss the details of metrics, KPI’s and reporting on SLA’s within Inforonics managed services and technical support operations.

Inforonics SupportCast Weekly – Episode 8: Managing Operations from Inforonics on Vimeo.

Download MP3 version here: Inforonics Supportcast Weekly Episode 8: Managing Operations

Inforonics SupportCast Weekly – Episode 7: Visual Operations

Inforonics — Fri, 08 Oct 2010 15:24:56 +0000

Visual Operations is concept that allows decision makers to take action based on visual outputs. Responding to graphs, charts, dashboards, and visual cues is a fundamental in a multi-tenant, multi-client support organization like Inforonics. How do we get these dashboards to show us the right amount of data in an action oriented way? Tune in as Inforonics CIO, Matt Hooper, walks through the knowledge mapping of Data, Information, Knowledge and Wisdom.

Inforonics Supportcast Weekly – Episode 7: Visual Operations from Inforonics on Vimeo.

Download MP3 version here: Inforonics Supportcast Weekly Episode 7: Visual Operations

Inforonis SupportCast Weekly – Episode 6: Service Level Agreements

Inforonics — Tue, 05 Oct 2010 01:17:20 +0000

SLA’s, Service Level Agreements, a term that makes most IT professionals cringe. Are SLA’s really our enemy? Some sort of stick to keep us in line, or are they a protection for us, allowing us to justify resources or increase capabilities to meet the needs of both parties in a mutually beneficial way? Listen in as Inforonics’ VP of Client Services, Jeff Roberts, and CIO, Matt Hooper, talk about “Dynamic SLA’s”, and how they can meet the needs of the business and work as an enabler and not a moderator between service sponsors and service providers.

Inforonics Supportcast Weekly – Episode 6: Service Level Agreements from Inforonics on Vimeo.

Download MP3 version here: Inforonics Supportcast Weekly Episode 6: Service Level Agreements

Inforonics Supportcast Weekly – Episode 5: Enterprise Monitoring and Alerting

Inforonics — Tue, 28 Sep 2010 15:52:58 +0000

“Just monitor the service, and if it goes down fix it.” You have to love the pointed-hair boss comments like this. Activities like monitoring and alerting are essential to Incident Management, Event Management and Problem Management processes. The completion and execution of such activities are extremely difficult. The complexity in instrumentation, collection, reporting, filtering and alarming is great, yet we sometimes expect it to be easy. Listen in as Inforonics CTO and VP of Infrastructure, Ted Wilbur, and CIO, Matt Hooper, discuss how Inforonics approaches enterprise management from a manage service provider perspective.

Inforonics Supportcast Weekly – Episode 5: Enterprise Monitoring and Alerting from Inforonics on Vimeo.

Download MP3 version here: Inforonics Supportcast Weekly Episode 5: Monitoring and Alarming

Value of IT Outsourcing

ciohooper — Tue, 21 Sep 2010 16:07:55 +0000

We’ve been discussing how IT can become more innovative from a CIO perspective. Part of that requires us to understand what IT is spending its time on and is that effort aligned with the core business. The other area that needs investigation is management of your functional areas. Are they delivering IT services in the most cost effective manner? For example, many organizations have network and server management teams who care for the networks/servers and the associated devices. Yet, no one owns the carrier services of the network, or replacement services for the servers. It is important that you have a reliable responsive network, but not that you own every component required for service delivery. With that mindset some have looked at server and network monitoring services, patch management services, and basic restore services as commodities and chosen to find providers for theses services as well. Delegating that management to someone else may be more cost effective and may actually give you more control. More control? How?

RightSourcing

IT is faced with many pressures, one of which is to innovate. Not all groups have the capability to build the organizational strength and diversity needed to keep up with business demands. Credit Suisse CIO Karl Landert stated in his July 2009 interview with Wall Street and Technology that agility and efficiency are key to building long term success. Landert built his empire through organizational changes and recognizing the value of contingent forces to augment his core competencies. In order to remain cost-effective and forward thinking in your service delivery you must build a service provider network that is focused on enablement of multiple providers. Focusing on building and managing this service network of providers will allow more time for proactive engineering and architecture work that will provide stable and innovative results. A few things these proactive resources can focus on are Event Management and Portfolio Management.

Event Management prioritizes incidents and problems allowing the right focus by the right resources at the right time. Portfolio and project management focuses development. Combining these brings you a long way towards enabling the business and technology convergence that drives long term success.

Next article we will tackle the “real value of Change Management”.

Inforonics Supportcast Weekly – Episode 4: Metrics, Metrics, Metrics

Inforonics — Fri, 17 Sep 2010 16:17:38 +0000

The great thing about automation and robots are they do what they are told to do. The challenge, however, arises when an introduction of a variant requires a change, a shift in behavior that will adjust to the right course of action. When managing an IT Enterprise, or dealing with end-users for customer and technical support, the world is full of variants. Constant adjustment, intervention and direction setting are required. Metrics are thus key to knowing where changes need to be made, and if they are the right changes. ITIL, ITSMF, HDI and others have taught us the sooner you can identify a trend, the sooner you can head off a negative outcome. Listen in this week as Inforonics Scott Taylor, Support Architect, and Matt Hooper, CIO, discuss the metrics they use to manage the IT enterprise and customer support functions.

Inforonics Supportcast Weekly – Episode 4: Metrics, Metrics, Metrics from Inforonics on Vimeo.

Audio Version

Links to MP3 Version here: http://inforonics.files.wordpress.com/2010/09/isw-episode4.mp3