Preparing a Security Metrics Framework
Security Metrics is a key tool in the business of Information Security. But it has often been debated that objective measurement of security related parameters is by and large difficult and when possible, rather inaccurate. However, with security having emerged as a design criterion in most organizations, systems and software, and financial consequences of security violations having become extremely high, there is growing acceptance amongst most CISOs and even the CIO/CTOs about the need for security metrics. It is also a prevalent view that these metrics should be aligned to business KRAs.
But the agreement almost ends there and there are differences of opinion regarding many aspects. In most organizations, an independent Information Security stream evolves in due course of time and is often carved out of the IT department. Thus, besides having business KRAs-aligned security metrics, presence of apex level directives is essential as a key facilitator for the security metrics program.
With that in place, the next step would be to prepare an exhaustive body of metrics which cover all areas of the operational landscape. To arrive at the set of metrics that best fit your organization, we need to answer a few very relevant questions:
a. Which metrics are relevant for the organization? Answer to this question will lead to weeding out those metrics which add bulk to the metrics program but no value. In security metrics, more is not necessarily better. Also, there may be a tendency to propose metrics based on the ease of availability, the so called criterion of STTCBM (“Security Things That Can Be Measured”). However, ideally we should like to see a small collection of quantitative values that are a good characterization of the current security state of an organization as well as a measure of its dynamic rate of change (hopefully improvement) over time. For executive audiences this is critical. The best candidates for such metrics should first focus on outcomes as opposed to the policies, people, processes, and technologies that have been put in place to achieve those outcomes. For example: Number of Security Incidents of varying severity for a given period in the entire organization, Mean Time between Security Incidents and Mean Time for closure of a Security Incident.
b. Are these measurable? In the scope of Security Metrics, we have several tangible or objective things such as number of virus incidents or number of people trained etc. And we also have certain intangible things such as level of security awareness, effectiveness of Risk Assessment etc. It is relatively easy to measure tangible aspects. But measuring intangible things poses several challenges and it is easy to give into the temptation of sticking to only those which can be measured. Given the intangible but vital nature of security awareness, it is definitely worth putting effort into the measurement of such subjective factors, rather than relying entirely on easy-to-measure but largely irrelevant objective factors. Measuring anything makes it easier to drive improvements but measuring the wrong things leads to improving the wrong things.
c. What Business goals do they serve? The hard part is not measurement per se but is figuring out what parameters to measure and what needs to be done to make them measurable. Information security is a complex field with ramifications throughout the organization. It is unrealistic to expect to find a few simple measures. It is important to think long and hard about what actually needs to be improved before building a measurement system, or at least before casting it in stone. First of all one should be absolutely clear about the purpose of the measurements, how it will be reported and what use do they serve. We should know who will ‘consume’ them, and what we expect the consumers to do with them. If you have a clear purpose in mind (e.g. to inform management about the information security program, to justify the investment and/or to improve the program’s effectiveness), it will help determine which metrics you need.
Overcoming Challenges
After the above questions have been answered, we then have to initiate the process of data collection from the relevant systems or processes, analyse this data and process it to formulate the metrics. Some challenges may be found at this stage. There may be challenge of getting stakeholders buy in for collecting metrics data from the source(s). And once that is surmounted an even bigger challenges could be automating these metrics.
The concerned departments whose operations come under the scanner may have issues in making the data for metrics available while the Information Security department remains thirsting for information. Nothing paves the way smoother for a Security Metrics program than having direct support from top management. Every security conscious organization should have an apex body or a council where security matters are tabled in a regular periodicity. This forum then serves as the driving force behind not only the Information Security governance and policy implementation processes but also that of the Security Metrics program. The security metrics should be handled by a metrics program manager under the CISO and the process should be periodically audited independently.
Metrics are very good if you can obtain them in an automated way. It is of importance because errors inherent in human intervention can be avoided; also the program becomes sustainable as well as scalable. Another critical aspect is independence. When automated the data would come independently from the source of collection rather than provided by the owner of the source. Essentially this would avoid a conflict of interest. However, automating metrics collection/reporting has its own pitfalls, the main one being a bias towards metrics that can be collected and analyzed in an automated fashion, which implies using data extracted from the IT systems. Unfortunately, some of the most interesting aspects of Information Security are the people and not the systems. For example, the effectiveness of policies and awareness activities can be measured by observing employee's changing behaviors. And unfortunately very few of those changes can be easily measured.
Objective and Relevance
There are many advantages associated with security metrics such as facilitating risk quantification leading to management decisions, optimum resource allocation etc. However, one of the main goals is that metrics should lead to positive change in behavior. While they should focus on generating and presenting data which point to the deficiencies that need to be plugged, it must be remembered that the very behaviors that these metrics want to amend have come into existence not purely out of users uninformed habits. Often, it is because of the way IT is rolled out in a company. And almost always, it is a cultural issue; culture of the organization as also of its people. It is clearly apparent that security is not a technology problem; we may have the most complex mathematics and cryptography; but all the security in the world may not help. Security is a social issue; it is a trust issue. The gaming industry and marketing industry are very adept at manipulating peoples’ behaviors. We may actually need to involve Security Psychologists and may be some marketing people to prepare a security marketing campaign to motivate people to do the right thing. And the right thing is of course; that which has been measured to be the most desirable and effective behavior for the security issue at hand.
Information security, like risk, is a notoriously difficult area to measure, the main problem being how to measure the ‘lack of incidents’. Some concerns in doing this are enumerated below:
• If the information security risk analysis is accurate, and if effective information security controls are implemented, security incidents should be avoided, or at least their number and severity should reduce. (That’s primarily what we’re trying to achieve through information security, though it is not all. There are other important benefits too: strong information security controls improve management’s confidence, making it easier to enter into new business ventures that would otherwise be too risky to consider. For some like financial services, security is absolutely valuable for their core business and therefore a fundamental brand value.)
• If we simply measure the number and severity of incidents, we will have some numbers to play with but what will those numbers actually tell us?
o If the numbers are lower than before we started the information security program, we could claim success but what if the number and severity of incidents had fallen anyway? For example, we see fewer website defacements now than a decade ago: is that because websites are better protected now (control improvement) or because there are fewer hackers actively targeting website defacements (threat reduction)? Or in fact is it because website defacements are no longer as newsworthy as they once were, in other words there has been no real improvement in our controls but we don’t hear about the control failures?
o If the numbers are higher than before, does that necessarily mean our controls are ineffective? Or could it mean that the threats and impacts have increased and we have not kept pace?
• It is practically impossible to measure objectively what could have happened had we not improved our information security controls. The issue is sometimes one of conjecture.
Conclusion
Meanwhile, work on development of an ISO standard on Information Security Management (Measurement) has started from 2005 onward and continues apace. Publication of this standard is expected soon. The standard is intended to help organizations measure and report the effectiveness of their Information Security Management Systems, covering both the security management processes (defined in ISO/IEC 27001) and the security controls (ISO/IEC 27002). The purpose of the Information security management measurements development and implementation process, defined in this Standard is to create a base for each organization to collect, analyse, and communicate data related to ISMS processes. This data is ultimately to be used to base ISMS-related decisions and to improve implementation of ISMS.
