Skip to Content

PRAGMATIC Security Metrics

Applying Metametrics to Information Security

By W. Krag Brotby, Gary Hinson

Auerbach Publications – 2013 – 512 pages

Purchasing Options:

  • Add to CartHardback: $69.95
    978-1-43-988152-1
    January 8th 2013

Description

Other books on information security metrics discuss number theory and statistics in academic terms. Light on mathematics and heavy on utility, PRAGMATIC Security Metrics: Applying Metametrics to Information Security breaks the mold. This is the ultimate how-to-do-it guide for security metrics.

Packed with time-saving tips, the book offers easy-to-follow guidance for those struggling with security metrics. Step by step, it clearly explains how to specify, develop, use, and maintain an information security measurement system(a comprehensive suite of metrics) to help:

  • Security professionals systematically improve information security, demonstrate the value they are adding, and gain management support for the things that need to be done
  • Management address previously unsolvable problems rationally, making critical decisions such as resource allocation and prioritization of security relative to other business activities
  • Stakeholders, both within and outside the organization, be assured that information security is being competently managed

The PRAGMATIC approach lets you hone in on your problem areas and identify the few metrics that will generate real business value. The book:

  • Helps you figure out exactly what needs to be measured, how to measure it, and most importantly, why it needs to be measured
  • Scores and ranks more than 150 candidate security metrics to demonstrate the value of the PRAGMATIC method
  • Highlights security metrics that are widely used and recommended, yet turn out to be rather poor in practice
  • Describes innovative and flexible measurement approaches such as capability maturity metrics with continuous scales
  • Explains how to minimize both measurement and security risks using complementary metrics for greater assurance in critical areas such as governance and compliance

In addition to its obvious utility in the information security realm, the PRAGMATIC approach, introduced for the first time in this book, has broader application across diverse fields of management including finance, human resources, engineering, and production—in fact any area that suffers a surplus of data but a deficit of useful information.

Visit Security Metametrics. Security Metametrics supports the global community of professionals adopting the innovative techniques laid out in PRAGMATIC Security Metrics. If you, too, are struggling to make much sense of security metrics, or searching for better metrics to manage and improve information security, Security Metametrics is the place. http://securitymetametrics.com/

Reviews

Like all books on metrics, PRAGMATIC Security Metrics: Applying Metametrics to Information Security makes the statement that "you can't manage what you can't measure". The authors claim that other books on information security metrics discuss number theory and statistics in academic terms. This title promises to be light on mathematics and heavy on utility and is meant as a how-to-do-it guide for security metrics.

As to the title, PRAGMATIC is an acronym for the basis of the method of the book, in using metrics that are predictive, relevant, actionable, genuine, meaningful, timely, independent and cost. After reading the first chapter, PRAGMATIC Security Metrics: Applying Metametrics to Information Security looks like it may live up to its promise of being able to use metrics not only to track and report performance but to identify problem areas and opportunities, and drive information security improvements. If so, this could be the metrics book a lot of information security professionals have been waiting for.

—Ben Rothke, CISSP, CISM, Information Security Manager, Wyndham Worldwide; and author of Computer Security: 20 Things Every Employee Should Know, writing on the RSA Conference Blog, www.rsaconference.com

Contents

Introduction

Why Have We Written This Book?

What’s Different about This Metrics Book?

Who Are We Writing This For?

Who Are We?

Krag Brotby

Gary Hinson

What We’ll Be Talking About

Defining Our Terminology

What We Expect of You, the Reader

Summary

Why Measure Information Security?

To Answer Awkward Management Questions

To Improve Information Security, Systematically

For Strategic, Tactical, and Operational Reasons

For Compliance and Assurance Purposes

To Fill the Vacuum Caused by Our Inability to Measure Security

To Support the Information Security Manager

For Profit!

For Various Other Reasons

Summary

The Art and Science of Security Metrics

Metrology, the Science of Measurement

Governance and Management Metrics

Information Security Metrics

Financial Metrics (for Information Security)

(Information Security) Risk Management Metrics

Software Quality (and Security) Metrics

Information Security Metrics Reference Sources

Douglas Hubbard "How to Measure Anything" (Hubbard 2010)

Andrew Jaquith: Security Metrics (Jaquith 2007)

NIST SP 800-55: Performance Measurement Guide for Information Security (NIST 2008)

Debra Herrmann: Complete Guide to Security and Privacy Metrics (Herrmann 2007)

Krag Brotby: Information Security Management Metrics (Brotby 2009a)

Lance Hayden: IT Security Metrics (Hayden 2010)

Caroline Wong "Security Metrics: A Beginner’s Guide" (Wong 2012)

ISO/IEC 27004: Information Security Management–Measurement (ISO/IEC 27004 2009) 3.7.9 CIS Security Metrics (CIS 2010)

ISACA

Specifying Metrics

Metrics Catalogs and a Serious Warning About SMD

Other (Information Security) Metrics Resources

Summary

Audiences for Security Metrics

Metrics Audiences Within the Organization

Senior Management

Middle and Junior Management

Security Operations

Others with Interest in Information Security

Metrics Audiences From Without the Organization

Summary

Finding Candidate Metrics

Preexisting/Current Information Security Metrics

Other Corporate Metrics

Metrics Used in Other Fields and Organizations

Information Security Metrics Reference Sources

Other Sources of Inspiration for Security Metrics

Security Surveys

Vendor Reports and White Papers

Security Software

Roll-Your-Own Metrics

Metrics Supply and Demand

Summary

Metametrics and the PRAGMATIC Approach

Metametrics

Selecting Information Security Metrics

PRAGMATIC Criteria

6.3.1 P = Predictive

6.3.2 R = Relevant

6.3.3 A = Actionable

6.3.4 G = Genuine

6.3.5 M = Meaningful

6.3.6 A = Accurate

6.3.7 T = Timely

6.3.8 I = Independent

6.3.9 C = Cost

Scoring Information Security Metrics against the PRAGMATIC Criteria

Other Uses for PRAGMATIC Metametrics

Classifying Information Security Metrics

6.6.1 Strategic/Managerial/Operational (SMO)Metrics Classification

6.6.2 Risk/Control Metrics Classification

6.6.3 Input–Process–Output (Outcome) Metrics Classification

6.6.4 Effectiveness and Efficiency Metrics Classification

6.6.5 Maturity Metrics Classification

6.6.6 Directness Metrics Classification

6.6.7 Robustness Metrics Classification

6.6.8 Readiness Metrics Classification

6.6.9 Policy/Practice Metrics Classification

Summary

150+ Example Security Metrics

Information Security Risk Management Example Metrics

Information Security Policy Example Metrics

Security Governance, Management, and Organization Example Metrics

Information Security Financial Management Metrics

Information Security Control-Related Metrics

Metrics for Business Alignment and Relevance of Controls

Control Monitoring and Testing Metrics

Financial Information Security Metrics

Information Asset Management Example Metrics

Human Resources Security Example Metrics

Physical Security Examples

IT Security Metric Examples

Access Control Example Metrics

Software Security Example Metrics

Incident Management Example Metrics

Business Continuity Management Examples

Compliance and Assurance Metrics Examples

Summary

Designing PRAGMATIC Security Measurement System

Brief History of Information Security Metrics

Taking Systems Approach to Metrics

Information Security Measurement System Lifecycle

Summary

Advanced Information Security Metrics

High-Reliability Metrics

Indicators and Proxies

Key Indicators

Key Goal Indicators (KGIs)

Key Performance Indicators (KPIs)

Key Risk Indicators (KRIs)

Critical Success Factors (CSFs)

Targets, Hurdles, Yardsticks, Goals, Objectives, Benchmarks, and Triggers

Summary

Downsides of Metrics

Numbers Don’t Always Tell the Whole Story

Scoring Political Points through Metrics

Implausible Deniability

Metrics Gaps

On Being Good Enough

What Not to Measure

Summary

Using PRAGMATIC Metrics in Practice

Gathering Raw Data

Sampling

Automated Data Sources

Observations, Surveys, and Interviews

Online or In-Person Surveys

Scoring Scales

Audits, Reviews, and Studies

Data Analysis and Statistics

Data Presentation

General Considerations

Analytical Tools and Techniques

Reporting Tools and Techniques

Presentational Tools and Techniques

Graphs, Figures, Diagrams, and Illustrations

Drawing Attention to Specific Issues

Using, Reacting to, and Responding to Metrics

Periodic versus Event-Driven Reporting

Summary

Case Study

The Context: Acme Enterprises, Inc.

Information Security Metrics for C-Suite

Information Security Metrics for the CEO

Information Security Metrics for the CIO

Information Security Metrics for the CISO

Information Security Metrics for the CFO

Information Security Metrics for the VP of Production

Information Security Metrics for the VP of Marketing

Information Security Metrics for Management and Operations

Information Security Metrics for External Stakeholders

Acme’s Information Security Measurement System

Summary

Conclusions

Take-Home Lessons from This Book

On Pragmatism and Being PRAGMATIC

On Giving You the Confidence and Skills to Have a Go

On Improving the Quality of Your Management Information through Metametrics

On Improving Metrics of All Sorts

Your Chance to Advance the Profession and the Practice of Metrics

An Action Plan to Take Away

Summary

Appendix A: PRAGMATIC Criteria

Appendix B: Business Model of Information Security (BMIS)

Appendix C: Capability Maturity Model (CMM)

Level 1–Initial

Level 2–Repeatable

Level 3–Defined

Level 4–Managed

Level 5–Optimizing

Appendix D: Example Opinion Survey Form

Security Awareness Survey on Malware

Appendix E: SABSA Security Attributes Table

Appendix F: Prototype Metrics Catalog

Appendix G: Effect of Weighting the PRAGMATIC Criteria

Appendix H: ISO27k Maturity Scale Metrics

Appendix I: Sample Management Survey

Appendix J: Observer Bias

Appendix K: Observer Calibration

Appendix L: Bibliography

Author Bio

Krag Brotby has 30 years of experience in the area of enterprise computer security architecture, governance, risk, and metrics and is a Certified Information Security Manager (CISM) and Certified in the Governance of Enterprise Information Technology qualifications. Krag is a CISM trainer and has developed a number of related courses in governance, metrics, governance-risk-compliance (GRC), and risk and trained thousands on five continents during the past decade.

Krag’s experience includes intensive involvement in current and emerging security architectures, IT and information security metrics, and governance. He holds a foundation patent for digital rights management and has published a variety of technical and IT security-related articles and books. Brotby has served as principal author and editor of the Certified Information Security Manager Review Manual(ISACA 2012) since 2005, and is the researcher and author of the widely circulated Information Security Governance: Guidance for Boards of Directors and Executive Management(ITGI 2006), and Information Security Governance: Guidance for Information Security Managers(ITGI 2008a) as well as a new approach to Information Security Management Metrics(Brotby 2009a) and Information Security Governance; A Practical Development and Implementation Approach(Brotby 2009b).

Krag has served on ISACA’s Security Practice Development Committee. He was appointed to the Test Enhancement Committee, responsible for testing development, and to the committee developing a systems approach to information security called the Business Model for Information Security (BMIS). He received the 2009 ISACA John W. Lainhart IV Common Body of Knowledge Award for noteworthy contributions to the information security body of knowledge for the benefit of the global information security community.

Krag is a member of the California High Tech Task Force Steering Committee, an advisory board for law enforcement. He is a frequent workshop presenter and speaker at conferences globally and lectures on information security governance; metrics; information security management; and GRC and CISM preparation throughout Oceania, Asia, Europe, the Middle East, and North America. As a practitioner in the security industry for three decades, Krag was the principal Xerox BASIA enterprise security architect and managed the proof-of-concept project, pilot, and global PKI implementation plan. He was a principal architect of the SWIFT Next Gen PKI security architecture; served as technical director at RAND Corporation for the cyber assurance initiative; as chief security strategist, was the PKI architect for TransactPlus, a J.P. Morgan spinoff; and developed policies and standards for a number of organizations, including the Australian Post Office and several U.S. banks.

Recent consulting engagements include security governance projects for the Australia Post, New Zealand Inland Revenue, and Singapore Infocom Development Agency. Clients have included Microsoft, Unisys, AT&T, BP Alyeska, Countrywide Financial, Informix, Visa, VeriSign, Digital Signature Trust, Zantaz, Bank Al-Bilad, J.P. Morgan Chase, KeyBank, Certicom, and Paycom, among others. He has served on the board of advisors for Signet Assurance and has been involved in significant trade secret theft cases in the Silicon Valley.

Gary Hinson—Despite his largely technical background, Dr. Gary Hinson, PhD, MBA, CISSP, has an abiding interest in human factors—the people side as opposed to the purely technical aspects of information security and governance. Gary’s professional career stretches back to the mid-1980s as both a practitioner and manager in the fields of IT system and network administration, information security, and IT auditing. He has worked for some well-known multinationals in the pharmaceuticals/life sciences, utilities, IT, engineering, defense, and financial services industries, mostly in the United Kingdom and Europe. He emigrated to New Zealand in 2005 and now lives on a "lifestyle block" surrounded by more sheep than people.

In the course of his work, Gary has developed or picked up and used a variety of information security metrics. Admittedly, they didn’t all work out, but such is the nature of this developing field (Hinson 2006). In relation to programs to implement information security management systems, for example, Gary had some success using conventional project management metrics to guide the implementation activities and discuss progress with senior managers. However, management seemed curiously disinterested in measuring the business benefits achieved by their security investments despite Gary having laid out the basis for measurement in the original business cases. And so started his search for a better way.

Since 2000, Gary has been consulting in information security, originally for a specialist security consultancy in London and then for IsecT Ltd., his own firm. Gary designed, developed, and, in 2003, launched NoticeBored (www.NoticeBored.com), an innovative information security awareness subscription service. NoticeBored has kept him busy ever since, researching and writing awareness materials for subscribers covering a different information security topic each month. One of the regular monthly awareness deliverables from NoticeBored is a management-level awareness briefing proposing and discussing potential metrics associated with each month’s information security topic—for example, a suite of metrics concerning the management of incidents was delivered with a host of other awareness materials about incident management.

Gary has been a passionate fan of the ISO/IEC 27000-series "ISO27k" information security management standards since shortly beforeBS 7799 was first released nearly two decades ago. He contributes to the continued development of ISO27k through New Zealand’s membership of SC27, the ISO/IEC committee responsible for them, although he arrived in NZ too late to influence ISO/IEC 27004:2009 on information security measurements, unfortunately (we have more to say on ’27004 below!). To find out what ISO27k can do for your organization, visit www.ISO27001security.com to explore the standards, find out about new developments, and join ISO27k Forum, the email reflector for a global user group.

Before all that, Gary was a scientist researching bacterial genetics at the universities of York and Leicester in the United Kingdom. He has long since lost touch with the cut and thrust of gene cloning, DNA fingerprinting, and all that, but despite recently discovering his creative streak through NoticeBored, the rational scientist and metrician still lurks deep within him. So seven years of university study was not a total waste after all.

Name: PRAGMATIC Security Metrics: Applying Metametrics to Information Security (eBook)Auerbach Publications 
Description: By W. Krag Brotby, Gary Hinson. Other books on information security metrics discuss number theory and statistics in academic terms. Light on mathematics and heavy on utility, PRAGMATIC Security Metrics: Applying Metametrics to Information Security breaks the mold. This is the ultimate...
Categories: Management of IT, Computing & IT Security, IT Security