For several years I have been honored to be a guest at the annual Forum on Financial Information Systems and Cybersecurity: A Public Policy Perspective organized at the University of Maryland by Larry Gordon, EY Alumni Professor of Managerial Accounting and Information Assurance; Martin Loeb, professor of accounting and information assurance and a Deloitte & Touche Faculty Fellow; and, William Lucyshyn, research professor and the director of research at the Center for Public Policy and Private Enterprise. The organizers bring in other researchers and practitioners with a variety of perspectives from around the world, creating a collegial and informative forum. Besides being quite informative, the forum has paid dividends in other ways for me through the networking that is always encouraged by the organizers. One such dividend was the introduction to Professor Shouhuai Xu in 2019 and then re-meeting Professor Shouhuai Xu this year. Professor Xu cordially accepted my invite to do an update interview on his research in cybersecurity dynamics which we did an interview about in 2019. According to Professor Xu, his ambitious research in cybersecurity dynamics includes three areas of focus or pillars:
- Cybersecurity First-Principle Modeling: We have established some basic frameworks for understanding whether or not, and when, cybersecurity will be manageable and measurable.
- Cybersecurity Data Analytics: We have established some basic frameworks for forecasting cyber threats hours ahead of time, similar to weather forecasting.
- Cybersecurity Metrics: We have systematized the gap between the current body of metrics and where we need to be. We have been systematically designing and exploring cybersecurity metrics, broadly defined to include resilience metrics and agility metrics.
For this update interview we dive deeper into the third pillar – Cybersecurity Metrics, specifically a metrics framework called SARR. Find out what SARR means and how it applies to cybersecurity dynamics in the interview below.
Spotlight on Professor Shouhuai Xu
» Title: Gallogly Chair Professor in Cybersecurity, Founding Director of the Laboratory for Cybersecurity Dynamics, Department of Computer Science, University of Colorado, Colorado Springs
» Website: https://xu-lab.org/
» LinkedIn: linkedin.com/in/shouhuai-xu-95b576
Read his bio below.
June 21, 2022
Chris Daly, Active Cyber™ – What is the SARR framework and what is its significance or importance? What are some challenges in the areas of metrics that it addresses?
Shouhuai Xu, Professor in Cybersecurity, University of Colorado, Colorado Springs – SARR stands for Security, Agility, Resilience, and Risk. The SARR framework is the first ever unification of these four concepts or perspectives. The framework allows the user to examine whether the assumptions made by the system model, the threat model, and the trust model are violated or not.
This underlying connection among these different models has not been recognized until now. With SARR, we now have a systematic way of thinking to tackle the holy-grail challenge of cybersecurity metrics and quantification. The importance of metrics can never be overstated because without accurate and relevant measures, we cannot adequately achieve, for example, Quantitative Risk Management, Trustworthy Decision-Making in Cyber Operations, and Cost-Effective or Optimal Investment, which is indeed the status quo. The framework paves a pretty clear way moving forward, in terms of: How should we systematically define metrics? How should we relate the metrics? How should we use them in a consistent fashion? We are now fully engaged in tackling these difficult, but important, problems.
Active Cyber™ – Many enterprises are mired in compliance-based assessments and metrics. How does SARR move enterprises to a new perspective of risk and cyber readiness? What is your perspective of metrics when taking a compliance view versus a readiness view?
Professor Xu – These are excellent questions! Compliance is a good practice to prevent some known weaknesses that can be exploited by attackers. However, it is well recognized that compliance does not imply security. From a technical point of view, the compliance approach is qualitative at best. The SARR framework aims at defining metrics that quantify cybersecurity from the four perspectives in a systematic manner. These quantitative measures can be used as input to quantitative decision-making algorithms, such as the ones in the Cybersecurity Dynamics framework we discussed last time. I would like to highlight that both the metrics and the decision-making algorithms embrace the dynamics view of cybersecurity because cyber threats dynamically evolve over time. That is, they cannot be done once-and-for-all, but must explicitly accommodate the evolution of vulnerabilities, attack capabilities, and defense capabilities.
Active Cyber™ – What is so difficult about developing and collecting cybersecurity measures and metrics?
Professor Xu – This is another excellent question! There are many technical reasons for this. For example, we can easily measure the length of an object, because it is static in the sense that its length does not change over time; in cybersecurity, dynamics is the norm and “things” are rarely static. This reiterates why the dynamics view of cybersecurity is fundamental.
As another example, cybersecurity as a discipline has not been supported by rigorous experiments, which are done in other disciplines (e.g., Physics or Chemistry). This can be evidenced by the lack of quality cybersecurity data, especially data from a holistic perspective (i.e., data recording what happened to a network including the computers, at the finest granularity possible). Piecing all factors together, we do not have “good intuitions” that can help address the problem of cybersecurity metrics.
Active Cyber™ – There are dozens of frameworks for security and risk metrics. What is unique about the SARR approach? How does it lead to better security? What types of management decisions is it intended to handle or improve?
Professor Xu – To be precise, I’d say that there are many frameworks that can give some qualitative advice or recommendations. However, these recommendations are not sufficient in making decisions. To see this, let’s consider a simple example: telling one that today’s weather is “cold” is not enough to give good advice on what one should wear, because there can be a substantial difference between the temperature being 30 vs. 5 degrees. The SARR framework can guide the definition of metrics to quantify cybersecurity in a systematic way, so that the measurements can be used as input to decision-making algorithms to produce quantitative recommendations on improving cybersecurity in a cost-effective or optimal fashion. Of course, there are many technical problems that need to be tackled, such as: What if the measurements contain errors? What if the algorithms incur some errors? These problems manifest the notion of uncertainty, which is inherent in making decisions and also needs to be quantified.
Active Cyber™ – It is important for metrics to reveal the effectiveness of cyber strategies that are designed to enhance critical infrastructure agility and resilience, e.g., can a system reconfigure dynamically to keep producing services when there is an impactful event, or is it totally dependent on the event to be fixed? When quantifying cyber metrics, how does SARR handle the significance of interdependencies and cascading effects of cyber events?
Professor Xu – This is once again an excellent question! Defining metrics to quantify properties of critical infrastructure is indeed one of the drivers that contributed to the birth of the SARR framework. The SARR framework focuses on defining “what we need to measure” which is very different from “what we can measure.” The resulting metrics can be used as parameters in advanced models, such as the ones in the Cybersecurity Dynamics framework, to derive effective defense strategies and orchestrate cyber defense operations. While interdependencies and cascading effects represent substantial challenges, we have obtained significant results towards ultimately tackling them, but much more research needs to be done. Please refer to my website for more information.
Active Cyber™ – One of the problems that seem to plague security metrics is inconsistent definition of measures and metrics. Measuring port scans is a good example of this common problem. Port scans are often identified by intrusion detection systems (IDSs), but each IDS uses its own proprietary algorithms for identifying port scans, so activity identified as a port scan by one IDS may not be identified as such by another. Do you see industry and the research communities reach agreement on what and how to measure to formulate consistent quantitative assessments of security, risk, agility and resilience?
Professor Xu – Yes! The example you mentioned indeed highlights the importance of defining metrics in a systematic way and measuring them in a consistent fashion. The SARR framework, and the research following it, would produce a systematic set of metrics, which collectively specify “what we need to measure” because they are important (e.g., for decision-making purposes). Once we know the metrics we need to measure, we can design consistent methods to measure them and we can design algorithms to leverage them for various application purposes (e.g., decision-making).
Active Cyber™ – What types of analytic or data collection tools are necessary to make SARR work effectively?
Professor Xu – It would be ideal that we have access to a Cyber Range-like environment where we can conduct various kinds of attack/defense experiments by red-blue-white teams, collecting data at various levels of abstractions (e.g., network traffic, host logs at fine granularities), and using data to validate theoretical cybersecurity models (e.g., the ones in the Cybersecurity Dynamics framework). This is reminiscent of how Physical Experiments are conducted to validate Physics Theories. I believe that cybersecurity needs to use a similar route.
Active Cyber™ – How does SARR relate to your previous research in Cybersecurity Dynamics and how does it move your overall concepts of security forward? What other research or frameworks does it build upon?
Professor Xu – SARR elaborates the metrics pillar of the Cybersecurity Dynamics framework, of which the other pillars are cybersecurity first-principle modeling and cybersecurity data analytics. This explains why SARR focuses on defining “what metrics we need to measure” and relating the resulting metrics in a consistent manner. These metrics will be used as parameters in the models of the other two pillars of the Cybersecurity Dynamics framework to characterize, predict, and prescribe the evolution of the attack/defense interactions (e.g., what the defender should do in order to lower the percentage of compromised computers in a network to be below a desired threshold).
Active Cyber™ – How does SARR remain flexible to deal with technological change over time?
Professor Xu – I think that good metrics are relatively stable and would not need to be frequently revised because of technological changes. Indeed, our research has been pursuing such metrics, while seeking metrics that are necessary to accommodate technological changes over time. This view is inspired by, and inherited from my background in Cryptography, where good security definitions (e.g., security of digital signature schemes) do not change with how a cryptosystem is designed or the computational problem on which a cryptosystem is based.
Active Cyber™ – What are the next steps in your research to move SARR forward?
Professor Xu – We are fully engaged in applying the SARR framework to guide us to define systematic sets of metrics. I hope to demonstrate its usefulness through some killer applications in the near future.
Thank you Professor Xu for this update on your research. I believe your research will find many applications as the field of cybersecurity metrics is quite important as we move to autonomous decision-making for our cyber defenses with AI/ML as well as informing our cyber investment strategies. I look forward to following your progress and that of your colleagues at University of Colorado on this exciting research.
And thanks to my subscribers and visitors to my site for checking out ActiveCyber.net! Let us know if you are doing some interesting research or have a cybersecurity product you would like discussed on Active Cyber™. Please give us your feedback because we’d love to know some topics you’d like to hear about in the area of active cyber defenses, authenticity, PQ cryptography, risk assessment and modeling, autonomous security, digital forensics, securing OT / IIoT and IoT systems, Augmented Reality, or other emerging technology topics. Also, email chrisdaly@activecyber.net if you’re interested in interviewing or advertising with us at Active Cyber™.
| About Professor Shouhuai Xu Shouhuai Xu is the Gallogly Chair Professor in Cybersecurity, Department of Computer Science, University of Colorado Colorado Springs (UCCS). Prior to joining UCCS in 2021, he was with Department of Computer Science, University of Texas at San Antonio. He introduced a systematic approach, dubbed Cybersecurity Dynamics, to modeling and quantifying cybersecurity from a holistic perspective. His research has won several awards, including the 2019 worldwide adversarial malware classification challenge organized by the MIT Lincoln Lab. His research has been funded by AFOSR, AFRL, ARL, ARO, DOE, NSA, NSF and ONR. He co-initiated the International Conference on Science of Cyber Security (SciSec) and is serving as its Steering Committee Chair. He has served as Program Committee co-chair for several international conferences. He is/was an Associate Editor of IEEE Transactions on Dependable and Secure Computing (IEEE TDSC), IEEE Transactions on Information Forensics and Security (IEEE T-IFS), and IEEE Transactions on Network Science and Engineering (IEEE TNSE). More information about his research can be found at https://xu-lab.org. |
