My recent SOC Service Manager experience got me to thinking about cyber tool sprawl and the return on investment that my customer was getting for their cyber investment. There is easily over 30 different cyber tools in their portfolio which is not unlike many large organizations these days. And it seemed during my tenure that more of my cyber operations resources were directed to cyber tool break-fix and maintenance than to optimizing their effective operation and contribution to risk reduction in the enterprise. These tool issues stemmed from two causes, in my opinion:
1) the customer’s desire to provide a tool to try to cover every threat vector; and, to protect every asset location and every different asset type; and,
2) a lack of focus on technology refresh / end-of-life. The thirst for cyber solutions doesn’t seem to get quenched either, as new tools are always being evaluated and added to the portfolio but no tool seems to leave.
Generally these tools all serve a purpose to protect the enterprise, and even though the risks to the enterprise are seemingly reduced, I wonder at what cost and to what extent? Could the cyber investment be more effective? What is the overall tool-cost-to-risk-reduction ratio? How much duplication of protection was there? What gaps exist in the coverage of risks? Sometimes it didn’t seem like the risk level had materially diminished with the cumulative operation of all the tools. There often seemed to be a lack of actionable situational information regarding asset cyber posture and operational state, or ambiguity around the attribution and authority for actions performed on the network, or uncertainty regarding the current threat status or potential impact of a threat to the mission. The lack of situational information regarding risk status made it difficult to discern where cyber investments need to be reallocated since it is hard to prioritize investments against where the greatest risk lies when you don’t know how much and where the risk is.
One reason for this conundrum, in my view, is that the cyber workforce skill levels seem to lag from the time a tool first gets introduced into the enterprise to the time a resource is actually trained and proficient in the use of the tool. In fact, lack of familiarity with the tool – its proper configuration, control, and operational standards – often occurred due to frequent personnel turnover, lack of training, and variation in approved baselines from location to location or tool to tool. These inexperienced operator issues and the lack of global standards sometimes created greater risks and occurrences of downtime through misconfigurations, misinterpretations of data, or incorrect security policy changes.
These problems have also been noted by Dr. Gene Spafford, the founder and executive director of Purdue University’s Center for Education and Research in Information Assurance and Security (CERIAS), “… poorly coded software combined with growing network complexity has increased the attack surface at many organizations. This has resulted in using all these [security] tools on a regular basis because the underlying software is not trustworthy.” All too often security tools do not integrate well with each other and only cover a subset of the assets that need to be protected. These coverage gaps result in loss of situational awareness, difficulties in defending against attacks, and slow response processes.
The problems that resulted from the lack of tool experts were further complicated by practices that relied on functional silos to manage and conduct work activities. In general, IT organizations have varying budgets and requirements leading to functional silos. Silos promote specialized (discipline specific) data sources, models, and methods to inform decision-making and guide work activities. However, the disjointed models of the different silos for my customer tended to inhibit collaborative decision-making and hampered accurate and holistic views of the cyber terrain. Cross-discipline communications about issues were through static, watered-down models or tool data subject to varying interpretation—“This is what I am seeing. What are you seeing?” The lack of a common language to describe what is happening, along with a common view of events and a common system model, created delays in diagnosing or troubleshooting problems and hid incidents from view, as well as hindered other operational processes.
One benefit of cyber tool investment often highlighted by the vendor is an increase to the speed-to-capability for responding to problems and incidents. However, IT operations are governed by ITSM processes and ticketing procedures. While necessary for managing change and understanding performance, these ITSM tools, processes and procedures tend to drive the response pace and may slow the tempo of IT operations. In my experience, playbooks for any event response often required a great deal of manual intervention with the workflow triggered and maintained by a ticketing system. Mitigations were often delayed as tickets got escalated and lost amid confusing hand-offs between functional silos and non-interoperable tools. Often, change management boards were required prior to making modifications to systems. This manually-intensive workflow system created significant latencies in incident responses, potentially allowing cyber adversaries time to pivot and to burrow into the network.
So in addition to improving workforce training and experience, I believe that more attention is needed towards security system engineering to manage cyber investment effectively. I believe a set of “living models” reflecting a cross-discipline view of the cyber enterprise is a must-have to begin to understand how to [re-]allocate cyber investments. The models should be centered on assessing risk to the mission or business area and to providing situational awareness about the relationships between mission, people, technology, and process. The following chart summarizes some of the types of models and information involved in such cross-discipline views. 
My vision is for a model that is centered on the development and management of an operational Mission Model of the different mission flows and dependencies. An ontology such as the DoDAF Meta Model (DM2) or the Mission and Means Framework (MMF) could be used as the foundation for such a model. Dependencies flow down from the mission objectives to the tasks, assets, flows, and threats related to accomplishing the mission. Each element of the model could be tagged with attributes, for example, mission assets can be tagged with attributes such as readiness, asset value, and identity for use in the model.
To make this “living model” real to the cyber analyst or operator, I would use it to evaluate potential attacks and protections (including new tools, security policies, and controls), while using live or simulated event data that captured the dynamics of the cyber terrain. As shown in the following figure, a series of models would be constructed and tied together by a scenario that reflects the critical activity sequence and timing of actions in the scenario.
• Defense objectives would be identified, modified, and allocated to sensors based on heuristics to create the sensor grid.
• An adversarial grid would also be developed based on the MITRE Att@ck model and the exposures related to the enterprise attack surface.
• A Network Model describes the computational and communication assets that are within the scope of the decision space, including their capacity limits and current configuration. A Vulnerability Graph, based on a network model, would be used to depict the influence of attacks and capture intrusion propagation.
• A Mission Model incorporates the objectives and “demands” or requirements that are to be met; policy, budget, or operational constraints; and the operational tasks and resource dependencies needed to support the mission. A mission could be defined simply as a “workload” or a more complex business process. The Mission Dependency graph (Mission-Task-Asset-Threat map) would capture the attacks’ impacts toward system objects as they relate to “missions.” A Bayesian network can be constructed on top of the MTA to infer the probabilities of missions being tainted.
• A Control Model describes the security controls. I would use Cybersecurity Framework Profiles to map the protections / controls to threats. The degree to which the control provided mitigation of the attack would need to be evaluated and captured as part of the model.
An executable set of models as described could provide a valuable “what if” analysis tool for understanding cyber posture and potential threats, while constructing a viable defense plan for mitigating attacks. It therefore can provide a foundation for the deployment of second generation security automation and orchestration (SAO) tools. AI and machine learning tools coupled together with the mod-sim aspects of an executable architecture such as this could enrich the automated synthesis of alternative actions and contextual-based decision-making capabilities, thereby reducing the need for a human-in-the-loop, and create intelligent dynamic systems that understand what is normal, what is not normal, and the ramifications of both action and inaction – or, change and non-change – within and around the cyber terrain.
Overlaying a cost model and/or a value model to such an executable reference model also can help pinpoint where investment may be needed to fill a cyber gap. The value model could capture the asset value [e.g., value of data at a specific time stored on a particular node of the network], workload value [value of a processing stream or computational process], or mission value [value of a set of workloads distributed over time and/or distance, and designed for a specific business purpose]. The cost model would include the cost of a protection or control used to defend the network or recover from an attack. These additional parameters are needed to measure returns on investment.
Tools such as ScienceLogic SL1, Scalable Networks’ EXata, Magic Draw and System Architect can help to build such an architecture model, to enable cyber tool portfolio analysis, help to identify better allocation of cyber investments, and to ultimately, help stem the tide of cyber tool glut. It is unfortunate that you may need to invest in yet another tool, but such a tool and set of models could be constructed to solve many different problems. For example, such a high fidelity model combined with machine learning and artificial intelligence can also help to automate the performance assessment, forecasting, and prognostics processes over the system life cycle – identifying issues before they become problems.
What tools are you using to evaluate your cyber investment? Are you considering the application of AI or mod-sim tools to complement your understanding of your cyber posture and risk profile? What unique issues are you facing when it comes to cyber investment? Drop us a comment to let us know your thoughts on this important topic.
And thanks for checking out ActiveCyber.net! Please give us your feedback because we’d love to know some topics you’d like to hear about in the area of active cyber defenses, PQ cryptography, risk assessment and modeling, autonomous security, securing the Internet of Things, or other security topics. Also, email marketing@activecyber.net if you’re interested in interviewing or advertising with us at ActiveCyber.
