ActiveCyber Interview with Rob Frazier – Certified Splunk Architect

Discover How Splunk Can Help You Build Active Cyber Defenses: Rob Frazier, Certified Splunk Architect and accomplished security professional reveals how Splunk can play a vital role in combating threats as part of active cyber defense in this recent interview with ActiveCyber.

Rob and I first met in the early 1990’s when we were co-workers at the Department of State Computer Security Lab (to be truthful he was my boss). We crossed paths later at IBM (where the tables were turned and I was his manager) and many times since. I have always been amazed by Rob’s ability to quickly develop a high level of proficiency with new tools, with Splunk being the latest in his tool bag. Actually Splunk has become a must have active cyber defense tool across many enterprises for the reasons that Rob explains in the interview that follows. So read on and learn why Splunk is making a difference in proactive cyber defenses.

Spotlight on Rob Frazier, Certified Splunk Architect

» Title: Certified Splunk Architect, Blue Canopy, Virginia
» Email: belka@att.net
» Website: https://www.xakephet.com
» Linkedin: https://www.linkedin.com/in/rob-frazier-a498385
Read his bio below.

December 1, 2015

Chris Daly, ActiveCyber: How did you get started on your Splunk journey and what problems were you trying to solve at the time that you felt Splunk could support?

Rob Frazier: I was working as a Security Manager for a Government Customer. They were having some terrible performance problems. I knew something about performance from working with Fusion-IO and how their products boosted performance. I decided to take a look and see if I could help diagnose the problem.

Since it was a government customer, there were restrictions on bringing in the software I was used to using for performance analysis. Instead, I discovered they were using a software platform called Splunk to aggregate logs. I did a little research and realized that I could correlate logs across all the elements of the Enterprise to help with performance tuning. During one meeting where parties were debating where the problem lay in performance, I said “I don’t know where the problem lies, but here is what the data says the problem is.” I presented some Splunk dashboards that pinpointed the performance issues. The next thing I knew, I was both the security manager and the data analytics guy.

ActiveCyber: Can you comment on what traits and skills you have learned are essential to becoming a really good Splunk analyst or architect?

Frazier: I’ve always said that to be a good security architect, you cannot just be the best at one technology; you have to be the second or third best at all technologies. Splunk is very similar. As a data analysis platform, the more you know about the interactions of technology platforms and the data exhaust that is the log files, the better you become at correlating data and finding gold. Splunk lets you look inside the mountains of data that our modern world produces and find not only new insights into operational data about IT systems, but customer behavior, trends in technology, and much more. To be a good Splunk architect, you have to understand technology, what logs tell you, and most important, be creative in how you look at data through the Splunk platform.

ActiveCyber: Today Splunk is considered much more than a log aggregation tool and has found a niche as a significant utility in the active cyber intelligence space. What do you see as the real advantages provided by Splunk in the areas of cyber threat analysis and sharing?

Frazier: Splunk is awesome in security detection, analysis and investigation. First, Splunk can be configured to learn what is normal in your environment, how the ecology of your environment interacts, and when there are events that go out of the bounds of normal. Second, Splunk stores log events over time. You can correlate events from all across your enterprise that took place a few seconds ago to over weeks and months. With those capabilities for correlation and analysis, it is possible to detect patterns, discern trends and behaviors that are normal or abnormal. Lastly, when an event is detected that needs investigation, all the data is in the Splunk indexes which allows you to dig into the details of when it began, what systems were affected, what the impact was, and what you need to do for remediation.

ActiveCyber: Splunk is well known for its wide support for connectors to sensors and log sources to ingest unnormalized data for analysis. But how well does Splunk handle workflows with third party threat sources to drive context and understanding up (e.g., STIX/TAXII); and, M2M interaction to direct mitigation actions?

Frazier: Splunk has an add-on product called Splunk Enterprise Security, now in version 4.0. In addition to robust dashboards that put security trends and events on a single pane of glass, ES 4.0 is a well-developed tool that allows analysts to not only conduct investigative searches and correlations, but to add notes to the output as they move to the next phase of the investigation. What is built by Splunk is an investigator’s case file with notes, correlated data, and search results. This enables investigators to have a step-by-step record of actions taken and results found that can be shared between team members and provide the foundation for evidence gathering.

ActiveCyber: How does Splunk perform machine learning to contribute to profiling adversarial behavior with respect to identified system attacks in an operational mission context?

Frazier: Splunk ES 4.0 incorporates two key technologies for machine learning. The first is Extreme Search which was developed by Sciatica, Inc. and bought by Splunk. Extreme Search lets Splunk learn data models of normalized machine behavior through comparing events coming into the Splunk indexers from log files. Extreme Search can spot changes in patterns and behavior and alert on those changes. It also allows analysts to change the parameters of the data model. Splunk notes how the data models evolve to change its “normal” profile as well as respond to human intervention to modify the data model.

The next innovation comes from another purchase by Splunk of Caspida. Caspida brings behavior analysis of machines, users, patterns, etc. to Splunk. It can be applied to detect zero day attack behavior, as well as information flows that are seen due to the presence of advanced persistent threats and command and control malware. Caspida can also be used in counter- fraud and counter-intelligence applications by looking for anomalous user behavior.

Putting my creative Splunk hat on, I can envision behavior models from Caspida that could be used in detecting anomalous behaviors in the Internet of Things (IoT), controlling SCADA systems, energy production systems, monitoring HVAC for energy efficiency, and even manufacturing systems maintenance and quality assurance applications. Splunk is limited almost solely by the data scientist’s imagination.

ActiveCyber: Splunk supports the concept of “tagging” data. Can you explain how this works and how it helps cyber analysts to assess events scattered across time, across the network, and across the cyber kill chain?

Frazier: Splunk uses a Common Information Model (CIM). This is a data catalog that defines actions, terms, and work products in a common dictionary. The Splunk analyst can take logs from all kinds of systems and tag the data against definitions in the CIM. For example, user names that are “jdoe” “johns” “does” “john.doe” and so on can all be tagged with metadata “under name.” The same applies to terms such as “src”, “source” or “dest”, “destination”, etc. Splunk has a lot of apps that will predefine and normalize terms and tags for major vendor products to match the CIM. And Splunk has a built-in tool for taking one-off data sources, XML, application source code, etc. and defining the tag in those sources and mapping it to the CIM.

ActiveCyber: Visualization and proactive monitoring of cyber events are essential active cyber capabilities for managing situational awareness. How does Splunk deliver these capabilities and what advantages does it offer over traditional SIEM tools?

Frazier: The Splunk Enterprise Security product has a number of predefined dashboards that model information coming in from various sources. These dashboards are constantly updated like the dashboard of an airplane, being fed from data models, as well as recurring and real-time searches. Instead of altitude, speed, fuel, artificial horizon and such, Splunk is measuring in real or near real-time things like failed log-ins, malware signatures, network traffic, etc. At the same time, Splunk dashboards indicate trends, rising or falling, as well as severity of the reported event.

The advantage Splunk has is it is highly configurable. Using tools like Extreme Search and Caspida, the analyst can redefine normal and abnormal behavior, allowing for very accurate alerting with low false positives. And if an analyst determines the thresholds for alerts were set too low, the data is in the Splunk indexes to go back and to rerun correlation and searches with new or different thresholds. Splunk is also very easy to add new panels to dashboards, allowing analysts to slice data in different ways depending on conditions and circumstances,

ActiveCyber: Managing the security context across virtualized endpoints is essential to enabling various benefits from an active cyber defense perspective, such as agility and flexibility for incident responses. Please describe the type of visibility Splunk can provide in virtual environments and how this visibility can enhance active cyber defenses.

Frazier: Splunk works within any virtual machine just as it would a physical server. Splunk has a robust way of managing an infinite number of endpoints into its indexing and searching infrastructure. Splunk scales linearly, just add more indexing capacity and searching capacity. Additionally, Splunk has a VMWare app that allows Splunk to monitor hypervisor activity. This gives Splunk the ability to compare the functioning and resources of the hypervisor’s host and how it affects the performance of the VM being hosted on the hypervisor.

ActiveCyber: Netflow analysis offers promise in detecting insider threats and the presence of malicious software such as botnets by providing insights about payloads, session information, errors, DNS, etc. What capabilities does Splunk offer when dealing with streaming data and what advantages does Splunk offer in this context?

Frazier: Splunk has a free app called Streams. The Streams client resides on the endpoints in the Splunkforwarder (process in Linux, service in Windows) on the monitored machine. Streams can be turned on or off by the analyst using Splunk’s management tools to begin a packet capture on demand that is then forwarded to the Splunk indexers. The Streams app can be used to detect patterns of packets, alert on packet payload, etc. Once captured and ingested into the indexes, the Stream data can be correlated with other events in the Enterprise. In this way, streamed data can be compared by what was in the stream, the packets, the payload and what happened before, during, and subsequent to the stream of data captured. It is a very slick security and trouble-shooting tool.

ActiveCyber: What are the advantages that Splunk provides over cloud-based storage and processing platforms such as Hadoop and can Splunk complement such environments?

Frazier: Splunk is highly scalable. Data comes into Splunk via indexers, which distribute the logs and data across multiple indexer servers. This works much like Hadoop. The Splunk model of licensing is by the amount of data indexed. In massive big data environments, this could be quite costly. Splunk has developed apps to work in both the Hadoop and AWS environments to allow customers to create huge “data lakes” of petabytes of data that Splunk can then search.

In the case of Hadoop, Splunk has a paid app called HUNK that puts a connection between the Splunk infrastructure and Hadoop to allow Splunk to search Hadoop clusters. Using HUNK, Splunk can search and pull data into Splunk for analysis and not ingest the data into the indexers (and count against the license). If the analyst would like to ingest the data for later analysis, HUNK can be configured to draw the data discovered in a search and then ingest it into the indexers for later analysis and correlation, dashboards etc. The Splunk AWS app works in a similar fashion.

Well Rob – that is a lot to “ingest!” Thanks for sharing your tremendous insight into the workings of Splunk and its impressive cyber analytic and connective features. I believe many of ActiveCyber’s readers will reap what you have sowed to construct better and more proactive cyber defenses.

And thanks for checking out ActiveCyber.net! Please give us your feedback because we’d love to know some topics you’d like to hear about in the area of active cyber defenses. Also, email marketing@activecyber.net if you’re interested in interviewing or advertising with us at ActiveCyber.