Currently, there is no formalized or standardized framework for measuring most things in CTI, including how to measure your data collection strategies. Inspired by the recent CTI maturity model blog from EclecticIQ, I drafted version 1.0 of a Collection Management Framework Maturity Model. This model aims to provide some measurements of your collection program to gauge where you need to spend more time and resources to better formalize.

With some maturity models, we look at things from the various stages, which are Initial, Repeatable, Defined, Managed, and Optimizing. These various levels are then measured against four categories:

• Data Source Identification

• Data Collection

• Data Quality

• Data Integration

Data Source Identification is the process by which you can identify new sources of data that are relevant to your specific intelligence needs. By asking yourself some questions like “Do I need a data source of Telegram postings to track threat actors” or “Do I need netflow to identify communications between threat actors and my network” you can better understand your needs. These should be related to your intelligence requirements, since those are what drives your intelligence program. If you find you are collecting data but not fully using it, what value does it really provide you, plus it could be costing you a good chunk of change to keep. Lastly, by having intelligence requirements properly defined, you can understand your collection needs and what data is required to satisfy those requirements.

Data Collection is the actual process of collecting the data once you understand what data you need to perform your intelligence functions. How well can you collect, parse, enrich, and utilize the data you are interested in? This area covers those measurements. As you move across the levels of maturity, you go from being able to query data ad-hoc through the source or tool itself and move to having integrations with the data so that you can automate the collection and ingestion into internal tools.

Next is Data Quality, which is a key component to having a good collection program in place. Without quality data, you are not set up for success, and this could severely impact your reporting. As we move across the maturity levels, you’ll find yourself going from one-off spot checks of the data quality to ensure it’s accurate, relevant, and timely with enough context to make informed decisions and to have automated checks of the data. At the more mature stages, you’ll be able to have automated quality checks and an integrated feedback loop of the data for instances where the data does not meet the criteria set forth.

Lastly, Data Integration is how well you can integrate the data into your processes and systems What good is data if it sits in a silo and can only be interacted with ad-hoc queries? As you evolve your maturity, you’ll go from being able to do manual data pulls into your SIEM or TIP, to having automated ingestion of the data that goes through proper checks and validation before making its way into your tools and systems.

The image below is one way to visualize your maturity model, and it shows you what data sources are required to answer each Intelligence Requirement. By understanding what IRs require a certain set of data, you can better prioritize your collection efforts for that particular source. Additionally, if you notice that none of your IRs require a particular source, you can use that to inform your budgetary needs for the new year. These are great metrics to use to inform various aspects of your program.Mapping your maturity model to Intelligence Requirements

This model is just a start and one instance of a maturity model for how you collect data and use it in your intelligence program. It’s meant to be molded around what works best for your use cases, and at the very least, to provide a starting point for CTI shops to understand how well they can collect, process, and integrate the data into their ecosystem.