Technology: The Five Critical Challenges in Threat Intelligence Sharing
Published: 10/05/2017 12:00 pm
Automated threat intelligence sharing is not new but it is still in its early years. During the past several years, the industry has invested in machine generation and machine consumption of tactical threat data.
Most data consists of event logs and indicators of compromise such as file hashes, suspicious URLs, and IP addresses. These indicators are very time sensitive, and lose value almost immediately. At the same time, the volume and quality of this data creates new challenges. It is hard to identify high-quality, actionable indicators among the flood of information, making triage difficult for security analysts.
Although the industry has built tactical intelligence sharing capabilities, especially among each company’s own products, the industry still fails at sharing high-level, contextually rich intelligence, such as advanced campaigns, at a meaningful level and with other industry participants.
Five critical challenges face security vendors and organisations that want to incorporate this valuable intelligence into their security operations. They are volume, validation, quality, speed, and correlation.
During the past few years, the deployment of enhanced and verbose security sensors and defences has resulted in a high volume of data fed into threat intelligence tools. Big data analytics and machine-learning tools consume this data and add their analyses to it. The net effect is an improvement in internal capabilities to detect potential attacks and a marked increase in internal threat detection, but a massive signal-to-noise problem remains to be solved. Although the systems are getting better at detection, we have not yet seen a corresponding improvement in the capability of human analysts to triage, process, and act on the intelligence. Vendors are working on solutions to address this problem, from access monitors on sensitive data to sophisticated sandboxes and traps that can resolve contextual clues about a potential attack or suspicious event.
Disinformation and fake news are not new. Adversaries may file false threat reports to mislead or overwhelm threat intelligence systems. As a result, it is essential to validate the sources of shared threat intelligence, from both inside and outside the organisation.
Outside validation is perhaps the more obvious requirement, ensuring that incoming threat intelligence is being sent by legitimate sources and has not been tampered with in transit. This is typically accomplished with encryption, hashes, and other methods of digitally signing content. Internal validation is a different problem, not so much validating the sources as analysing and evaluating the content to determine if it is a legitimate attack or a noisy distraction to draw attention and resources away from a quieter, stealthier threat.
Related to source validation is the quality of the information we share. Legitimate sources can send anything from definitive indicators of attack or compromise to their entire event feed, which may be of little or no relevance to the receiver. Although more threat intelligence is generally better, much of it is duplicated, and too much low-quality intelligence is of little value. Many threat exchanges are coming online, but they are only as good as their inputs and sensors.
Vendors need to re-architect security sensors to capture and communicate richer trace data to help decision-support systems identify key structural elements of a persistent attack. Filters, tags, and de-duplication are critical intake tasks to automate in order to increase the value of threat intelligence and make it actionable.
An early, promising effort to improve threat intelligence quality will come online in 2017 through the Cyber Threat Alliance. Threat intelligence coming from CTA members will be automatically scored for its quality, and members will be able to draw out threat intelligence only if they have provided sufficient quality input.
The speed of transmission, or more accurately, the latency between a threat detection and the reception of critical intelligence, is also an important attribute. Intelligence received too late to prevent an attack is still valuable, but only for the clean-up process. This is one reason why open and standardised communication protocols, designed and optimised for sharing threat intelligence, are essential to successful threat intelligence operations.
The propagation of attacks between systems happens within a minute or two of a machine’s being compromised, so communications between sensors and systems within the enterprise have to operate in near real time. Meanwhile, advanced persistent threats and sophisticated, targeted campaigns often go after multiple organisations in the same vertical market, so communications from one organisation to another, usually involving an intermediary or exchange, have to take place within a few hours of the first indication of an attack.
Finally, as threat intelligence is received, correlating the information – while looking for patterns and key data points relevant to the organisation – is the most critical step. Although some organisations treat the raw data as a proprietary or competitive advantage, the ability to collect data is not a critical factor. It is the processing that turns data first into intelligence and then into knowledge that can inform and direct the security operations teams.
The ability to validate data in near real time, correlate it across multiple operating systems, devices, and networks, use it to triage the event, prioritise the investigation, and scope the response is critical to provide effective detection and corrections actions. The goal is to leverage technologies and machine capabilities to triage event data, distil it into high-quality events, and scope and prioritize the incidents so that security analysts can focus their attention on the highest-risk items.
Together, these issues describe threat intelligence sharing’s ‘last mile’ problem: taking this information and converting it to controlled action. To cover this last mile we need to find better ways to share threat intelligence between a vendor’s products and with other vendors, improve methods to automatically identify relationships between the intelligence collected, and employ machine assistance to simplify triage.
By Vincent Weafer
Vice president, McAfee Labs