A new class of security threat, advanced persistent threats (APTs), are on the rise. In an APT, an attacker uses sophisticated techniques to stealthily gain unauthorized access to a system, and stay there for a long period of time. While these attacks take a long time to execute, their high-value targets, such as corporations and nation-states, often suffer big monetary and reputational damages.

According to research from IBM and The Ponemon Institute, the average amount of time to detect a security threat in 2021 was 207 days, and roughly 280 days to contain it. When log data is locked away after its initial retention window, these threats can linger for even longer than that. Rather than risk the high cost of these security incidents, organizations can look to log analytics for threat hunting. But short retention windows won’t work to identify and eliminate the potential of APTs and other lingering threats.

The most successful hunt teams rely on large-scale log data aggregation and analysis.

Average time to detect a security threat in 2021


Average time to contain a security threat in 2021

At a minimum, threat hunters need access to data sources that give them visibility into host and network activities, as well as telemetry data collected by the security solutions that are currently in place in their environment (such as a security information and event management, or SIEM, system).

Using analytics tools like Kibana and others, threat hunters can conduct a wide range of queries, perform data correlations, and create data visualizations that help uncover hidden insights within their data sets. Threat hunting involves investigating a potential attack scenario, rather than following up on an alert that existing security observability tools might have generated.

Without the clear-cut evidence that would trigger an alert, threat hunting requires the hunter to gather intelligence by conducting various analyses on the data in the environment. The most successful hunt teams rely on large-scale log data aggregation and analysis.

This log data can come from many sources, such as:

  • Proxies
  • DNS queries
  • Firewalls
  • NetFlow records
  • SSL/TLS and other certificate repositories
  • Access logs from cloud services
  • System event logs from endpoints
  • Windows Event logs
  • Windows Registry keys
  • Endpoint detection and response (EDR) tools
  • Application server logs
  • Email transaction logs
  • System audit records
How do you distinguish normal behavior from anomalous behavior within your applications and servers using this log data?

Hunters must always put new evidence in the context of a historical baseline condition. One way to do this effectively is to create visuals that span a large timeframe. For example, a long-term graph would make it easier to recognize a sudden spike in network traffic. Pairing a centralized log management solution with a SIEM can make this process far more efficient.

Beyond threat hunting alone, security operations (SecOps) teams can use log data to investigate known issues. During a security event, security teams often turn to IT and DevOps teams to gain access to more log data to determine the root cause of an incident.

Another common use case for logs are security audits, which can help:


security problems and gaps, or system weaknesses


a security baseline or determine if security training is adequate


with internal or external requirements such as PCI, GDPR and SOC 2 requirements

And More!

Pairing a centralized log management solution with a SIEM can make this process far more efficient