CTINEXUS: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models

CTINexus is a novel framework that automatically transforms unstructured cyber threat intelligence into a rich, connected cybersecurity knowledge graph (CSKG). By leveraging in-context learning with large language models, it extracts relevant cybersecurity entities and their relationships from threat reports—without requiring extensive labeled data or specialized rule sets. CTINexus adapts quickly to new and evolving threats, consolidates information through hierarchical entity alignment, and infers hidden links among distant parts of a report to build comprehensive, high-quality cybersecurity knowledge graphs for more effective threat monitoring and analysis.

News:

Feb 2025: This paper has been accepted by Euro S&P 2025.
Aug 2024: We now support open-source models, including the Llama and QWen series, through Ollama.

System Overview

Overview of CTINEXUS. CTINEXUS comprises three phases. Phase 1, Security Triplet Extraction, enables endto-end extraction of cybersecurity triplets using in-context learning of LLM. Phase 2, Hierarchical Entity Alignment, reduces the redundancy of CSKG through coarse-grained grouping and fine-grained clustering. Phase 3, Long-Distance Relation Prediction, connects disjoint subgraphs by identifying central nodes and performing relation inference.

Comparation with baselines

CSKGs extracted by EXTRACTOR, TTPDrill, LADDER, and CTINEXUS for a real-world CTI report. EXTRACTOR,TTPDrill, and LADDER tend to produce incomplete and fragmented subgraphs, lacking comprehensive contextual connections. In contrast, CTINEXUS constructs a more integrated and comprehensive CSKG, with key information extracted and entities linked, providing a clearer and more complete representation of the threat profile.

Takeaways

Outperforms Existing Methods. CTINEXUS significantly surpasses two state-of-the-art baselines—EXTRACTOR (syntax/semantic role-based) and LADDER (fine-tuned model). For entity–relation (“triplet”) extraction, CTINEXUS achieves an F1 score of 87.65% versus EXTRACTOR’s 62.29%. In entity recognition, CTINEXUS attains an F1 score of 90.13%, compared to LADDER’s 71.13%.
High-Quality Knowledge Graph Construction. Beyond extracting more accurate triplets, CTINEXUS excels at creating cleaner, more connected knowledge graphs. Its hierarchical entity-alignment step removes redundant mentions with an F1 score above 99%, and its “long-distance relation” inference step adds missing links among distant entities with an F1 near 91%.
Effectiveness of Optimized In-Context Learning (ICL).
- Using two or three demonstration examples in the prompt yields strong gains, especially in recall, while avoiding excessive prompt length.
- Sorting demonstration examples in ascending order of similarity (so the most similar example is placed last) consistently boosts performance, reflecting a “recency bias” in large language models.
- When inferring entity labels, increasing from a few (4–8) demonstration examples improves accuracy, but adding too many examples can plateau or slightly reduce performance.
Choice of Backbone Model. CTINEXUS was tested with GPT-4, GPT-3.5, Llama3, and Qwen2.5. GPT-4 provides the highest accuracy (e.g., 10+ percentage points above GPT-3.5 in triplet extraction), though it is also more expensive in tokens and runtime.
Embedding-Based Entity Merging. For consolidating duplicates, a similarity threshold of 0.6 and the “text-embedding-3” model yield the best precision (≈99.6%) without harming recall.
Efficiency Considerations. GPT-4 approximately doubles inference time and has higher token usage compared to GPT-3.5. Even so, CTINEXUS remains practical—most steps require only one LLM query per report, and the hierarchical merging uses lightweight embeddings rather than repeated LLM calls.

Citation

We hope our work serves as a foundation for further LLM applications in the CTI analysis community. If you find it helpful for your research, please consider citing our paper! ❤️

@inproceedings{cheng2025ctinexusautomaticcyberthreat,
  title={CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models},
  author={Yutong Cheng and Osama Bajaber and Saimon Amanuel Tsegai and  Dawn Song
  and Peng Gao },
  booktitle={2025 IEEE European Symposium on Security and Privacy (EuroS\&P)},
  year={2025}
}