Document Type

Thesis - Open Access

Award Date

2026

Degree Name

Master of Science (MS)

Department / School

Electrical Engineering and Computer Science

First Advisor

Chulwoo Pack

Abstract

Retrieval-Augmented Generation (RAG) systems aim to enhance the capabilities of language models by retrieving external knowledge during inference. However, the process of retrieving knowledge introduces new security vulnerabilities to these models, specifically through the possibility of retrieval poisoning attacks in which adversarial documents are added to the system knowledge base to poison the language model’s responses. Although there have been a variety of defenses proposed for retrieval poisoning attacks, limited work has been done in the area of anomaly detection in embedding spaces as a general defense against multiple retrieval poisoning attack strategies. This thesis investigates two embedding-space anomaly detection approaches that can be employed to defend RAG systems against retrieval poisoning attacks. The first approach employs Local Mahalanobis Distance to identify anomalies in the retrieved document embeddings, while the second approach employs Spectral Analysis to identify anomalies in the similarity graph of the system’s retrieved documents. PoisonedRAG and PoisonCraft, two representative retrieval poisoning attacks, were reproduced on the Natural Questions and HotpotQA datasets using the Contriever dense retriever. These attacks were evaluated using a variety of neighborhood sizes to determine the Area Under the Receiver Operating Characteristic Curve (AUROC), False Positive Rate (FPR), True Positive Rate (TPR), Precision, Recall, F1-score, and Attack Success Rate Reduction (ASR-R) metrics. Furthermore, the reproduced EcoSafeRAG semantic filtering defense was evaluated on these reproduced retrieval outputs to allow for a direct comparison between the semantic-based filtering defense and the embedding space anomaly detection framework. The results of the poisoning attacks on the anomaly detection methods reveal that each attack has a different set of anomaly signatures. Spectral Analysis achieved the strongest detection rates for PoisonedRAG attacks, with AUROC values of 0.7979 on the Natural Questions dataset and the largest reduction in the success of the attack with minimal false positive rates. Local Mahalanobis Distance achieved the best results against PoisonCraft attacks, with AUROC values of 0.8789 for the Natural Questions dataset and 0.8007 for the HotpotQA dataset, indicating that embedding-based point anomaly detection is effective against optimization-based attacks. Finally, the results of the attack against EcoSafeRAG indicate that embedding-based anomaly detection has a lower false positive rate than semantic filtering methods, especially for adversarial documents. The results of these experiments indicate that no single method of anomaly detection is optimal for every type of poisoning attack. However, by detecting anomalous documents at both the embedding and graph level, it is possible to create a more robust defense than is provided by any semantic filtering method alone. Thus, implementing a system that performs anomaly detection at multiple stages of RAG models can provide an effective defense against both conventional and optimization-based poisoning attacks.

Publisher

South Dakota State University

Download

Share

COinS
 

Rights Statement

In Copyright