Automated Abuse Detection for Email with User-Controlled Privacy
Citadel is a Chrome extension for Gmail that detects abusive emails and helps users take action while preserving privacy. The design is grounded in prior literature and a mixed-methods study (interviews + survey) examining user preferences for moderation approaches and privacy concerns regarding human vs. automated systems.
Platform: Chrome extension for Gmail, permission-based install; no storage of personal emails on our servers.
Control: Threshold-per-sender, block lists, and “accurate / inaccurate” feedback loops.
Support: Notification window + “Contact Trusted People” for quick outreach after detection.
Preferences for moderation approaches and privacy concerns informed Citadel’s design.
40% preferred automated moderation, 53% preferred a combined automated + human approach, and 7% preferred human-only moderation. A chi-square test indicated significant differences in privacy concern between human-only and automated conditions.
51% preferred combined moderation, 41% automated, and 8% human-only—mirroring interview trends with strong privacy concerns tied to human moderation.
Gmail was the primary email platform for participants (≈93% in interviews; ≈89% in survey), motivating a Chrome extension delivery.
Abusive emails surface in an accessible notification pane with severity indicators, keeping the inbox uncluttered.
One-click outreach to pre-selected contacts (friends/relatives) when support is needed after detection.
Users tune tolerance levels per sender; feedback (“Accurate?” / “Inaccurate?”) adaptively adjusts thresholds over time.
Instantly block a sender; future emails are moved to trash and kept out of the inbox. Manage block lists in the settings view.
Citadel went through two iterations. The first design validated demand for notifications and trusted contacts. The second introduced sender-specific thresholds and blocking, based on insights that some users tolerate language from friends but not strangers.
Deep neural network (CNN + LSTM + dense layers) detects abusive content. A secure database stores user profile & trusted contact metadata—never personal emails.
Due to privacy constraints around email storage, training used the Kaggle Jigsaw toxic comment dataset (159,571 items; labels: toxic, severe toxic, obscene, threat, insult, identity hate).
Inception-inspired convolutional branch + LSTM for long-range dependencies; character- and word-level inputs to handle context and misspellings. Test accuracy: ~98.4% on a 9,571-item split.
Works within Gmail (with user permission). Shows notifications with severity, “Ask for help!”, “Block Contact”, and sender-threshold feedback controls.