News Aggregator Duplicate Story Merger

WHAT IT DOES

News Aggregator Duplicate Story Merger scans the page for news article headlines and links, identifying duplicate stories across different sources by analyzing headline similarity and URL patterns. When duplicates are found, the script merges them into grouped entries with a badge showing the number of sources covering the same story. Each duplicate group displays a single representative story with an expandable indicator that reveals the full list of sources when clicked, allowing you to see all coverage of a story without the visual clutter of repeated headlines. Redundant duplicate entries are collapsed or hidden, keeping your news feed clean and organized while preserving access to all original sources.

WHERE IT RUNS

This script operates as a userscript within the BotGentz extension on any webpage that displays news articles, including news aggregators, RSS readers, and individual news sites. It functions on any modern browser that supports the BotGentz extension, including Chrome, Firefox, Edge, Safari, and Opera. The script runs entirely client-side, processing the page's DOM without sending any information to external servers. It works across all domains and adapts to different news site structures automatically.

HOW TO USE

After installation, the script automatically scans the page for articles and merges duplicate stories based on headline similarity. Each merged group appears with a numbered badge indicating how many sources are covering the same story. Click the arrow (›) next to any grouped story to expand the list of sources, showing each source name and a clickable link to the original article. The primary story remains visible with a "+X more" count. Use the panel to adjust the similarity threshold, toggle highlighting, enable or disable auto-collapse of source lists, or switch between automatic merge and manual highlight-only modes. Click "Scan Page Now" to re-scan if new content loads.

THE MECHANISM THAT MAKES IT WORK

Unlike simple duplicate finders that rely on exact URL matching, this script uses a hybrid similarity detection algorithm that combines keyword frequency analysis and word overlap scoring. The algorithm extracts significant words from each headline, ignoring common stop words, and calculates a similarity score between 0 and 1 based on the Jaccard index of overlapping word sets. Headlines with scores above the user-defined threshold are grouped together. The script intelligently selects the primary story based on source priority (configurable in settings) or the first encountered headline. URL pattern analysis extracts domain names to identify sources without relying on external APIs. The DOM traversal system identifies article elements using a cascading selector approach, detecting headlines, links, and article containers across diverse site structures, ensuring compatibility with most news platforms.

THE PANEL

The script panel is built using BotGentz's BGPanel framework and provides comprehensive control over the merger's behavior. The panel opens as a floating overlay that can be dragged anywhere on your screen, with a clean dark theme matching the tool's aesthetic. It includes dropdown menus for similarity threshold selection (50-90%) and merge mode (automatic or manual highlight-only). Toggle switches enable or disable the merger, highlight duplicate groups, auto-collapse source lists, and show source count badges. Action buttons allow you to scan the page immediately or reset all settings to defaults. The panel collapses to a small icon when not in use and expands when clicked, remaining accessible but unobtrusive. Press the Escape key at any time to close the panel.

PLEASE NOTE

This script requires the free BotGentz extension to be installed and activated in your browser. The extension is available from the official BotGentz website. The script's duplicate detection relies on headline text similarity and may occasionally group unrelated stories with similar wording or miss duplicates with very different headlines. You can adjust the similarity threshold in the panel to fine-tune sensitivity. The script modifies the page DOM by replacing duplicate entries with grouped containers; the "Undo" button on each group restores the original articles. No data is collected, stored, or transmitted by this script. All settings are persisted locally using GM_setValue. If a site's HTML structure changes significantly, the script may need updates to maintain compatibility.