Overview
This scalable automation system is built around twelve independent Python web scrapers that continuously gather information, filter it intelligently, and deliver AI-generated summaries directly to users’ inboxes. The system is managed through a custom Windows-based control panel, enabling intuitive operation even for non-technical users.
The Challenge
Monitoring numerous websites manually was time-consuming, inconsistent, and prone to missed updates. The goal was to create a unified, automated solution that could:
- Scrape multiple sites simultaneously
- Detect relevant content using dynamic keyword lists
- Re-format content into publication-ready summaries
- Operate reliably with minimal supervision
The Solution
The modular scraping system consists of twelve Python-based scrapers, each monitoring a distinct source, managed through a Tkinter desktop control panel. Features include:
- Start or stop scrapers individually
- Adjust file paths for different scraper versions
- View scraper status and logs in real-time
- Configure which email receives output or error alerts
The interface is packaged as a standalone Windows executable for non-technical users.
Key Features
- Keyword-Driven Detection: Triggers only when relevant content appears.
- Smart URL Deduplication: Prevents duplicate triggers by logging processed URLs.
- AI-Enhanced Summarisation: Extracts key text and rewrites it into publication-ready summaries.
- Error Handling & Alerts: Sends email alerts and pauses before retrying on errors.
- Media Extraction: Attaches images or document links found on the page.
- Flexible Configuration: All output destinations and paths are configurable through the GUI.
Results & Impact
This automation drastically reduced human workload while improving response times and consistency. The modular design allows easy integration of new scrapers for additional data sources.
Technology Stack
- Languages: Python
- Libraries: Requests, BeautifulSoup, smtplib, Tkinter
- AI Integration: OpenAI API
- Packaging: PyInstaller (Windows executable)
- Output: Automated email delivery via SMTP
Future Development
The next stage is a web-based dashboard for remote monitoring, allowing users to view scraper activity, manage configurations, and access results via a secure browser. Early development of this feature is underway.
