Deploy Now

Stars

166

Forks

19

Watchers

0

Developer links

MuckScraper

Reading the news should not mean opening twelve tabs about the same event, so MuckScraper folds every outlet's version into one story and shows you how the coverage differs. A scheduler fetches RSS feeds several times a day, with optional NewsAPI and GNews keys adding more sources, and each article is embedded and matched against recent stories so wire copies and follow-ups land together instead of flooding your feed. Every story carries a bias balance: how many left, center, and right outlets covered it, with each outlet rated from AllSides data or scored by the model where no rating exists. Local language models write the story summary and a deeper report comparing how each side frames the event, what is contested, and what nobody is mentioning, while guardrails reject cut-off summaries and figures that never appeared in the sources. Opinion columns stay in their own lane, linked to the stories they argue about and rounded up separately, never counted as news coverage. Twice a day the system publishes a balanced edition of twenty stories, and every past edition stays readable alongside full-text search across your whole archive. Reader, scraper, and admin roles decide who can simply read, who can trigger fetches, and who manages feeds, prompts, and users. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

MuckScraper
MuckScraper
MuckScraper
MuckScraper
MuckScraper
MuckScraper

Benefits

  • One Story, Every Outlet
  • Embedding-based grouping folds a dozen articles about the same event into a single story, detecting wire copies by text overlap, so your feed shows events rather than repetition.
  • Bias You Can See
  • Each story shows its left, center, and right coverage balance, using AllSides ratings where available and model scoring elsewhere, making slanted or one-sided coverage obvious at a glance.
  • Summaries Stay on Your Hardware
  • Ollama models on your own machine generate summaries and deep reports from real reporting, with quality checks that flag invented figures, so reading habits never leave your server.
  • Editions, Not Endless Feeds
  • Twice-daily balanced editions of twenty stories replace infinite scrolling, while past editions, opinion roundups, and full-text search keep the entire archive browsable on your own schedule.

Features

  • Embedding Story Grouping
  • Articles are embedded with nomic-embed-text into PostgreSQL with pgvector and compared against recent stories, with a language model adjudicating borderline matches and text overlap catching wire copies.
  • Local LLM Analysis
  • Ollama by default, or Gemini, Groq, and OpenAI-compatible endpoints, generates story summaries, per-article analysis, and deep reports comparing coverage, with guardrails rejecting truncated or unsupported output.
  • Bias Ratings and Balance
  • Outlet bias comes from AllSides lookups with model-based scoring as fallback, and every story displays its left, center, and right coverage counts plus an option to fetch missing perspectives.
  • Scheduled Feed Ingestion
  • A scheduler pulls RSS feeds on configurable full and fetch-only runs, resolves Google News links to publishers, and optionally enriches sources with NewsAPI and GNews keys.
  • Roles and Public Reading
  • Reader, scraper, and admin roles, optional OpenID Connect sign-in, and an opt-in public read mode control exactly who can read, fetch, configure, and manage users.

Apps Similar to MuckScraper