Crawler

DeepBriefBot

DeepBriefBot is the crawler of DeepBrief, an app that writes a personal financial brief every morning for retail investors. It identifies itself with this user agent:

DeepBriefBot/0.1 (+https://deepbriefapp.com/bot)

What it fetches

• RSS and Atom feeds of news publishers.
• Your robots.txt, read before any article page and cached for 24 hours. If it cannot be reached because of a server error, the bot fetches nothing from your site.
• An article page only when its feed item has no usable description. A short summary is stored; the full text is not kept. No browser, no JavaScript execution.
• Public data APIs for market indicators.

When

Twice a day, at 04:00 and 16:00 Paris time.

Why we fetch

Every chapter of a DeepBrief brief cites its sources and links to the original article: readers who want more go to the publisher.

How to block it

Add these lines to your robots.txt. The bot honours them at its next run.

User-agent: DeepBriefBot
Disallow: /

Questions or a request about your site: support@deepbriefapp.com