NwlyBot

NwlyBot is the crawler of Nwly, an Arabic news reader that groups the same story from many outlets and sends readers to them.

Operator: Mohamed Radwan. Documentation: https://api.nwly.site/bot.

What it fetches

Public publisher article pages, article text, metadata, images, feeds, sitemaps and public article endpoints. Nwly retains article content for news indexing, grouping related coverage, search, summaries and reading in the app. Articles retain attribution and a link to the original publisher.

Retained headlines, summaries and article openings can also be used to train Nwly's local news-category classifier.

This is automated collection operated by Nwly, rather than an agent acting only on a person's current request. Content is retained and reused; it is not used only for an immediate answer. This bot does not log in to private publisher accounts or purchase access to subscription content.

How it behaves

• User-Agent: Mozilla/5.0 (compatible; NwlyBot/0.2; +https://api.nwly.site/bot)
• Direct crawler IP addresses: 199.241.137.175
• Requests are signed with Web Bot Auth (HTTP Message Signatures). Key directory: https://api.nwly.site/.well-known/http-message-signatures-directory, key id v4ugBq9sCR4k6XfyG1IDcW2_3vukvX2HJYZyXHg46nA.
• This identity describes requests identifying as NwlyBot. Browser rendering and third-party retrieval services are separate access paths and do not inherit this bot's verification.

Publisher controls and contact

The product token for publisher robots.txt rules is NwlyBot. A site-wide disallow rule is:

User-agent: NwlyBot
Disallow: /

Or write to [email protected] — to be allowed, limited or removed.