Danbooru

403 When fetching RSS with Discord Bot

Posted under Bugs & Features

ibnu

Hi, I'm having trouble fetching an RSS feed from the Danbooru website using self-hosted MonitoRSS bot (https://github.com/synzen/MonitoRSS) on my server. Problem is, I keep getting a 403 error message (last time it work about 2 months ago). I've tried adding login and api_key on the URL parameters, but that doesn't seem to work. Changing the user agent is not an option since it would affect all my other feeds.

Thank you!

Reply
kittey

No, sorry. One of the admins will have to look into that.

A single request every 5 minutes should be fine, so I’d guess that MonitoRSS is banned. Maybe you can pause all your other feeds and change your user agent just to find out if that’s the problem?

Reply
bipface

assuming you take kittey's suggestion and confirm that the user-agent is indeed the cause of your problem,

potential workaround would be to send the feed requests through a reverse-proxy which alters the user-agent before forwarding them to danbooru — haproxy would be ideal for that purpose,

one other thing is that in the MonitoRSS source i see usage of a config field called feedUserAgent ( here ), which almost looks and sounds like a feed-specific option to configure the user-agent …
that might be an angle worth investigating (however I can't see any other reference to it nor any documentation for it)

Reply
nonamethanks

If you're getting 403 then it's a cloudflare issue, and as far as i know changing the user agent to something unique is the only way to fix it.

IIRC we had issues in the past with RSS users doing stupid things like checking feeds ten times a second forever so not sure if @evazion added harsher filters for them.

Reply
evazion

I unbanned the MonitoRSS user agent. As I recall, it was banned before because the hosted version was generating unreasonable volumes of traffic (think: trying to check hundreds of tags once per minute) and it was rotating IPs every few hours, which made it impossible to ban just the user or instance responsible. I may have to ban it again if it becomes a problem again. The way to not get banned is to use a unique user agent so that if your bot causes problems I know who to yell at and don't have to resort to banning entire IP ranges or user agents.

PS: The rate limit for RSS feeds is 1 per 2 seconds. This is enough to check 30 tags per minute, or 1800 tags per hour. But most feed readers are badly behaved and don't have proper rate limiting or error handling. So people do things like put in a thousand tags, then the feed reader tries to check them all at once instead of spacing them out. Then most of them fail, and the feed reader ignores the errors and keeps trying anyway instead of backing off. So if you're going to follow large numbers of tags, make sure your feed reader can handle it and won't do stupid things like try to check 1000 tags per minute.

Updated

Reply
ibnu

evazion said:

I unbanned the MonitoRSS user agent. As I recall, it was banned before because the hosted version was generating unreasonable volumes of traffic (think: trying to check hundreds of tags once per minute) and it was rotating IPs every few hours, which made it impossible to ban just the user or instance responsible. I may have to ban it again if it becomes a problem again. The way to not get banned is to use a unique user agent so that if your bot causes problems I know who to yell at and don't have to resort to banning entire IP ranges or user agents.

PS: The rate limit for RSS feeds is 1 per 2 seconds. This is enough to check 30 tags per minute, or 1800 tags per hour. But most feed readers are badly behaved and don't have proper rate limiting or error handling. So people do things like put in a thousand tags, then the feed reader tries to check them all at once instead of spacing them out. Then most of them fail, and the feed reader ignores the errors and keeps trying anyway instead of backing off. So if you're going to follow large numbers of tags, make sure your feed reader can handle it and won't do stupid things like try to check 1000 tags per minute.

thanks, but I still got 403 with original MonitoRSS UA ("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0"). As kittey said, changing UA works, and I have raised an issue for the MonitoRSS developer. Right now, I think I'll stick with Danbooru user agent and if I have problem with other feeds, I'll just deploy second bot. I don't think I am smart enough tinker with the bot's code or set up reverse proxy. Or if I give you my server ip address would fix the issue, please let me know.

Reply
1