Posted under Bugs & Features
Much like two months or so ago, grabber seems to be experiencing issues. Except this time using the API key, and the fix outlined in topic #22838 do not remedy the situation.
I am unsure if I'm doing something wrong. Other sources for grabber return results as normal. It's limited to only Danbooru as far as I can tell.
Any help in fixing is appreciated.
Updated
nonamethanks said:
We were getting hammered pretty hard earlier, so api requests using fake user agents like a browser's are being blocked now. Set the user agent to something unique, like the url to your danbooru account, and it should work again.
I attempted to name it a couple of different things like "Grahf" or "Grahf Danbooru" and it still gives the same result. I am unsure if I'm doing something wrong. For some reason trying to put my user url it won't accept it.
Same with BID [ Bulk Image Downloader ] [ since no Linux version, run in a Virtualbox windows thing, downloaded to a shared folder so I can see the images normally. Slower with the extra overhead, it mostly works ]
The last half day, it displays the results on the screen / so obviously sees them / but won't download -- the internal log just has image not found for each line.
Incidentally, the configuration tool for BID has 'load cookies from' [browser --- in the virtual box ], and defaults to Internet Explorer. Now I can't imagine using IE, but wretched old Microsoft is now yanking it out of Windows without asking/informing the owner....
lkjh098 said:
It looks like the site is doing cloudflare checks for everything, even unique user-agents and requests with API keys. It would be really nice if requests with API keys were excluded.
Requests go through Cloudflare before even reaching the site so there is no way to know if an API key is valid or not at the Cloudflare level. Just send a real User-Agent and you'll be fine. Actual "unique" ones work, pretending you are a browser does not.
Bots or scripts that try to impersonate browsers are now blocked. I get too much traffic from people doing things like trying to scrape the entire site one post at a time, or checking the front page 10 times per second, or checking hundreds of tags every few minutes. It's hard to control this traffic because people do things like use tons of IPs to avoid rate limits, or rotate IPs to avoid bans, or spoof browsers to blend in with normal traffic. So now anything pretending to be a browser is blocked so I can better keep track of API traffic.
You can either set your user agent to something unique, that's not pretending to be a browser, or if you can't do that, ask the developer of the program to.
Talulah said:
Requests go through Cloudflare before even reaching the site so there is no way to know if an API key is valid or not at the Cloudflare level. Just send a real User-Agent and you'll be fine. Actual "unique" ones work, pretending you are a browser does not.
You don't need to verify the API key at the Cloudflare level, if it's invalid it will be rejected once it hits the actual server. Assuming the goal is to get actual results rather than just a denial-of-service attack, that should be enough to stop whoever's hammering the server.
lkjh098 said:
You don't need to verify the API key at the Cloudflare level, if it's invalid it will be rejected once it hits the actual server. Assuming the goal is to get actual results rather than just a denial-of-service attack, that should be enough to stop whoever's hammering the server.
if you're suggesting to disallow unauthenticated API requests, that would simply encourage the scripters to scrape HTML instead
bipface said:
if you're suggesting to disallow unauthenticated API requests, that would simply encourage the scripters to scrape HTML instead
I think the idea was more like “if the request contains an API key, don’t make Cloudflare block it”. If there’s no API key, have Cloudflare process it as usual.
kittey said:
I think the idea was more like “if the request contains an API key, don’t make Cloudflare block it”. If there’s no API key, have Cloudflare process it as usual.
in that case, while i can't speak for evazion i would guess it's not going to happen due to the complexity of setting up such rules -- noting that authentication credentials can be found in at least 4 different places:
?login=x&api_key=y)Authorization: headerCookie: headerx-http-method-override)Well... In short, is there a remaining way to search anything on the site with a third party software and through a VPN ?
why third party software : in order to post-filter results (since the gold accounts don't exist anymore and we're limited to 2 tags search, it's basically the only way to get relevant results...)
why VPN : my country is unfriendly to Hentai searches, since the internet providers are, by law, forced to record all the navigation datas and store it 10 years, and since bestiality/furry/loli/guro/non consensual...etc. is illegal (by law there's no difference between reality and art...)... getting a single result in these categories could give me a ticket to having the police knock at my door at any moment within 10 years... So a VPN is pretty much mandatory.
I mean, most websites block VPN unless we have an account and login, but here, having an account and API keys don't help :/
bipface said:
in that case, while i can't speak for evazion i would guess it's not going to happen due to the complexity of setting up such rules -- noting that authentication credentials can be found in at least 4 different places:
- URL parameters (
?login=x&api_key=y)Authorization:headerCookie:header- request body (in the case of
x-http-method-override)
Even only allowing URL parameters with the API key would be a significant improvement over the current situation.
Re B.I.D., see a solution on my post topic #23530 .