Danbooru

Failbooru

Posted under General

I've deployed the new calculator now. No guarantees it'll have better performance overall but I think it'll scale better. Will have to wait a few days for the cache to reprime itself.

I've also eliminated most of the counts in the user profiles. It makes it more of a hassle to drill down to view the numbers, but at least the page will load now. I don't think it's tenable to have that many real time calculations on one page.

albert said:
I've also eliminated most of the counts in the user profiles. It makes it more of a hassle to drill down to view the numbers, but at least the page will load now. I don't think it's tenable to have that many real time calculations on one page.

What about putting the numbers back up there and just updating them every X hours? Or making some way to view your own stats, at the least. I miss the numbers...

Seeing as how all the tag counts are gone, though, I guess caching wasn't enough? Losing those numbers hits even harder. Can't tell well-used from under-used/new tags.

Updated

The tag counts missing from the post view pages doesn't bother me so much so long as they are available through the tags interface (which they still are).

The loss of user statistics is a shame though. It'd be nice if there was a way to check them via the API or some other lesser used page (off the main profile page). So long as they aren't polled too often, I wouldn't think they should pose too much of a load hit. Also, as RaisingK says, even if they are cached and only refreshed on an hourly or daily basis, they would still be valuable.

The statistics are useful as well to get an idea of a user's general activity before considering invitations and such.

I haven't been able to access /post_tag_history for 2 days now, this makes it hard to figure out if someone's just dumb or a vandal. (Note that I have only been trying for two days it may have been down for far longer than that.)

Updated

Just for the record, the tag counts missing from the post view and especially the search pages bothers me a lot. Please find a way to reinstate it in some form, even if it's just for privileged and higher or some unfair tyrannical bullshit like that.

0xCCBA696 said: Just for the record, the tag counts missing from the post view and especially the search pages bothers me a lot.

I think it's just temporary, so that Albert can hunt-down what's been causing all these problems. Sort of like when user profiles had Simple and Advanced views for a few days.

Regarding the numbers, there are now two cases:

- Show the global post count. I don't consider this useful since many common tags have over 10,000 counts. Numbers that high don't convey much information and just add noise.
- Show the post count for related tags. Unfortunately, the switch to a sample-based related tag calculator means I can only calculate the counts for the 150 most recent posts, so the number you see won't match the number you get when you drill down. I predict people will complain about the confusion, so I decided it's better to just drop the number altogether.

albert said:
Show the global post count. I don't consider this useful since many common tags have over 10,000 counts. Numbers that high don't convey much information and just add noise.

But the counts for "uncommon" tags are useful. If a tag only has one or two posts, the tag might be a mistake or something that needs more use. And even for tags that aren't that rare, the counts still provide measures of relative popularity.

Updated

Okay, so as much as I can see, the changes did more or less nothing to improve the situation. I'm able to see my own profile now, which I couldn't before, but that's the only profile I can in fact access. All the other ones time out. I still get a lot of failboorus in many other places, too.

Albert, maybe we could, for once, concentrate on determining what exactly went wrong instead of shotgunning every useful feature in sight in the hope it will somehow fix things? You said the logs indicate the related tags to be the problem, but do you have any idea how come it came down crashing suddenly? Was there a particular surge in traffic? Registrations? You just added a new server recently, so how come it went from drastically better to unusable in such a short time? Also the related tags calculations were already being cached, so why were they so expensive? Another unusual thing is that with the site being so painful to use, there should be enough reduction in traffic to restore to within the operational limits, which would result in an oscillating pattern. But that's not what we observed, it went down and stayed there.

I'd really like this to be analysed and solved properly, based on the observed data, instead of attempting yet another blind poking in fifty different places, which results in no noticeable improvement, but many noticeable regressions and fresh bugs.

RaisingK said:
But the counts for "uncommon" tags are useful. If a tag only has one or two posts, the tag might be a mistake or something that needs more use. And even for tags that aren't that rare, the counts still provide measure of popularity.

This.
According to forum #38464, global counts are helpful for identifying whether posts you look at were tagged correctly or not.

  • only high counts → try finding at least one more specific tag to make this post stand out,
  • extremely low count → possible typo or something that needs an alias to the more used tag describing the same thing.

Missing the numbers here makes things much harder to read, and using the tags page to keep track of everything each time would just be too long to the point it's not even useful for that matter.

EDIT: the post counts on artist searches are also super useful to get a precise glimpse of what an artist drew (stats like 80-75-30-17-14 don't mean the same as 35-33-32-30-26, etc.).

> Now for a dumb question—apologies in advance, but what exactly are related tags?
I'm asking this because I guess I never used them (so I'm not even sure what they are besides the links you get under the tag edit box), and from what I read they're apparently stressing the servers a lot.
Blindly assuming that other users aren't making an efficient use of them either, and that they still generate server calls even when not used, perhaps this part is too much hassle for their actual use?

Updated

Related tags are those that show up when you click "related tags" on the update or edit interface. Also (and probably more importantly for this thread) they are/were used on searches to show which tags are most relevant to the query you searched for and how many times each showed up within that set of search results.

I haven't been using the site much this weekend since I'm on a slow connection at the moment, but I can see how the loss of this information too would be unfortunate.

Bapabooiee said:
I think it's just temporary, so that Albert can hunt-down what's been causing all these problems. Sort of like when user profiles had Simple and Advanced views for a few days.

I certainly hope so. It'd be a shame if things like that have to be crippled in order to make the site usable.

Although I still can't see my own profile I have noticed that page loading time has decreased and I'm no longer getting errors when uploading.

There's something refreshingly clean about not having the post counts next each tag. I'm not saying it should stay that way (for reasons mentioned by other members) but there is a certain appeal to it.

Shinjidude said:
Also (and probably more importantly for this thread) they are/were used on searches to show which tags are most relevant to the query you searched for and how many times each showed up within that set of search results.

... Damn. Didn't realize it was part of the same thing. *headdesk*
Well sorry for the inconvenience.

If nothing else, it would be useful to indicate tags that have single-digit usage, because that tells you you probably made a typo or some such thing when you put the tag in (or that the tag is named differently, whatever.)

For tag counts, maybe set a threshold so after a tag reaches a certain number of posts (1,000? 5,000? 10,000?) it stops counting them individually and just says "1000+" or whatever. This way the tags with low numbers still stick out, which is important for reasons RaisingK and Cyberia-Mix mentioned.

And for the user page, is it possible to make it so the "View" button loads the numbers right there onto the user profile? That's actually what I was expecting to happen at first, but instead it just takes you directly to the entire history page.

1 2 3 4 5 6 7 8 12