Oh god. Not again. Right?
In a recent post on The Animachronism, there was a nice little comparision between Good Reads and MAL rating categories. As a stimulus, I thought I could elaborate a bit on a 10-level suggestive system, and wound up bouncing thoughts around with ghostlightning (GL). For reference, we were on the concept of rating experience as compared to rating an item in general; probably a whole separate discussion could be spawned from that idea.
My initial proposed system was overly-positive, having double the levels for positive ratings (6), one neutral, and three negative. GL expanded/modified and came up with a 5 positive and 5 negative; well-balanced.
Overly-Positive System
- Pathetic - A rating of pity
- Fallible - Well below accepted.
- Unacceptable - Inadequate, insufficient.
- Ordinary - Of no special quality.
- Acceptable - Standard.
- Commendable - Worth the experience, in time/energy.
- Exceptional - Above the norm.
- Excellent - Superb representation of the medium
- Essential - Essential experience within the medium
- Transcendent - Transcending a given medium
5-to-5 System (credits to GL)
- Waste - Wasted my time and hate myself for it.
- Discouraging - Begin to question why I like this medium.
- Undesired - Will watch again only if I admit I like the wrong kind of pain.
- Displeasure - Suffered throughout.
- Negative - Felt like I should have enjoyed myself but could not overcome my problems.
- Positive - Enjoyed myself despite having problems.
- Solid - Enjoyed a lot and have little problem saying so.
- Validating - Reminds me why I like this medium in general.
- Excellent - A superlative experience that can eclipse my preferences in the medium.
- Orgasmic - I came, and will orgasm every single time I watch.
Make of these what you will. Right, so after a bit, I attempted to hybridize, which caused me some mind exhaustion. This morning (on the EST) I sat down with a few relatives who have listened on ratings for some time now, and gave them some example rating lists (including MAL and ANN) for thought. After some coffee, snacks, and smirks:
- Wasted - Complete waste of time and mistake on my part.
- Despairing - A turnoff towards the medium.
- Poor - I suffered and will watch again only if I admit liking the wrong kind of pain.
- Unfavorable - Unable to overlook my issues during the experience.
- Ordinary - Just. Neutral. Not positive, not negative.
- Acceptable - Enjoyed, but with some issues.
- Good - Enjoyable and do not mind saying so.
- Great - A reminder why I enjoy this medium.
- Excellent - Able to eclipse my preferences within the medium.
- Superlative - Exceptional in and beyond the medium.
At first glance, there are still more positive levels than negative, but I feel there is a special kind of lensing inherent. If we take levels 5 and 10 out of the list, we are left with 4 positives and 4 negatives; balance. So why discard these levels in considering balance?
Ordinary, is neutral, so there is no effect on balance. On the other hand, Superlative is the most positive, the highly-debated 10. My personal stance on 10s I don’t have a stance, nor a 10, ‘kay I’ll make something up is that there is no reason not to have 10s, but at the same time, the occurrence of them should somewhat represent an exponential decay from 6 to 10 1. it’s a 10 don’t cheapen it >_<! Considering this should-be rarity is outside of the Gaussian distribution, it can be discarded.
Also assisting in the rarity of 10, is the terminology. For those who may be reading and do have 10s, do all of them fit the Superlative description? Exceptional beyond the medium? I feel, in most cases, about 60% of 10s would become 9s 2. It’s not a bad thing, only a redefinition; being stuck to the notion of omg but it’s a 10 is silly.
Well what about a 1 rating, can’t that be thrown out? No, because the Zero rating has already be discarded :P.
Light, fun, nothing serious.
Notes
There are merely thoughts and pondering. I, myself, still prefer rating with arbitrary numbers, by subjective quality high-to-low (doesn’t matter what a 10 is, but it was an experience greater than a 9… drops the requirement of having any specific number of tiers).
1 - It’s a 10, why cheapen it >_<
2 - General guess.
Hmm, interesting viewpoints. I think my personal rating system falls closer to gl’s 5×5.
I’ve noticed that for me, there’s a very small difference between my 7’s and 8’s. I wouldn’t have too much problem swapping between those 2 ratings. But then I have a pretty substantial gap between 8 and 9. In fact, that gap might be slightly bigger than the gaps between my 9’s and 10’s.
On the crap side, the difference between a 4 and 5, may as well be the difference between a 1 and 5. I’ve never ranked anything under a 4, because if it gets that bad, I’ll usually drop it before I feel like I’ve watched enough to give it a fair rating.
In the end, it’s all really relative though.
@RP Relative, very true. Ratings should be taken individually anyhow, and if they are to be used for overall stats, do it properly and not just via hard numbers (not considering every rating instance by a user to calculate a single item rating) (I’ve gone over calculations before, but it’s not really important for raters to “know” all the details).
I like that self-observation about the distances between some of your rating levels. Optimally you should be allowed more levels to make the distances balanced (the “points” going from level to level should degrade smoothly). As I’ve mentioned previously, if a rater wants 15 levels great, do it, but at the same time it should be translatable to other users (which is possible).
After hearing GL, I’d say dropped shows are fair game, even with the negative rating. So why not.
Whether a user uses defined categories, re-labels with their own meanings, or uses a larger/smaller spectrum, there should still be some fundamental understandings:
1 - Two items, one with a higher rating is self-explanatory.
2 - Loading down a rating level desensitizes the meaning of the rating.
*In a perfect situation, each item has exactly one level, and all levels are related higher/lower with each other (It’s painful to try and rate everything on it’s own level, so we lose accuracy by grouping on tiers, not necessarily bad).
Re: dropped shows, I prefer to withhold a rating decision until I get a sense of how the show’s doing over a handful of episodes (sort of a weighted average). I’m reticent to “penalize” a show without giving it a “fair shake.” But that’s more of a personal thing, then a universal rule of thumb. I can definitely see the rationale for rating it anyway, and sometimes I wouldn’t mind just to spite the show’s rating.
The 1st fundamental understanding makes sense to me, but with the 2nd, are you saying that having too many shows with a 7 (for example), desensitizes the meaning of the 7?
I could see that. If you go to the logical exclusion and say everything is “7″ or above, then “7″ itself will lose meaning. Suddenly, then “6″ takes on a whole new meaning, and if you see fit to reward something with “6″, people will see you as saying, well, this is much, much worse than everything else.
I think the implication is that ratings that are handed out rarely are the ones that have the greater impact. It’s like the idea of advertisements trying to come up with new superlatives to describe their products. If everyone starts by saying “this product is good, you should buy it”, after a while people will be inoculated to the word “good” and think, when they hear it “well, ‘good’ is fine, but it’s not that great”. So advertisers will move on to “great”, then “wonderful”, then “terrific”, then “sublime”, etc, and each time one of these words becomes overused, it’s impact and original meaning will be reduced. Which is why I think it’s makes sense to keep the ratings that you want to use to make a statement in rare use. If you want people to sit up and take notice when you award something with a “10″, then the way you do that is to only award it extremely rarely. That’s why “10″ isn’t redundant if used extremely rarely… if anything, it’s meaning and impact increases the less it’s used. Same goes for “1″ or, as we have on our site “0″ (which is even more rare than “10″).
Coincidentally, the rating system we have on our site compares very closely to the “hybridized” system, except that we also have a “0″. That kinda lends it to a sense of symmetry, with “5″ for “Average” at the middle. It doesn’t work that way, though, we are, after all, anime fans, so the weight of the distribution is on the positive side. I’m not sure I agree with the idea that there should be an exponential decay between “6″ and “10″. On our site, the most numerous rating is “7″, and there’s a linear drop-off between “7″ and “9″ and a fast drop off between “9″ and “10″. Personally, I’d have preferred “6″ to be the most numerous rating, but that kinda can’t be helped. Reviewers prefer to review the series they like.
@RP Personally, I don’t like to rate things I haven’t completed, but I can understand when some do rate dropped series. ^^
Well there is no defining too many but generally, yes. For instance, an extreme case (or possibly base-case), a user who rates everything the exact same number. This is the inverse of the optimal rating, where every item has it’s own level.
No matter what the “value” of the rating may be, the user’s ratings are flat. What would one think if they saw a user who rated everything a 10? Whether the ratings are serious or not, it does not help in figuring out where items stand relative to each other, and is quite similar to “crying wolf.” If that makes sense.
From another’s point-of-view, how useful are those ratings…
Given, this is not necessarily the user’s fault, should not matter to the user (unless they intend to make their ratings useful) and could be attributed to the limits of a fixed rating system.
@Sorrow-kun Hey Hey!
I think the implication is that ratings that are handed out rarely are the ones that have the greater impact.
Precisely.
linear drop-off between “7″ and “9″ and a fast drop off between “9″ and “10″.
I did say exponential decay, but this is similar enough. The main idea is that certain ratings do become rarer at some point in the scale.
Also, review ratings within a closed system, such as a review site or blog tend to be accompanied with explanations, which is advantageous imo. Open-rating systems such as ANN, etc have a much harder time, because individuals cannot represent the way the would rate things or what a certain value means to the specific user.
This is a big stimulus for myself in creating a method of dynamic open-rating, where users may rate with any number of levels as they please, but still be relevant system-wide (math math math) simply by following the basic two principles stated (though no.2 requires more explanation).
A GL-style rating-the-experience scale has many merits, and one I’d also like open, public systems (ANN, MAL, etc.) to adopt, if not at least shared among individuals, but it should be pointed out that there are pitfalls too. Namely, reactions heavily based on impressions from the last episode, or batch of most recent ones seen. Other times, it’s difficult to separate a show’s experience from outside influences, like who you experienced it with, what was being discussed on the interwebs or at the water cooler, the span of time over which you saw/read the whole work, things of that nature.
There were plenty of times I’d mark something on MAL an 8, 9, or 10, only to later reflect more (harder?) and pull back the score because it was swayed more by a great finale than the whole, pure experience.
@kadian1364 I feel reactionary ratings are natural, and if the rater wishes to express more about the “environment” of the experience, there are plenty of places to blog or express. Ratings are not as textured as expression in words.
Ratings should be allowed constant adjustment as well. If you glance back at this post, you’ll see one of the main advantages of dynamic rating (if you change one rating, you have the ability to change all other ratings fluidly). Example:
If suddenly a rater feels the need for a level between 6 and 7, they can rate a new item as a 7, and shift anything which was 7 or above, upward.
Another rare case would be if something comes along that blows all other experiences away. Yea, so it’s a 10, but shouldn’t it be rated higher than other 10s. So rating an 11 is totally viable.
There isn’t alway a need to do it, but the ability is there. It also doesn’t matter if two user’s scales have more or less levels; it’s transformable.
I know, sorry.
I like the focus on the quality of the experience, which ties into the whole ‘experience is not bound to medium’ thing. Though I suppose the idea of works which are ‘exceptional in and beyond the medium’ might tangle the hypothetical raters up in questions about the relative status of different media. (If a book is exceptional for its medium, is it likely to be automatically superior to all television, and so on . . .)
Since I was writing about the ways that two sites present their respective rating systems, rather than how or why we should actually rate things, I didn’t really feel best-placed to judge between them and/or other regimes . . . but I’m inclined to agree that a high-to-low system with an indefinite, expanding number of tiers is more practical, though it takes a bit of time to get my head around it.
@IKnight
You make a great point about ‘exceptional’ state. I sort of threw it out there, simply based on what I felt a 10, the highest rating, could possibly mean. Although it is the ‘experience’ being rated, it’s tough to break through user preference. A person enjoying a book at the same level as enjoying a movie might rate the experience of the book higher than that of the movie because they prefer the enjoyment of reading vs watching over the course of a couple hours. It’s a predicament of trying to have a 10 equal a 10 for every medium; it came out that way, but wasn’t quite my stimulus.
Rather than ‘being exceptional in every medium’ I feel a 10, for myself, is something that can be enjoyed/satisfying even beyond the medium; not necessarily a 10 outside the realm, but enjoyable. It’s almost like a recommendation. My personal thoughts on recommendation would be similar; if I recommend something, it is most likely across all media.
Case in point: Cowboy Bebop. Perhaps the only thing I would recommend without a thought. It possesses a level of cultural familiarity non-anime fans should be able to identify with, a fresh exposition, great character dynamics, etc etc.
Within the medium, the experience is a 10. Outside the medium and if I wasn’t an anime-enthusiast, I still feel it’s a satisfying experience.
It’s kind of difficult to push that meaning of a 10 on others, but that’s how the dynamic rating system comes into play. The concept is rather tough to think through, but it’s quite the same as a fixed-point system for the most part. It’s extremely difficult to rate everything on it’s own tier in an infinite list, but defining what level X means ahead of time helps to sort these thoughts.
Allowing users to define their rating system is optimal, and what matters in the overall is something of this nature: “the height at which an item appears in a list, over the system of all lists, determines the relation of said items to all other items in the system.”
If an item is in the top tier on every list in which it appears, it should be on the top tier in the overall… almost, it is surely more complex, but that is the general idea.
Rating—serious business.
Don’t mind me; just dropping a “I’m not dead” comment.
Oh aurabolt. You are pro-AniBlogger-Commenter now XD
Ratings are serious, but maybe they shouldn’t be. I dunno, sometimes I just kinda go with it. Sure if someone is a reviewer for a professional review facility (ie. AllMediaGuide) then the rating isn’t really open to change. Most cases, pliable ratings are okay