It's not about the average. It's about a more accurate rating system. As I said before, rating something 7 of 10 is more accurate then rating something 4 of 5.
As you've been told, that's a misuse of the word "accurate." You're making the common mistake of confusing accuracy with precision. Accuracy is about whether you have the right answer, while precision is about how narrow the margin of error is. For instance, if you say that the value of pi is somewhere between 3 and 4, that's accurate but not at all precise. But if you say that the value of pi is 3.9658751356884309, then that's extremely precise but not at all accurate. Increasing precision does not necessarily increase accuracy.
Indeed, too much emphasis on precision can work against accuracy. For instance, the common belief that normal human body temperature is precisely 98.6 degrees Fahrenheit is quite wrong. In fact, it varies from person to person and fluctuates over the course of the day. What medical science actually determined was that human body temperature has an average value of approximately 37 degrees Celsius. And then, when that was converted into Fahrenheit, someone made the mistake of taking that accurate but approximate figure as a precise one -- as
exactly 37 C, which converts to exactly 98.6 F. It would be more accurate to say that body temperature is about 99 F, give or take, but the presence of the decimal point created a false perception that it was a precise and unchanging figure, rather than something that naturally varies. Thus, too much precision created a belief that was inaccurate.
In this case, we're talking about personal opinions. It's problematical to talk about "accuracy" for something like that, because there is no objective "right" answer for matters of taste. Heck, I've often read a book a second time and found I liked it substantially better or worse than when I read it the first time. Even a single individual's tastes are variable enough that a broad and imprecise set of ratings is more likely to be accurate -- i.e. to encompass the right answer -- than a more granular rating that may include your reaction on one reading but exclude it on another. So your belief that more precision equals more accuracy is pretty much backward. Part of being accurate is acknowledging the margin for error.
Learn to love the five point system, it's here to stay 
