I'm tweaking a lossy encoder. Some tweaks increase file size and improve output quality at the same time (or reduce file size and reduce quality). How can I measure whether those changes are beneficial overall? (i.e. check if it's a Pareto improvement)
Is there a commonly accepted metric for this? I'm looking for a metric that would give me a single number that I could use to automatically compare different versions of the encoder with each other.
Edit: I need results in less than 10 seconds, and metric must be able to notice improvements by less than 1%. I deliberately don't want any sort of manual testing. I'm OK with imperfections of objective quality metrics (they're good enough for testing small incremental improvements).