The past few days the concept of frametime tests - the way in which we began testing GPUs a while back - has become a hot topic. Both Nvidia and AMD have taken an active interest in the subject which has led to passionate discussions and exciting new insights. During the Game Developers Conference 2013 in San Francisco this week, I spoke to a number of the main players. Considering frametime tests have become an important component of the Hardware.Info graphics card test suite, we are of course very interested in the latest developments in this area.
If you're not familiar with the term "frametime test", please read our in-depth article here. In short, by analyzing how long it takes to render each frame of a 3D game, you get a much more accurate reflection of real-life performance. A slow-down in rendering a single frame can cause stuttering in a game, even while the average frames per second can be quite high. The traditional indicator for graphics cards in benchmarks has always been fps, but since that is an average both over time and of the time it takes to render a given amount of frames within a second, it doesn't always tell the whole story.
We found that the results of the frametime tests correspond closer to the actual gaming experience than if you look purely at the average fps. When we tested Far Cry 3 back in December, we noted that "In our experience you need about 60 fps to run Far Cry 3 smoothly. We also noticed that the game doesn't run great on AMD cards. Even when they are fast enough and achieve more than 60 fps we still noticed stutters. Judging from forums, we are not the only ones to experience this."
And when we performed frametime tests with the Radeon HD 7970 and GTX 680, we saw the exact same thing.
Frametime tests confirmed our own subjective experience. Far Cry stutters with AMD cards.
Before frametime tests, we more or less had to state that an average of 60 fps was required for a guaranteed smooth gaming experience. However, in practice certain games are very smooth at much lower fps as well. That this is possible was also confirmed by frametime tests. For example, when we tested Battlefield 3 with Ultra settings on the Radeon HD 7870 and the GeForce GTX 660 the average fps would lead you to believe that the game is on the edge of being playable. However, the frametime tests confirmed our subjective hands-on experience, that the game played just fine despite the seemingly low average frame per second count.