The benchmark utility for Final Fantasy XV Windows Edition dropped last week, and eager gamers immediately started testing their own systems against the daunting requirements of Square Enix's upcoming monster release. Unfortunately it quickly became apparent that not all was well in the sate of Lucis.
For one, the benchmark tool had few discrete quality controls to speak of, just resolution and broad quality settings - Lite, Standard and High. There's no indication of how these settings differ through the tool, and it's only on external sites where this information is available. It turns out that each quality tier enabled specific NVIDIA GameWorks features such as Hairworks and Turfworks, immediately reducing the tools' relevance to non-NVIDIA GPU owners who would choose to have these features disabled.
The pesky buffalo, here rendered correctly
Secondly, some great investigative work by GamersNexus (ironically using Ansel, which is compatible with the benchmark) uncovered evidence that certain Hairworks-intensive buffalo models were being rendered, despite not featuring or even being visible within the scene. That's a significant red flag that could be evidence of an attempt to cripple performance on a certain subset of GPUs (i.e. Radeon). Emphasis on could... it could just as easily be down to sloppy or rushed implementation.
Third, the benchmark tool doesn't provide FPS numbers or frame times, just a aggregated number meant to represent performance. This score is then compared against previously determined results to indicate High, Medium or Low performance level, as if that is at all meaningful or relevant when assessing anticipated in-game performance.
Oh, and the benchmark runs aren't identical; far from it in fact. It's especially noticeable in the combat segment, but NPC movement and camera angles differ from run to run which results in different scores. Two runs we performed back-to-back differed by 90pts, and that's just the briefest of investigations.
Finally, the in-benchmark experience is horrifically bad, stuttering regularly even on mid-range current generation hardware at standard settings. While such configurations my achieve a 'High Performance' result, evidence would suggest that your in-game experience would be terrible.
These criticisms came in like a flood over the weekend, and all credit to Square Enix for responding quickly.
Square Enix official response via Facebook:
We would like to thank the gaming community for all the feedback on our recently released FFXV WINDOWS EDITION benchmark tool.
Your feedback will help us fine-tune the performance of the game.
We wanted the benchmark tool to approximate the level of visual quality and performance you can expect from your PC environment when running FINAL FANTASY XV WINDOWS EDITION.
A Level Of Detail (LOD) issue has been discovered that affects the benchmark scores.
The benchmark also suffers from stuttering; both of the issues will be addressed in the shipping game.
While the settings in the benchmark are largely pre-selected for simplicity’s sake, the final game will feature highly customizable settings with On/Off options for a wide variety of settings.
The benchmark will give you an idea of how beautiful the game will be upon release, but for the reasons stated above, the benchmark may not accurately reflect the game’s final performance.
We would like to thank the gaming community for all the feedback on our recently released FFXV WINDOWS EDITION benchmark tool.
Your feedback will help us fine-tune the performance of the game.
We wanted the benchmark tool to approximate the level of visual quality and performance you can expect from your PC environment when running FINAL FANTASY XV WINDOWS EDITION.
A Level Of Detail (LOD) issue has been discovered that affects the benchmark scores.
The benchmark also suffers from stuttering; both of the issues will be addressed in the shipping game.
While the settings in the benchmark are largely pre-selected for simplicity’s sake, the final game will feature highly customizable settings with On/Off options for a wide variety of settings.
The benchmark will give you an idea of how beautiful the game will be upon release, but for the reasons stated above, the benchmark may not accurately reflect the game’s final performance.
Good for: deliciously rendered pancakes. Bad for: pretty much everything else.
While that clears up what we can expect in regards to graphical options in the final game, the question remains whether the tool is fit for purpose and the conclusion has to be that it currently isn't. Sufficient information or configuration options are not provided, it doesn't generate an output result which is easy to parse, and the scenes rendered are not indicative of the final game in any aspect except frame by frame visual fidelity.
We praise the developers for releasing this benchmark, but frankly pre-ordering the game based on the information and tools provided would be extremely unwise.
