Xiaomi caught cheating at Mi 8's DxOMark camera benchmark

Xiaomi has emerged as one of the most successful smartphone companies in recent times. The company has especially hit gold in India where it recently dethroned Samsung to have the highest market share of any smartphone manufacturer. However, it's not unusual for such a high level of success to tow with it an ample amount of controversy. Xiaomi is no stranger to controversy, having been the perpetrator of issues ranging from breach of privacy to invasive ad injection. To add to those, it has now been discovered that Xiaomi has CHEATED, or at least attempted to manipulate scores on DxOMark’s camera benchmark test for the Mi 8.

The Mi 8 was launched in May earlier this year. At launch, Xiaomi did nothing to hide how proud it was of the Mi 8's DxOMark score of 99 (greater than even the Pixel 2's score of 98!). XDA developer/camera modder defcomg, who is especially known for his work on Google Camera mods, through his data mining efforts, has now discovered that Xiaomi has used scene recognition to identify DxOMark’s camera testing scenes and enhanced the output for those specific scenes, to illegitimately achieve high scores. This was done by setting filters for those scenes in a file (/vendor/lib/camera/com.qti.tuned.semco_imx363.bin) that appears to be a database for scene based colour correction values.

At the time of writing this article, we were able to confirm the existence of references to said tuning in the same file taken from a recent version of Mi 8’s stock ROM. Digging through the file using a hex editor, we could see that there were at least 190 different references to DxOMark’s testing conditions. Some examples of the references we identified include:

1. "DXO Office Yellow Wood Table"

2. "green water in DXO"
3. "For DXO office table-1"

4. "For DXO office table-2"
5. "DXO office Red plastic board"

(i) What we think happened:

Going by what defcomg says, we believe Xiaomi has used scene recognition data based on DxO's test scenes from past camera benchmarks (refer section (iii) for more info) to recognize those scenes when the camera was being tested and applied well tuned white balance values for those specific scenes, making the output images look close to perfect in terms of colour reproduction. That way, they would be granted high scores, as the camera would seem well tuned, while in reality only those specific scenes were tuned to near perfection. To explain in more simple terms, this is quite similar to how the so-called 'AI ' modes most OEMs these days include on their devices' camera applications work, an image recognition data model (this data model contains the key points with which to recognize an image) is used to recognize a scene and apply filters specific to those scenes so that the output image would look better than when shot normally. Only, in this case, Xiaomi has used this technique to specifically enhance the output of DxO's test scenes to manipulate scores.

We attempted to match the DxO strings with test scenes from DxO's review, to support our claims. Here are a few examples:

1. "DXO Office Yellow Wood Table"

Image source: DxOMark

2. "green water in DXO"

Image source: DxOMark

3. "For DXO office table-1" (could be any of the tables), "For DXO office table-2" (could be any of the tables), "DXO office Red plastic board" (the red board is placed on the shelf adjacent to the blue wall)

Image source: DxOMark

Being well aware of how tall our claims are, we are including concise instructions to reproduce the proof on which we base our claims. You are the judge, people!

(ii) Steps to reproduce:

1. Get a copy of the file com.qti.tuned.semco_imx363.bin from /vendor/lib/camera/com.qti.tuned.semco_imx363.bin on a Mi8 (requires root access). For the sake of convenience, I am linking the file here (linked file is from a dump of Mi 8's stock ROM based on Android version 9).

2. Open the file using a hex editor, I’d recommend using Hex Workshop.

3. Press Ctrl+F, set the type to 'Text String', set value to 'dxo' (we are searching for references to DxOMark), make sure ‘Find all instances’ is checked and press OK.

4. You should now be able to view all of the references to the DxO test conditions Xiaomi has prepped the camera to detect, as shown in the screenshots below (screenshots are random, in no specific order).

Click here to open the image in a new tab. This screenshot of Hex Workshop includes references to "For DXO office table-1", "For DXO office table-2," and "green water in DXO"

Click here to open the image in a new tab. This screenshot of Hex Workshop includes references to "DXO office Red plastic board"

(iii) Why this was possible:

From this incident, it becomes apparent that Xiaomi has exploited the fact that DxOMark uses similar test scenes to evaluate cameras by preparing scene recognition data for test scenes DxO has used before, and is likely to use, and applied specific enhancements to those scenes, thereby effectively manipulating scores. This could be avoided by using different test scenes for different devices, but that would lead to inconsistencies in the evaluation process. The only way DxOMark could truly ensure something like this doesn't happen again is by requiring that the device's camera software undergo a preliminary screening process.

DxOMark uses similar test scenes to evaluate different devices. Left: DxOMark review of the ZenFone 5, Right: DxOMark review of the Galaxy Note 9. Notice how similar the test scenes are. Click here to open the image in a new tab

DxOMark uses similar test scenes to evaluate different devices. Left: DxOMark review of the Mi 8, Right: DxOMark review of the OnePlus 6. Notice how similar the the test scenes are. Click here to open the image in a new tab

Needless to say, this discovery is quite scandalous and extremely disappointing, seeing how Xiaomi aggressively promoted the alleged superiority of the Mi 8’s camera to that of other mainstream flagships and the recent announcement of the establishment of a dedicated camera department. We, at AndroGuider, hope Xiaomi and/or DxOMark acknowledge this fiasco. Also, seeing how simple it is to manipulate DxO's benchmark from this incident, who knows how many other OEM's have adopted similar practices (we are investigating!). We have reached out to Xiaomi for comments on this issue. Let us know what you think in the comments and stay tuned for updates!