Tuesday, October 13, 2015

Quickie#3 CM12.1 vs Exodus ROM : SQLite performance

Having two devices with similar hardware helps a lot for "real-time" comparisons of different ROM at a given time. Here we are comparing CM12.1 (20151012 nightly) with Exodus ROM (based on 12.1 ... 20151012) regarding SQLite databases performances, this is not like a full ROM versus, simply a comparison on a very specific point.


The devices
Similar Hardware ? yes, and no...

The two devices are both XT1032, with pretty similar hardware, but with some differences regarding memory :
*Device 1Device 2
ROMCM12.1Exodus
eMMC8GB Sandisk16GB Toshiba
/data file systemF2FSF2FS
DRAM1024MB Samsung S41024MB Hynix S4

(does this adds a significant bias? hmmm maybe, but it won't explain the gigantic differences seen later)


Benchmarking
AndroBench 4.1

Here i am testing pure SQLite performances with AndroBench 4.1 on both devices using BFQ scheduler (default CM and Exodus I/O scheduler since quite a while)-results in Transactions Per Seconds (TPS)-



Exodus-BFQ CM12-BFQ

average stdev average stdev
SQLite Insert 1241,40 21,19 116,22 4,48
SQLite Update 632,82 12,41 107,78 2,82
SQLite Delete 952,57 16,57 118,21 5,81


Looks like there's quite a big difference... isn't it?
In day to day usage i don't feel much gap between those two ROMs, but this specific benchmark shows that Exodus had more in-depth optimizations regarding SQLite database operations

Discussing bias:
  • Memory brand is different, your results are messed up: No, in fact other devices (aka Moto E 1st gen [XT1022, eMMC: Hynix, DRAM 1024MB Hynix S4, Snapdragon200] or MotoG 2d gen) show similar results on CM12.1 (or even lower for the MotoE). So even if the brand may have a little impact on performance, it won't explain the 10^1 between results.
  • As Processing unit has a fairly big impact on I/O performances, both devices are featuring a Snapdragon 400 @1.2GHz, Interactive governor, MP decision enabled.
  • Is choosing BFQ as scheduler impacts results? and why choosing it? BFQ is the default scheduler for both CM12.1 and Exodus, for this simple reason there is no point testing another in first-line. Plus it could be interesting to see if the optimizations have been directly applied to this specific scheduler or if they are more system-wide. On previous tests i couldn't show any difference in performance between BFQ and Fiops regarding SQLite Update or Delete operations.
  • SQLite databases are not reflecting real-life performance though, so this only shows a very little part of everything -but should make benchmarks addicts happy ;-) -