Ext3 / ext4 extended options with soft raid and simple performance tests.
I guess almost every linux admin heard that when you creating ext filesystem over software raid you should consider using extended mkfs options. Two important options for that operation are: stride and stipe-width, which you should calculate for your setup of raid (number of disks, level, chunk size). You can calculate all those parameters each time you create ext2/ext3/ext4, but there are lot of ready calculators to use. I'd prefer using this one. Cause it's simple and fast. But do those options really matter or this is just cosmetic change in performance?
Let's check it.
My favourite disk benchmark software is bonnie++. I always use it to test raid, filesystems or hardware performance. Bonnie++ has lot of options and switches, which gives you chance to measure I/O parameters you are interested in. At the moment I'm rather curious about continuous speed of disk subsystem (read/write big files), rather then random operations or read/write big amount of small files. Of course random reads/writes are very important when tuning database I/O subsystem, so we gonna look at that too. Strange thing is that I've almost never seen production database system on software raid (not sure why ppl prefer using hardware ones).
But let's go test some performance.
Test environment was my local (home) storage system. It's PC with C2D E8200, 4GB RAM, 3*1,5TB Seagate (ST31500341AS) soft raid5. It had ubuntu 10.04 at that moment, and bonnie++ was 1.96 (default ubuntu).
I've created dedicated 10GB LVM volume for tests, other disk I/O was quieted down. And I've run 4 tests: ext3 with default options, ext4 with default options, ext3 with proper extended options and ext4 with proper extended options. I've used default bonnie++ options (2*RAM disk space) with -f switch to skip character i/o tests which are not important to me. Here are results (file create test results are not shown cause system was too fast to evaluate proper results - with default file test values):
So apart of testing extended option impact, I've tested ext3/ext4 performance difference too. So let's read the results.
As you can see test used 8GB of data to check performance.
I've skipped i/o character tests so first result is block sequential write speed in KB/s. So we got 104.4 MB/s for default ext3, 141.1 MB/s for default ext4, 105.6 MB/s for proper ext3 and 144 MB/s for proper ext4. Quite nice speeds, but as you can see there are almost no performance impact between default and proper stride and stipe-width and differences are statistical like. But there is big difference in favour of ext4 (about 40%) in block writes to I/O system. That's worth to remember. You can also see that CPU usage is lower for ext4, worth to remember too.
Next result is sequential rewrite (read and write) in KB/s. So we got 59.2 MB/s, 86, 60 and 85.5 respectively. Again proper extended options gave nothing, but ext4 is 40% faster than ext3. CPU usage is higher for ext4 this time, but it's not killing machine CPU.
Then we got sequential block read in KB/s. And results are 225.7, 234.2, 225.4 and 236.2 respectively. And again extended options gave nothing, but ext4 is about 5% faster. CPU usage at almost same level.
Next is random seeks result (look at bonnie++ readme if you interested what it does - and you should be). This is amount of randoms seeks and results are 384, 397, 369, 394. Almost the same and this is no surprise, cause the biggest impact for this value have raid configuration. And for each test raid configuration was the same, so no difference. So far looks like ext4 is better choice that ext3 and extended mkfs options don't have impact at performance.
Disk subsystem was to fast for default file tests, that's why I cut them out. Secondly I'm not that interested in multiple file operations at the moment, so I haven't bothered to run tests again (yeah, they took some time to complete).
But than there is latency (maximum amount of time for a single operation), again character i/o results are missing. Results shows that sometimes ext3 is 20 times faster that ext4. But those are maximum latency values, and considering read/rewrite/write results average latency is lower for ext4, cause it have better overall results. So those maximum latencies don't really matter im my opinion and probably they appear very rarely during tests.
The thing that bothers me most here, is that extended options are not giving any performance impact. But I remember testing storage subsystems I/O of some servers (IBM, HP and FS), using their disks in software raids and there were huge impact (at least 20%) when extended options have been added. Maybe LVM mixes things up, but I guess I've always used LVM for those tests. Anyway results show that using ext4 is better idea when you work with sequential operations, and I prefer using it in my private environments. But I mostly work with CentOS/RHEL on my production servers and there is no official support for ext4 yet (it has technology preview status). I know you can use it, but looks like you can run into troubles when something goes wrong and you want some support. Looks like RHEL6 gonna have full official support for ext4, so I can wait a bit.
Anyway I still recommend using extended options when creating ext3/4 filesystem on software raids and still personally use them. Those results show something different (that there's no impact), but I remember my other results (guess I can find them someday). And I use ext4 for my personal data/servers cause I'm the only responsible person for those data. Gonna use ext4 with my company's production servers when it officially hit RHEL and guess gonna give it a spin on some tests servers.
One final note. Ext3/4 extended options are fit to your current raid setup. So if you change you raid layout (grow raid to new disks, change raid level, etc.) you have to rebuild your filesystem with new options. Which means backup data, make filesystem from scratch with new options and then restore data from backup (haven't heard about doing it with existing ext).
Next test coming up is huge comparison of performance of different raid levels with different amount of disks. So stay tuned.