Gateway NV52 (AMD) - OS Benchmarks
The final aspect of operating system performance we want to look at involves common OS tasks. We measure how long it takes to perform the routine operations of booting, shutting down, hibernating, resuming, suspending, and waking your computer using the four test operating systems. Unlike other test results, there was a huge degree of variability in some of the tests. We ran each test at least five times, and we took the best result - the reason being that some of the results were completely out of line with the average result, so including one or two results where the system took four times as long it didn't seem logical. Ubuntu and Windows 7 didn't show as much variation as XP and Vista, particularly in the sleep/wake testing, but the hibernate/resume test once again varies quite a bit between test runs.
The variability remained even after following a relatively straightforward test procedure. First we booted each laptop using the test OS (timing how long it took). Then we let the system stabilize for at least 15 minutes, with no extra applications running and the system sitting idle at the desktop. Finally, we performed one of the other tests - shut down, hibernate/resume, or sleep/wake. Obviously, you can't test resume time without first hibernating, and you can't test wake time without first putting the system to sleep, but otherwise we made sure to shut down the system between test runs.
We also need to mention that Windows XP has an inherent advantage in the hibernate/resume testing because we are using a 32-bit OS instead of a 64-bit OS. That means the system doesn't make use of all 4GB of memory, and the hibernation file is 3GB. Writing 3GB of data will obviously be easier than writing 4GB of data, especially when you're dealing with a 5400 RPM laptop hard drive.
The results are rather scattered this time, with certain functions favoring one OS and others looking better on a different OS. Boot times favor Windows XP, followed by Windows 7 which takes 11% longer. Vista trails Windows 7 by 10%, and Ubuntu brings up the rear at 5% longer than Vista. Shutdown times are a different matter, where Ubuntu takes the lead. Vista and Windows 7 are essentially tied, taking 55-60% longer to shut down than Ubuntu. Windows XP brings up the rear this time, at over 2.5 times as long as Ubuntu, although we have to say that this appears to be a driver issue as the result for the NV58 is radically different. When you shut down Windows XP on the NV52, there's a delay of about 15 seconds where there is no apparent activity, after which the hard drive activity light begins blinking. It would seem that without that handicap, Windows XP would be roughly the same as Vista and Windows 7.
As already stated, Windows XP 32-bit has an advantage for the hibernate/resume testing because it doesn't have as much addressable RAM. It ends up walking away with the hibernate/resume performance crown, followed by Windows 7. That said, the time to hibernate in Windows 7 is 34% longer than Windows XP and resume time is 32% longer; Win7 has 33% more memory, so a 32-bit version of Windows 7 ought to be as fast as Windows XP (or a 64-bit XP as slow as Win7). The gap between Windows 7 and Windows Vista is extremely large for hibernation time, with Vista taking 65% longer to hibernate and 46% longer to resume. Ubuntu is almost tied with Vista on the hibernation time, but it takes 50% longer to resume - about twice as long as Win7 - making it by far the slowest OS for this particular test.
We need to take a moment to discuss boot times and resume times, however, as there's more going on than meets the eye. When booting or resuming a system, the computer needs to perform a Power On Self Test (POST). That takes around nine seconds on these Gateway systems, and once that's complete the boot loader is activated which then loads the OS. Ubuntu uses GRUB and that appears to add an extra ~10 seconds relative to Windows. Without that time penalty, Ubuntu would be far more competitive in boot time (about equal to Win7), although it would still be in last place in the resume test. There are also laptops that have a fast boot feature so that they don't perform a lengthy POST, which can shave another 6 or 7 seconds off the boot/resume times. This is definitely a feature we would like to see implemented on more systems going forward, particularly laptops where users routinely boot or resume several times per day.
Wrapping up the OS performance analysis, the sleep/wake times were highly variable on the Windows XP and Vista operating systems. This is especially true if you put the system to sleep right after waking it up, booting, resuming, or any other complex task; in those cases, it might take over a minute to go back to sleep. Ouch! It may also take over 20 seconds for Windows to go to sleep if you have a bunch of applications open - that's about six times as long to put your laptop to sleep, which can be very annoying if you just want to pack up and go. You might think that you can just put your system to sleep and throw it in its bag and walk off, but sometimes a glitch will prevent a laptop from going to sleep so we like to make sure our laptops actually shut off before putting them in an enclosed space. We have heard of a few instances where a laptop has apparently overheated because it didn't go to sleep and a user put it in a laptop bag, resulting in a dead system. That's some food for thought anyway. Ubuntu and Windows 7 did a lot better, with the worst-case result being around twice as long as the best time.
Sleep times are a tie between Ubuntu, XP, and 7 at just under 5 seconds. Vista took noticeably longer even in our best-case result, trailing XP by 59%. Wake times mix things up quite a bit and show a lot less variation between runs, with Ubuntu falling to dead last and Vista and 7 essentially tied at ~3.5 seconds. XP takes 46% longer than Vista to wake and Ubuntu is 55% longer than XP (over twice as long as Win7). Again, though, variability in sleep times makes us prefer Windows 7 and Ubuntu followed by XP, with Vista being the worst at consistently going to sleep without a large delay.
106 Comments
View All Comments
JarredWalton - Tuesday, September 22, 2009 - link
I don't think it works quite like that. If you set it to 0%, I believe that's the minimum CPU speed (i.e. 5.25 x 200MHz on the NV52 and 6 x 200MHz on the NV58), while the higher percentage may try to target a maximum speed. 100% would be the normal CPU speed, but would 50% be half-way between minimum and maximum?I'd have to investigate more, but I do remember testing with CPU-Z and seeing CPU clocks go well above the 50% mark. I think at best it's approximate, as you suggest, and how accurate it is likely varies greatly with the CPU - and even BIOS options.
trochevs - Tuesday, September 22, 2009 - link
Looking the starting times (startup and resume) I have the feeling that Ubuntu has some kind of problem on your hardware. I have quite bit experience with Dell and Lenovo and Ubuntu 9.04 is always faster to boot compare to any Windows. It is not only my experience, but I am doing test on one senior citizen and one teenager. They both agree with my observation. You should press Alt-F1 during the boot and check for any errors. Gateways could have some additional peace of hardware that does not work correctly under Ubuntu and the kernel has to wait to time-out.In regards out of the box experience you should get hold of the system that is optimized for Linux (Ubuntu) just like the Gateway is optimized for Windows. www.system76.com or Dell http://www.dell.com/content/topics/segtopic.aspx/u...">http://www.dell.com/content/topics/segt...s=19&... would be good start. Then you don't have to fool with drivers. I would love to see how much optimization has been done by System76 and Dell.
ekul - Tuesday, September 22, 2009 - link
It's true; ubuntu 9.04 boots very quickly and 9.10 will be even better. On my netbook and my desktop 9.04 boots in a lot less then 30 seconds. I agree something isn't quite right with the boot procedure.The other thing to keep in mind for linux boot times is once the desktop is displayed the system is fully up. No background loading, no delayed startups. On Vista and 7 I find after the desktop appears it will take another 30-45 seconds before the HD is done reading and the system is responsive
oyabun - Tuesday, September 22, 2009 - link
Thank you very much for an enlightening article. I am one of the people who indeed care for battery life!Regarding your testing methodology where you drain each battery over and over again, wouldn't it be more efficient to take the battery out of the equation (and physically remove it) completely? Just measure the Wh consumed by the power brick during 30 or 60 minute runs and extrapolate to the capacity of the battery. That would greatly reduce your testing times.
You should of course measure at the DC end of the transformer, otherwise you should factor in it's efficiency.
You could even calibrate the whole procedure with a single battery powered run. It certainly beats what you are currently subjecting yourself to! :-)
Keep up the good work!
JarredWalton - Tuesday, September 22, 2009 - link
My experience is that laptops typically switch to different power states on AC vs. DC power, even if you have all the settings the same. It's possible to estimate battery life, but I do like to do "real world" testing where possible. Anyway, it's not a bad idea and I may do a follow up article at some point looking at just the power numbers. Taking the power transformer efficiency out of the equation isn't something I'm equipped to do right now, unfortunately. I can measure power at the outlet... and that's it. And it's only accurate to ~1W there so I'd need a better device than my current Kill-A-Watt.BTW, have you ever stared at a small Kill-A-Watt display while running tests? Frankly, running battery tests where I can walk away and collect the results later is less painful all around! :)
oyabun - Wednesday, September 23, 2009 - link
Of course a Kill-A-Watt won't cut it! You need a datalogger on a separate PC and a power gauge, logging the total energy consumption over any period of time unattended. With such a setup you would be able to measure from the DC side by splicing the wires leading from the transformer to the notebook. And, naturally, a datalogger support more than one gauge, so you could measure in parallel.I understand what you mean when you say that power profiles behave differently under AC. It is possible though your (and mine) experience is based on Windows XP. Perhaps Windows 7 are more consistent.
oyabun - Wednesday, September 23, 2009 - link
I didn't see the comments by Kibbles before. Our posts convey the smae message!And I concur, the power supply testing team appears to have the tools for the job.
strikeback03 - Wednesday, September 23, 2009 - link
Yeah, IIRC they're also in Europe, while Jarred is in Seattle or somewhere out west.JarredWalton - Wednesday, September 23, 2009 - link
There's more to it than that, but suffice it to say I don't have power testing equipment and it's not high on my list of priorities right now. Rough estimates are sufficient on the power side of the equation, since you get whatever power brick the laptop comes with. It's not like you can upgrade to a more efficient power brick with a Dell laptop.Kibbles - Tuesday, September 22, 2009 - link
"maybe borrow it from the powersupply setting team"I meant "powersupply testing team".
Also for the convenience of being able to walk away. As long as you get a voltmeter with logging capability, you can leave it to do it's thing and just pull up the logs.