Conclusion: Round 3?
After looking at the initial Internet battery life results from both tests, one thing is abundantly clear: using live websites with variable content isn't the best way of doing things. One of our original test websites appears to have become less strenuous during the past month or so (perhaps one less Flash ad is showing), and there's always a risk of radically different results if/when a site undergoes a redesign. As much as it pains us to say this, we have to consider all our Internet battery life results suspect at this point. Worst-case, they may be off by as much as 10%.
Going forward, we are working to create some test webpages where the content will be strictly controlled - i.e., identical between test runs. We expect there will always be some slight variation between test runs, but the present variation of up to 10% in some cases is far too high. We could run each test half a dozen times and take the median result, or throw out the high and low scores and average the remaining results. However, running even two tests on each laptop for each test configuration can rapidly result in several weeks of testing and we certainly don't want to triple the number repetitions for each test when they can take several hours to complete.
At any rate, we have started testing with a third set of websites, and hopefully the results will remain consistent for all OSes - and web browsers. We should probably mention that the results in those browser battery life tests are also suspect at this point. We completed those tests several weeks ago, and at the time the results between test runs appeared to be relatively consistent, but we're no longer even 90% confident in those results. We will follow up once we have confirmed our latest testing procedure. Our plan is to start with Windows Vista, perform several test runs on each laptop, confirm that the results are consistent - i.e. no more than 2-3% variation between test runs for a given power profile - and then retest the other operating systems. Once we have tested Windows XP, Win7, and Ubuntu we will reinstall Vista and repeat the tests one last time to confirm that there has been no variation over the course of a couple weeks of testing (or however long it takes). Then we'll think about looking at more browser battery life tests.
Unlike the Internet battery life results, we can discuss DVD battery life results with relative confidence. We use the same DVD, and variations between test runs have always been consistent. We plug in a set of headphones and use Windows Media Player for the DVD playback test. Windows 7 was clearly the winner in this particular test on the Gateway NV52, beating Windows XP by around 6% and Windows Vista by 22%. Again, however, changing to a different platform muddies the waters. On the Gateway NV58, Windows Vista is clearly in last place; XP offers about 21% more battery life and Win7 offers 18% better battery life. That said, this time Windows XP beats Windows 7 by around 3%.
We also said that we weren't going to focus on AMD versus Intel, since we've already looked at that aspect of battery life on these laptops. However, there's no getting away from the fact that the Intel platform offers substantially more battery life. Over 30% more time for each battery test means we only get three or four runs per day compared to four to six runs. (Note that it takes another 90 minutes or so between tests to recharge the battery, and we're not always around to immediately plug the laptops in at the completion of a test.) That means it takes the NV58 anywhere from 4.5 to 6 hours per test compared to 3.5 to 5 hours for the NV52 - and we do need to sleep at some point.
The average increase in battery life over all of the tests we performed so far is 33%, and that's taking into account the fact that Ubuntu closes the gap quite a bit between the two platforms. Throw out that Ubuntu result (only 13% in Intel's favor) and the average battery life lead increases to 37%. Why should we throw out Ubuntu? If you took the time to read this article, you already know that Ubuntu is consistently the worst battery life of the tested operating systems. As much as people like to complain about Windows, manufactures have worked a lot more on optimizing battery life performance for Microsoft's OSes. Then again, as we repeatedly mention in laptop reviews, Apple's OS X is in a league of its own when it comes to battery life. Not surprisingly, having full control over your operating system and hardware can give you a real advantage when it comes to laptop mobility.
The final topic to address is OS performance. Again, we have to pretty much throw Ubuntu out of the running. These are by no means high-performance laptops, but surfing the Internet using Firefox on Ubuntu makes you think you're running an Intel Atom netbook instead of an entry-level notebook. By no means are we discounting Linux in general, and it's still very difficult to beat the price (free); however, we think the vast majority of users will be more than happy to pay the cost for a Windows operating system. Looking at performance on the Windows OSes, once again there are some interesting trends to note. In PCMark05, Windows XP consistently scores higher in every individual test than Vista and Win7; the exception to this statement is the 2D Transparency test, which makes it look like Windows XP is old and outdated. Generally speaking, Windows XP just feels a little snappier than Vista on these laptops, probably due in part to the fact that it has a smaller memory footprint. On the other hand, Windows 7 is the clear victor in general system performance. We would love to have PCMark Vantage results from Windows XP, but unfortunately that's not going to happen. As it stands, Windows 7 outperformed Windows Vista by 15-20% in PCMark Vantage; the individual multitasking test results in PCMark05 also favor Windows 7 over Vista by an average of around 5%.
Without running a reasonable number of game benchmarks, we are not prepared to make any statements about the 3D graphics performance of any of the Windows operating systems. The various 3DMark tests show little to no difference on the NV52, but the NV58 shows differences of 3-19% in the earlier versions and a whopping 84% difference between Vista and Win7. Considering that these notebooks are anything but fast when it comes to gaming performance - IGPs still don't perform very well compared to discrete solutions - and the variations we see may simply come down to driver optimizations. We will leave any firm conclusions about differences in overall Windows performance among the various OSes to the desktop people for the time being. As a laptop OS, we would definitely take Win7 over Vista, but despite UI improvements we still find Windows XP to be more than adequate for most users. Secure? No, but still adequate if you know what you're doing. :-)
At this point, you're probably wanting more information, and admittedly we've only scratched the surface. How do other browsers fare on other operating systems? What about a better Linux distro than Ubuntu? What happens if we use FlashBlock - or a similar add-on - with these other browsers? Can we improve battery life by using a different media player or codec? What about Mac OS X, including differences between Snow Leopard and Leopard - and Tiger, Panther, and maybe even Jaguar if we want to go nuts? Give us time, because if there's one thing we know for sure it's that conducting battery life tests can take a while to complete.
To Be Continued….
Facebook
Twitter
RSS
106 Comments
View All Comments
nortexoid - Wednesday, September 23, 2009 - link
I'd like to see a test done using "regular" office apps (openoffice, acrobat reader, etc.) and NO internet browsing. (Yes, that's how I use my computer off the plug usually.) It would weed out the performance usage hit taken by Ubuntu when flash is running.It might also be interesting to do a test with tweaked systems, e.g. by tuning Jaunty with PowerTop or similar apps. How does each OS perform when FULLY optimized for battery life (without sacrificing features or much performance, of course)?
7Enigma - Wednesday, September 23, 2009 - link
I know there are users (Jarred apparently you are one of them) that run a system without antivirus/spyware, but you are (or at least should be) in the minority. Linux distro's apparently can get away without it, but on ANY Windows box it's a MUST HAVE. Also disabling those other services (while good at reducing variables) again undermines the system's protection and comparisons to a general usage scenario.In your article I do not think you mentioned what you tweaked (but I could have overlooked this). From my initial impression when reading the article you took both OS' as they were default installed and then tested from there. I think you skewed the results badly in the favor of the Windows platforms by doing this, and I say this as a Windows-only user (never used Linux in any flavor). The first thing I do after a fresh install (still on Vista) is turn off all of those programs you did, along with a host of other services/eye candy using BlackViper's Vista Tweaks. This significantly speeds up the OS in all aspects but can hardly be considered most users configs.
At first reading these comments I was firmly on the "sour grapes" to all of the Linux users complaining about what distro was used, or why X wasn't tweaked by going to a website and reading a tech article, but now I kind of agree with them. Your experience with Windows and lack of experience with Ubuntu had you setting up one for failure before the first test was even run.
Either you test both OS' as they are default installed (driver incompatibilities aside), or you need to have a Linux semi-guru set up your Ubuntu box. My recommendation is the former, as the latter has so many variables it's probably not worth testing in the first place.
Aside from the Ubuntu portion I enjoyed the article. It was very interesting to see how the different power profiles jockeyed for position. I would like to second an earlier commment that asked for idle time to shutdown numbers. Let's be honest, most of us are not going to surf continuously from 100% to dead battery. Rather we are more likely to use the laptop for a bit and then walk away, and come back later. It also will give a good indication of those background tasks impact on battery life (if you don't disable them in your config). I have a sneaky suspicion XP might look very good as it seems to be quite a bit more bare-bones than Vista/7.
Thanks again.
code65536 - Tuesday, September 22, 2009 - link
An Intel engineer explained it some years ago in a blog posting: XP's max battery basically throttles the CPU's frequency down all the time. For example, on my Core2Duo, it'd mean that the CPU will operate at 800MHz all the time, even when it's busy. On the other hand, if Windows gives Speed Step a free hand and lets Speed Step determine the speed, then the CPU will operate at 2GHz when busy, and slow to 800MHz only when it's idling. According to this Intel engineer, it makes no sense to have it always throttled to 800MHz, because that means that tasks will take longer to finish, and the CPU will spend less time idling (which is when the CPU *really* saves power), and he labeled the max battery mode as the "what the hell are you doing?" mode.Drizzt321 - Tuesday, September 22, 2009 - link
I completely understand your complaints on the length of time required to run the test, plus recharge the battery. Have you considered buying a 2nd or 3rd battery and finding an external charger? Or would that kill the results of the test because the battery would be different?PepperPot2 - Tuesday, September 22, 2009 - link
I fail to believe you included the default grub time in the boot as this to do so would be unrepresentative of real use. Who would sit there and wait for it to time out rather than just press the enter key triggering the boot process? Anyway a default install of ubuntu (where it is the only OS on the machine) doesn't show the grub menu, only a 2 second alert to allow you to bring it up if you want. I then immediately boots.My experience with ubuntu is totally contrary to the conclusions you just posted about speed. We've have ~20 installs of Ubuntu 9.04 at work on old machines (7 yrs old) to a machine I built 1 month ago. In all instances the staff find it it is smoother and quicker to use than the dual boot copy of vista or XP.
The boot time on XP is clear nonsense, when ubuntu boots to the desktop you can use it almost immediately. XP is typically totally unusable for ~30 seconds while it loads more crap in the background. Vista is worse.
jasperjones - Tuesday, September 22, 2009 - link
That's true, XP and Vista are pretty unresponsive for a minute or so after boot on your average entry-level laptop. OTOH, Ubuntu is snappy once you see the GNOME desktop. 9.04 is called jaunty ducy ;)However, the same can be said of Win 7. It keeps loading stuff after you see the desktop, but that doesn't make the system feel sluggish--not at all
JarredWalton - Tuesday, September 22, 2009 - link
The Windows boot times are stopped when I see the default system try icons appear (WiFi connectivity, volume, etc.) so the system is pretty much usable at that point. As for GRUB, it's a 2 second timeout, which I could bypass by pressing enter twice, but that still leaves around 8 seconds from the POST to the point where it looks like Ubuntu is actually loading. What's going on during that time? If we subtract POST times, then Ubuntu would look even worse. I can see about tweaking GRUB to skip straight to loading the OS and see if that helps as well... when I get time.jasperjones - Tuesday, September 22, 2009 - link
FYI--one can simply configure GRUB to not wait for user input by editing /boot/grub/menu.lst and changing the second or third line from "timeout x" to "timeout 0"But I generally completely understand the argument not to change too many things from the default installation of Ubuntu, as it's doubtful that the average user would do it.
maveric7911 - Tuesday, September 22, 2009 - link
I would love to see a properly installed and setup distribution of linux in this review (not bashing love that you included linux at all). As others stated above Archlinux would be a pretty good choice. All packages come optimized and things like native 64bit flash and other native 64 bit applications are apart of the stable repository "no adding repos necessary". Also as mentioned, not using the hardware accelerated driver of ati or nvidia will take a big hit on power right away. I'm always available to assist with any linux testing and/or questions.HP
Sr Linux Engineer
andrewaggb - Tuesday, September 22, 2009 - link
What I would like to see is Jared run all the tests on his version of Ubuntu, then let a linux expert have some ssh love to tweak it all up, and then let Jared run the tests again and see how much difference he finds.