Methods
As mentioned above, testing the KillerNIC was interesting to say the least. Not only did we have to devise our own custom test methods, but we also we had to spread out our testing to eliminate as many variables as possible. The guys at Bigfoot probably hate us for it (kidding), but we believe that the extra time was well spent as it ensured a more scientific procedure. The variables we had to eliminate - or do our best to eliminate - were human error, even slight test variance, cable internet fluctuations, and dissimilar hardware and software configurations.

To combat these variables, we first did something that we do in most of our reviews: tested a whole bunch of times. You will see that we only tested three games with the KillerNIC, but each of these games was tested virtually non-stop for about two full days. This allowed us to do several things. The first benefit of a two-day testing time is the ability to retest several times to minimize the effect of human error and procedural variation. These problems are inherent to any experiment, and usually the only way to truly verify results is to run the test again and again to see if they are repeatable. The second benefit of this procedure is that by dedicating one day to testing the system without the card, and one day to testing the system with the card, we minimize the effect of the cable internet signal being diminished with more people using it. Cable service providers in all areas suffer from the same sort of “peak-usage” hours as all utility companies, and running each test at the same time as the corresponding test from the previous day allowed us to replicate, in really the only way possible, the usage fluctuation present.

The second thing we did was keep the same system for all tests. In their in-house tests, Bigfoot used two identical machines, one with the Killer and one without, and tested them side by side to get their performance numbers. While this technique is very strong and is probably the best way to test the KillerNIC, the possibility of performance variations caused by software problems is increased with the more systems you add to the testing equation. With one system for all testing, every single hardware and software variable can be kept constant with the exception of the ones being tested. Although this may or may not be the best way to test the KillerNIC, we hope you feel that it is at least a valid method.

More specifically, the games we tested were Counter-Strike 1.6, Counter-Strike: Source, and F.E.A.R multiplayer. We feel that while all of these games are first person shooters, they represent a fairly decent sample of the different netcode used in games over the past 8 or so years. In all of these games, testing consisted of getting FPS values over 10 minute intervals of gaming. Each of these tests was carried out 9 times, meaning an hour and a half of gaming, in order to get a solid performance number. The same test would then be done the next day with the KillerNIC card in use. Furthermore, we tested each game on two different internet servers, one from Los Angeles and one from Chicago, to provide for an almost local result and a cross-country result. This means that we completed a total of 6 hours of FPS testing for each of the three games. For testing ping times, we used a method of sampling from the in-game console screen (TAB) at 10 second intervals while standing around alone in the server. We did this for 10 minutes, attaining 60 values for latency. This test was repeated three times with and without the KillerNIC. The values achieved by this test are likely to be more constant than the other tests we ran, but they do not represent a real-world gaming scenario. To facilitate this, we also endured a 30 minute gaming session, and sampled ping times every 60 seconds. This test was also repeated three times for each game, with and without the KillerNIC.

We would like to take this opportunity to thank the guys at NuclearFallout (#Nuclearfallout on irc.gamesurge.net) for providing us with the servers we used for our testing. Their stable bandwidth and great customer support was extremely helpful during our tests.