The result of the survey will be more interesting after reading some important points about Load testing and Load testing tools that may be interesting even before beginning evaluation of particular tool.
1. There is a lot of tools for load testing from free to very expensive tools. For example LoadRunner from Mercury Interactive may cost $40,000 for only 50 concurrent users. The real price may grow up to $150,000 with reasonable number of concurrent users.
2. Is the tool you choose suitable for your application and your environment? What is the evaluation and learning period for the tool?
3. These tools support only certain protocols (be careful if you are working with Tuxedo) and platforms. SilkPerformer currently supports only Windows NT and Windows 2000.
4. Do the tools own processes, running at the same time with testing, have an influence on the applications performance? Assume about 3 MB of RAM for every concurrent user. In my case 512 MB of RAM did not allowed me to run more then 100 users.
5. Do not forget that tools have their own bugs. In our case we had a real time money transaction and the tool we tried could only send requests but didnt wait for a system response.
6. It is necessary to pay attention (and every experienced QA person understands it) to the fact that without a proper testing scenario and definition of the load points of system architecture all the results may be not precise.
7. Pay attention on the exact influence of the mirror sites and the network system on your tests. Do you load the production environment or only your own network system?
8. Scaling down the production environment for creating Load test environment is a great idea. But is there a linear correlation between the results in the testing and real environments? Pay great attention to all details during designing and creating of testing environment.
9. Nortel wishes everyone had optical pipes running everywhere, the equivalent of adding a 20 meters width channel build across Toronto. What happens in traffic when a stranger is trying to move from one location to another? Does anybody have an idea? Even Sherlock Holms will not investigate this problem.
Results of this survey show that most problems can be found using from 2 to 10 concurrent users or VU (virtual users). Increasing the number of VU until 200-250 by buying additional licenses did not discover additional problems. 250 VU may help to find problems sooner without thinking and studying the application, but this is not the right method from my point of view. All discovered problems were related to database locks, pure OO design, coding and implementation. Some researches show that it is possible to find out the maximum number of simultaneous connections by writing a code. Connections must be created for every of the following load points: Database, Web Server, Application Server, Router, Firewall, Hosting Companys Network. This method allows testing the system for scalability. While running this code, I suggest testing applications manually for performance and functionality. These allow finding bottlenecks and checking scalability of the system, without using expensive tools and with the same accuracy. To check what ports you need to load simply type netstat na | more in command prompt. Now you can use different types of loading:
1. Send garbage data to an open file sharing port 139 on windows machine.
2. Use UDP to load ports 69, 161, 7070
3. Use TCP to load Port 21 (FTP)
4. In additional you can try to load Port 23 Telnet, Port 25 SMTP, Port
666 Doom and of course Port 80 HTTP
For loading you can use free tools for ports scanning like NmapNT or Nmap or create different types of packets and be smarter making packets none of which have a fragment offset of zero or be set to incorrect values. You can try to oversize ping packets.
Let us begin developing our own drivers. Let us build testing environment that allows us not only to find maximum load for any loading point of our system but also be sure what is going on during testing and in production.
And use tools like WebWatch. See "Software We Offer" page