Amazon-EC2.md

@@ -168,3 +168,40 @@ IPs so that we won't be charged anymore. See "Stopping the Testing Framework".
   Warning: this will totally clear the instance and its associated volume. Be sure to save unfinished task before doing so. In particular save a copy of the test results to Amazon S3. See also "Maintenance of the Testing Machine" if you need to update the AMIs.
   * Right Click on an instance and select terminate.
   * Check "Release Elastic IPs", Yes Terminate
+
+Organization of test execution
+
+In this section, we find a way to optimize the test execution on EC2. We would like to test the four main rendering engines (Gecko, Presto, Webkit, Trident) and their supported output modes (HTML-CSS+STIX, HTML-CSS+TeX, SVG). For Internet Explorer, we also have 4 document compatibily modes to test. This gives 3+2+3+9=17 tasks to execute:
+
+- {Firefox} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux
+- {Opera} × {HTML-CSS+TeX, SVG}: Linux
+- {Chrome} × {HTML-CSS+STIX, HTML-CSS+TeX, SVG}: Linux (with Selenium 1 API) or Windows (Webdriver)
+- {Internet Explorer} × [ {HTML-CSS+STIX, HTML-CSS+TeX} × {Quirks, IE7, IE8, IE9} ∪ {(SVG, IE9)} ]: Windows
+
+Some points to consider:
+
+* Pricing is per instance-hour consumed for each instance, from the time an instance is launched until it is terminated. Each partial instance-hour consumed will be billed as a full hour.
+* Execution of Linux is cheaper but not possible for all browsers. Moreover the task handler is installed on Linux, so when you execute a test on a Windows instance you also need to run a Linux instance at the same time.
+* More powerful hardware is faster but more expensive.
+* One can execute several EC2 instances in parallel but that's less convenient to set up and manage.
+* Webdriver is faster, more reliable and allow parallel execution on the same machine. However, it is not supported yet for Chrome+Linux and only one instance of Internet Explorer can be executed at a time.
+* Execution on Internet Explorer is the slowest and there are many configurations to test.
+* The testing framework can not handle too many tasks at once, for otherwise it automatically stops unresponsive tasks. We have to oversee the execution and run interrupted tasks manually again.
+* In addition to test execution, you need time to set up the testing framework, save the results to AmazonS3 and stop the testing framework.
+
+Combinations experimented:
+
+- Running Linux Standard Small On-Demand Instances, the 5 Firefox&Opera tasks.
+  Interruption: It seems hard for the testing framework to handle everything. Tasks are interrupted too often.
+- Running Linux Standard Small On-Demand Instances, the 3 Firefox tasks, followed by the 2 Opera tasks.
+  Interruption: Opera tasks have been interrupted twice.
+  Estimated Execution time: ~100min (Firefox) + ~45min (Opera).
+  Estimated cost: $0.240
+- Running Linux High-CPU Medium On-Demand Instances , the 5 Firefox&Opera tasks.
+  Interruption: one Opera task has been interrupted twice.
+  Estimated Execution time ~40min.
+  Estimated cost: $0.165
+- Running Linux Standard On-Demand Instances and 2 Windows High-CPU On-Demand Instances.
+  Linux (5 tasks): Firefox, Opera
+  Windows1 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+STIX}, {Chrome}×{HTML-CSS+STIX, HTML-CSS+TeX}
+  Windows2 (6 tasks): {Quirks, IE7, IE8, IE9}×{HTML-CSS+TeX},  {IE9,Chrome}×{SVG}