Faster Functional Tests
Sun, Nov 16, 2014Functional tests
Functional tests are one of the tests that provide the most benefit and pay themselves off earlier than other types of tests. The Symfony documentation defines the functional test as so:
Functional tests check the integration of the different layers of an application (from the routing to the views). They are no different from unit tests as far as PHPUnit is concerned, but they have a very specific workflow:
- Make a request;
- Test the response;
- Click on a link or submit a form;
- Test the response;
- Rinse and repeat.
Too slow
One issue with functional tests is that they are slow. Since a functional test requires testing the entire stack, the performance benefits of mocking out services are not available. There are many approaches to speeding up functional tests but in this post we’ll focus on the major bottleneck: the database.
A highly recommended way of structuring your functional tests is to rebuild the database per test. By starting with a pristine database based on some dataset, you reduce the amount of issues you may run into with tests sharing state with each other. If for some reason you don’t do this, this post probably won’t apply to you as much.
A functional test for an application using Doctrine
<?php
class TestTheTesting extends WebTestCase
{
public function setUp()
{
$this->createDatabase();
$this->client = self::createClient();
$this->container = $this->client->getContainer();
$metadatas = $this->getMetadatas();
if (!empty($metadatas)) {
$tool = new \Doctrine\ORM\Tools\SchemaTool(
$this->container->get('doctrine.orm.entity_manager')
);
$tool->dropSchema($metadatas);
$tool->createSchema($metadatas);
}
}
}
Lowest hanging fruit
One quick speed up is to use SQLite for your database when running tests. In this example, we are using Doctrine as a persistence layer so switching out the database is just a simple configuration change. Overriding the doctrine configuration in the test specific config file (config_test.yml):
config_test.yml
...
doctrine:
dbal:
default_connection: default
connections:
default:
driver: pdo_sqlite
memory: true
db_name: %database_name%_test
charset: UTF8
...
In my own tests, this saw a significant speed boost from 4-5 seconds per test to about 1.
Database pooling
A slightly more involved approach with maximum environment parity, is using a database pool.
The database pool approach front-loads the majority of the work before the tests are ever ran.
Standard Approach
Database Pooling
The “fetch database from pool” is significantly faster than creating a database on the fly. Although the same amount of work is done (and more due to the pooling overhead) this new method makes the database creation step asynchronous. Since testing during development is sporadic happening in small bursts, we are able to use this fact to get a significant speed increase.
Note: My particular setup involves multiple databases, the database used in pooling for this case is being populated via a MySQL dump file. The test results above are the database managed by Doctrine
So how do we implement such a thing?
Create the pool filler
A database pool should have an interface to get a item from the pool and a way to fill the pool. Below is one implementation that is MySQL specific:
DatabasePool.php
<?php
class DatabasePool
{
public function __construct($host, $port, $user, $password)
{
$this->pdo = new \PDO(
"mysql:host=$host;port=$port",
$user,
$password
);
$this->host = $host;
$this->port = $port;
$this->user = $user;
$this->password = $password;
}
/**
* Returns the name of the database that is in a pristine condition
*
* @return string
*/
public function get()
{
$pooledDatabases = $this->getPool();
if (count($pooledDatabases) == 0) {
throw new \RuntimeException("Database pool is empty");
} else {
return array_pop($pooledDatabases);
}
}
/**
* Drops the database that was used from the pool
*/
public function drop()
{
$this->pdo->exec("DROP DATABASE {$this->get()}");
}
/**
* Fills the pool with the given number of databases
*
* @param int $size The number of pristine databases to create
*
* @return string
*/
public function fillPool($size)
{
if ($size <= 0) {
throw new \UnexpectedValueException('Pool size must be greater than 0');
}
$result = $this->getPool();
$delta = $size - count($result);
if ($delta < 0) {
foreach ($this->getPool() as $pooledDatabase) {
$this->pdo->exec("DROP DATABASE {$pooledDatabase}");
}
$delta = $size;
}
for ($i = 0; $i < $delta; $i++) {
$databaseName = 'test_'.str_replace('.', '_', microtime(true));
$this->createDatabase($databaseName);
}
$pooledDatabases = $this->getPool();
return array_pop($pooledDatabases);
}
/**
* Returns all the database names that are within the pool
*
* @return array
*/
private function getPool()
{
$sql = <<<SQL
SELECT `schema_name` FROM information_schema.schemata
WHERE `schema_name` LIKE 'test_%' ORDER BY `schema_name` DESC;
SQL;
return $this->pdo->query($sql)->fetchAll(\PDO::FETCH_COLUMN);
}
/**
* Creates a pristine database with the given name
*
* @param string $name
* @return string
*/
private function createDatabase($name)
{
// Your database creation code here
}
}
Pool management is tracked using a naming scheme of the pooled databases, in this case its test_<MICROTIMESTAMP>.
Create a command to fill your database pool
With your new DatabasePool you need some way to invoke it and have it run along side your tests in a seperate process. A Symfony command would handle that nicely:
FillPoolCommand.php
...
protected function execute(InputInterface $input, OutputInterface $output)
{
$pool = new DatabasePool(
$this->getContainer()->getParameter('database_host'),
$this->getContainer()->getParameter('database_port'),
$this->getContainer()->getParameter('database_user'),
$this->getContainer()->getParameter('database_password')
);
$pool->fillPool($input->getArgument('pool_size'));
if ($input->getOption('watch')) {
while(true) {
$pool->fillPool($input->getArgument('pool_size'));
sleep(5);
}
}
}
...
You run this command in the background while you are developing and you will always have fresh databases to work with.
Update your tests to use the database pool
Now you need to update your tests to use the newly created database pool. By using the fact that config_test.yml overrides the configuration for the test environment we are able to create a custom database connection that pulls database names from the pool. We also use the handy expression language that Symfony supports in its configuration files:
config_test.yml
...
services:
database_pool:
class: DatabasePool
arguments:
- %database_host%
- %database_port%
- %database_user%
- %database_password%
doctrine.dbal.default_connection:
class: Doctrine\DBAL\Portability\Connection
factory_class: Doctrine\DBAL\DriverManager
factory_method: getConnection
arguments:
- driver: %database_driver%
host: %database_host%
port: %database_port%
dbname: @=service('database_pool').get()
user: %database_user%
password: %database_password%
The magic lies here
dbname: @=service('database_pool').get()
The dbname for the doctrine connection is evaluated on each test run based on the the DatabasePool::get() method.
Fin
By doing these improvements I was able to shave down my test run time from 39 seconds for 10 tests down to 2.1 seconds. This only captures the amount time savings if I just stared at the screen while the tests were running. I haven’t looked into the amount of time savings from not being able to open up reddit because my tests were already done.