Reporting with the Functional Testing Framework

The Functional Testing Framework (FTF) provides a reporting tool, which logs failures or any other information for you during test run.

The following image demonstrates example of a general flow.

The event manager is a core component which:

• dispatches events
• gets a list of observers
• notifies observers depending on read configuration and preset

Event manager

The event manager is defined in the \Magento\Mtf\System\Event\EventManager class that:

• is an entry point to the event management system
• fetches configuration and observers
• handles events and passes them to observers
• notifies selected observers according to an event tags array and configuration

phpunit.xml configuration

In <magento_2_root_dir>/dev/tests/functional/phpunit.xml, you can set a preset to use and a directory to store reports.

Set a preset

Set a preset, which is a list of dispatched events and an observers to handle them:

1
2
3
<php>
    <env name="events_preset" value="<preset_name>" />
</php>

Replace <preset_name> with a name of preset that you want to use.

Set a reporting directory

Set the value of <env name="log_directory"> to the directory in which to store your reports.

1
2
3
<php>
    <env name="log_directory" value="<your_directory_path>" />
</php>

The default path is <magento_2_root_dir>/dev/tests/functional/var/log.

events.xml configuration

An event preset specifies observers and dispatched events handled by them. <magento_2_root_dir>/dev/tests/functional/etc/events.xml contains a list of event presets.

Format of a preset:

1
2
3
4
5
6
7
8
9
<preset name="...">
    <observer class="Magento\Mtf\System\Observer\...">
        <tag name="..." />
        <tag name="..." />
    </observer>
    <observer class="other\observer\class\path\...">
        <tag name="..." />
    </observer>
</preset>

Example::

1
2
3
4
5
6
7
8
9
10
11
12
13
<preset name="custom">
    <observer class="Magento\Mtf\System\Observer\SourceCode">
        <tag name="exception" />
        <tag name="failure" />
    </observer>
    <observer class="Magento\Mtf\System\Observer\Screenshot">
        <tag name="exception" />
        <tag name="failure" />
    </observer>
    <observer class="Magento\Mtf\System\Observer\CurlResponse">
        <tag name="curl_failed" />
    </observer>
</preset>

Explanation of the example:

Each time the "exception" and "failure" events are dispatched, HTML code and screen capture of the current web page are logged (see Observers section for details). If the "curl_failed" event is dispatched, a corresponding cURL response is saved in an HTML file in the magento/<module>/<test_case>/<variation>/curl-response directory inside the report directory.

Initially, event presets are defined in the FTF in <magento_2_root_dir>/dev/tests/functional/vendor/magento/mtf/etc/events.xml (open the events.xml on GitHub repository). It is not recommended to edit this file. You can extend the initial list or add new presets in <magento_2_root_dir>/dev/tests/functional/etc/events.xml. All changes are merged automatically.

Observers

An observer is a PHP class which defines actions under Magento instance, browser, test run, and so on.

The list of ready-to-use observers is the following:

Tags

A tag contains name of an event. When you want to process any event by a particular observer, you need to:

1. Dispatch the event.
2. Add a tag with name of the event to required observer in corresponding event preset.

In terms of XML, it is represented as an element <tag /> in events.xml. <tag /> is a child element of an <observer> element. See the following example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<observer class="Magento\Mtf\System\Observer\ClientError">
    <tag name="page_changed" />
    <tag name="exception" />
</observer>
<observer class="Magento\Mtf\System\Observer\Log">
    <tag name="exception" />
    <tag name="failure" />
</observer>
<observer class="Magento\Mtf\System\Observer\SourceCode">
    <tag name="exception" />
    <tag name="failure" />
</observer>
<observer class="Magento\Mtf\System\Observer\Screenshot">
    <tag name="exception" />
    <tag name="failure" />
</observer>
<observer class="Magento\Mtf\System\Observer\CurlResponse">
    <tag name="curl_failed" />
</observer>

As you can see, a tag contains one required attribute name, where a name of event must be assigned.

Event dispatching

A method that is used to dispatch events is defined in \Magento\Mtf\System\Event\EventManagerInterface. The FTF uses its default implementation \Magento\Mtf\System\Event\EventManager::dispatchEvent().

1
$this->eventManager->dispatchEvent(['your_event_tag'], [$your_input_parameters]);

It has two arguments:

• Array of event tags. Event tags specify the name of event that is dispatched. It is used as a tag in event preset.
• Input parameters. The parameters used by observers as input parameters. For example, a cURL response.

Example of use:

1
2
3
4
5
6
<?php

if (!strpos($response, 'data-ui-id="messages-message-success"')) {
    $this->_eventManager->dispatchEvent(['curl_failed'], [$response]);
    throw new \Exception('Product creation by curl handler was not successful!');
}

Examples

The following examples explain how to use the reporting tool on practice.

• Create a preset
• Edit a preset
• Create and apply a custom observer

Create a preset

The following example shows how to add a custom preset.

Task: Create a preset that logs only a web page HTML code and its screenshot when a test run is failed.

Reports:

• HTML code
• screenshots

What is needed:

• observers:
  • \Magento\Mtf\System\Observer\SourceCode
  • \Magento\Mtf\System\Observer\Screenshot
• events:
  • failure

Solution:

Step 1. Dispatch the failure event in the test case. This event already exists and is added to the code where the FTF processes the test failure.

Step 2. Open <magento_2_root_dir>/dev/tests/functional/etc/events.xml.

Step 3. Add a preset with required observers and tags.

1
2
3
4
5
6
7
8
<preset name="custom">
    <observer class="Magento\Mtf\System\Observer\SourceCode">
        <tag name="failure" />
    </observer>
    <observer class="Magento\Mtf\System\Observer\Screenshot">
        <tag name="failure" />
    </observer>
</preset>

Edit a preset

The following example shows how to edit the base preset.

Task: Take a screenshot before click, after click, and when a value is set.

Reports:

• screenshots

What is needed:

• preset:
  • base
• observers:
  • \Magento\Mtf\System\Observer\Screenshot
• events:
  • click_before
  • click_after
  • set_value

The base preset is stored in the FTF <magento2>/dev/tests/functional/vendor/magento/mtf/etc/events.xml. To add or change any setting, edit <magento2>/dev/tests/functional/etc/events.xml, which is merged with the one in the FTF.

Solution:

Step 1. Dispatch the events click_before, click_after and set_value in your code where applicable.

Step 2. In the base preset, add required observer and event tags.

1
2
3
4
5
6
7
8
9
<preset name="base">
    ...
    <observer class="Magento\Mtf\System\Observer\Screenshot">
        <tag name="click_before" />
        <tag name="click_after" />
        <tag name="set_value" />
    </observer>
    ...
</preset>

Create and apply a custom observer

You can create your own observer using existing examples.

General implementation rules:

• An observer must implement ObserverInterface.
• Put the class in <magento_2_root_dir>/dev/tests/functional/lib/Magento/Mtf/System/Observer.

The following example shows how to use a custom observer in the example with the \Magento\Mtf\System\Observer\WebapiResponse observer.

Task: Create observer that logs WebAPI responses containing exceptions. Use the observer when a fixture is saved.

Reports:

• Log in JSON

What is needed:

• preset:
  • base
• observer:
  • \Magento\Mtf\System\Observer\WebapiResponse
• event:
  • webapi_failed

Solution:

Step 1. Create an observer class \Magento\Mtf\System\Observer\WebapiResponse that stores incoming events in JSON files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
<?php
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */

namespace Magento\Mtf\System\Observer;

use Magento\Mtf\System\Event\Event;

/**
 * Observer for obtaining response of webAPI  handler.
 */
class WebapiResponse extends AbstractObserver
{
    /**
     * File name of response source.
     */
    const FILE_NAME = 'webapi_response.log';

    /**
     * Collect response source artifact to storage.
     *
     * @param Event $event
     * @return void
     */
    public function process(Event $event)
    {
        $directory = $this->createDestinationDirectory('webapi-response');
        $this->logger->log(
            json_encode($event->getSubjects()[0]),
            $directory . '/' . $event->getFileIdentifier() . '.json'
        );
    }
}

Step 2. Dispatch an event webapi_failed in the \Magento\Tax\Test\Handler\TaxRule\WebApi::persist() handler for failed responses.

1
2
3
4
5
6
7
8
9
10
11
12
public function persist(FixtureInterface $fixture = null)
{
    // Implementation of the method
    // ...
    if (empty($response['id'])) {
        $this->eventManager->dispatchEvent(['webapi_failed'], [$response]);
        throw new \Exception('Tax rule creation by Web API handler was not successful!');
     }

    return ['id' => $response['id']];
}

Step 3. Add the observer and the tag to the base preset in events.xml.

In <magento_2_root_dir>/dev/tests/functional/etc/events.xml, add to a preset <preset name="base"> an observer <observer class="Magento\Mtf\System\Observer\WebapiResponse"> with a tag <tag name="webapi_failed" />:

1
2
3
4
5
6
7
<preset name="base">
...
    <observer class="Magento\Mtf\System\Observer\WebapiResponse">
        <tag name="webapi_failed" />
    </observer>
...
<preset />