Reporting
The Magento Functional Testing Framework provides two types of reporting:
- Inline reporting that you can view in the terminal as you run
mftf
orcodecept
CLI commands. - HTML reports that you can view using the Allure Framework after a test run completes.
When you run a test, MFTF copies all reporting artifacts to the dev/tests/acceptance/tests/_output
subdirectory in the Magento root directory.
The directory contains:
allure-results/
that is a directory generated and served by the Allure Framework.failed
that is a text file containing relative paths to failed tests after the last test run. The paths are relative todev/tests/acceptance/
..html
and.png
files that are screenshots of fails in HTML and PNG formats. To cleanup the_output/
directory, remove them manually.
The mftf
tool logs output continuously to the dev/tests/acceptance/mftf.log
file.
Command line
MFTF reports about its progress during test run when you run the mftf
CLI tool with run:test
or run:group
commands.
The report can contain three main parts:
- Pre-run checks:
- Environment check, such as PHP warnings, etc.
- XML test validation like deprecation warnings such as missing required components in XML tests.
- Codeception report which is the progress report for each test.
- Total results of the test run such as number of tests, assertions, and failures.
To manage the level of verbosity, use -v
or --verbose
flag in the mftf
commands.
To enable verbosity using the codecept
commands, refer to the Codeception Console Commands.
The following sections demonstrate an example interpretation of a complete log separated into chunks.
Pre-run check report
First, MFTF returns DEPRECATION
warnings alerting you that required test components are missing in XML test definitions.
1
2
3
4
DEPRECATION: Test AdminFilteringCategoryProductsUsingScopeSelectorTest is missing required annotations.{"testName":"AdminFilteringCategoryProductsUsingScopeSelectorTest","missingAnnotations":"stories"}
DEPRECATION: Test AdminAbleToShipPartiallyInvoicedItemsTest is missing required annotations.{"testName":"AdminAbleToShipPartiallyInvoicedItemsTest","missingAnnotations":"stories"}
DEPRECATION: Test AdminRemoveProductWeeeAttributeOptionTest is missing required annotations.{"testName":"AdminRemoveProductWeeeAttributeOptionTest","missingAnnotations":"stories"}
Generate Tests Command Run
Generate Tests Command Run
indicates that test generation is finished and tests are able to be executed.
Test execution report
A test execution report is generated by Codeception and includes configuration information, scenario execution steps, and PASSED/FAIL verdict after each test completion.
General information
The general information can be useful for MFTF contributors, but can be ignored by a test writer.
Let’s consider the general part of the following test execution report:
1
2
3
4
5
6
==== Redirecting to Composer-installed version in vendor/codeception ====
Codeception PHP Testing Framework v2.3.9
Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors.
Magento\FunctionalTestingFramework.functional Tests (2) ------------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTes
After the test generation command (mentioned in the previous section), MFTF delegates control to the vendor/codeception
tool, which is the Codeception PHP Testing Framework
of version 2.3.9
that uses PHPUnit
of version 6.5.13
.
The tool runs 2 Tests
using the configuration defined in the functional
suite under the Magento\FunctionalTestingFramework
namespace.
The corresponding configuration file is acceptance/tests/functional.suite.yml
.
It enables Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, ...
Passed tests
The next chunk of the log reports about test execution of the first test:
1
2
3
4
5
6
7
8
9
10
11
12
AdminLoginTestCest: Admin login test
Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest
Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest
Scenario --
I am on page "/admin/admin"
I fill field "#username","admin"
I fill field "#login","123123q"
I click ".actions .action-primary"
I wait for page load 30
I close admin notification
I see in current url "/admin/admin"
PASSED
The running test is AdminLoginTestCest
, which is Admin login test
(this text is generated from the test name but with the Cest
part excluded).
Its test signature is Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest
that matches a className:methodName
format using namespaces.
A path to the corresponding Test
is tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest
(relative to the acceptance/
directory).
Scenario
lists the tests steps as they run during test execution, ending with the successful test verdict PASSED
.
It means that all test steps were processed as expected.
Failed tests
The second test fails with the following report:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test
Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest
Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest
Scenario --
I magento cli "config:set admin/security/use_form_key 1"
Value was saved.
I magento cli "cache:clean config full_page"
Cleaned cache types:
config
full_page
I am on page "/admin/admin"
I wait for page load
I fill field "#username","admin"
I fill field "#login","123123q"
I click ".actions .action-primary"
I wait for page load 30
I close admin notification
I click "//li[@id='menu-magento-backend-stores']"
I wait for loading mask to disappear
I click "#nav li[data-ui-id='menu-magento-config-system-config']"
I wait for page load
I see current url matches "~\/admin\/system_config\/~"
I see "#something"
I save screenshot
FAIL
I magento cli "config:set admin/security/use_form_key 0"
Value was saved.
I magento cli "cache:clean config full_page"
Cleaned cache types:
config
full_page
I am on page "/admin/admin/auth/logout/"
--------------------------------------------------------------------------------
The general test details and scenario has the same format as in the Passed test.
The interesting part starts near the FAIL
line.
1
2
3
I see "#something"
I save screenshot
FAIL
When a test step fails, MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs.
I save screenshot
follows the failing test step I see "#something"
in our case.
A screenshot of the fail goes at the acceptance/tests/_output
directory in both PNG and HTML formats:
Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html
Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png
The file name encodes:
Magento
namespace- with the
AcceptanceTest
test type - generated as a part of the
_default
suite - defined at the
Magento_Backend
module - implemented in the
AdminMenuNavigationWithSecretKeysTestCest
PHP class - with the
AdminMenuNavigationWithSecretKeysTest
test name - and execution status
fail
Actions after FAIL
are run as a part of the after
hook of the test.
Test result report
After MFTF completed test execution, it generates a general report about test results along with detailed information about each fail.
1
2
3
4
5
6
7
--------------------------------------------------------------------------------
DEPRECATION: Calling the "Symfony\Component\BrowserKit\Client::getInternalResponse()" method before the "request()" one is deprecated since Symfony 4.1 and will throw an exception in 5.0. /Users/.../magento2ce/vendor/symfony/browser-kit/Client.php:208
Time: 52.43 seconds, Memory: 16.00MB
There was 1 failure:
---------
First you see warnings and deprecations.
The DEPRECATION
here is thrown by an MFTF dependency (Symfony) that is out of the scope for test writers and should be considered by MFTF contributors.
If you encounter this type of reporting, report an issue.
Then, MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM.
And, finally, that there was 1 failure
.
Next, the report provides details about the test failure.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
---------
1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test
Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest
Step See "#something"
Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../
--> DASHBOARD
SALES
CATALOG
CUSTOMERS
MARKETING
CONTENT
REPORTS
STORES
SYSTEM
FIND PARTNERS & EXTENSIONS
Configuration
admin
1
Store View: Default Config
What is this?
Save Config
Country Options
State Options
Locale Options
Store Information
Store Name
Store Phone Number
Store Hours of Operation
Countr
[Content too long to display. See complete response in '/Users/dmytroshevtsov/Projects/vagrant/vagrant-magento/magento2ce/dev/tests/acceptance/tests/_output/' directory]
--> contains "#something".
Scenario Steps:
23. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:54
22. // Cleaned cache types:
config
full_page
21. $I->magentoCLI("cache:clean config full_page") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52
20. // Value was saved.
19. $I->magentoCLI("config:set admin/security/use_form_key 0") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:50
18. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:63
-
1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test
- the failed Codeception test is AdminMenuNavigationWithSecretKeysTestCest. It references to the PHP class that implemented the failed test. -
Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest
- the test is implemented in the AdminMenuNavigationWithSecretKeysTest test method of the tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php file under<magento root>/dev/tests/acceptance/
. It matches the corresponding test defined in XML that is AdminMenuNavigationWithSecretKeysTest defined in<test name="AdminMenuNavigationWithSecretKeysTest">...</test>
-
Step See "#something"
- the failing test step is the see action with the #something selector. It would correspond the<see selector="#something" ... />
test step in the XML defined tests. -
Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../
- the fail occurred on the web page<MAGENTO_BASE_URL>/admin/admin/system_config/index/key/678b7ba922c.../
.
1
2
3
--> ...
[Content too long to display. See complete response in '/../../magento2/dev/tests/acceptance/tests/_output/' directory]
--> contains "#something".
The web page is too long to be reported in the CLI, and it is stored at ‘/../../magento2/dev/tests/acceptance/tests/_output/’. Search the web page by test name AdminMenuNavigationWithSecretKeysTest. The failing test assertion is that the web page contains contains a CSS locator #something.
Finally, the report finishes with fairly self-descriptive lines.
1
2
FAILURES!
Tests: 2, Assertions: 3, Failures: 1.
MFTF encountered failures due to the last test run, that included 2 tests with 3 assertions. 1 assertion fails.
Allure
Each time you run tests, MFTF appends an XML file with results at the tests/_output/allure-results/
directory.
The official Allure Test Report documentation is well-covered, so we’ll list only the CLI commands that you would need for your day-to-day work.
To generate the HTML Allure report in a temporary folder and open the report in your default web browser:
1
allure serve dev/tests/acceptance/tests/_output/allure-results/
To generate a report to the allure-report/
at the current directory:
1
allure generate dev/tests/acceptance/tests/_output/allure-result
To generate a report to a particular directory, use the -o
option:
1
allure generate dev/tests/acceptance/tests/_output/allure-result -o dev/tests/acceptance/tests/_output/allure-report
To launch the generated report in a web browser:
1
allure open dev/tests/acceptance/tests/_output/allure-report
By default, Allure generates reports in the allure-report/
at the current directory.
For example, if you run the command without -o
flag while you are in the magento2/
directory, Allure will generate a report at the magento2/allure-report/
directory.
1
allure generate dev/tests/acceptance/tests/_output/allure-result
Example of file structure after the command run:
1
2
3
4
5
6
magento2
├── allure-report
├── app
├── bin
├── dev
├── ...
And if you run the open
command with no arguments while you are in the same directory (magento2/
):
1
allure open
Allure would attempt to open a generated report at the magento2/allure-report/
directory.’
%}
To clean up existing reports before generation (for example after getting new results), use the --clean
flag:
1
allure generate dev/tests/acceptance/tests/_output/allure-result --clean
Refer to the Reporting section for more Allure CLI details.