Magento 2.1.18 is the final 2.1.x release. After June 2019, Magento 2.1.x will no longer receive security patches, quality fixes, or documentation updates.
To maintain your site's performance, security, and PCI compliance, upgrade to the latest version of Magento.

Test suite in the Functional Testing Framework

Overview

In the Functional Testing Framework (FTF), you can run a group of tests, which is called test suite. A test suite is a collection of tests, grouped by a specified rule, that is used to test a custom scope of functionality.

You can group the test cases, variations or constraints in any combination during the test run.

There are two rule types available:

  • allow, specifying what must be included during the test run
  • deny, specifying what must be excluded during the test run

The rules for a test case are defined in a separate .xml file. (Recommended naming: use lowercase letters and underscore as a separator). One file contains rules for one test suite. All files are stored in the <magento2_root_dir>/dev/tests/functional/testsuites/Magento/Mtf/TestSuite/InjectableTests directory by default. Only one test suite can be run at a time.

The example of the default test suite:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<?xml version="1.0"?>
<!--
/**
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/mtf/Magento/Mtf/TestRunner/etc/testRunner.xsd">
    <rule scope="testcase">
        <allow>
            <tag group="test_type" value="acceptance_test" />
        </allow>
        <deny>
            <tag group="stable" value="no" />
        </deny>
    </rule>
    <rule scope="variation">
        <allow>
            <tag group="test_type" value="acceptance_test" />
        </allow>
        <deny>
            <tag group="stable" value="no" />
        </deny>
    </rule>
</config>

This set of rules selects functional tests that accepts the following criteria:

  • WITH the tag const test_type = 'acceptance_test'
  • EXCEPT test cases with the tag const stable = 'no'

AND runs variations

  • WITH the tag test_type:acceptance_test
  • EXCEPT variations with the tag stable:no”.

Learn more details in next topics.

Configure phpunit.xml

Define the test suite to be run in the <magento2_root_dir>dev/tests/functional/phpunit.xml:

1
2

<env name = "testsuite_rule" value = <test_suite_name> />
<env name = "testsuite_rule_path" value = <test_suite_directory> />

The default test suite is <magento2_root_dir>/dev/tests/functional/testsuites/Magento/Mtf/TestSuite/InjectableTests/basic.xml.

In phpunit.xml:

1
2

<env name="testsuite_rule" value="basic" />
<env name="testsuite_rule_path" value="Magento/Mtf/TestSuite/InjectableTests" />

Run your test suite

To run a test suite enter the following commands from your terminal:

1

cd <magento2_root_dir>/dev/tests/functional

1

vendor/bin/phpunit testsuites/Magento/Mtf/TestSuite/InjectableTests.php

Rule

Each test suite can be defined by the rules that allow or deny running of a test case, a variation, or a constraint.

The only attribute of a rule node is the scope, which enables you to use the following options:

scope = "testsuite"

This scope enables you to filter functional tests using the following criteria:

Option Description Occurrence Example
<class> Apply a rule to the test case with the specified class name. multiple <class value = "Magento\Catalog\Test\TestCase\Product\CreateSimpleProductEntityTest" />
<module> Apply a rule to all test cases from the specified module. Some test cases may refer to other modules using merging functionality of variations, fixtures etc. You can restrict such reference to other modules adding the strict="1" argument. The default value is strict="0". multiple <module value = "Magento_Tax" strict="1" />
<namespace> Apply a rule to all test cases with the specified namespace. multiple <namespace value = "Magento\Catalog\Test\TestCase\Product" />

The namespace filter example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?xml version="1.0"?>
<!--
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/mtf/Magento/Mtf/TestRunner/etc/testRunner.xsd">
    <rule scope="testsuite">
        <allow>
            <namespace value="Magento\Catalog\Test\TestCase\Product" />
        </allow>
    </rule>
</config>

The module filter example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<?xml version="1.0"?>
<!--
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/mtf/Magento/Mtf/TestRunner/etc/testRunner.xsd">
    <rule scope="testsuite">
        <allow>
            <module value="Magento_Catalog" strict="1" />
            <module value="Magento_Cms" />
        </allow>
    </rule>
</config>

The class filter example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<?xml version="1.0"?>
<!--
/**
 * Copyright © 2015 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="../../../../../vendor/magento/mtf/Magento/Mtf/TestRunner/etc/testRunner.xsd">
    <rule scope="testsuite">
        <allow>
            <class value="Magento\Catalog\Test\TestCase\Product\CreateSimpleProductEntityTest" />
            <class value="Magento\ConfigurableProduct\Test\TestCase\CreateConfigurableProductEntityTest" />
        </allow>
    </rule>
</config>

scope = "testcase"

In this scope, you can group test cases using tags.

You can use group and value parameters in the test case scope. In a test case, they are provided as a constant name and its value respectively. See the following example:

  • Any tags that are used in the test case should be added to the beginning of a class definition:
1
2
3
4
5

/* tags */
const TEST_TYPE = '3rd_party_test_deprecated';
/* end tags */
  • The tag in the rule:
1
2
3
4
5
6
7

<rule scope="testcase">
    <deny>
        <tag group="test_type" value="3rd_party_test_deprecated" />
    </deny>
</rule>

A test case can contain multiple tag groups, and a group can have multiple values. For example:

1
2
3
4
5
6
7

/* tags */
const MVP = 'yes';
const DOMAIN = 'PS';
const TEST_TYPE = 'extended_acceptance_test, 3rd_party_test_deprecated';
/* end tags */

scope = "variation"

You can assign a tag node to a data set variation. This enables you to use customized sets of variations during the test run. You can allow to use in the test run only that variations which are specified with tag, or deny to use them.

You can use group and value parameters in the variation scope. In a variation, they are provided in the following format:

1
2
3
4
5

<variation ...>
    <data name="tag" xsi:type="string">group_1:value, group_2:value</data>
    <data ...>
</variation>

For example, you have a data set with the following variation:

1
2
3
4
5
6
7
8

<variation name="CreateSimpleProductEntityTestVariation3" summary="Create product with special price and custom options(fixed price)">
    <data name="tag" xsi:type="string">test_type:extended_acceptance_test</data>
    <data name="product/data/url_key" xsi:type="string">simple-product-%isolation%</data>
    ... ... ...
    <constraint name="Magento\Catalog\Test\Constraint\AssertProductSaveMessage" />
    <constraint name="Magento\Catalog\Test\Constraint\AssertProductInGrid" />
    ... ... ...
</variation>

By using the <allow> element, you can create a rule to use only the CreateSimpleProductEntityTestVariation3 variation:

1
2
3
4
5

<rule scope="variation">
    <allow>
        <tag group="test_type" value="extended_acceptance_test" />
    </allow>
</rule>

or use all variations except the CreateSimpleProductEntityTestVariation3 variation:

1
2
3
4
5

<rule scope="variation">
    <deny>
        <tag group="test_type" value="extended_acceptance_test" />
    </deny>
</rule>

scope = "constraint"

You can select constraints from the variation that will be run after a test flow using tags and rules with scope="constraint".

A tag has two parameters: group and value. In a constraint, they are provided as a constant name and its value respectively. See the following example:

  • Any tags that are used in the constraint should be added to the beginning of a class definition:
1
2
3
4
5
6
7
8
9
10

class AssertProductView extends AbstractConstraint
{
    /* tags */
    const SEVERITY = 'low';
    /* end tags */

    // ...
}
  • The rule that allows constraints with this tag only:
1
2
3
4
5

<rule scope="testcase">
    <allow>
        <tag group="severity" value="low" />
    </allow>
</rule>
  • The rule that allows all constraints except those having this tag:
1
2
3
4
5

<rule scope="testcase">
    <deny>
        <tag group="severity" value="low" />
    </deny>
</rule>
Updated
© 2023 Magento. All rights reserved.