Sourcing Engine

The sourcing engine does many tasks to help support the flow of orders in the Order Management System (OMS). It searches for the best possible sourcing option and configures the best possible sourcing configuration for the next successive line in an order.

The sourcing engine:

• Supports direct or batch mode, which is configurable at the client or order level.
• Processes waves of orders for a specific merchant.
• Receives sourcing requests via the service bus.
• Fetches and understands store/aggregate/source relationships.
• Deals with order errors across the system.
• Queries orders at any time and maintains a record of order statuses as they occur in your OMS.
• Immediately prunes sourcing configurations that do not adhere to configured client restrictions (filter rules).

We retain historical sourcing information for three months.

Filters and scoring

The sourcing engine implements filter rules and scoring rules to begin the determination process for the best sourcing option(s).

See Option scoring for more information.

Scoring in the sourcing engine

The sourcing process, configured at the aggregate level, consists of the following four steps:

1. Batch sorting
2. Option generation
3. Option scoring

4. Best option selection

   

Batch sorting

The sourcing engine can process sourcing requests in direct or batch modes. This is a configurable option at the client level, or through an specific request.

Sourcing requests that are processed in direct mode are immediately sourced. Sourcing requests that are processed in batch mode are batched together with all other sourcing requests using the same mode and processed together at specific intervals (which must be configured at the client level beforehand).

An in-store pickup (ISPU) order is sourced, regardless of the configuration to speed up the preparation process of the order and customer pickup.

Batch mode allows the sourcing engine to optimize the usage of source stock across a range of requests to ensure the priority options are more likely to be sourced successfully, and to minimize overall shipments and backorders.

Batch sorting is currently based on three criteria:

• Age
• Shipping method
• Request size (line count)

To minimize splits, sort by request size first, followed by shipping method and age. To ensure orders are shipped as soon as possible, sort by age, size, and then shipping method.

These definitions are enabled and configured in your System Integrator (SI) Portal, which is not yet accessible externally. Please contact Magento Support for assistance.

Option generation

The Sourcing Engine researches the best available sourcing option, based on the client’s restrictions, by generating all viable options. Then, the Sourcing Engine explores all the possible ways to source a specific order depending on these available sources and selects the best option.

In this way, the Sourcing Engine scores each option and retains the sourcing option with the best score. Then, the result is returned to your OMS as the sourcing response along with the other 10 best scored options for reporting purposes (an admin can see the best scored options generated for each order).

Option scoring

The sourcing engine works with two different types of rules:

• Filter Rules—Strict yes/no votes on a given sourcing option provide restrictions for what a customer allows. Examples of filter rules include Max Splits and Source provides required services.

  All rules (filter and scoring alike) can be enabled and disabled by the client in their SI Portal. During its preparation phase, the sourcing engine queries the SI Portal and requests the configuration for each rule. This configuration can include rule-specific information, such as the exact limit for the maximum number of shipment filter rule. They all also contain an enabled flag, by which the client can choose to use or not use any given rule.

• Scoring Rules—Scoring rules help you evaluate the sourcing options based on customer criteria (if applicable). Each enabled rule provides a score for a given sourcing option. Scores range from 10.0 (best) to 0.0 (worst) and the set of scores across all options are scaled to this range to ensure that the best options receive 10.0 and the worst receive 0.0.

  See the Stock aggregates topic for more information about configuration.

  The Sourcing Engine scores in two phases. In the first phase, it looks at all the generated sourcing options and allows each rule to gather any meta information that needs to score. This meta information is often used to scale the scoring results to the mentioned range of 0.0 to 10.0.

  Scaling is important to ensure that rules can be fairly evaluated against each other. In most cases, a customer wishes to score options across multiple rules/criteria. For instance, while grouping shipments in a single order may be important, the customer might also wish to prioritize certain warehouses.

  These different rules are not equally important. To handle these situations, it is possible for each client to specify the importance of each rule, called weight values. These weight values are, for practical purposes, bundles—meaning you can attach a weight of 10,000,000 to one rule and a weight of 1 to another.

  This weighting system provides flexibility in controlling your sourcing decisions!

  

Scoring example

A client’s order line has a total of 10 items. When reviewing all the possible sourcing options, the sourcing engine finds that the fewest number of possible shipments is four and the most possible shipments is nine.

The client has weighted the minimize shipments rule as a 10.0, to prioritize options with the fewest number of shipments. They also score a 0.0 to options with the most number of shipments found.

Any option with four shipments receives a score of 10.0 (the highest score, since they are focusing on minimizing shipments). Any option with nine shipments receives a score of 0.0 (the lowest score, because that is the least desirable shipment situation). Options with between four and nine shipments are scaled accordingly. For instance, an option with seven shipments receives a score of 4.0.

The client also decides to prioritize a source priority distance rule based on the distance of the warehouse from the recipient. In this client’s situation, orders that require more shipments, such as eight or nine shipments, are coming from local stores. The distance rule scores options with the most shipments a 10.0, because they are coming from local warehouses, and scores options with the fewest shipments 0.0, because they are coming from remote warehouses.

But this client wants to minimize shipments over prioritizing shipments from local warehouses, so they change the weight of the minimize shipments rule to 2.0 and the weight of the source priority rule to 1.0. When computing the overall score, the sourcing engine determines the weighted average of all rule scores. The rules with higher weights will have more influence on scoring. Since the minimize shipments rule has twice the power over the source priority rule, that factors into the overall calculation.

This weighting system provides flexibility in controlling sourcing decisions. The client feels that minimizing shipments is important but the source priority is still a concern. Though they can achieve a single shipment by using the lowest priority source (warehouses far from the recipient), they prefer to split the order into multiple shipments and ship from their two highest priority warehouses (which are closer to the recipient).

Available sourcing rules

These are the available sourcing rules:

• Configurable maximum number of splits allowed for each order
• Weighted rule—Minimize splits for a single order
• Weighted rule—Prioritize sources by rankings
• Weighted rule—Prioritize sources by stock availability
• Weighted rule—Prioritize sources by distance
• Weighted rule—Preference for sources in the same shipping zone as the delivery address
• Rule Weight definition for the range of five values (Lowest, Low, Medium, High, Highest)
• Prevent splits across bundled products
• Flag that allows the engine to split the bundle after reaching a threshold of failed attempts to source the request. The bundle must still be completely sourced in order to successfully source the request.
• Prevent sourcing in case some of the lines don’t have stock, and hold back the entire order as backorder.
• Flag that allows the engine to ignore the “Prevent partial sourcing rule” after reaching a threshold of failed attempts to source the request
• Capabilities—Source lines to sources that provide required service
• Capabilities—Source lines to sources that provide required shipping method (additional paramenter is available to make standard shipping to be always assumed true)
• Filter shipping zones by defining a zone mapping and assign each source to a specific zone. Orders will be sourced only to sources that are in the same shipping zone as the defined shipping address (based on two configurable criterias state or country). If a source is not part of any defined zone it can be used as a fallback.

To learn more about sourcing and scoring see Order details.

Rules that can be overwritten at order level

Some rules can be overwritten at the order level, including sourcing decisions and bundle rules.

Force sourcing decision at the order line level

To force a sourcing decision, you can define the source at the order line level. If the source_location is defined then other available sourcing options are ignored. Then, lines are forced to the defined source.

If the stock is unavailable in the source_location, the line is back-ordered until there is available stock for this source.

Overwrite bundle rule

The currently configured bundle rules can be overwritten with a custom attribute provided at the bundle line level.

See our Bundles page for more information about bundle rules.

Overwrite order splitting rule

Configured rules determine if an order can be split or overwritten with a custom attribute provided within the magento.sales.order_management.create message.

Example

{
    "order": {
        "id": "001",
        "store": "STORE",
        "language": "en_US",
        "vat_country": "US",
        "origin": "WEB",
        "origin_date": "2019-01-09T10:44:54+01:00",
        "custom_attributes": [
                    {
                        "name": "mom_srcrr_FilterNoPartialSource",
                        "value": "SHIPTOGETHER"
                    }
                ],
        "lines": [
            {
                "id": "1234",
                "line_number": 1,
                "payment_reference": "1",
                "product_name": "Test product",
                "product_type": "PHYSICAL",
                "shipping_address_reference": "johndoepers",
                "sku": "sku-1"
            }
        ]
    }
}

Configure allocation waves

Using the OMS Admin, you can define when a batch process should be run (at a maximum once per hour), and to indicate which source will be considered active in each one of the executed batches.

This configuration is only defined in UTC.

A configuration defines the sorting used to process batched orders based on:

• Size
• Age
• Shipping method

The client can use the allocation waves in two different modes:

• Batch mode—Client sourcing in batch mode provokes all new orders in the sourcing queue and runs the sourcing logic at the defined time. For these clients, you can define which source should be considered active in a specific wave (for example, during one wave several sources you do not want them to be considered, and you want to use other available options).
• Direct mode—Clients sourcing in direct mode use the waves to process orders in the exception queue (backorders). In this case, all sources are always assumed to be active for all waves (if the configuration is overwritten at the source level it will be ignored).

You can see all the completed sourcing attempts for each order and available stock in the selected source during a wave. If the order cannot be sourced, a list of reasons is displayed.

The wave schedule is also used to process the backorder queue. Each wave checks that the stock for queued orders is available and in that case sources the backorders.

Allocation wave times can be configured either at the root level or at the aggregate level. At root level this implies that all existing sources associated to the aggregate are considered as available during each wave.

To disable a specific source for a specific time and remove the the configuration under the specific source node:

1. Selecting the source in the Scope dropdown.
2. Navigate to the configuration, Sourcing > Active Wave Times.
3. Deselect the DEFAULT input field.

4. Click the trash icon to remove the wave.

   

If root does not have any configured wave, but there are configured waves at the aggregate level, this means that none of the sources automatically inherit the setting (it is disabled). In this case, each one of the sources must define the exact same waves that exist for the aggregate level. To optimize the creation of configurations, definite it in the MetaSource so that it is inherited by all children sources.

Integration order flow

The following are example order flows illustrating the four steps previously explained.