Magento Integration Bus

The Magento Integration Bus (MIB) is an API gateway which provides a common communication layer for different services to interact with the Order Management System (OMS). The MIB provides services such as basic services discovery capabilities, an authorization and authentication mechanisms for your OMS, as well as a synchronous or asynchronous broadcast communication mechanisms.

The MIB acts as a centralized API gateway and messaging broker, which receives requests from everywhere and routes them to their correspondent services. These services are addressed using unique identifiers.

Every service registers itself in the MIB and provides basic information about itself: service unique identifier, its URL, topics it wants to subscribe to, and secret. This process helps the MIB to learn about new integrations and gives instructions about where and how messages should be delivered.

Keep in mind that all API fields are case-sensitive by default.

Transport protocol

The MIB uses JSON RPC 2.0 over HTTP as a simple and widely spread set of protocols.

There are a few exceptions to how the MIB handles a JSON RPC call:

1. Batch requests are not currently supported
2. Parameters encoded as an object, "params":{"foo":"bar"}), are supported, but "params":["bar"] is not.
3. Currently, only HTTP 1.0 and HTTP 1.1 are supported.

See https://www.jsonrpc.org/specification for more information about JSON RPC.

Communication types

The MIB was designed to provide different mechanisms for services to communicate among themselves. There are three communication types:

• Synchronous
• Asynchronous
• Broadcast

Synchronous (one-to-one)

Synchronous communication is when one service makes a call to another and then waits for a response. In this case the MIB plays the role of a proxy server, which simply forwards the request and the response between two sides.

Asynchronous (one-to-one)

Asynchronous communication is when one service makes a call to another without waiting for a response. In this case, the MIB just acknowledges receipt of the request and delivers message separately, maybe with a little delay. The MIB will also retry delivery of a message, in case of an error. If, for instance, your service delegates a call to the MIB, once the MIB took responsibility for delivering the message (acknowledging it), your service can safely assume that the message will be eventually delivered.

Broadcast (one-to-many)

Broadcast communication is when one service broadcasts a message to either none, or many, other services, depending on their interest in the message topic. Every service can provide a list of topics it wants to subscribe to in the registration call. Broadcast is similar to the asynchronous call: the MIB will simply acknowledge that broadcast is received and queued. Then, the message will be delivered and retried in case of an error.

Endpoints

The MIB provides multiple endpoints for different services and communication types. The MIB base URL resembles:

Example of a URL

https://mib.mom.magento.com/luma/

Base endpoint /

A base endpoint provides an API to interact with the MIB itself. This is the endpoint you should use for making registration and discovery calls. A list of all available methods available can be found on the #magento.service_bus.remote specifications page.

Example of a base endpoint

https://mib.mom.magento.com/luma/

Synchronous endpoint /remote/{app-id}

A synchronous endpoint proxies every call to a given service. A call to endpoint /remote/hello-app will instruct the MIB to look for the URL for a service with the ID hello-app. If such service is registered, the MIB will make a RPC call with exactly same parameters to this URL and return a response.

Example of a synchronous endpoint

https://mib.mom.magento.com/luma/remote/oms

Asynchronous endpoint /delegate/{app-id}

Asynchronous endpoints (or delegate endpoints) drop every call into a queue and immediately acknowledge the request. For example, a call to endpoint /delegate/hello-app will instruct the MIB to look for the URL for a service with the ID hello-app. If such service is registered, the MIB will drop the message into an internal queue, and reply to the original request with null, meaning the request was accepted and persisted.

Example of an asynchronous endpoint

https://mib.mom.magento.com/luma/delegate/oms

Broadcast endpoint /events

A broadcast endpoint broadcasts every RPC call received to all subscribers. For example, an RPC to the magento.service_bus.explained method will be delivered to any service subscribed to the topic which matches the method name.

Example of a broadcast endpoint

https://mib.mom.magento.com/luma/events

Date and time

OMS Specifications use a date and time encoded in an ISO-8601 datetime format.

Example of an ISO-8601 datetime format

{
    "exp": "ge",
    "args": [
        {"exp": "field", "args": [{"value": "origin_date"}]},
        {"value": "2017-01-01T00:00:00+00:00", "type": "datetime"}
    ]
}

Authorization

MIB uses OAuth2 to authorize all incoming HTTP requests. Before making any calls to the MIB you need to receive an OAuth2 token. OAuth2 is only valid for a limited amount of time before expiring, so you need to develop your service to dynamically fetch newer tokens in case a 401 error code is received.

Once you receive a token, send it in an Authorization header: Authorization: Bearer TOKEN-HERE. In the examples below, the placeholder TOKEN is used to mark where OAuth2 token should be.

Fetching token

Contact Magento Support to receive your OAuth2 Server URL, Client ID, and Client Secret.

It takes one HTTP call to receive an OAuth2 token:

curl -X POST https://auth-stg.bcn.magento.com/oauth/token \
 -F grant_type=client_credentials \
 -F client_id=CLIENT-ID \
 -F 'client_secret=CLIENT-SECRET'

Then, you will receive 2xx response with the following content:

{"access_token":"QlhjQclLI0TSS8UcHms4pCLUugq2Aris654HknyH","token_type":"Bearer","expires_in":3600}

Your OAuth2 token is in the field named access_token.

Registration

Registration is a process in which an API calls to add or update information about a service connected to the MIB. If you try to register a service with an ID that already exists in the MIB, it overrides the previously registered record.

For every attempt to register a new service, a probe is made against the registered URL using the OPTIONS HTTP verb. The server should reply with an empty body including the header X-Magento-Service-Bus: *. This process is called URL probing and should prevent the MIB from making calls to URLs which are not meant to be used as MIB endpoints.

Example of headers that the prober sends within the request

`Access-Control-Request-Headers`: Authorization,Content-type,X-Magento-Service-Bus
`Access-Control-Request-Method`: POST
`Origin`: https://api.mcom.magento.com
`User-Agent`: Service-Bus/1.0
`X-Signature`: sha1=fb0b1d1b1eaa6c08324bcd64b71fb71370610e1c

Registration is performed during the service deployment process, or periodically by some scheduled process, for example, every hour. You can also perform a registration manually, although it is preferred to automate the process.

All parameters for a registration call are described in the API specifications.

Example bash

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "warehouse-integration-example",
        "url": "https://warehouse-integration.mom/api"
    }
}'

Unregistration

Unregistration process allows you to remove unused integration. It can be performed using “unregister” command.

Unregistration will cause undelivered messages to be removed, and it won’t be possible to restore them later. Although unregistration does not guarantee that all messages will be fully removed, if you register another service with the same name, some of the messages still may be delivered within a short period of time.

Example bash

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.unregister",
    "params": {
        "id": "warehouse-integration-example"
    }
}'

Health checks

The OMS performs health checks of integrations every hour. Health checks help identify active integrations that are not working correctly and are generating unnecessary traffic.

They validate if an integration is responding within one second, which signifies that the integration is healthy. If the response takes longer, the integration is considered not healthy. In this case, after 48 hours the integration will be deactivated.

Commerce and OMS connectivity

You can see health check reports for assessing the health of the Magento Commerce (MC) and OMS integration connectivity.

The available checks include:

• Prune message job is running
• System has incoming messages
• Success rate on message communication is over a rational threshold

You can view the MC OMS connectivity report in System > System Report > New Report > OMS Connector in the MC Admin.

There are two different types of health check statuses available in the report:

• ERROR—Indication that something is not working properly; may require further investigation and/or action
• WARNING—Prompts you to do more research about a potential issue, but no immediate action is currently required

Discovery

The MIB provides an API to fetch a list of all registered services with their meta information. The response will contain the same information as was provided during registration, but secret. You can use the discovery call for debugging, to verify if your service was properly registered or to find out what services are connected to the MIB.

All parameters for the discovery calls are described in the API specifications.

Example bash

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.discover"
}'

Synchronous communication

As mentioned above, the MIB provides a proxy functionality for synchronous calls.

Assuming there is a warehouse-integration-example service registered with the same parameters as in the registration example, we can send a synchronous call.

Example bash

curl -X POST https://mib.mom.magento.com/luma/remote/warehouse-integration-example -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}'

Then, the MIB will make a POST request to URL to https://warehouse-integration.mom/api (URL used in registration call) with following payload:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}

The reply will be forwarded back.

Asynchronous communication

Asynchronous communication allows you to leverage MIB retry capabilities and deliver messages, even if the service is not available at the moment of the call.

Assuming there is a warehouse-integration-example service registered with the same parameters as in the registration example, we can send an asynchronous call.

Example bash

curl -X POST https://mib.mom.magento.com/luma/delegate/warehouse-integration-example -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}'

You may notice that the call is exactly the same as for synchronous communication, only URL is different.

The MIB will reply with an error, in case there is any or null meaning the request was accepted:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

Then, the MIB will make a POST request to https://warehouse-integration.mom/api (the URL used in the registration call) with the following payload:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.warehouse.ship",
    "params": {
        "request_id": "100"
    }
}

Finally, the MIB will put the message back to the queue in case an error occur at any moment (DNS failure, network failure, non-2xx HTTP response, or other).

Broadcasting

The MIB provides a simple publish/subscribe API for events. Any service can list topics it wants to receive and once a message with a given topic is published to the broadcasting endpoint, it will be delivered to a given API.

In the API call below a service with the ID event-subscriber subscribes for magento.foo and magento.bar events.

Example bash

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "event-subscriber",
        "url": "https://event-subscriber.mom/",
        "subscribes": [
            "magento.foo",
            "magento.bar"
        ]
    }
}'

Then, any other service could publish an event using the broadcast endpoint: https://mib.mom.magento.com/luma/events.

Example bash

curl -X POST https://mib.mom.magento.com/luma/events -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.foo",
    "params": {
        "foo": "bar"
    }
}'

MIB will reply with an error, in case there is any or null meaning the request was accepted:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

Then, the MIB will make a POST request to https://event-subscriber.mom/ (the URL used in the registration call) with the following payload:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.foo",
    "params": {
        "foo": "bar"
    }
}

If an error occurs, the MIB will apply the same retry mechanism as for asynchronous communications.

Verify delivery

Once your service is registered as an endpoint for the MIB, it accepts any payload sent to the URL you configured. For security reasons, you probably want to limit requests to those coming from the MIB. There are a few ways to do this:

• Opt to whitelist requests from certain IP address.
• Set up a secret token and validate incoming requests (this is an easier method).

Setting up a secret

The MIB uses a signature with pre-shared secret. It means MIB will use a secret (a string provided during registration) to sign all requests to your endpoint, and your service would need to verify that this signature is correct.

You should provide a secret for your endpoint during its registration, for example like this:

Example bash

curl -X POST https://mib.mom.magento.com/luma/ -H'Authorization: Bearer TOKEN' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.service_bus.remote.register",
    "params": {
        "id": "warehouse-integration-example",
        "url": "https://warehouse-integration.mom/api",
        "secret": "foo"
    }
}'

You can generate it dynamically during registration and then store it in some persistent storage, so you can validate the signature. Or, alternatively, you can put it in a configuration file.

Validating requests

The service bus will add a special X-Signature-SHA256 header in every request to URLs registered with a non-empty secret.

If a URL is registered with an empty secret, the header is not set. This SHA-256 signature is an improved security measure. We strongly recommend you use it.

Signature SHA-256

There is a new shared secret signature verification, SHA-256. We strongly recommended using this SHA signature, a much secure signature than the legacy SHA1 signature. A X-Signature-SHA256 header is included in the delivery with a hash value:

Example hash

`X-Signature-SHA256: 7ec3619ea7f921721b6d8bec83ea2d0c6d557e0b164daf577e62c8cc88ed74ae`.

The hash is generated from the request body of the delivery and a salt (secret) provided during registration. The previous X-Signature header and hash are still present for backward compatibility.

This header allows registered endpoints to validate that the sender of a message (for OMS, usually Service Bus) is an accepted communication gateway.

Verification of this header includes:

• The header value will be the hash itself. It skips the addition of the hash method in the value.
• The value of X-SIGNATURE-SHA256 is the SHA-256 hashing of the message body (or empty string if there is no body), plus the shared secret (as opposed to SHA-1).

Example .yaml with SHA-256 signature

$secret = "foo";
$raw = file_get_contents("php://input"); // request body

$signature = isset($_SERVER['HTTP_X_SIGNATURE_SHA256']) ? $_SERVER['HTTP_X_SIGNATURE_SHA256'] : null;

if ($signature === null) {
    return false; // Request is not signed!
}

$hash = hash_hmac("sha256", $raw, $secret);
return hash_equals($hash, $signature);

Signature SHA1

The system also sends the SHA1 signature to allow backward compatibility:

• X-Signature-SHA256
• X-Signature-SHA1 (backward compatibility)

The SHA1 signature is a legacy functionality, but it remains in our system for backward compatibility. We strongly recommend using the SHA-256 signature.

Contact Support for assistance in removing your SHA1 signature.

The SHA1 header is a hash in HEX format prefixed with sha1=:

Example hash

`SHA1=9f00d52ea157870e92a975070e277748230443a3`.

The hash is generated from the request body and a secret provided during registration.

This signature SHA1 functionality is currently deprecated.

Here is a basic PHP code which verifies the signature:

Example .yaml with SHA1 signature

$secret = "foo";
$raw = file_get_contents("php://input"); // request body

$signature = isset($_SERVER['HTTP_X_SIGNATURE']) ? $_SERVER['HTTP_X_SIGNATURE'] : null;

if ($signature === null) {
    return false; // Request is not signed!
}

$hash = "sha1=".hash_hmac("sha1", $raw, $secret);
if ($hash != $signature) {
    return false; // Signature does not match
}

return true; // Signature is valid!

Error processing

The MIB uses HTTP protocol in a transport layer. You should handle cases when the HTTP protocol gives an error response (status code is different than 200) and process them accordingly. Normally, an error in the HTTP protocol would mean misconfiguration or some connectivity issue.

HTTP status code examples

Other errors should be handled according to HTTP specification.

Once you receive HTTP response with status code 200, it means there was no errors on transport layer, you should proceed to parse response body as JSON RPC response.

The JSON RPC response may still indicate an error. Check JSON RPC specification for more details.

Error codes used by the MIB

Error notification for asynchronous messages

While sync messages will get an immediate response on whether the message was successfully processed, asynchronous messages have to follow another path due to the nature of such delayed processes.

An integration may choose to send a specifically crafted HTTP header with each request made to the Magento API in order to be notified if a message fails. Such headers will contain the name of the integration itself that needs notification (usually the same integration that sends the message):

Example yml

`X-Error-Reply-To: <integration-name>`

An asynchronous request is made directed at the OMS setting up the X-Error-Reply-To header so integration mdc is notified of any particular error:

Example bash

curl -X POST https://mib.mom.magento.com/luma/delegate/oms -H'X-Error-Reply-To: mdc' -d'
{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.sales.order_management.create",
    "params": {
        "incorrect": "payload"
    }
}'

Immediate response from the Magento API would be (as expected for all async requests):

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}

As expected, such messages will fail upon arrival to the OMS, and if the X-Error-Reply-To header is set, the Magento API will attempt to notify the registered mdc service of said failure via the following message:

Example JSON

{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "magento.common.error_management.notify",
    "params": {
        "topic": "magento.sales.order_management.create",
        "message": "Validation failed:\n [order] This value should not be blank.",
        "payload": "{\"incorrect\": \"payload\"}"
    }
}

Where the params structure will be:

• topic: Original topic processing where error occurred
• message: Error message
• payload: Payload of message that triggered the error (as a string)

Retry policies

The MIB will retry delivery of the messages that were not delivered or accepted by the external server. The system will retry to deliver them until it receives a valid JSON RPC response from the server. This JSON RPC error response should not correspond with any of the error codes in the error codes processing section, or be equal to 0.

Retries occur for up to 48h, since the original message is not being delivered, with a maximum incremental delay (or back off) of 1 hour between retries:

• First retry happens after 30 seconds of the original message.
• Then, retries after an incremental back off of 1.5 times up to a maximum of 1 hour between retries.

Example of a retry scenario

30 seconds, 45 seconds, 68 seconds, 102 seconds… Up to 3600 seconds.

Retries stop when original message is aged 48 hours.

Retry policies occur when:

• A DNS error occurs while resolving the destination URL.
• A connection refused error occurs while connecting to the remote host.
• The HTTP response has a status code different than 200.
• A JSON RPC error response occurs with error code -32000.

The OMS does not retry delivery of a message when:

• A JSON RPC result response occurs.
• A JSON RPC error response with the error code -32601 occurs.

Disable integration

It is possible that a client integration does not accept messages from the MIB. If this happens for a long period of time, it creates a high increment in failed deliveries due to our retry policies. It is recommended to disable this integration to avoid costs associated with unnecessary message deliveries.

To disable a client integration, only applicable in production environments:

• Notify the merchant if the integration keeps failing after 1 day. In staging environments, integration is disabled after these 24 hours.
• Disable the client integration if the delivery failure ratio is equal or bigger than a 90% failed deliveries for a period of 7 days.

Delivery failure ratio is the percentage of failed deliveries of a message between the client integration and the MIB.

See the Customer support topic for more information about submitting a Magento Support request.

Understanding rate limits

Rate limiting is a very common practice that the OMS employs to control the rate of traffic, and to prevent system overloads, service attacks, spread request spikes, and overfilling of the queue. Rate limits help us ensure stability of the system while not affecting the user experience.

There are default rate limits noted in the Specifications page. You can also find rate limits in the OMS Admin:

1. Navigate to System > Other settings > API Integrations in the Admin.
2. Click an integration in the Currently registered integrations section.

3. View the rate limit info in the Parameters section near the bottom.

   

Bursts, rate limits of 10 requests per second, are how the OMS processes requests during short intervals of time. The total amount of messages can never exceed the defined rate limit, however.

Monitoring on your OMS-integrated systems will help you identify peaks of traffic or a high number of rejected messages due to exceeding rate limits. You can regulate the rate of requests by implementing a throttling process so that the requests are distributed evenly over a time period.

Rate limiting does not differentiate between incoming and outgoing messages. Messages are only differentiated via their individual rate limit, which can be seen in the description of the message. Some messages do not have a specified rate limit, which means that a rate limit has not been set for that particular message.

Your rate limits and configuration may be different than what is noted in the specifications, depending on your individual situation. Contact Magento Support for more information.

When your defined rate limit is reached, the OMS will reply with a 429: Too Many Requests error. The messages will not be imported, and the sending system must parse the error and resend the message.

Mail notifications

In certain events, an integration owner can configure its endpoint information to receive certain updates via email.

Currently supported notifications

• Endpoint disabled: Whenever a given endpoint is disabled (either manually or through the health-check process) a notification email will be sent to the configured address so the owner can check and re-enable it if necessary.

Magento Commerce Connector notification email setup

For integrations that use the MC Connector this notification email can be configured in the env.php configuration file inside the serviceBus.labels[] key:

Example bash

return [
    ...
    'serviceBus' => [
      'url' => 'http://sample-mcom-url:8080/LUMA/',
      'secure_endpoint' => true,
      'labels' => [
        ...
        // Email address to send notifications to
        'magento.notification_email' => 'some@email.com',
        ...
      ],
    ],
    ...
)

After updating that file, run bin/magento setup:upgrade to trigger registration again and update the OMS API with the new label being added.

If you need to disable these notifications, remove the label and perform the process again so registration information is updated.

Manual registration

For custom made integrations which do not use the MDC Connector, the process is the same as described in the registration section, adding the magento.notification_email configuration to the labels section.

Example JSON

{
    "JSON RPC":"2.0",
    "method": "magento.service_bus.remote.register",
    "id": 1,
    "params": {
        "id": "name-of-integration",
        "url": "http://url-of-remote-endpoint/",
        "contracts": [
        ],
        "subscribes": [
        ],
        "labels": {
            "magento.notification_email": "some@email.com"
        }
    }
}

If you need to disable these notifications, remove the label and perform the process again so registration information is updated.

Alarm email

In the event of service bus not being able to deliver a certain message to an integration, there’s an alarm system in place that checks hourly for non delivered messages and creates a report sent to a configured e-mail address.

Magento Commerce Connector alarm email setup

For integrations that use the MC Connector this alarm email can be configured in the env.php configuration file inside the serviceBus.labels[] key:

Example yml

return [
    ...
    'serviceBus' => [
      'url' => 'http://sample-mcom-url:8080/LUMA/',
      'secure_endpoint' => true,      'labels' => [
        // Email address to send alarms to
        'magento.alarms_email' => 'some@email.com',
      ],
    ],
    ...
);

This parameter can include more than one email address if needed, concatenating them with pipes:

Example yml

'magento.alarms_email' => 'some@email.com|another@mail.com'

After updating that file, run bin/magento setup:upgrade to trigger registration again and update the OMS API with the new label being added.

Manual registration

For custom made integrations which do not use the MDC Connector, the process is the same as described in the registration section, adding the magento.alarms_email configuration to the labels section.

Example JSON

{
    "JSON RPC":"2.0",
    "method": "magento.service_bus.remote.register",
    "id": 1,
    "params": {
        "id": "name-of-integration",
        "url": "http://url-of-remote-endpoint/",
        "contracts": [
        ],
        "subscribes": [
        ],
        "labels": {
            "magento.alarms_email": "some@email.com"
        }
    }
}

For multiple emails to be notified, concatenate them with pipes as before "magento.alarms_email": "some@email.com|another@email.com"