Earlier this year, we announced that we had partnered with an authorised Electronic Money Institution (EMI) to launch Yapily Virtual Accounts, a brand new product offering that enables Yapily’s customers to make and receive open banking payments into an unlimited number of Virtual IBAN (vIBAN) accounts across multiple currencies.
As an engineering team, this was an incredibly exciting project to work on for many reasons. Here are just a few:
- This was a greenfield project that gave us the opportunity to build several new micro-services from the ground up (we use the Spring Reactive stack)
- We got to work collaboratively with a new cross-border engineering team, with team members split across the UK and Spain
- We could test out a new use case for another Yapily Beta service, Webhook Notifications
While, like with any project, we encountered a few interesting challenges along the way, one of our favourites was how to solve one of the most well-known pain points with open banking payments: how to provide real-time payment settlement notifications.
A quick overview of open banking payment statuses
Before we get into the details of how we built real-time settlement notifications, it’s worth reviewing why this use case is so important in the first place.
Below you’ll see an example of an open banking payment response made using the Yapily API. While the HTTP status of the response is 201 - Created, the status of the payment is still PENDING.
At this point, the originating bank has accepted the request as a valid payment instruction, but it has not yet processed it. Therefore, there is still opportunity for the payment to not be completed. This could be because…
- The sending account may have insufficient funds
- The bank’s compliance controls could delay or prevent the payment from being made
- The payment could breach some other business rules that the bank does not enforce immediately at the time of the request
- The payment may not be properly routed to the receiving bank
- The receiving bank could reject the request entirely
…amongst others!
The reality is, relying on synchronous responses for asynchronous processes does not ensure that the funds will arrive in the receiving account. For Third Party Providers (TPPs) who provide payment services to merchants, this can be problematic. Why? Naturally, merchants will want confirmation that the funds have landed in their bank account before releasing the purchased goods.
In the absence of Virtual Accounts and webhooks, an available alternative used by our clients is to poll our Get Payment Details endpoint, continually checking the status of the payment until it reaches a final state (e.g. “Completed” or “Failed”):
While this approach can be somewhat helpful, many banks will never communicate a truly final status for an open banking payment. This could be due to non-instant payment schemes like regular SEPA or BACS not supporting this, or potential reporting constraints at the originating bank.
Polling is also an inefficient use of resources: each new incoming connection must be established, the HTTP headers must be parsed, a query for new data must be performed, and a response (often with no new data to offer!) must be generated and delivered.
‘Payment Guaranteed’: How we built real-time settlement notifications for open banking payments
As we’ve seen so far, open banking payments are not always guaranteed and merchants risk non-settlement. By integrating with our EMI’s Banking-as-a-Service (BaaS) infrastructure to provide virtual accounts to our clients, we are able to leverage features which aren’t available via traditional open banking APIs, such as integrating with their payment completed webhooks.
Instead of sending money directly to a merchant’s bank account, our client is instead able to direct a payment into a virtual account by providing the virtual account details as the payee of an open banking payment request. In doing so, we are able to provide notifications to our clients when the funds from a payment made using the Yapily API have arrived in the receiving virtual account.
However, simply passing on the webhook notification from our BaaS provider isn’t hugely helpful without understanding which specific open banking payment the webhook relates to.
This is where payment reconciliation comes in - the process during which we match the transaction details provided in the webhook from our BaaS provider back to the original open banking payment from the Yapily API. Ultimately, this process proved to be quite tricky, since seemingly comparable fields from both representations of the payment can appear quite different.
As an example, lets take a look at a few fields which appear on both the webhook containing the transaction details from the receiving virtual account, and the open banking payment from Yapily’s API:
Primary ID
The payment ID provided in an open banking payment is different from the transaction ID for the transaction on the virtual account, which is controlled by the receiving account’s institution. This means that it’s impossible to match an ID from an open banking payment with an ID from the corresponding transaction.
While an end-to-end identifier is provided at time of payment initiation to the originating bank, and this is meant to travel with the payment (including to the receiving bank account - the Yapily Virtual Account here), this is not fool proof. The identifier is not always correctly passed between banks, or made available by the receiving bank.
Payment Reference
Payment reference is a value that is controlled by the TPP (or even perhaps the merchant or payment services user (PSU) themselves), and therefore a reference is never guaranteed to be unique. Institutions may also trim a provided reference to meet a certain length requirement, or append additional characters to increase the length. This could result in distinct differences in the payment reference on the open banking payment compared to the one which the institution returns.
Moreover, if we wanted to enforce using the payment reference for unique identifiers, it would mean that this field would no longer be available for more descriptive references that could be of use to the Payer or Payee to identify the payment on their account statements.
Currency and Amount
In the case of international payments, even the currency and amount for a payment could appear differently on the receiving account’s statement. The payment could have been made in GBP, but when arriving in a EUR account, appear as the converted amount and currency. The result? A non-match between the open banking payment and the transaction on the receiving account.
Clearly, we were unable to rely on a single common field to help us match an open banking payment to a payment settlement webhook from our BaaS provider.
The technical solution
Our solution needs were clear. We needed to maximise our ability to find a single matching payment for each payment settlement webhook received by our BaaS provider, and do so as efficiently as possible and in the most scalable way.
Therefore, we settled (no pun intended) on analysing a combination of payment identifiers, and ranked the combinations on the likelihood that they matched the open banking payment and the receiving account’s transaction. However, given the number of combinations of “matches” that were possible, it was not scalable to run an analysis of all possible match combinations at the time of receiving the payment settlement webhook - each of which would have required complex queries of our payments database - to try and find the original open banking payment.
To prevent this, we elected to pre-process the relevant payment fields when the payment is initially created. By leveraging an event-driven architecture, when a successful payment has been made, a message is published to a Pub/Sub topic. The consumer (our payment reconciliation service in this case) receives this message, filtering only for payments where the payment destination is a virtual account, and begins processing the message body which contains the details from the payment response. We generate a hash for all the possible match combinations we have deemed to be appropriate for identifying the original open banking payment, which are then persisted to a database alongside the open banking payment ID from the response, waiting for reconciliation.
Then, at some point in the future when we receive notification from our BaaS provider that a payment has settled on a virtual account (typically less than a second later), we apply the same hashing logic to the payment identifiers from the body of the webhook, starting with the highest ranked match - in other words, the combination which would produce a perfect match. We query our hash database looking for that exact hash, and if we find a match, we return the open banking payment ID associated with that hash. If not, we apply the same process to the next most highly ranked hash until we find a match.
Finally, we build the event object to be included in the webhook which we send to a client via our Notifications Service to notify them of the settled transaction, including both the details from the settled transaction on the virtual account, alongside the open banking payment ID corresponding to the original payment response.
What’s next
While we’re really proud of what we’ve built to enable real-time settlement for our Beta release of Yapily Virtual Accounts, we’re looking forward to applying more institution-specific intel toward our reconciliation logic over time.
For example, we may begin to recognise that Bank ABC always truncates their payment references to a specific length, meaning that we will often not find a match based on payment reference. Instead, we could disregard any matching combinations which depend on reference, and only generate the hashes for combinations which we know are likely to produce a match. This will continue to increase the speed at which we are able to deliver settlement notifications to clients
Conclusion
We hope you enjoyed this blog post which covered some of the reasons why we built real-time settlement notifications for open banking payments, how we designed our technical solution to be efficient and scalable, and what we’re hoping to achieve in future.
To learn more about Virtual Accounts, check out our API documentation.
If these kinds of problems sound interesting to you, check out our Yapily Careers page for current openings.
If you are interested in Yapily Virtual Accounts, please speak to your dedicated Customer Success Manager or get in touch with us to book a demo.