Wednesday, September 28, 2016

Introducing the Data Exchange Framework for Sitecore

Some Sitecore implementations can have a requirement to integrate with other third party systems. A common example of this might be a CRM which would sync data to/from Sitecore experience database (contacts). Other examples might be the syncing of product data into Sitecore commerce or simply creating/updating Sitecore items based on a legacy system.

For those of us who have implemented or inherited such integrations in the past, the common problem was often around no real structure and possibility of error. Thankfully Sitecore have released the Data Exchange Framework.

What is the Data Exchange Framework?

To put it simply, it's a framework which handles the transfer of data between two systems. Commonly Sitecore would be one of the two systems (either the source or target). However it is possible to use the framework to handle data exchange between two independent systems - a case for this might be syncing data between a finance system and training management application.

Full documentation for the Data Exchange Framework is available on the Sitecore website - and this is a great example of the new documentation format Sitecore will be using moving forward.

What are it's uses?

Well, the great thing about it being a framework is that the potential uses are really open to your business requirements. Some potential examples I have done in the past whilst working on Sitecore projects, that would be cases for using the Data Exchange Framework are:

  1. Sync course/enrollment data from a training management system. The Sitecore website could then handle new enrollments, existing enrollments and even personalisation based on your course history.
  2. Syncing sales data from Sitecore to a finance system. 
  3. Syncing contact history (products current subscribed to and claims history) from a legacy internal system.
An actual example currently available from Sitecore that uses the Data Exchange Framework is the Dynamics CRM Connect module. The following image is an example of the pipelines (units of work) which this module can perform.


Being that CRM focuses around customers (aka contacts in Sitecore), you will notice that a lot of these pipelines revolve around contacts in the xDb. Other implementations of the Data Exchange Framework might revolve around item creation or update for example.

How does it work?

Without going into too much detail here, the basic gist of the Data Exchange Framework is as follows.

Pipelines are units of work that are performed and are usually related to data transfer. These pipelines can contain multiple Pipeline Steps, which are a series of tasks run in a given order. A Pipeline Processor runs the given pipeline by; deciding which steps to run, running these steps and handling errors. Each pipeline step can also have a Pipeline Step Processor which is the logic used to run that step. The pipeline step might be though of as the configuration (such as the connection string for a CRM), where the pipeline step processor then uses that configuration (step) to actually read/write data in the CRM. There is also a Pipeline Context which allows for data to be shared across multiple pipeline steps in a given pipeline.

Pipelines are run via a Pipeline Batch - which can contain one or multiple pipelines to be run. This wrapper maintains logs of the pipelines running (and when) and can be called via the content editor or as a scheduled job. Of course this then means that there is a Pipeline Batch Processor which handles the logic of which pipelines to run, actually running them and handling any errors.

Going back to my example of Syncing sales data from Sitecore to a finance system, there would likely be the following structure:
  • Update financial system (pipeline)
    • Read data items from Sitecore (pipeline step)
    • Loop through items (pipeline step)
    • Update target system (pipeline step)
This would all then be kicked off via a pipeline batch called sync financial data, which would be run on a schedule (every 30 minutes for example).

Getting started

It all sounds rather complex and may come off as gibberish, however the best advice I can give is to take a read through the Data Exchange Framework documentation. Then actually install the module into a clean Sitecore development environment, along with Sitecore's Dynamics CRM Connect module. The CRM connect module provides a good working example which can help one to grasp the concepts of the framework and provide background for implementing it yourself.

No comments:

Post a Comment