Wednesday, May 9, 2018

Sitecore Experience Commerce - A minion to import/sync catalogs and products

At the time of release, Sitecore Experience Commerce came with an API method which could be used to import and export commerce data. It used a zip containing many JSON files, each of which was a catalog, sellable item, category or relationship of some type. This is handy for once off exports of data which could then be imported into another environment (perhaps developer machines). But for a real world environment, something more robust is required and that's where a custom import minion comes in.

Introduction to minions

In Sitecore Commerce a minion can be thought of as a scheduled task, which runs in the minions commerce engine environment and has access to the full commerce context. 

The habitat minions environment which comes out of the box with commerce (defined in the PlugIn.Habitat.CommerceMinions-1.0.0.json file) has minions defined such as the full index minion and the incremental index minion. These two minions both handle indexing of the commerce data into SQL. The incremental index minion is scheduled to run every 5 minutes and looks for new or updated items to index and the full index minion is not scheduled and can be called via the run minion API call.

Creating a commerce minion and defining it's schedule has been covered in an earlier post. Much like a standard Sitecore pipeline it has a run method which is where things kick off (including initialization). From here you are able to call commerce pipelines (either your own custom or the out of the box ones). 

A pipeline can be consider a wrapper for a number of blocks which will contain the actual code and logic. As an example, there might be a need to get a Sellable Item which already exists inside Sitecore commerce. This is done using the pipeline IGetSellableItemPipeline. If you were to look inside the log files of one of your commerce engine environments (NodeConfiguration_Deployment01 log files in particular), you would see this pipeline is configured as follows:
IGetSellableItemPipeline (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.GetSellableItemInitializeBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.ProductArgument)     ------------------------------------------------------------     Plugin.Catalog.GetSellableItemBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Inventory.ResolveSellableItemInventoryInformationBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.EnsureSellableItemPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Availability.EnsureSellableItemAvailabilityPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.ICalculateSellableItemPricesPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Availability.IPopulateItemAvailabilityPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.AddSellableItemToContextBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
It takes in an argument of ProductArgument and will output a SellableItem. You can see that the pipeline itself calls a number of various blocks, each of which performs it's own piece of logic/code. In fact this particular pipeline actually calls additional pipelines (such as ICalculateSellableItemPricesPipeline) which again call their configured blocks (and perhaps further pipelines).

The import minion structure

If you are looking to build an import/sync minion then you obviously have the data available in some format. In my case, I was building a sync minion and used SQL server to provide delta information of the data for each run of the minion. This was structured into several key areas (each of which could be accessed and actioned in a separate block of my import minions custom pipeline. These are the required actions I identified:
  1. Added category
  2. Added sellable item
  3. Added ranging (a given sellable item appearing in a given catalog)
  4. Removed ranging (a given sellable item no longer appearing in a given catalog)
  5. Deleted sellable item
  6. Deleted category
  7. Updated sellable item
Therefore I structured my minion as follows:
  • Import Minion
    • Import Pipelines
      • ProcessAddedCategories
      • ProcessAddedSellableItems
      • ProcessRanging
      • ProcessDeranging
      • ProcessDeletedSellableItems
      • ProcessDeletedCategories
      • ProcessUpdatedSellableItems
Note that this is a cut down example, from what I actually built as there was a large amount of business requirements which dictated the needs to separate catalogs for distinct stores each of which required their own price books and price cards to handle separate pricing at each.

The import minion code

The example commerce minion code solution I have placed on GitHub is a great start for this project, as it comes with the minion itself along with a pipeline and block. That means that it only would need the extra blocks created as defined above.

I'm not going to provide a full working example of an import or sync minion, but instead will show the commerce (out of the box) pipelines you would call for some of the key interactions (creating a sellable item for example). These pipelines would be called as part of the relevant block section inside a loop (for example the loop of categories to add inside ProcessAddedCategories).

The reason we call these commerce pipeline (as opposed to the service calls to the API) is the fact that those APIs will actually themselves call the pipeline code. This saves going over the wire and dramatically will cut down on processing time.

To see which pipelines are available, head over to the NodeConfiguration_Deployment01 log files in one of your commerce instance engines logs folder (C:\inetpub\wwwroot\CommerceMinions_Sc9\wwwroot\logs in my case). A search through this for keywords such as SellableItem as an example, can show you whats available. From there it's a bit of trial and error to correctly call the pipeline. 

The following code snippets and commerce minions will require access to the Sitecore Commerce nuget repository. You should add this to Visual Studio: https://sitecore.myget.org/F/sc-commerce-packages/api/v3/index.json.

Adding a SellableItem

This example shows how to create a basic sellable item and once completed will return a SellableItem object. 
using Sitecore.Commerce.Core;
using Sitecore.Framework.Pipelines;
using System;
using System.Linq;
using System.Threading.Tasks;
using Sitecore.Commerce.Plugin.Catalog;

namespace Plugin.MyProject.Import
{
    class ImportMinionBlock : PipelineBlock<MinionRunResultsModel, MinionRunResultsModel, CommercePipelineExecutionContext>
    {
        private readonly ICreateSellableItemPipeline _createSellableItemPipeline;

        public ImportMinionBlock(ICreateSellableItemPipeline createSellableItemPipeline)
        {
            _createSellableItemPipeline = createSellableItemPipeline;
        }

        public override async Task<MinionRunResultsModel> Run(MinionRunResultsModel arg, CommercePipelineExecutionContext context)
        {
            try
            {
                var result = await _createSellableItemPipeline.Run(new CreateSellableItemArgument("MyProdID", "TestProduct", "Test Product", "This is a description of my test product"), context);

                if (result != null) // Null here means it failed to add
                {
                    SellableItem addedItem = result.SellableItems.FirstOrDefault(); // This is the item which has just been added
                }
            }
            catch (Exception)
            {
                // Some sort of failure - for example trying to add a duplicate product
                throw;
            }

            return arg;
        }
    }
}
The key pipeline in this case is the ICreateSellableItemPipeline.


Editing/updating a SellableItem

Now lets continue on from the previous example and add some additional data to our SellableItem.
using Sitecore.Commerce.Core;
using Sitecore.Framework.Pipelines;
using System;
using System.Linq;
using System.Threading.Tasks;
using Sitecore.Commerce.Plugin.Catalog;

namespace Plugin.MyProject.Import
{
    class ImportMinionBlock : PipelineBlock<MinionRunResultsModel, MinionRunResultsModel, CommercePipelineExecutionContext>
    {
        private readonly ICreateSellableItemPipeline _createSellableItemPipeline;
        private readonly IPersistEntityPipeline _persistEntityPipeline;

        public ImportMinionBlock(ICreateSellableItemPipeline createSellableItemPipeline, IPersistEntityPipeline persistEntityPipeline) 
        {
            _createSellableItemPipeline = createSellableItemPipeline;
            _persistEntityPipeline = persistEntityPipeline;
        }

        public override async Task<MinionRunResultsModel> Run(MinionRunResultsModel arg, CommercePipelineExecutionContext context)
        {
            try
            {
                var result = await _createSellableItemPipeline.Run(new CreateSellableItemArgument("MyProdID", "TestProduct", "Test Product", "This is a description of my test product"), context);

                if (result != null) // Null here means it failed to add
                {
                    SellableItem addedItem = result.SellableItems.FirstOrDefault(); // This is the item which has just been added

                    addedItem.Brand = "My Brand";
                    addedItem.Manufacturer = "My Manufacturer";
                    addedItem.ListPrice = new Money("USD", 9.99m);

                    var saveResult = _persistEntityPipeline.Run(new PersistEntityArgument(addedItem), context);
                }
            }
            catch (Exception)
            {
                // Some sort of failure - for example trying to add a duplicate product
                throw;
            }

            return arg;
        }
    }
}
The key pipeline used here is the IPersistEntityPipeline, and in fact can be used to persist (or save) any commerce entity of which you make changes to.

Final thoughts

Hopefully this blog post has provided a good base for anyone who is looking to create a Sitecore minion to handle importing and syncing products. As someone who has built a complete sync minion here are a few further tips:
  • Work with subsets of product data instead of testing you minion with larger data sets.
  • Test out how the minion will handle any corrupted data - that commerce server might not like
  • Logging is your best friend, as in many cases an exception on a given pipeline will stop the whole minion run
  • The example Habitat data that comes with commerce is your best friend, and is an example of what your end product should look like.
Any further questions, feel free to comment below or contact me on twitter @RyanBaileyNZ.

No comments:

Post a Comment