Tuesday, May 29, 2018

Sitecore and Azure Search - Conflict between type of incomming field and field in schema

When attempting to rebuild the web/master indexes for a Sitecore 9 instance using Experience Commerce and Azure Search as the provider, the logs were filled with the following fatal errors:
FATAL Could not add field {B775BCA9-2605-4D60-B367-A0C354BE2504} : default value for indexable sitecore://master/{2F5990B9-E6F2-4D2E-9179-7851789EB66C}?lang=da&ver=1Exception: System.ApplicationExceptionMessage: Conflict between type of incomming field 'Edm.String' and field in schema 'Edm.Boolean'Source: Sitecore.ContentSearch.Azure at Sitecore.ContentSearch.Azure.Schema.CloudSearchIndexSchemaBuilder.AddFields(IndexedField[] fileds) at Sitecore.ContentSearch.Azure.Schema.CloudSearchIndexSchemaBuilder.AddField(String fieldName, Object fieldValue, CloudSearchFieldConfiguration configuration, CultureInfo culture)   at Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilder.AddField(String cloudName, Object fieldValue, CloudSearchFieldConfiguration cloudConfiguration, Boolean append, Boolean fieldIsEmpty)   at Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilder.AddField(String fieldName, Object fieldValue, CloudSearchFieldConfiguration cloudConfiguration, Boolean append) at Sitecore.ContentSearch.Azure.CloudSearchDocumentBuilder.AddField(IIndexableDataField field)   at Sitecore.ContentSearch.AbstractDocumentBuilder`1.CheckAndAddField(IIndexable indexable, IIndexableDataField field)
Strangely enough, this bug was supposedly fixed in version 8.2 of Sitecore Experience Platform. However it's likely the addition of new fields in later version that were not fixed up have caused it to come back.

The issue boils down to the fact that Azure search will store a type against a given field name. So in the case above, the field {B775BCA9-2605-4D60-B367-A0C354BE2504} is called default value and is of type single-line text (string). A quick search for other fields called default value, leads to 3 others, including one of type checkbox (boolean) and another of type integer.

That's where the issue comes in, the first occurrence of the field name will set the type in stone. So the only fix (if you wish to stop these errors) is to exclude the "duplicate fields" from the index.

Below is a patch file which has cleaned up many of these errors for me, I have not found any adverse affects (as these are often irrelevant system fields as opposed to rich text content).
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
<sitecore role:require="Standalone or ContentManagement" search:require="Azure">
    <contentSearch>
      <configuration>
        <indexes>
          <index id="sitecore_master_index">
            <configuration>
              <documentOptions ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/documentOptions">
                <exclude hint="list:AddExcludedField">
                  <NameConflictA tag="{EA2A0983-36EA-4957-B613-6F23EA5CDAD4}">{EA2A0983-36EA-4957-B613-6F23EA5CDAD4}</NameConflictA>
      <NameConflictG tag="{3F67BE6A-F3F0-41C5-B379-C6AF29685666}">{3F67BE6A-F3F0-41C5-B379-C6AF29685666}</NameConflictG>      
      <NameConflict1 tag="{FF1383BB-F095-4958-9B6D-E555DB653C44}">{FF1383BB-F095-4958-9B6D-E555DB653C44}</NameConflict1>
      <NameConflict2 tag="{D660A7A4-3F42-4953-A13F-CFCA850BDFEA}">{D660A7A4-3F42-4953-A13F-CFCA850BDFEA}</NameConflict2>
      <NameConflict3 tag="{7DC7CD49-9B23-4D44-B21E-FA5443224A50}">{7DC7CD49-9B23-4D44-B21E-FA5443224A50}</NameConflict3>
      <NameConflict4 tag="{B775BCA9-2605-4D60-B367-A0C354BE2504}">{B775BCA9-2605-4D60-B367-A0C354BE2504}</NameConflict4>
      <NameConflict5 tag="{E43E4E5F-6B45-4E3B-8D85-85AB87D2CC12}">{E43E4E5F-6B45-4E3B-8D85-85AB87D2CC12}</NameConflict5>
      <NameConflict6 tag="{CEDA3E7F-7406-40D0-A154-7CF7E3E7E85B}">{CEDA3E7F-7406-40D0-A154-7CF7E3E7E85B}</NameConflict6>
      <NameConflict7 tag="{4F17697F-BD00-4D76-B4F5-1B53C0C8D884}">{4F17697F-BD00-4D76-B4F5-1B53C0C8D884}</NameConflict7>
      <NameConflict8 tag="{BEFC61BF-8CFD-4478-9F5C-8422E6E2F005}">{BEFC61BF-8CFD-4478-9F5C-8422E6E2F005}</NameConflict8>
      <NameConflict9 tag="{4F892959-FDE6-4020-8569-2CF934BA412C}">{4F892959-FDE6-4020-8569-2CF934BA412C}</NameConflict9>
      <NameConflict10 tag="{85E2D73B-F11A-4BD7-9826-86BB191925C5}">{85E2D73B-F11A-4BD7-9826-86BB191925C5}</NameConflict10>
      <NameConflict11 tag="{EA45EDF7-CD8B-433F-99A6-EBFE3C7CE3A5}">{EA45EDF7-CD8B-433F-99A6-EBFE3C7CE3A5}</NameConflict11>
      <NameConflict14 tag="{1B9D1028-C20C-407B-9DCD-AFFE86A6F793}">{1B9D1028-C20C-407B-9DCD-AFFE86A6F793}</NameConflict14>
      <NameConflict15 tag="{09147FB2-EBFB-4949-8C8E-26A424409D5E}">{09147FB2-EBFB-4949-8C8E-26A424409D5E}</NameConflict15>
      <NameConflict16 tag="{3DD612C4-3AA7-48A9-9F46-FD8DE9AE3ACA}">{3DD612C4-3AA7-48A9-9F46-FD8DE9AE3ACA}</NameConflict16>
      <NameConflict17 tag="{F917C951-1F75-4D62-B14A-BC6888D7EECA}">{F917C951-1F75-4D62-B14A-BC6888D7EECA}</NameConflict17>
      <NameConflict18 tag="{290DEFD0-7B9A-4541-A326-87C44BEE5309}">{290DEFD0-7B9A-4541-A326-87C44BEE5309}</NameConflict18>
      <NameConflict19 tag="{59B79BAF-4F30-4ECA-961D-4AFAB14D157C}">{59B79BAF-4F30-4ECA-961D-4AFAB14D157C}</NameConflict19>
      <NameConflict20 tag="{9EDD2FCC-04AE-4230-84B4-C46C963F2284}">{9EDD2FCC-04AE-4230-84B4-C46C963F2284}</NameConflict20>
                </exclude>
              </documentOptions>
            </configuration>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>
Ignore the mismatched field names, this was after some cleanup.

Sitecore and Azure Search - An index can have at most 1000 fields

When attempting to run a rebuild of the master and web indexes in a PAAS environment of Sitecore 9 using Azure search, as the search was nearing the end, the following error would stop the index in it's tracks.
Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallExceptionMessage: {"error":{"code":"","message":"The request is invalid. Details: definition : Invalid index: The index contains 1087 field(s). An index can have at most 1000 fields.\r\n"}}
As it turns out, Azure Search can only index a maximum of 1000 unique field names. Annoyingly enough, the OOTB Sitecore 9 with Experience Commerce appears to actually go over this limit - hence the error.

There are a couple of configuration patch files that come OOTB that attempt to cull down the number of fields by excluding a whole lot of them:

  1. App_Config/Include/Sitecore.XA.Foundation.Search.Azure.Index.Master.ExcludeFields.config
  2. App_Config/Sitecore/ContentSearch.Azure/Sitecore.ContentSearch.Azure.Index.Master.ExcludeFields.config
Which is a start, but it means the next step is for you to create your own version of one of these field exclude configs and start adding your custom fields (and any others you can find for that matter). In my case it took a number of deploys, because as the index was able to progress further the error kept rearing it's ugly head. Ultimately I settled on 200 odd excludes, which were made up with a combination of my custom fields and OOTB Sitecore fields.

A handy tip is to keep the Sitecore show configuration page open, and before you exclude a field, do a quick search on that page to ensure it has not already been excluded.

This isn't an elegant solution, but I will update this page if a patch is provided.

Sitecore and Azure Search - contains a term that is too large to process

Similar to the PostFailedForSomeDocumentsException error, when I was attempting to rebuild the master index on a Sitecore 9 instance in Azure PAAS, the following error would appear in the logs:
{"key":"a1c7786e795d5d2f16525bda3d7087b9","status":false,"errorMessage":"Field 'content_1' contains a term that is too large to process. The max length for UTF-8 encoded terms is 32766 bytes. The most likely cause of this error is that filtering, sorting, and/or faceting are enabled on this field, which causes the entire field value to be indexed as a single term. Please avoid the use of these options for large fields.","statusCode":400},
I was able to trace the content_1 field back to the Sitecore.ContentSearch.Azure.DefaultIndexConfiguration.config configuration file to the field definition for _content (where the main item content is stored).
<field fieldName="_content" cloudFieldName="content_1" searchable="YES" retrievable="NO" facetable="NO" filterable="NO" sortable="NO" boost="1f" type="System.String" settingType="Sitecore.ContentSearch.Azure.CloudSearchFieldConfiguration, Sitecore.ContentSearch.Azure" />
As suggested by Azure search I set filterable, sortable and facetable to NO on this field. It would be unlikely that I would need any of these features on a field containing the bulk of the indexes text.

Furthermore I also found the same error with a mysterious field name 'script':
"errorMessage":"Field 'script' contains a term that is too large to process. The max length for UTF-8 encoded terms is 32766 bytes. The most likely cause of this error is that filtering, sorting, and/or faceting are enabled on this field, which causes the entire field value to be indexed as a single term. Please avoid the use of these options for large fields."
This field wasn't as easily traced, so I simply added a number of fields called script the to the list of fields to not be indexed. A patch file is shared below for the master database with Azure Search.
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
<sitecore role:require="Standalone or ContentManagement" search:require="Azure">
    <contentSearch>
      <configuration>
        <indexes>
          <index id="sitecore_master_index">
            <configuration>
              <documentOptions ref="contentSearch/indexConfigurations/defaultCloudIndexConfiguration/documentOptions">
                <exclude hint="list:AddExcludedField">
      <scriptField tag="{CD8DA5E2-3B65-4A14-B7A6-9F41181CE172}">{CD8DA5E2-3B65-4A14-B7A6-9F41181CE172}</scriptField>
      <scriptField2 tag="{3AC4B854-6CF9-4B30-9D52-1E5518AFF0E8}">{3AC4B854-6CF9-4B30-9D52-1E5518AFF0E8}</scriptField2>
      <scriptField3 tag="{DF23B990-55C1-401F-8F77-4698EDBD6FA9}">{DF23B990-55C1-401F-8F77-4698EDBD6FA9}</scriptField3>
      <scriptField4 tag="{FF1383BB-F095-4958-9B6D-E555DB653C44}">{FF1383BB-F095-4958-9B6D-E555DB653C44}</scriptField4>
      <scriptField5 tag="{CEDA3E7F-7406-40D0-A154-7CF7E3E7E85B}">{CEDA3E7F-7406-40D0-A154-7CF7E3E7E85B}</scriptField5>
      <scriptField6 tag="{C0A5BD91-E658-46CE-B631-77CE337D8E6E}">{C0A5BD91-E658-46CE-B631-77CE337D8E6E}</scriptField6>
      <scriptField7 tag="{B1A94FF0-6897-47C0-9C51-AA6ACB80B1F0}">{B1A94FF0-6897-47C0-9C51-AA6ACB80B1F0}</scriptField7>
      <scriptField8 tag="{CD8DA5E2-3B65-4A14-B7A6-9F41181CE172}">{CD8DA5E2-3B65-4A14-B7A6-9F41181CE172}</scriptField8>
                </exclude>
              </documentOptions>
            </configuration>
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

Monday, May 28, 2018

Sitecore and Azure Search- PostFailedForSomeDocumentsException error

When attempting to rebuild the master index of a Sitecore 9 instance in PAAS, the indexing would freeze and the following error was in the logs:
Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.PostFailedForSomeDocumentsException
Message: Partial success for insert or update. Some documents succeeded, but at least one failed.
This traces back to the number of items being sent to Azure Search in a given patch. Inside the Sitecore.ContentSearch.Azure.DefaultIndexConfiguration.config configuration file is a node called cloudBatchBuilder with a child of maxDocuments. Decreasing this value from the default (1000 at the time of writing), should resolve this error.

Tuesday, May 22, 2018

Sitecore Experience Commerce - Service Proxy execute compilation error

When attempting to call my custom API in the commerce engine, I was getting a compilation error on the Execute method of a service proxy call.
Cannot convert from 'Microsoft.OData.Client.DataServiceActionQuerySingle<MyModel>' to 'Microsoft.OData.Client.DataServiceActionQuery<MyModel>' 
At first I thought it was a duplicate method across both the shops and ops proxy. But as it turns out, I was expecting a return type of collection and the proxy was expecting a single return object (hence the single at the end of DataServiceActionQuerySingle).

This was resolved by going back into the commerce engine where I configured the API and changing the return type from:
configuration.ReturnsFromEntitySet<ExtendedPricingDetail>("Api");
To:
 configuration.ReturnsCollectionFromEntitySet<ExtendedPricingDetail>("Api");
Then a re-generation of the service proxy and all was working as expected.

Monday, May 21, 2018

Sitecore Experience Commerce - Creating a custom API in the Commerce Engine

Due to a customization in how my Sitecore Experience Commerce implementation structured pricing, there was a requirement for a custom API which returned this additional data. In my post Drilling into storefront rendering code and commerce engine pipeline code, I dug down to the proxy executing the API GetBulkPrices. In this post I am going to create my own custom implementation GetExtendedBulkPrices.

The return model

First I needed to create a model, which I wanted this method to return.
using System;
using System.ComponentModel.DataAnnotations;

namespace Plugin.MyProject.PriceDetail.Model
{
    public class ExtendedPricingDetail
    {
        [Key]
        public string ProductId { get; set; }
        public Decimal OriginalPrice { get; set; }
        public int Flybuys { get; set; }
        public string PromotionCode { get; set; }
        public string PromotionText { get; set; }

        public ExtendedPricingDetail()
        {
            ProductId = string.Empty;
            OriginalPrice = 0.00M;
            Flybuys = 0;
            PromotionCode = string.Empty;
            PromotionText = string.Empty;
        }
    }
}
The key attribute was required as it would error otherwise.

The command

I needed a command for my custom API to call. A command will generally not contain any business logic and will simply be used to call a pipeline. This post won't go as deep as pipelines, however my empty commerce minion code sample post has a pipeline example.
using Plugin.MyProject.PriceDetail.Model;
using Plugin.MyProject.PriceDetail.Pipelines;
using Sitecore.Commerce.Core;
using Sitecore.Commerce.Core.Commands;
using Sitecore.Commerce.Plugin.Catalog;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace Plugin.MyProject.PriceDetail.Commands
{
    public class GetExtendedBulkPricesCommand : CommerceCommand
    {
        private readonly IExtendedPricingDetailPipeline _pipeline;

        public GetExtendedBulkPricesCommand(IExtendedPricingDetailPipeline getSellableItemPipeline, IServiceProvider serviceProvider) : base(serviceProvider)
        {
            _pipeline = getSellableItemPipeline;
        }

        public async Task<IEnumerable<ExtendedPricingDetail>> Process(CommerceContext commerceContext, IEnumerable<string> itemIds)
        {
            using (CommandActivity.Start(commerceContext, this))
            {
                CommercePipelineExecutionContextOptions contextOptions = commerceContext.GetPipelineContextOptions();
                var items = new List<ExtendedPricingDetail>();

                foreach (var current in itemIds)
                {
                    ProductArgument productArgument = ProductArgument.FromItemId(current);
                    if (!productArgument.IsValid())
                    {
                        string str = await contextOptions.CommerceContext.AddMessage(commerceContext.GetPolicy<KnownResultCodes>().Error, "ItemIdIncorrectFormat", new object[1]
                        {
                            (object) current
                        }, string.Format("Expecting a CatalogId and a ProductId in the ItemId: {0}.", (object)current));
                    }
                    else
                    {
                        var priceDetail = await _pipeline.Run(productArgument, contextOptions.CommerceContext.GetPipelineContextOptions());
                       
                        if (priceDetail != null)
                        {
                            items.Add(priceDetail);
                        }
                    }
                }

                return items.AsEnumerable();
            }
        }
    }
}
As you can see, this example calls my custom pipeline IExtendedPricingDetailPipeline.

The API Controller

Now we need a controller with an action which can act as an API method, this code is a standard MVC example.
using Sitecore.Commerce.Core;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Web.Http.OData;
using Plugin.MyProject.PriceDetail.Commands;

namespace Plugin.MyProject.PriceDetail.Controller
{
    public class ApiController : CommerceController
    {
        public ApiController(IServiceProvider serviceProvider, CommerceEnvironment globalEnvironment): base(serviceProvider, globalEnvironment)
        {
        }

        [HttpPut]
        [Route("GetExtendedBulkPrices()")]
        public async Task<IActionResult> GetExtendedBulkPrices([FromBody] ODataActionParameters value)
        {
            if (!ModelState.IsValid)
                return (IActionResult)new BadRequestObjectResult(ModelState);
            if (!value.ContainsKey("itemIds") || !(value["itemIds"] is JArray))
                return (IActionResult)new BadRequestObjectResult((object)value);
            JArray jarray = (JArray)value["itemIds"];

            var bulkCommand = Command<GetExtendedBulkPricesCommand>();
            var res = await bulkCommand.Process(this.CurrentContext, jarray != null ? jarray.ToObject<IEnumerable<string>>() : (IEnumerable<string>)null);

            return (IActionResult)new ObjectResult(res);
        }
    }
}

In this example we are calling this ApiController, which means it follows an example of a Shops API method. Much like the GetBulkPrices method, that means the Postman call URL for this API would be {{ServiceHost}}/{{ShopsApi}}/GetExtendedBulkPrices(). It takes in an input of a list of strings and will output my custom model defined above.

This simply calls my command (as defined above).

Registering Exerything

Inside the Configure Sitecore class for the given plugin which contains these newly added classes, they need to be configured. This is done as follows (and does not show the Pipeline configuration):
public void ConfigureServices(IServiceCollection services)
{
    var assembly = Assembly.GetExecutingAssembly();
    services.RegisterAllCommands(assembly);

    services.Sitecore().Pipelines(config =>
        config
            .ConfigurePipeline<IConfigureServiceApiPipeline>(c => 
            {
                c.Add<RegisterPricingApiBlock>();
            })
    );
}
There is a single method which will register all commands, and added is a reference to RegisterPricingApiBlock. This is where the API and it's added entities are registered/built up for the oData service layer (and therefore commerce proxy used from the front-end). The code for this class is:
using System.Threading.Tasks;
using Microsoft.AspNetCore.OData.Builder;
using Plugin.MyProject.PriceDetail.Model;
using Sitecore.Commerce.Core;
using Sitecore.Framework.Conditions;
using Sitecore.Framework.Pipelines;

namespace Plugin.MyProject.PriceDetail.Pipelines.Blocks
{
    class RegisterPricingApiBlock : PipelineBlock<ODataConventionModelBuilder, ODataConventionModelBuilder, CommercePipelineExecutionContext>
    {
        public RegisterPricingApiBlock() { }

        public override Task<ODataConventionModelBuilder> Run(ODataConventionModelBuilder modelBuilder, CommercePipelineExecutionContext context)
        {
            Condition.Requires(modelBuilder).IsNotNull($"{base.Name}: The argument can not be null");

            modelBuilder.AddEntityType(typeof(ExtendedPricingDetail));
            modelBuilder.EntitySet<ExtendedPricingDetail>("ExtendedPricingDetails");

            var configuration = modelBuilder.Action("GetExtendedBulkPrices");
            configuration.CollectionParameter<string>("itemIds");
            configuration.ReturnsCollectionFromEntitySet<ExtendedPricingDetail>("Api");

            return Task.FromResult(modelBuilder);
        }
    }
}
Here the entity (custom model) and API method have been registered, which now allows them to be called via Postman and accessed via the Service Proxy project.

Calling the API via Postman

The API call can be a copy of one of the existing examples that come with the SDK. In my case, as I was using the same input (string array) as the GetBulkPrices API call, I simply copied this and updated the action name.

Calling the API via Postman


Updating the Service Proxy

Rob Earlam, has a great explanation up about what the Service Proxy is. Basically it's used to be able to share models between the .NET Core commerce Engine and standard .NET Sitecore website. However as I have added a custom model, I will need this to re-generate at the Service Proxy level.

The prerequisite for this is to install the OData v4 Client Code Generator plugin into Visual Studio 2017.

Now you simply add the Sitecore.Commerce.ServiceProxy project to your solution and under connected services are the ops and shops services which have a right click option to update (as long as a commerce engine instance is listening on port 5000). There is a JSON configuration if required.

Regenerating the Service Proxy

Conclusion

This was a simple example of how a custom API endpoint can be added to the commerce engine to call custom code and return custom models. This code could be slightly modified/expanded to actually extend a given model in the commerce engine layer.

Sitecore Experience Commerce - The entity does not have a key defined

After creating a custom command, API and return entity, I was getting the following error when starting the commerce engine in debug mode.
The entity 'MyModel' does not have a key defined
at Microsoft.AspNetCore.OData.Builder.ODataModelBuilder.ValidateModel(IEdmModel model)
The fix was to add the [Key] attribute to one of the attributes of my model. This can be found under the System.ComponentModel.DataAnnotations model.
public class ExtendedPricingDetail
{
 [Key]
 public string ProductId { get; set; }
 public Decimal CurrentPrice { get; set; }
 public Decimal OriginalPrice { get; set; }
 public string PromotionCode { get; set; }
 public string PromotionText { get; set; }
 public bool OnSpecial { get; set; }

 public ExtendedPricingDetail()
 {
  ProductId = string.Empty;
  CurrentPrice = 0.00M;
  OriginalPrice = 0.00M;
  PromotionCode = string.Empty;
  PromotionText = string.Empty;
  OnSpecial = false;
 }
}

Sitecore Experience Commerce - Command won't resolve in commerce engine

I have a custom API in the commerce engine which was able to be called via Postman. However when debugging the API, it was unable to get a copy of my command. The line of code which was failing was:
var myCommand = Command<MyCustomCommand>();
This was returning null. What was missing that the command was not properly being registered, this is done in the  ConfigureSitecore class of the plugin which contains the command and is achieved with the following line of code:
services.RegisterAllCommands(assembly);
After this the command was able to be accessed and run correctly.

Thursday, May 10, 2018

Sitecore Experience Commerce - Brand facet is splitting text by spaces

During an implementation of a product list in Experience Commerce, I found that the brand facet was splitting the brand's text at the occurrence of spaces. So a brand of "My Brand" would appear as two separate facet items: "my" and "brand".

The field was set to "UN_TOKENIZED" which should have not allowed this to happen. Nevertheless, the fix is to change the fields returnType from "test" to "string". Then rebuild the web and master indexes for your Sitecore instance. Below is a patch file to make this change (for SOLR search provider).

    
        
            
                
                    
                        
                            
                        
                    
                
            
        
   

Wednesday, May 9, 2018

Sitecore Experience Commerce - A minion to import/sync catalogs and products

At the time of release, Sitecore Experience Commerce came with an API method which could be used to import and export commerce data. It used a zip containing many JSON files, each of which was a catalog, sellable item, category or relationship of some type. This is handy for once off exports of data which could then be imported into another environment (perhaps developer machines). But for a real world environment, something more robust is required and that's where a custom import minion comes in.

Introduction to minions

In Sitecore Commerce a minion can be thought of as a scheduled task, which runs in the minions commerce engine environment and has access to the full commerce context. 

The habitat minions environment which comes out of the box with commerce (defined in the PlugIn.Habitat.CommerceMinions-1.0.0.json file) has minions defined such as the full index minion and the incremental index minion. These two minions both handle indexing of the commerce data into SQL. The incremental index minion is scheduled to run every 5 minutes and looks for new or updated items to index and the full index minion is not scheduled and can be called via the run minion API call.

Creating a commerce minion and defining it's schedule has been covered in an earlier post. Much like a standard Sitecore pipeline it has a run method which is where things kick off (including initialization). From here you are able to call commerce pipelines (either your own custom or the out of the box ones). 

A pipeline can be consider a wrapper for a number of blocks which will contain the actual code and logic. As an example, there might be a need to get a Sellable Item which already exists inside Sitecore commerce. This is done using the pipeline IGetSellableItemPipeline. If you were to look inside the log files of one of your commerce engine environments (NodeConfiguration_Deployment01 log files in particular), you would see this pipeline is configured as follows:
IGetSellableItemPipeline (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.GetSellableItemInitializeBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.ProductArgument)     ------------------------------------------------------------     Plugin.Catalog.GetSellableItemBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Inventory.ResolveSellableItemInventoryInformationBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.EnsureSellableItemPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Availability.EnsureSellableItemAvailabilityPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.ICalculateSellableItemPricesPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Availability.IPopulateItemAvailabilityPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.AddSellableItemToContextBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
It takes in an argument of ProductArgument and will output a SellableItem. You can see that the pipeline itself calls a number of various blocks, each of which performs it's own piece of logic/code. In fact this particular pipeline actually calls additional pipelines (such as ICalculateSellableItemPricesPipeline) which again call their configured blocks (and perhaps further pipelines).

The import minion structure

If you are looking to build an import/sync minion then you obviously have the data available in some format. In my case, I was building a sync minion and used SQL server to provide delta information of the data for each run of the minion. This was structured into several key areas (each of which could be accessed and actioned in a separate block of my import minions custom pipeline. These are the required actions I identified:
  1. Added category
  2. Added sellable item
  3. Added ranging (a given sellable item appearing in a given catalog)
  4. Removed ranging (a given sellable item no longer appearing in a given catalog)
  5. Deleted sellable item
  6. Deleted category
  7. Updated sellable item
Therefore I structured my minion as follows:
  • Import Minion
    • Import Pipelines
      • ProcessAddedCategories
      • ProcessAddedSellableItems
      • ProcessRanging
      • ProcessDeranging
      • ProcessDeletedSellableItems
      • ProcessDeletedCategories
      • ProcessUpdatedSellableItems
Note that this is a cut down example, from what I actually built as there was a large amount of business requirements which dictated the needs to separate catalogs for distinct stores each of which required their own price books and price cards to handle separate pricing at each.

The import minion code

The example commerce minion code solution I have placed on GitHub is a great start for this project, as it comes with the minion itself along with a pipeline and block. That means that it only would need the extra blocks created as defined above.

I'm not going to provide a full working example of an import or sync minion, but instead will show the commerce (out of the box) pipelines you would call for some of the key interactions (creating a sellable item for example). These pipelines would be called as part of the relevant block section inside a loop (for example the loop of categories to add inside ProcessAddedCategories).

The reason we call these commerce pipeline (as opposed to the service calls to the API) is the fact that those APIs will actually themselves call the pipeline code. This saves going over the wire and dramatically will cut down on processing time.

To see which pipelines are available, head over to the NodeConfiguration_Deployment01 log files in one of your commerce instance engines logs folder (C:\inetpub\wwwroot\CommerceMinions_Sc9\wwwroot\logs in my case). A search through this for keywords such as SellableItem as an example, can show you whats available. From there it's a bit of trial and error to correctly call the pipeline. 

The following code snippets and commerce minions will require access to the Sitecore Commerce nuget repository. You should add this to Visual Studio: https://sitecore.myget.org/F/sc-commerce-packages/api/v3/index.json.

Adding a SellableItem

This example shows how to create a basic sellable item and once completed will return a SellableItem object. 
using Sitecore.Commerce.Core;
using Sitecore.Framework.Pipelines;
using System;
using System.Linq;
using System.Threading.Tasks;
using Sitecore.Commerce.Plugin.Catalog;

namespace Plugin.MyProject.Import
{
    class ImportMinionBlock : PipelineBlock<MinionRunResultsModel, MinionRunResultsModel, CommercePipelineExecutionContext>
    {
        private readonly ICreateSellableItemPipeline _createSellableItemPipeline;

        public ImportMinionBlock(ICreateSellableItemPipeline createSellableItemPipeline)
        {
            _createSellableItemPipeline = createSellableItemPipeline;
        }

        public override async Task<MinionRunResultsModel> Run(MinionRunResultsModel arg, CommercePipelineExecutionContext context)
        {
            try
            {
                var result = await _createSellableItemPipeline.Run(new CreateSellableItemArgument("MyProdID", "TestProduct", "Test Product", "This is a description of my test product"), context);

                if (result != null) // Null here means it failed to add
                {
                    SellableItem addedItem = result.SellableItems.FirstOrDefault(); // This is the item which has just been added
                }
            }
            catch (Exception)
            {
                // Some sort of failure - for example trying to add a duplicate product
                throw;
            }

            return arg;
        }
    }
}
The key pipeline in this case is the ICreateSellableItemPipeline.


Editing/updating a SellableItem

Now lets continue on from the previous example and add some additional data to our SellableItem.
using Sitecore.Commerce.Core;
using Sitecore.Framework.Pipelines;
using System;
using System.Linq;
using System.Threading.Tasks;
using Sitecore.Commerce.Plugin.Catalog;

namespace Plugin.MyProject.Import
{
    class ImportMinionBlock : PipelineBlock<MinionRunResultsModel, MinionRunResultsModel, CommercePipelineExecutionContext>
    {
        private readonly ICreateSellableItemPipeline _createSellableItemPipeline;
        private readonly IPersistEntityPipeline _persistEntityPipeline;

        public ImportMinionBlock(ICreateSellableItemPipeline createSellableItemPipeline, IPersistEntityPipeline persistEntityPipeline) 
        {
            _createSellableItemPipeline = createSellableItemPipeline;
            _persistEntityPipeline = persistEntityPipeline;
        }

        public override async Task<MinionRunResultsModel> Run(MinionRunResultsModel arg, CommercePipelineExecutionContext context)
        {
            try
            {
                var result = await _createSellableItemPipeline.Run(new CreateSellableItemArgument("MyProdID", "TestProduct", "Test Product", "This is a description of my test product"), context);

                if (result != null) // Null here means it failed to add
                {
                    SellableItem addedItem = result.SellableItems.FirstOrDefault(); // This is the item which has just been added

                    addedItem.Brand = "My Brand";
                    addedItem.Manufacturer = "My Manufacturer";
                    addedItem.ListPrice = new Money("USD", 9.99m);

                    var saveResult = _persistEntityPipeline.Run(new PersistEntityArgument(addedItem), context);
                }
            }
            catch (Exception)
            {
                // Some sort of failure - for example trying to add a duplicate product
                throw;
            }

            return arg;
        }
    }
}
The key pipeline used here is the IPersistEntityPipeline, and in fact can be used to persist (or save) any commerce entity of which you make changes to.

Final thoughts

Hopefully this blog post has provided a good base for anyone who is looking to create a Sitecore minion to handle importing and syncing products. As someone who has built a complete sync minion here are a few further tips:
  • Work with subsets of product data instead of testing you minion with larger data sets.
  • Test out how the minion will handle any corrupted data - that commerce server might not like
  • Logging is your best friend, as in many cases an exception on a given pipeline will stop the whole minion run
  • The example Habitat data that comes with commerce is your best friend, and is an example of what your end product should look like.
Any further questions, feel free to comment below or contact me on twitter @RyanBaileyNZ.

Friday, May 4, 2018

Sitecore Experience Commerce - Drilling into storefront rendering code and commerce engine pipeline code

After importing my own catalog containing various categories and sellable items into Sitecore Experience Commerce. I found that pricing was displaying as $0 for all items instead of resolving via price cards.

The problem rendering

Sitecore Experience Commerce - Product list product
The presentation details of the search results page, showed that the rendering which was incorrectly displaying pricing was the 'Product List' rendering.

Search results page renderings
This rendering then traced back to the following controller: Sitecore.Commerce.XA.Feature.Catalog.Controllers.CatalogController found inside the Sitecore.Commerce.XA.Feature.Catalog DLL.

On the load of the page, the product list also populates after page load via a JavaScript call. The network tab of the console in chrome shows this call as GetProductList.

Experience Commerce - API call

Tracing the website code

Using dotPeek we are able to de-compile the DLL and view the code contained inside the controller.

Controller source 
The main controller action of ProductList (and therefore it's model source of ProductListRepository.GetProductListRenderingModel()) is not the real source of this data, as it actually comes via an API call. However that same ProductListRepository has a method called GetProductListJsonResult and this is where the API call gets it's data from. Looking into this method leads down a rabbit hole as follows:
  1. AdjustProductPriceAndStockStatus
  2. CatalogManager.GetProductBulkPrices
  3. PricingManager.GetProductBulkPrices
  4. PricingServiceProvider.GetProductBulkPrices
  5. RunPipeline<GetProductBulkPricesRequest, GetProductBulkPricesResult>("commerce.prices.getProductBulkPrices", request);
    1. Resolves to Sitecore.Commerce.Engine.Connect.Pipelines.Prices.GetProductBulkPrices
    2. Then calls Proxy.Execute<SellableItemPricing> ... GetBulkPrices
  6. Sitecore.Commerce.Engine.Container.GetBulkPrices
    1. References the URI: "/GetBulkPrices"
  7. Sitecore.Commerce.Plugin.Catalog.ApiController.GetBulkPrices
  8. IGetSellableItemPipeline
After all of that digging it finally leads to a commerce engine pipeline being run. This is where the actual logic will query the commerce data and attempt to resolve the price.

Tracing Commerce Engine code

How the commerce engine works is that a given pipeline such as IGetSellableItemPipeline will call a number of blocks (which is where code the actual logic sits) or even another pipeline. To see what code a given pipeline is actually calling, head over to the logs folder of one of your commerce engine instances (C:\inetpub\wwwroot\CommerceAuthoring_Sc9\wwwroot\logs for example) and open up the latest NodeConfiguration log file.

Performing a search in the file for IGetSellableItemPipeline returns the following:
Sitecore.Commerce.Plugin.Catalog
IGetSellableItemPipeline (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.GetSellableItemInitializeBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.ProductArgument)
     ------------------------------------------------------------
     Plugin.Catalog.GetSellableItemBlock (Sitecore.Commerce.Plugin.Catalog.ProductArgument => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Inventory.ResolveSellableItemInventoryInformationBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.EnsureSellableItemPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Availability.EnsureSellableItemAvailabilityPoliciesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.ICalculateSellableItemPricesPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Availability.IPopulateItemAvailabilityPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.AddSellableItemToContextBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
What this means is that for each call to IGetSellableItemPipeline, it will run each of these children blocks and pipelines in turn. Because I am interested in how pricing is resolved for a sellable item, the ICalculateSellableItemPricesPipeline is the where this happens. This pipeline is configured as:
Sitecore.Commerce.Plugin.Catalog
ICalculateSellableItemPricesPipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.GetSellableItemCatalogBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.ICalculateSellableItemSellPricePipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.ICalculateSellableItemListPricePipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
     ------------------------------------------------------------
     Plugin.Catalog.ReconcileSellableItemPricesBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
In my case, as I am not looking to debug list price and instead pricing via price cards the ICalculateSellableItemSellPricePipeline is what is relevant to investigate. This pipelines is configured as:
Sitecore.Commerce.Plugin.CatalogICalculateSellableItemSellPricePipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.CalculateSellableItemSellPriceBlock (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Plugin.Catalog.ICalculateVariationsSellPricePipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)
Which finally leads to a code block CalculateSellableItemSellPriceBlock and this is where the true logic to calculate the price for a given sellable item is.

Replacing and debugging Sitecore Commerce Engine code

The great thing about the Sitecore Commerce Engine, is that it is fully customizable. Any of the pipelines above can be removed, replaced and even reconfigured (in terms of what blocks/pipelines the pipeline run). In my case I would like to replace the CalculateSellableItemSellPriceBlock with my own version (the same code taken via DotPeek) so that it can be debugged and changed if required.

This is easily achieved by using the Commerce Engine source solution (which is what is deployed to the 4 commerce engine roles in IIS). This is the Sitecore.Commerce.Engine project and comes with the SDK when you install Experience Commerce (in my case Sitecore.Commerce.Engine.SDK.2.1.10). The Sitecore.Commerce.Engine project out of the box will reference a number of plugins, such as the braintree payment provider - Plugin.Sample.Payments.Braintree. In my case I have a number of custom plugins to extend commerce entities such as Price Cards and Sellable items. 

The first step is to create the replacement block inside a plugin project. I have called this example: CalculateSellableItemSellPriceBlockCustom
namespace MyPlugin.Pipelines.Blocks
{
    [PipelineDisplayName("Catalog.block.calculatesellableitemsellprice")]
    public class CalculateSellableItemSellPriceBlockCustom : PipelineBlock<SellableItem, SellableItem, CommercePipelineExecutionContext>
    {
        private readonly IResolveActivePriceSnapshotByCardPipeline _resolveSnapshotByCardPipeline;
        private readonly IResolveActivePriceSnapshotByTagsPipeline _resolveSnapshotByTagsPipeline;
        private IGetCatalogPipeline _getCatalogPipeline;

        public CalculateSellableItemSellPriceBlockCustom(IResolveActivePriceSnapshotByCardPipeline resolveSnapshotByCardPipeline, IGetCatalogPipeline getCatalogPipeline, IResolveActivePriceSnapshotByTagsPipeline resolveActivePriceSnapshotByTagsPipeline)
          : base((string)null)
        {
            this._resolveSnapshotByCardPipeline = resolveSnapshotByCardPipeline;
            this._resolveSnapshotByTagsPipeline = resolveActivePriceSnapshotByTagsPipeline;
            _getCatalogPipeline = getCatalogPipeline;
        }

        public override async Task<SellableItem> Run(SellableItem arg, CommercePipelineExecutionContext context)
        {
            // Code removed
            return arg;
        }
    }
}
Inside each plugin is a ConfigureSitecore class, it's in here that we can configure the replacement of the out of the box CalculateSellableItemSellPriceBlock block to my custom CalculateSellableItemSellPriceBlockCustom block.
public void ConfigureServices(IServiceCollection services)
{
 var assembly = Assembly.GetExecutingAssembly();
 services.RegisterAllPipelineBlocks(assembly);

 services.Sitecore().Pipelines(config =>
  config
   .ConfigurePipeline<ICalculateSellableItemSellPricePipeline>(c =>
   {
    c.Replace<CalculateSellableItemSellPriceBlock, CalculateSellableItemSellPriceBlockCustom>();
   })
 );
}
Now the code can be debugged and customized as required by a given commerce instances business rules. If you open up the node configuration after deploying the commerce engine, you can see that the out of the box pipeline is not referenced and the custom one is instead:
Sitecore.Commerce.Plugin.CatalogICalculateSellableItemSellPricePipeline (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)     ------------------------------------------------------------     Myproject.Pipelines.Blocks.CalculateSellableItemSellPriceBlockCustom (Sitecore.Commerce.Plugin.Catalog.SellableItem => Sitecore.Commerce.Plugin.Catalog.SellableItem)

Conclusion

Ultimately I traced my particular error back to a timezone issue (which stopped price card snapshots resolving correctly). However this post serves as an example of the processes required to trace back issues from the Commerce Experience Accelerator components back to the Sitecore Commerce Engine.