Wednesday, September 28, 2016

Sitecore Authentication on CES Discovery service failed

Any good Sitecore developer periodically checks their production logs for any error messages or worrying warnings. Whilst doing this on an 8.2 implementation, I noticed the following warning:
ManagedPoolThread #3 09:15:45 WARN Authentication on CES Discovery service failed.
Exception: System.Net.WebException
Message: The remote server returned an error: (403) Forbidden.
Source: System
at System.Net.HttpWebRequest.GetResponse()
at Sitecore.CES.Client.WebClient.ExecuteRequest(String requestUri)
at Sitecore.CES.Client.ResourceConnector`1.Request(String endpoint, Object[] parameters)
at Sitecore.CES.Discovery.EndpointSource.GetEndpoint(String serviceName)
The line before this in the logs states:
INFO  Job started: Update device detection DB
Which  traces back to device detection in Sitecore which is a paid service (since Sitecore 8.1). This can be disabled by using the following patch file:
<configuration xmlns:patch="">
      <setting name="DeviceDetection.Enabled">
        <patch:attribute name="value">false</patch:attribute>
It will then begin to insert the following (more graceful) warning :)
WARN  Device detection component is disabled
For those users who want to use this service, remember to enable the connections to and on the firewall.

Introducing the Data Exchange Framework for Sitecore

Some Sitecore implementations can have a requirement to integrate with other third party systems. A common example of this might be a CRM which would sync data to/from Sitecore experience database (contacts). Other examples might be the syncing of product data into Sitecore commerce or simply creating/updating Sitecore items based on a legacy system.

For those of us who have implemented or inherited such integrations in the past, the common problem was often around no real structure and possibility of error. Thankfully Sitecore have released the Data Exchange Framework.

What is the Data Exchange Framework?

To put it simply, it's a framework which handles the transfer of data between two systems. Commonly Sitecore would be one of the two systems (either the source or target). However it is possible to use the framework to handle data exchange between two independent systems - a case for this might be syncing data between a finance system and training management application.

Full documentation for the Data Exchange Framework is available on the Sitecore website - and this is a great example of the new documentation format Sitecore will be using moving forward.

What are it's uses?

Well, the great thing about it being a framework is that the potential uses are really open to your business requirements. Some potential examples I have done in the past whilst working on Sitecore projects, that would be cases for using the Data Exchange Framework are:

  1. Sync course/enrollment data from a training management system. The Sitecore website could then handle new enrollments, existing enrollments and even personalisation based on your course history.
  2. Syncing sales data from Sitecore to a finance system. 
  3. Syncing contact history (products current subscribed to and claims history) from a legacy internal system.
An actual example currently available from Sitecore that uses the Data Exchange Framework is the Dynamics CRM Connect module. The following image is an example of the pipelines (units of work) which this module can perform.

Being that CRM focuses around customers (aka contacts in Sitecore), you will notice that a lot of these pipelines revolve around contacts in the xDb. Other implementations of the Data Exchange Framework might revolve around item creation or update for example.

How does it work?

Without going into too much detail here, the basic gist of the Data Exchange Framework is as follows.

Pipelines are units of work that are performed and are usually related to data transfer. These pipelines can contain multiple Pipeline Steps, which are a series of tasks run in a given order. A Pipeline Processor runs the given pipeline by; deciding which steps to run, running these steps and handling errors. Each pipeline step can also have a Pipeline Step Processor which is the logic used to run that step. The pipeline step might be though of as the configuration (such as the connection string for a CRM), where the pipeline step processor then uses that configuration (step) to actually read/write data in the CRM. There is also a Pipeline Context which allows for data to be shared across multiple pipeline steps in a given pipeline.

Pipelines are run via a Pipeline Batch - which can contain one or multiple pipelines to be run. This wrapper maintains logs of the pipelines running (and when) and can be called via the content editor or as a scheduled job. Of course this then means that there is a Pipeline Batch Processor which handles the logic of which pipelines to run, actually running them and handling any errors.

Going back to my example of Syncing sales data from Sitecore to a finance system, there would likely be the following structure:
  • Update financial system (pipeline)
    • Read data items from Sitecore (pipeline step)
    • Loop through items (pipeline step)
    • Update target system (pipeline step)
This would all then be kicked off via a pipeline batch called sync financial data, which would be run on a schedule (every 30 minutes for example).

Getting started

It all sounds rather complex and may come off as gibberish, however the best advice I can give is to take a read through the Data Exchange Framework documentation. Then actually install the module into a clean Sitecore development environment, along with Sitecore's Dynamics CRM Connect module. The CRM connect module provides a good working example which can help one to grasp the concepts of the framework and provide background for implementing it yourself.

Improving and optimising the performance of your Sitecore website

Statistics around user behavior and page load times is a good bit of background reading on how users perceive the performance of a web site. The standout pieces of information for me was that:
"A 1 second delay in page response can result in a 7% reduction in conversions."
 Along with:
"40% of people abandon a website that takes more than 3 seconds to load."
This then sets the scene for why optimisation of a Sitecore website might be important to the end-users. For Sitecore websites which have a sales/lead generation component, there will be less conversions if the performance is not optimised.

Optimise the front-end

The first thing I always do is take a look at the website using the YSlow tool. This is a browser extension that essentially gives a rating on how well a given page will perform. They key areas it looks at is around the number of HTTP requests, CDN, compression, location of JavaScript/CSS (page blocking), minification and so on. It's a great starting point to get Sitecore optimised and often makes one of the biggest differences to end users.

The main outputs after using this tool should be:

  • CSS/JS bundled and minimised where possible
  • Any third party fonts or JavaScript libraries (jQuery) using CDN where possible
  • Images optimised in terms of size and kept minimal (use sprites in your CSS for example)
  • Static and dynamic compression enabled (this makes a big difference with those JS libraries)
  • Keep the HTTP requests to a minimum, once under load these exponentially affect performance

Optimising the caches

Database caches: Inside Sitecore.config (or web.config) will be a database section for Master, Core and Web. These caches are used to store data coming from the SQL databases and will lead to less database calls and ultimately better performance. The default values are generally not enough for smaller Sitecore implementations and should be optimised in a testing environment for every Sitecore website you work on. The http://server/sitecore/admin/cache.aspx page can be used to monitor these caches and in turn work out the ideal values they should be set to. There are plenty of blog posts/guides on correctly configuring these cache values.

Prefetch caches: When Sitecore is starting up, one of the tasks is to prefetch some data from the various databases, ready to use. These caches are configured in the files available in Website\App_Config\Prefetch and by default will hit key templates, items and item children. Recently I discovered that for the web prefetch cache, Sitecore attempts to cache the default home item and it's children. As many implementations use a custom home item, this is an example of a cache configuration which could be changed. This cache will have a minor impact on startup time, which should be okay on production environments, but will affect developers on their environments.

HTML Output cache: When you use caching for sublayouts/renderings, this data is stored in the HTML output cache. On the Site element for your web site (found under <configuration><sitecore><sites>) there is a variable called htmlCacheSize which controls the size of this cache. By using the cache administration page mentioned above, you are able to see if this cache is adequately sized. Remember to check this page on content delivery servers in a scaled environment, as these are the ones mainly serving data from this cache.

Optimising the database

SQL Server Index Fragmentation: For existing Sitecore implementations which have been active for some time, one area to investigate is the fragmentation percentage of SQL Server. As Sitecore puts it:
As indexes age, insertion and deletion of noncontiguous data can take its toll and cause fragmentation to occur. This can happen in just a few days on a busy CMS database. Minor amounts of fragmentation won't generally hurt performance. But as the percentage of fragmentation increases, performance suffers dramatically.
Your SQL Server database administrator should be able to help run the Index Physical Statistics report, which will then show which indexes have a high percentage of fragmentation  (over 10% is bad). The defragment indexes maintenance plan being run will solve these issues but ensure to make backups beforehand.

Further reading

These are some of the key areas in which I focus on and have found to provide the most benefit to optimising the performance of a Sitecore website. Of course this assumes that the custom code used on the website itself is optimised and running well. 

Monday, September 26, 2016

Sitecore Lucene appending pipe character to field

While implementing blog functionality in Sitecore, using Lucene as a datasource, I noticed that when I output the title field onto the page, it was appending a piped character ("|") to the end.

This wasn't a front-end issue where a | was being output in the repeater, so I used Luke to open the physical search index to see what was actually being stored.

As it turns out there were two fields on the blog post template which had the name title. The second title field was empty which made the pipe character appear to be appended. The truth was that Lucene uses the pipe character to concatenate all data for fields of the same name.

Monday, September 19, 2016

The new Sitecore standard - Helix principals

As a consultant working with Sitecore for a number of years I have had the pleasure to be involved with a large number of Sitecore implementations. These ranged from complete end-to-end implementation to reviews of existing sites. This has allowed me to see Sitecore development principals from a large amount of developers across multiple countries. One surprising aspect of this is the true extent that developers have implemented Sitecore, and whilst it's often pretty good, there are plenty of live examples of John West's Sitecore worst practices.

Luckily Sitecore has recently released the Helix principals which provide: "A set of overall design principles and conventions for Sitecore development". Whilst this has also been released in tandem with the habitat Sitecore project example (which I was lucky enough to be on a team which released one if the first production examples), the topic of this blog post will be primarily on Helix.

So why follow Helix principals?

Well I think Sitecore puts it best on the Helix site itself:
Helix provide a set of guidelines for your Sitecore projects. The Habitat example provides you with a pre-built and tested set of common modules that you can use as an inspiration to your project. Both improve the efficiency of your projects, reduce costs and time to market. As more and more people and organisations adopt the Helix conventions and principles, it will become a Sitecore standard. This means that people who are familiar with the conventions or the Habitat example will be able to work more easily on other convention-based projects with minimal training. It will be easier for Sitecore Product Support to understand projects built using the conventions, enabling them to resolve issues more quickly.
Some of the key take-aways for me are around the project efficiency, setting a global Sitecore standard and of course allowing official support to better understand your project.

What does it cover?

Rather than me rewriting the documentation in a blog format, lets cover off the main areas covered in Helix.

Patterns, Principles and Conventions
  • Architecture Principles: Looks at a modular solution architecturally.
  • Visual Studio: How to setup and structure and Visual Studio solution with projects.
  • File and Disk Structure: The physical structure of solutions and modules.
  • Managing Sitecore Items: Covers managing items in development and deploying them.
  • Templates: The setup of templates and how to reference them in code.
  • Page Layout: A look at page layouts, renderings, data sources and of course placeholders.
  • Configuration and Settings: Strategies for configuration and how to work with Sitecore configuration files.
  • Multi-site and multi-tenant: How to create multi site or tenant (logical grouping such as business unit).
  • Language and culture support: Multi-language implementations of Sitecore and use of Dictionaries.
  • Security and workflows: How to handle permissions in Sitecore and workflows for content editors.
  • Working with code: Code formatting for Sitecore.
  • Visual Design and Theming: A look at the front-end stack for your Sitecore site.
DevOps and development lifecycle management
  • Development: Covers general development and version control.
  • Build and integration: Covers build options for Sitecore projects along with integration points.
  • Testing: Covers managing your tests, along with unit testing and other testing activities (acceptance testing for example).
  • Deployment: Deployment strategy for Sitecore and what should be deployed where.

Get started!

For those users with large existing implementations, this can seem like a daunting list of changes to make your Sitecore solution standardised. But we all have to start somewhere, so I recommend taking one piece at a time. Fixing up how you deal with configuration can make upgrades easier so is a good start. As is the testing and deployment to ensure you have control and quality for what is released into each environment.

For the lucky ones who are starting a clean Sitecore solution, you have the option of taking the Habitat solution and using it outright (or as a best practice example). Or as intended you can take the Helix principals and bring them into your development team as general rules/guiding principals.

Wherever you are at with your particular Sitecore solution, lets all get on board with Habitat and ensure that moving forward we are all producing quality Sitecore sites that are efficient, easy to maintain and best practice!