Sitecore media optimization with Azure Functions + Blob Storage + Magick.NET

In my previous post, I’ve explained how to configure the Blob Storage Module on a Sitecore 9.3+ instance. The following post assumes you are already familiar with it and you’ve your Sitecore instance making use of the Azure blob storage provider.

In this post I’ll show you how we can make use of Azure Functions (blob trigger) to optimize (compress) images on the fly, when those are uploaded to the media library, in order to gain performance and with a serverless approach.

Media Compression Flow

About Azure Functions and Blob Trigger

Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in Azure or third party service as well as on-premises systems. Azure Functions allows developers to take action by connecting to data sources or messaging solutions thus making it easy to process and react to events. Developers can leverage Azure Functions to build HTTP-based API endpoints accessible by a wide range of applications, mobile and IoT devices. Azure Functions is scale-based and on-demand, so you pay only for the resources you consume. For more info please refer to the official MS documentation.

Azure Functions

Azure Functions integrates with Azure Storage via triggers and bindings. Integrating with Blob storage allows you to build functions that react to changes in blob data as well as read and write values.

Creating the Azure Function

For building the blob storage trigger function I’ll be using Visual Code, so first of all make sure you have the Azure Functions plugin for Visual Code, you can get it from the marketplace or from the extensions menu, also from the link: vscode:extension/ms-azuretools.vscode-azurefunctions.

Install the extension for Azure Functions
Azure Functions Plugin

Before proceeding, make sure you are logged into your Azure subscription. >az login.

  1. Create an Azure Functions project: Click on the add function icon and then select the blob trigger option, give a name to the function.

2. Choose the Blob Storage Account you are using in your Sitecore instance (myblobtestazure_STORAGE in my case).

3. Choose your blob container path (blobcontainer/{same})

4. The basics are now created and we can start working on our implementation.

Default function class

Generated project files

The project template creates a project in your chosen language and installs required dependencies. For any language, the new project has these files:

  • host.json: Lets you configure the Functions host. These settings apply when you’re running functions locally and when you’re running them in Azure. For more information, see host.json reference.
  • local.settings.json: Maintains settings used when you’re running functions locally. These settings are used only when you’re running functions locally. For more information, see Local settings file.

Edit the local.settgins.json file to add the connection string of your blob storage:

local.settings.json

The function implementation

using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using ImageMagick;
using Microsoft.WindowsAzure.Storage.Blob;

namespace SitecoreImageCompressor
{
    public static class CompressBlob
    {
        [FunctionName("CompressBlob")]
        public static async void Run([BlobTrigger("blobcontainer/{name}", Connection = "myblobtestazure_STORAGE")] CloudBlockBlob inputBlob, ILogger log)
        {
            log.LogInformation($"C# Blob trigger function Processed blob\n Name:{inputBlob.Name} \n Size: {inputBlob.Properties.Length} Bytes");

            if (inputBlob.Metadata.ContainsKey("Status") && inputBlob.Metadata["Status"] == "Processed")
            {
                log.LogInformation($"blob: {inputBlob.Name} has already been processed");
            }
            else
            {
                using (var memoryStream = new MemoryStream())
                {
                    await inputBlob.DownloadToStreamAsync(memoryStream);
                    memoryStream.Position = 0;

                    var before = memoryStream.Length;
                    var optimizer = new ImageOptimizer { OptimalCompression = true, IgnoreUnsupportedFormats = true };

                    if (optimizer.IsSupported(memoryStream))
                    {
                        var compressionResult = optimizer.Compress(memoryStream);

                        if (compressionResult)
                        {
                            var after = memoryStream.Length;
                            var gain = 100 - (float)(after * 100) / before;

                            log.LogInformation($"Optimized {inputBlob.Name} - from: {before} to: {after} Bytes. Optimized {gain}%");

                            await inputBlob.UploadFromStreamAsync(memoryStream);
                        }
                        else
                        {
                            log.LogInformation($"Image {inputBlob.Name} - compression failed...");
                        }
                    }
                    else
                    {
                        var info = MagickNET.GetFormatInformation(new MagickImageInfo(memoryStream).Format);

                        log.LogInformation($"Image {inputBlob.Name} - the format is not supported. Compression skipped - {info.Format}");
                    }
                }

                inputBlob.Metadata.Add("Status", "Processed");
                
                await inputBlob.SetMetadataAsync();
            }
        }
    }
}

As you can see, I’m creating and async task that will be triggered as soon as a new blob is added to the blob storage. Since we’re compressing and then uploading the modified image, we’ve to make sure the function is not triggered multiple times. For avoiding that, I’m also updating the image metadata with a “Status = Processed“.

The next step is to get the image from the CloudBlockBlob and then compress using the Magick.NET library. Please note that this library also provides a LosslessCompress method, for this implementation I choose to go with the full compression. Feel free to update and compare the results.

Nuget references

So, in order to make it working we need to install the required dependencies. Please run the following commands to install the Nuget packages:

  • dotnet add package Azure.Storage.Blobs –version 12.8.0
  • dotnet add package Magick.NET-Q16-AnyCPU –version 7.23.2
  • dotnet add package Microsoft.Azure.WebJobs.Extensions.Storage –version 3.0.10
  • dotnet add package Microsoft.Azure.WebJobs.Host.Storage –version 4.0.1
  • dotnet add package Microsoft.NET.Sdk.Functions –version 1.0.38

Test and deploy

Now we have everything in place. Let’s press F5 and see if the function is compiling

Terminal output

We are now ready to deploy to Azure and test the blob trigger! Click on the up arrow in order to deploy to Azure, choose your subscription and go!

Azure publish

Check the progress in the terminal and output window:

Testing the trigger

Now we can go to the Azure portal, go to the Azure function and double check that everything is there as expected:

Azure function from the portal

Go to the “Monitor” and click on “Logs” so we can have a look at the live stream when uploading an image to the blob storage. Now in your Sitecore instance, go to the Media Library and upload an image, this will upload the blob to the Azure Storage and the trigger will take place and compress the image.

Media Library Upload
Azure functions logs

As we can see in the logs the image got compressed, gaining almost 15%:

2021-02-23T10:21:36.894 [Information] Optimized 6bdf3e56-c6fc-488b-a7bb-eee64ce04343 – from: 81147 to: 69158 Bytes. Optimized 14.774422%

Azure Blob Storage – With the trigger enabled
Azure Blob Storage – With the trigger disabled

Let’s check the browser for the final results

Without the trigger: the image size is 81147 bytes.

With the trigger: the image size is 69158 bytes.

I hope you find this useful, you can also get the full implementation from GitHub.

Thanks for reading!

How to enable Azure Blob Storage on Sitecore 9.3+

In this post I’m explaining how to switch the blob storage provider to make use of Azure Blob Storage. Before Sitecore 9.3, we could store the blobs on the DB or filesystem, Azure Blob Storage was not supported out of the box and even tough it was possible, it required some customizations to make it working, nowadays, since Sitecore 9.3 a module has been released and is very straightforward to setup, as you will see in this post.

By doing this we can significantly reduce costs and improve performance as the DB size won’t increase that much due to the media library items.

Resultado de imagen de azure blob storage

Introduction to Azure Blob storage

Azure Blob storage is Microsoft’s object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn’t adhere to a particular data model or definition, such as text or binary data.

Blob storage is designed for:

  • Serving images or documents directly to a browser.
  • Storing files for distributed access.
  • Streaming video and audio.
  • Writing to log files.
  • Storing data for backup and restore, disaster recovery, and archiving.
  • Storing data for analysis by an on-premises or Azure-hosted service.

Users or client applications can access objects in Blob storage via HTTP/HTTPS, from anywhere in the world. Objects in Blob storage are accessible via the Azure Storage REST APIAzure PowerShellAzure CLI, or an Azure Storage client library.

For more info please refer here and also you can find some good documentation here.

Creating your blob storage resource

Azure Storage Account

Create the resource by following the wizard and then check the “Access Keys” section, you’ll need the “Connection string” later.

Connection String and keys

Configuring your Sitecore instance

There are basically three main option to install the blob storage module into your instance:

  1. Install the Azure Blob Storage module in Sitecore PaaS.
    1. Use the Sitecore Azure Toolkit:
      1. Use a new Sitecore installation with Sitecore Azure Toolkit
      2. Use an existing Sitecore installation with Sitecore Azure Toolkit
    2. Use Sitecore in the Azure Marketplace (for new Sitecore installations only)
  2. Install the Azure Blob Storage module on an on-premise Sitecore instance.
  3. Manually install the Azure Blob Storage module in PaaS or on-premise.

This time I’ll be focusing in the last option, manually installing the module, doesn’t matter if it’s a PaaS or on-premise approach.

Manual installations steps

  1. Download the Azure Blob Storage module WDP from the Sitecore Downloads page.
  2. Extract (unzip) the WDP.
  3. Copy the contents of the bin folder of the WDP into the Sitecore web application bin folder.
  4. Copy the contents of the App_Config folder of the WDP into the Sitecore web application App_Config folder.
  5. Copy the contents of the App_Data folder of the WDP into the Sitecore web application App_Data folder.
  6. Add the following connection string to the App_Config\ConnectionStrings.config file of the Sitecore web application.
 <add name="azureblob" connectionString="DefaultEndpointsProtocol=https;AccountName=myblobtestazure;AccountKey={KEY};EndpointSuffix=core.windows.net"/>

7. In the \App_Config\Modules\Sitecore.AzureBlobStorage\Sitecore.AzureBlobStorage.config file, ensure that <param name="blobcontainer"> is the name you gave to the container after creating the resource.

Let’s test it!

If everything went well, then we can just test it by uploading a media item to the Sitecore media library

Let’s have a look now at the Storage Explorer in the Azure portal

Here we go, the image is now uploaded into the Azure blob storage, meaning the config is fine and working as expected.

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision – Part III.

In my previous post I’ve shared the custom image field implementation that makes use of the Azure Computer Vision service in order to crop and generate the thumbnails using AI. Please before proceed with this reading, make sure you already went through the previous posts: Part I and Part II.

Now, I’ll be sharing the last, but not least part of this topic, how to make it working in the front-end side, the media request flow and so on.

Image request flow

Image request flow
The image request flow

So, the request flow is described in the following graph, basically follows the normal Sitecore flow but with the introduction of the Azure Computer Vision and Image Sharp to generate the proper cropping version of the image.

AICroppingProcessor

This custom processor will be overriding the Sitecore OOTB ThumbnailProcessor. It’s basically a copy from the original code with a customization to check the “SmartCropping” parameter from the image request.

using Sitecore.Diagnostics;
using Sitecore.Resources.Media;
using System;
using Microsoft.Extensions.DependencyInjection;
using System.IO;
using System.Linq;
using Sitecore.Computer.Vision.CroppingImageField.Services;
using Sitecore.DependencyInjection;

namespace Sitecore.Computer.Vision.CroppingImageField.Processors
{
    public class AICroppingProcessor
    {
        private static readonly string[] AllowedExtensions = { "bmp", "jpeg", "jpg", "png", "gif" };

        private readonly ICroppingService _croppingService;

        public AICroppingProcessor(ICroppingService croppingService)
        {
            _croppingService = croppingService;
        }

        public AICroppingProcessor()
        {
            _croppingService = ServiceLocator.ServiceProvider.GetService<ICroppingService>();
        }

        public void Process(GetMediaStreamPipelineArgs args)
        {
            Assert.ArgumentNotNull(args, "args");

            var outputStream = args.OutputStream;

            if (outputStream == null)
            {
                return;
            }

            if (!AllowedExtensions.Any(i => i.Equals(args.MediaData.Extension, StringComparison.InvariantCultureIgnoreCase)))
            {
                return;
            }

            var smartCrop = args.Options.CustomOptions[Constants.QueryStringKeys.SmartCropping];

            if (!string.IsNullOrEmpty(smartCrop) && bool.Parse(smartCrop))
            {
                Stream outputStrm;

                outputStrm = Stream.Synchronized(_croppingService.GetCroppedImage(args.Options.Width, args.Options.Height, outputStream.MediaItem));
                args.OutputStream = new MediaStream(outputStrm, args.MediaData.Extension, outputStream.MediaItem);
            }
            else if (args.Options.Thumbnail)
            {
                var transformationOptions = args.Options.GetTransformationOptions();
                var thumbnailStream = args.MediaData.GetThumbnailStream(transformationOptions);

                if (thumbnailStream != null)
                {
                    args.OutputStream = thumbnailStream;
                }
            }
        }
    }
}

We need also to customize the MediaRequest to also take the “SmartCropping” parameter into account:

using Sitecore.Configuration;
using Sitecore.Diagnostics;
using Sitecore.Resources.Media;

using System.Web;

namespace Sitecore.Computer.Vision.CroppingImageField.Requests
{
    using System.Collections.Specialized;

    public class AICroppingMediaRequest : MediaRequest
    {
        private HttpRequest _innerRequest;
        private MediaUrlOptions _mediaQueryString;
        private MediaUri _mediaUri;
        private MediaOptions _options;

        protected override MediaOptions GetOptions()
        {
            var queryString = this.InnerRequest.QueryString;

            if (queryString == null || queryString.Count == 0)
            {
                _options = new MediaOptions();
            }
            else
            {
                SetMediaOptionsFromMediaQueryString(queryString);

                if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
                {
                    SetCustomOptionsFromQueryString(queryString);
                }
            }

            if (!this.IsRawUrlSafe)
            {
                if (Settings.Media.RequestProtection.LoggingEnabled)
                {
                    string urlReferrer = this.GetUrlReferrer();

                    Log.SingleError(string.Format("MediaRequestProtection: An invalid/missing hash value was encountered. " +
                        "The expected hash value: {0}. Media URL: {1}, Referring URL: {2}",
                        HashingUtils.GetAssetUrlHash(this.InnerRequest.RawUrl), this.InnerRequest.RawUrl,
                        string.IsNullOrEmpty(urlReferrer) ? "(empty)" : urlReferrer), this);
                }

                _options = new MediaOptions();
            }

            return _options;
        }

        private void SetCustomOptionsFromQueryString(NameValueCollection queryString)
        {
            this.ProcessCustomParameters(_options);

            if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping))
                    && !_options.CustomOptions.ContainsKey(Constants.QueryStringKeys.SmartCropping)
                    && !string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
            {
                _options.CustomOptions.Add(Constants.QueryStringKeys.SmartCropping, queryString.Get(Constants.QueryStringKeys.SmartCropping));
            }
        }

        private void SetMediaOptionsFromMediaQueryString(NameValueCollection queryString)
        {
            MediaUrlOptions mediaQueryString = this.GetMediaQueryString();

            _options = new MediaOptions()
            {
                AllowStretch = mediaQueryString.AllowStretch,
                BackgroundColor = mediaQueryString.BackgroundColor,
                IgnoreAspectRatio = mediaQueryString.IgnoreAspectRatio,
                Scale = mediaQueryString.Scale,
                Width = mediaQueryString.Width,
                Height = mediaQueryString.Height,
                MaxWidth = mediaQueryString.MaxWidth,
                MaxHeight = mediaQueryString.MaxHeight,
                Thumbnail = mediaQueryString.Thumbnail,
                UseDefaultIcon = mediaQueryString.UseDefaultIcon
            };

            if (mediaQueryString.DisableMediaCache)
            {
                _options.UseMediaCache = false;
            }

            foreach (string allKey in queryString.AllKeys)
            {
                if (allKey != null && queryString[allKey] != null)
                {
                    _options.CustomOptions[allKey] = queryString[allKey];
                }
            }
        }

        public override MediaRequest Clone()
        {
            Assert.IsTrue((base.GetType() == typeof(AICroppingMediaRequest)), "The Clone() method must be overridden to support prototyping.");

            return new AICroppingMediaRequest
            {
                _innerRequest = this._innerRequest,
                _mediaUri = this._mediaUri,
                _options = this._options,
                _mediaQueryString = this._mediaQueryString
            };
        }
    }
}

This code is very straightforward, it will basically check if the “SmartCropping=true” parameter exists in the media request, and then executes the custom code to crop the image.

The “Get Thumbnails” method limitations

As we can see in the official documentation, there are some limitations on the thumbnail generator method.

  • Image file size must be less than 4MB.
  • Image dimensions should be greater than 50 x 50.
  • Width of the thumbnail must be between 1 and 1024.
  • Height of the thumbnail must be between 1 and 1024.

The most important one is that the width and height cannot exceed the 1024px, this is problematic as sometimes we need to crop on a bigger ratio.

So, in order to make it more flexible, I’m doing the cropping using the Graphics library but getting the focus point coordinates from the “Get Area Of Interest” API method:

using Sitecore.Data.Items;
using Microsoft.Extensions.DependencyInjection;
using System.IO;
using Sitecore.DependencyInjection;
using Sitecore.Resources.Media;
using System.Drawing;
using System.Drawing.Imaging;
using System.Drawing.Drawing2D;

namespace Sitecore.Computer.Vision.CroppingImageField.Services
{
    public class CroppingService : ICroppingService
    {
        private readonly ICognitiveServices _cognitiveServices;

        public CroppingService(ICognitiveServices cognitiveServices)
        {
            _cognitiveServices = cognitiveServices;
        }

        public CroppingService()
        {
            _cognitiveServices = ServiceLocator.ServiceProvider.GetService<ICognitiveServices>();
        }

        public Stream GetCroppedImage(int width, int height, MediaItem mediaItem)
        {
            using (var streamReader = new MemoryStream())
            {
                var mediaStrm = mediaItem.GetMediaStream();

                mediaStrm.CopyTo(streamReader);
                mediaStrm.Position = 0;

                var img = Image.FromStream(mediaStrm);

                // The cropping size shouldn't be higher than the original image
                if (width > img.Width || height > img.Height)
                {
                    Sitecore.Diagnostics.Log.Warn($"Media file is smaller than the requested crop size. " +
                        $"This can result on a low quality result. Please upload a proper image: " +
                        $"Min Height:{height}, Min Width:{width}. File: {mediaItem.DisplayName}, Path{mediaItem.MediaPath}", this);
                }

                // if the cropping size exceeds the cognitive services limits, get the focus point and crop 
                if (width > 1025 || height > 1024)
                {

                    var area = _cognitiveServices.GetAreaOfImportance(streamReader.ToArray());
                    var cropImage = CropImage(img, area.areaOfInterest.X, area.areaOfInterest.Y, width, height);

                    return cropImage;
                }

                var thumbnailResult = _cognitiveServices.GetThumbnail(streamReader.ToArray(), width, height);

                return new MemoryStream(thumbnailResult);
            }
        }

        public string GenerateThumbnailUrl(int width, int height, MediaItem mediaItem)
        {
            var streamReader = MediaManager.GetMedia(mediaItem).GetStream();
            {
                using (var memStream = new MemoryStream())
                {
                    streamReader.Stream.CopyTo(memStream);

                    var thumbnail = _cognitiveServices.GetThumbnail(memStream.ToArray(), width, height);
                    var imreBase64Data = System.Convert.ToBase64String(thumbnail);

                    return $"data:image/png;base64,{imreBase64Data}";
                }
            }
        }

        private Stream CropImage(Image source, int x, int y, int width, int height)
        {
            var bmp = new Bitmap(width, height);
            var outputStrm = new MemoryStream();

            using (var gr = Graphics.FromImage(bmp))
            {
                gr.InterpolationMode = InterpolationMode.HighQualityBicubic;
                using (var wrapMode = new ImageAttributes())
                {
                    wrapMode.SetWrapMode(WrapMode.TileFlipXY);
                    gr.DrawImage(source, new Rectangle(0, 0, bmp.Width, bmp.Height), x, y, width, height, GraphicsUnit.Pixel, wrapMode);
                }
            }

            bmp.Save(outputStrm, source.RawFormat);

            return outputStrm;
        }
    }
}

Let’s see this in action!

After picking your picture in the AI Cropping Image field, it gets already cropped and you can see the different thumbnails. You can choose or change the thumbnails by updating the child items here: /sitecore/system/Settings/Foundation/Vision/Thumbnails.

Also note that you get an auto generated Alt text “Diego Maradona holding a ball” and a list of tags.

AI Cropping Image Field
AI Cropping Image Field

The results

This is how the different cropped images will look like in the front end. Depending on your front end implementation, you will define different cropping sizes per breakpoints.

In this following implementation, I’m setting the image as a background and using the option to render the image URL as follows:

<div class="heroBanner__backgroundWrapper">
    <div v-animate-on-inview="{class: 'animateScaleOut', delay: 10}"
         v-animate-on-scroll="{class: 'animateOverlay'}"
         class="heroBanner__background @Model.HeroClass" role="img" aria-label="@Model.GlassModel.ProductHeroImage.Alt"
         v-background="{
                        '0': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 600, Height = 600, OnlyUrl = true})',
                        '360': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 900, Height = 900, OnlyUrl = true})',
                        '720': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 1667, Height = 750, OnlyUrl = true})',
                        '1280': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 2000, Height = 900, OnlyUrl = true})'
                        }">
    </div>
Tablet Version
Tablet
Desktop Version
Desktop
Mobile Version
Mobile

Sitecore Media Cache

As I mentioned before, the cropped images are also stored in the media cache, as we can confirm by checking the media cache folder

The Sitecore media cache

Other usages and helpers

Sitecore HTML helper

You can use the @Sitecore.Html helper to render an image tag, as usual, or to generate just the URL of the image (src).

Code
@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new AdvancedImageParameters { Width = 600, Height = 600, AutoAltText = true })
Result
<img alt="a close up of a person wearing glasses"
 src="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=600&h=600&smartCropping=true&hash=C2E215FE2CF74D4C8142E35619ABB8DE">

Note: Have a look at the AdvancedImageParameters:

  • OnlyUrl: If true it will just render the image URL (for being used as src in the img tag).
  • AutoAltText: If true, the alt text will be replaced by the one generated from Azure IA.
  • Width and Height: int values, to specify the cropping size.
  • Widths and Sizes: If set, it will generate a srcset image with for the different breakpoints.
  • SizesTag and SrcSetTag: Those are mandatories if when using the previous settings.
Code
@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new 
AdvancedImageParameters {Widths = "170,233,340,466", Sizes = "50vw,(min-width: 
999px) 25vw,(min-width: 1200px) 15vw", SizesTag = "data-sizes", SrcSetTag = "data-
srcset", AutoAltText = true })
Result
<img alt="a close up of a person wearing glasses" data-sizes="50vw,(min-width: 
999px) 25vw,(min-width: 1200px) 15vw" data-
srcset="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=170&hash=1D04C1F551E9606AB2EEB3C712255651 
170w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=233&hash=DD2844D340246D3CF8AEBB63CE4E9397 
233w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=340&hash=3B773ACB5136214979A0009E24F25F02 
340w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=466&hash=424F7615FBECFED21F48DA0AE1FE7A5B 466w" 
src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==">

GlassMapper extension

At last, an extension method has been added in order to get the media URL from the image field.

Code
<img src="@Model.AiImage.GetImageUrl(600, 600)" />
Result
<img src="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=600&h=600&smartCropping=true&hash=C2E215FE2CF74D4C8142E35619ABB8DE">

Sitecore Package and code

Please find the whole implementation in my GitHub repo, also feel free to contribute 🙂

You can also download the Sitecore package from here. Note: It has been tested on Sitecore 8.2, 9.x and 10.x.

You can also get the Docker asset image from Docker Hub!

docker pull miguelminoldo/sitecore.computer.vision

That’s it! I hope you find it interesting and useful! Any feedback is always welcome!

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision – Part II.

In my previous post I’ve shared a quick overview on the Azure Computer Vision API service and it’s implementation. If you didn’t read it yet, please do before proceeding to this reading!

With the basics and the CognitiveServices in place, let’s move forward and create a custom image field that uses this service to handle the image cropping, tagging and alt text description, all with AI.

I’ll be sharing the whole implementation in GitHub later and also a package plugin, but let’s get into the implementation details first.

Custom Image Field

The first step is to create the custom field, for doing that, go to the core DB and duplicate the /sitecore/system/Field types/Simple Types/Image field item. Let’s call it “AICroppedImage“.

Keep everything as it is except the assembly and class fields

AICroppedImage Class

For the implementation, we just decompiled the code from Sitecore.Kernel (Sitecore.Shell.Applications.ContentEditor.Image) and made all our needed customizations.

using Sitecore.Configuration;
using Sitecore.Data.Items;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;
using Sitecore.Globalization;
using Sitecore.Resources.Media;
using Sitecore.Shell.Applications.ContentEditor;
using Sitecore.Web.UI.Sheer;
using System;
using System.IO;
using System.Text;
using System.Web;
using System.Web.UI;
using Microsoft.Extensions.DependencyInjection;
using System.Linq;
using Sitecore.Computer.Vision.CroppingImageField.Models.ImagesDetails;
using Sitecore.Computer.Vision.CroppingImageField.Services;
namespace Sitecore.Computer.Vision.CroppingImageField.Fields
{
    public class AICroppedImage : Image
    {
        private readonly string ThumbnailsId = Settings.GetSetting("Sitecore.Computer.Vision.CroppingImageField.AICroppingField.ThumbnailsFolderId");
        private readonly ICognitiveServices _cognitiveServices;
        private readonly ICroppingService _croppingService;
       
        public AICroppedImage(ICognitiveServices cognitiveServices, ICroppingService croppingService) : base()
        {
            _cognitiveServices = cognitiveServices;
            _croppingService = croppingService;
        }
        public AICroppedImage() : base()
        {
            _cognitiveServices = ServiceLocator.ServiceProvider.GetService<ICognitiveServices>();
            _croppingService = ServiceLocator.ServiceProvider.GetService<ICroppingService>();
        }
        protected override void DoRender(HtmlTextWriter output)
        {
            Assert.ArgumentNotNull((object)output, nameof(output));
            Item mediaItem = this.GetMediaItem();
            string src;
            this.GetSrc(out src);
            string str1 = " src=\"" + src + "\"";
            string str2 = " id=\"" + this.ID + "_image\"";
            string str3 = " alt=\"" + (mediaItem != null ? HttpUtility.HtmlEncode(mediaItem["Alt"]) : string.Empty) + "\"";
            this.Attributes["placeholder"] = Translate.Text(this.Placeholder);
            string str = this.Password ? " type=\"password\"" : (this.Hidden ? " type=\"hidden\"" : "");
            this.SetWidthAndHeightStyle();
            output.Write("<input" + this.ControlAttributes + str + ">");
            this.RenderChildren(output);
            output.Write("<div id=\"" + this.ID + "_pane\" class=\"scContentControlImagePane\">");
            string clientEvent = Sitecore.Context.ClientPage.GetClientEvent(this.ID + ".Browse");
            output.Write("<div class=\"scContentControlImageImage\" onclick=\"" + clientEvent + "\">");
            output.Write("<iframe" + str2 + str1 + str3 + " frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" width=\"100%\" height=\"128\" " +
                "allowtransparency=\"allowtransparency\"></iframe>");
            output.Write("<div id=\"" + this.ID + "_thumbnails\">");
            output.Write(GetThumbnails());
            output.Write("</div>");
            output.Write("</div>");
            output.Write("<div>");
            output.Write("<div id=\"" + this.ID + "_details\" class=\"scContentControlImageDetails\">");
            string details = this.GetDetails();
            output.Write(details);
            output.Write("</div>");
            output.Write("</div>");
        }
        protected override void DoChange(Message message)
        {
            Assert.ArgumentNotNull((object)message, nameof(message));
            base.DoChange(message);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
            if (string.IsNullOrEmpty(this.Value))
            {
                this.ClearImage();
            }
            SheerResponse.SetReturnValue(true);
        }
        protected new void BrowseImage(ClientPipelineArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            base.BrowseImage(args);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
        }
        protected new void ShowProperties(ClientPipelineArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            base.ShowProperties(args);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
        }
        public override void HandleMessage(Message message)
        {
            Assert.ArgumentNotNull((object)message, nameof(message));
            base.HandleMessage(message);
           
            string name = message.Name;
            if (name == "contentimage:clear")
            {
                this.ClearImage();
            }
            else if (name == "contentimage:refresh")
            {
                this.Update();
            }
        }
        private void ClearImage()
        {
            if (this.Disabled)
            {
                return;
            }
            if (this.Value.Length > 0)
            {
                this.SetModified();
            }
            this.XmlValue = new XmlValue(string.Empty, "image");
            this.Value = string.Empty;
            this.Update();
        }
        protected new void Update()
        {
            string src;
            this.GetSrc(out src);
            SheerResponse.SetAttribute(this.ID + "_image", "src", src);
            SheerResponse.SetInnerHtml(this.ID + "_thumbnails", this.GetThumbnails());
            SheerResponse.SetInnerHtml(this.ID + "_details", this.GetDetails());
            SheerResponse.Eval("scContent.startValidators()");
        }
        private string GetDetails()
        {
            var empty = string.Empty;
            MediaItem mediaItem = this.GetMediaItem();
            if (mediaItem != null)
            {
                var innerItem = mediaItem.InnerItem;
                var stringBuilder = new StringBuilder();
                var xmlValue = this.XmlValue;
                stringBuilder.Append("<div>");
                var item = innerItem["Dimensions"];
                var str = HttpUtility.HtmlEncode(xmlValue.GetAttribute("width"));
                var str1 = HttpUtility.HtmlEncode(xmlValue.GetAttribute("height"));
                ImageDetails imageDetails;
                using (var streamReader = new MemoryStream())
                {
                    var mediaStrm = mediaItem.GetMediaStream();
                    mediaStrm.CopyTo(streamReader);
                    imageDetails = _cognitiveServices.AnalyzeImage(streamReader.ToArray());
                }
                if (!string.IsNullOrEmpty(str) || !string.IsNullOrEmpty(str1))
                {
                    var objArray = new object[] { str, str1, item };
                    stringBuilder.Append(Translate.Text("Dimensions: {0} x {1} (Original: {2})", objArray));
                }
                else
                {
                    var objArray1 = new object[] { item };
                    stringBuilder.Append(Translate.Text("Dimensions: {0}", objArray1));
                }
                stringBuilder.Append("</div>");
                stringBuilder.Append("<div style=\"padding:2px 0px 0px 0px; text-align=left; \">");
                var str2 = HttpUtility.HtmlEncode(innerItem["Alt"]);
                var str3 = imageDetails.Description.Captions.FirstOrDefault()?.Text;
                if (!string.IsNullOrEmpty(str3) && !string.IsNullOrEmpty(str2))
                {
                    var objArray2 = new object[] { str3, str2 };
                    stringBuilder.Append(Translate.Text("AI Alternate Text: \"{0}\" (Default Alternate Text: \"{1}\")", objArray2));
                }
                else if (!string.IsNullOrEmpty(str3))
                {
                    var objArray3 = new object[] { str3 };
                    stringBuilder.Append(Translate.Text("AI Alternate Text: \"{0}\"", objArray3));
                }
                else
                {
                    var objArray4 = new object[] { str2 };
                    stringBuilder.Append(Translate.Text("Default Alternate Text: \"{0}\"", objArray4));
                }
                stringBuilder.Append("</br>");
                var objArray5 = new object[] { str3 };
                stringBuilder.Append(Translate.Text("Tags: \"{0}\"", string.Join(",", imageDetails.Description.Tags), objArray5));
                stringBuilder.Append("</div>");
                empty = stringBuilder.ToString();
            }
            if (empty.Length == 0)
            {
                empty = Translate.Text("This media item has no details.");
            }
            return empty;
        }
        private Item GetMediaItem()
        {
            var attribute = this.XmlValue.GetAttribute("mediaid");
            if (attribute.Length <= 0)
            {
                return null;
            }
            Language language = Language.Parse(this.ItemLanguage);
            return Sitecore.Client.ContentDatabase.GetItem(attribute, language);
        }
        private MediaItem GetSrc(out string src)
        {
            src = string.Empty;
            MediaItem mediaItem = (MediaItem)this.GetMediaItem();
            if (mediaItem == null)
            {
                return null;
            }
            var thumbnailOptions = MediaUrlOptions.GetThumbnailOptions(mediaItem);
            int result;
            if (!int.TryParse(mediaItem.InnerItem["Height"], out result))
            {
                result = 128;
            }
            thumbnailOptions.Height = Math.Min(128, result);
            thumbnailOptions.MaxWidth = 640;
            thumbnailOptions.UseDefaultIcon = true;
            src = MediaManager.GetMediaUrl(mediaItem, thumbnailOptions);
            return mediaItem;
        }
        private string GetThumbnails()
        {
            var html = new StringBuilder();
            var src = string.Empty;
            var mediaItem = this.GetSrc(out src);
            if (mediaItem == null)
            {
                return string.Empty;
            }
            html.Append("<ul id=" + this.ID + "_frame\" style=\"display: -ms-flexbox;display: flex;-ms-flex-direction: row;flex-direction: row;-ms-flex-wrap: wrap;flex-wrap: wrap;\">");
            var thumbnailFolderItem = Sitecore.Client.ContentDatabase.GetItem(new Sitecore.Data.ID(ThumbnailsId));
            if (thumbnailFolderItem != null && thumbnailFolderItem.HasChildren)
            {
                foreach (Item item in thumbnailFolderItem.Children)
                {
                    GetThumbnailHtml(item, html, mediaItem);
                }
            }
           
            html.Append("</ul>");
            return html.ToString();
        }
        private void GetThumbnailHtml(Item item, StringBuilder html, MediaItem mediaItem)
        {
            if (item.Fields["Size"]?.Value != null)
            {
                var values = item.Fields["Size"].Value.Split('x');
                var width = values[0];
                var height = values[1];
                int w, h;
                if (int.TryParse(width, out w) && Int32.TryParse(height, out h) && w > 0 && h > 0)
                {
                    var imageSrc = _croppingService.GenerateThumbnailUrl(w, h, mediaItem);
                    html.Append(string.Format("<li id=\"Frame_{0}_{1}\" style=\"width: {2}px; height: {3}px; position: relative; overflow: hidden; display: inline-block;border: solid 3px #fff;margin: 5px 5px 5px 0;\">" +
                        "<img style=\"position: relative;position: absolute;left: 0;top: 0;margin: 0;display: block;width: auto; height: auto;min-width: 100%; min-height: 100%;max-height: none; max-width: none;\" " +
                        "src=\"{4}\"><img /><span style=\"position: absolute;" +
                        "top: 0;left: 0;padding: 2px 3px;background-color: #fff;opacity: 0.8;\">{5}</span></li>", this.ID, item.ID.ToShortID(), w, h, imageSrc, item.DisplayName));
                }
            }
        }
    }
}

We’re basically modifying the way Sitecore renders the field with some small customizations, basically to add the thumbnails generated by the Azure Cognitive service and also the Alt and Tags texts.

Ok, so that’s very much it, let’s deploy our code and see how it looks in the Sitecore Content Editor. The only thing you need to do next, is create a template and make use of the newly created “AI Cropped Image” field.

Et Voila! The image field is now rendering a few thumbnails that gives you an idea of the final results when rendering the image in the front-end. As you can see, it gives also some tags and a description (“Diego Maradona holding a ball”) used as alt text, everything coming from the Azure AI service, awesome!

Make the field rendered to work as an OOTB Sitecore image field

Next step, is to make sure we can still using the Sitecore helpers for rendering this field. For making this possible, we want to customize the Sitecore.Pipelines.RenderField.GetImageFieldValue processor. Same as before, we decompile the OOTB code from Sitecore.Kernel and we make our updates there. Then just patch the config like that:

<pipelines>
  <renderField>
    <processor patch:after="processor[@type='Sitecore.Pipelines.RenderField.GetImageFieldValue, Sitecore.Kernel']"
                   type="Foundation.Vision.Pipelines.RenderAICroppingImageField, Foundation.Vision">
    </processor>
  </renderField>
</pipelines>

Here, we just need to add the newly created field type (AI Cropped Image) as a valid image field type by overriding the IsImage() method.

using Sitecore.Diagnostics;
using Sitecore.Pipelines.RenderField;
namespace Sitecore.Computer.Vision.CroppingImageField.Pipelines
{
    public class RenderAICroppingImageField : GetImageFieldValue
    {
        public override void Process(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            if (!this.IsImage(args))
            {
                return;
            }
            var renderer = this.CreateRenderer();
            this.ConfigureRenderer(args, renderer);
            this.SetRenderFieldResult(renderer.Render(), args);
        }
        protected override bool IsImage(RenderFieldArgs args)
        {
            return args.FieldTypeKey == "AI Cropped Image";
        }
    }
}

Make it working with GlassMapper

Now, we can do some quick updates to GlassMapper as well so we can benefit from the glass helpers. Let’s add a custom field mapper, again after decompiling Glass.Mapper.Sc.DataMappers.SitecoreFieldImageMapper, we can just extend it to work in the same way with the newly introduced AI Cropping Image field.

using Glass.Mapper.Sc;
using Glass.Mapper.Sc.Configuration;
using Glass.Mapper.Sc.DataMappers;
using Sitecore.Data;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
using System;
using Sitecore.Computer.Vision.CroppingImageField.Fields;
namespace Sitecore.Computer.Vision.CroppingImageField.Mappers
{
    public class AICroppedImageFieldMapper : AbstractSitecoreFieldMapper
    {
        public AICroppedImageFieldMapper(): base(typeof(AICroppedImage))
        {
        }
        public override object GetField(Field field, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var img = new AICroppedImage();
            var sitecoreImage = new AICroppedImageField(field);
            SitecoreFieldImageMapper.MapToImage(img, sitecoreImage);
            return img;
        }
        public override void SetField(Field field, object value, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var img = value as AICroppedImage;
            if (field == null || img == null)
            {
                return;
            }
            var item = field.Item;
            var sitecoreImage = new AICroppedImageField(field);
            SitecoreFieldImageMapper.MapToField(sitecoreImage, img, item);
        }
        public override string SetFieldValue(object value, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            throw new NotImplementedException();
        }
        public override object GetFieldValue(string fieldValue, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var item = context.Service.Database.GetItem(new ID(fieldValue));
            if (item == null)
            {
                return null;
            }
            var imageItem = new MediaItem(item);
            var image = new AICroppedImage();
            SitecoreFieldImageMapper.MapToImage(image, imageItem);
            return image;
        }
    }
}

We need also to create our custom field that inherits from Glass.Mapper.Sc.Fields.Image

using Glass.Mapper.Sc.Fields;
namespace Sitecore.Computer.Vision.CroppingImageField.Mappers
{
    public class AICroppedImage : Image
    {
    }
}

Last step is to add the mapper to the create resolver from the GlassMapperSCCustom.cs

public static  class GlassMapperScCustom
{
    public static IDependencyResolver CreateResolver(){
        var config = new Glass.Mapper.Sc.Config();
        var dependencyResolver = new DependencyResolver(config);
        // add any changes to the standard resolver here
        dependencyResolver.DataMapperFactory.First(() => new AICroppedImageFieldMapper());
        dependencyResolver.Finalise();
			
	return dependencyResolver;
    }
}

Custom Caching

In order to reduce the calls to the service, an extra layer of caching has been implemented. This cache, as any other Sitecore cache gets flushed after a publishing and the size can be easily configured through it’s configuration.

ComputerVision[CroppedImages] custom cache
...
   <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.CacheSize" value="100MB" />
    </settings>
    <events>
      <event name="publish:end:remote">
        <handler type="Sitecore.Computer.Vision.CroppingImageField.Caching.CacheManager, Sitecore.Computer.Vision.CroppingImageField" method="ClearCache" />
      </event>
      <event name="customCache:rebuild:remote">
        <handler type="Sitecore.Computer.Vision.CroppingImageField.Caching.CacheManager, Sitecore.Computer.Vision.CroppingImageField" method="ClearCache" />
      </event>
    </events>
...

In my next post, I’ll be sharing the front end implementation, the full media request flow and the customizations needed to make it working in your site. Stay tuned!

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision – Part I.

Images are nowadays a critical part on websites, specially with “mobile first” approach and responsive designs. Cropping images in a proper way is extremely important if you don’t want to destroy your website’s user experience. Imagine an e-commerce website offering a product that is not visible anymore when the user is browsing the site from a mobile device. :facepalm:

In this post I’ll share a way to solve this issue with the help of AI, more specifically using Azure Cognitive Services (Computer Vision).

Azure Computer Vision

The Computer Vision API provides state-of-the-art algorithms to process images and return information. For example, it can be used to determine if an image contains mature content, or it can be used to find all the faces in an image. It also has other features like estimating dominant and accent colors, categorizing the content of images, and describing an image with complete English sentences. Additionally, it can also intelligently generate images thumbnails for displaying large images effectively. For more details about the API, refer to the official documentation here. It gives also some some good examples in C#.

First Step: Create the Azure resource

Before being able to play with this awesome service, we’ve to create the resource, and good news: the free plan would be enough for your tests (20 calls/min – 5K calls/month):

Login to the Azure portal, and go to add a new resource, search for “Computer Vision” and as usual, follow the wizard in order to create it.

Then just go to the “Keys and Endpoint” section and get your key, endpoint and location. Let’s write those down, we’ll use later to connect to our API.

For this implementation I’ll be using the following methods:

  • Analyze Image: This operation extracts a rich set of visual features based on the image content.
  • Get Area of Interest: This operation returns a bounding box around the most important area of the image.
  • Get Thumbnail: This operation generates a thumbnail image with the user-specified width and height. By default, the service analyzes the image, identifies the region of interest (ROI), and generates smart cropping coordinates based on the ROI. Smart cropping helps when you specify an aspect ratio that differs from that of the input image.

Testing the endopins

We can now use Postman for testing the API endpoints and the results we get. This is very straightforward by following the documentation from MS:

  1. Do a POST or GET (depending on the service you want tot test), to the following URL: https://{yourComputerVisionService}.cognitiveservices.azure.com/vision/v2.0/{APIMethod}?{Params}
  2. Add the needed headers:
    • Ocp-Apim-Subscription-Key: Your app key from the “Keys and Endpoint” previous section.
    • Ocp-Apim-Subscription-Region: Your app region from the “Keys and Endpoint” previous section.
    • Content-Type: application/json
  3. Add the URL of the image in the “Body“.

Let’s do a test with the following image:

Get Thumbnail

As you can see, Computer Vision is retrieving a cropped version of the image by the width/height we passed as parameters (200×200). And it’s cropping in the right way keeping focus in the most important part of the picture.

Get Area Of Interest

Same as generating the thumbnail, it retrieves the coordinates of the area of interest. As the thumbnail generation has some limitations that I’ll explain later, I’ll be using this method to crop the image.

Analyze Image

Depending on the parameters we send to this method, it will return a lot of different elements after analyzing the image, such as tags, description, brands information, etc. I’ll be using this method for generating tags but also to give an automatic alt text to the image.

Service Implementation

Let’s now implement the API service. As an starting point, we’ve to create a service that will take care of the communication to the Computer Vision API:

The ICognitiveServices Interface:

using Sitecore.Computer.Vision.CroppingImageField.Models.AreaOfInterest;
using Sitecore.Computer.Vision.CroppingImageField.Models.ImagesDetails;
namespace Sitecore.Computer.Vision.CroppingImageField.Services
{
    public interface ICognitiveServices
    {
        ImageDetails AnalyzeImage(byte[] image);
        byte[] GetThumbnail(byte[] image, int width, int height);
        AreaOfInterestResult GetAreaOfImportance(byte[] image);
    }
}

The CognitiveServices Class:

using System;
using Newtonsoft.Json;
using Sitecore.Configuration;
using Sitecore.Diagnostics;
using System.Net.Http;
using System.Net.Http.Headers;
using Sitecore.Computer.Vision.CroppingImageField.Models.AreaOfInterest;
using Sitecore.Computer.Vision.CroppingImageField.Models.ImagesDetails;
using Sitecore.Computer.Vision.CroppingImageField.Caching;
using Sitecore.Computer.Vision.CroppingImageField.Extensions;
namespace Sitecore.Computer.Vision.CroppingImageField.Services
{
    public class CognitiveServices : ICognitiveServices
    {
        private readonly string _cognitiveServicesKey = Settings.GetSetting($"Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiKey", "");
        private readonly string _cognitiveServicesUrl = Settings.GetSetting($"Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiUrl", "");
        private readonly string _cognitiveServicesZone = Settings.GetSetting($"Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiZone", "");
        public ImageDetails AnalyzeImage(byte[] image)
        {
            var requestUri = _cognitiveServicesUrl + "analyze?" + Settings.GetSetting(
            $"Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.Analyze.Parameters", "");
            return CacheManager.GetCachedObject(image.GetHashKey() + requestUri, () =>
            {
                using (var response = this.CallApi(image, requestUri))
                {
                    if (response.IsSuccessStatusCode)
                    {
                        var result = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                        var responeData =
                            JsonConvert.DeserializeObject<ImageDetails>(result, new JsonSerializerSettings());
                        return responeData;
                    }
                    var errorMessage = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                    Log.Error(errorMessage, this);
                    return null;
                }
            });
        }
        public byte[] GetThumbnail(byte[] image, int width, int height)
        {
            var requestUri = _cognitiveServicesUrl +
                $"generateThumbnail?width={width}&height={height}&{Constants.QueryStringKeys.SmartCropping}=true";
            return CacheManager.GetCachedObject(image.GetHashKey() + requestUri, () =>
            {
                using (var response = this.CallApi(image, requestUri))
                {
                    if (response.IsSuccessStatusCode)
                    {
                        return response.Content.ReadAsByteArrayAsync().GetAwaiter().GetResult();
                    }
                    var errorMessage = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                    Log.Error(errorMessage, this);
                    return null;
                }
            });
        }
        public AreaOfInterestResult GetAreaOfImportance(byte[] image)
        {
            var requestUri = _cognitiveServicesUrl + "areaOfInterest";
            return CacheManager.GetCachedObject(image.GetHashKey() + requestUri, () =>
            {
                using (var response = this.CallApi(image, requestUri))
                {
                    if (response.IsSuccessStatusCode)
                    {
                        var result = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                        var responeData = JsonConvert.DeserializeObject<AreaOfInterestResult>(result, new JsonSerializerSettings());
                        return responeData;
                    }
                    var errorMessage = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                    Log.Error(errorMessage, this);
                    return null;
                }
            });
        }
        private HttpResponseMessage CallApi(byte[] image, string requestUri)
        {
            using (var client = new HttpClient())
            {
                client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", _cognitiveServicesKey);
                client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Region", _cognitiveServicesZone);
                using (var content = new ByteArrayContent(image))
                {
                    content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
                    return client.PostAsync(requestUri, content).GetAwaiter().GetResult();
                }
            }
        }
    }
}

The Config file:

      <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.ThumbnailsFolderId" value="{C3EC5BF1-2182-40AB-AEE7-B2AE3C292620}" />
      <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiKey" value="{YOUR_APP_KEY}" />
      <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiUrl" value="https://{YOUR_AZURE_SERVICE_URL}.cognitiveservices.azure.com/vision/v2.0/" />
      <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.ApiZone" value="{YOUR_ZONE}" />
      <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.Analyze.Parameters" value="visualFeatures=Brands,Categories,Description,Faces,Objects,Tags" />

So, now we have our Azure Computer Vision resource created, our code is ready and we can connect and play with it.

In the next post, I’ll be creating a custom Sitecore image field that makes use of this implementation to solves the cropping issues and also adds the alt text automatically generated to the image. I’ll be sharing the code in GitHub but also a plugin package, stay tuned!

Remote debugging your Sitecore instance running on Azure Web Apps

If you’re working on a Sitecore project that is hosted in Azure PaaS, surely you have found yourself in situations where there is a bug in production or UAT that you cannot easily reproduce in your local environment. Adding logging can also help in such situation, but sometimes this is not enough to troubleshoot your issue. So, debugging your Azure webapp instance seems to be the best and quick option to go and fix your bug!

First step, connect to Cloud Explorer

In Visual Studio, go to “main menu” -> “view” -> “Cloud Explorer“. You’ll get a list of resources that you have access within your Azure subscription. Search for the “App Services” section and choose the webApp you want to attach the debugger to. Note: choose the group by “Resource Types” option to make it easier to navigate.

You’ll get there the list of different resources types, this time expand the “App Services” as this is where your Sitecore webApp should show up.

Before debugging

In order to be able to debug our code, of course we need our dll to be compiled in “debug” mode. So, as the deployed instance must be compiled in “release” mode so you have to first override the dll you want to attach the debugger to.

Make sure you’re doing it with the same code base version that is deployed into the server you’re going to test.

We’ll also need to have the debugging symbols (*.pdb file) in place, so basically if you are debugging a feature called “MyProject.Feature.Navigation“, then make sure you move those files to your webApp (“/bin” folder):

  • MyProject.Feature.Navigation.dll (debug)
  • MyProject.Feature.Navigation.pdb

Note: you can use the same Cloud Explorer to copy those files to the server!

Web App settings

In order to make it working, make sure that the remote debugging is enabled on your web app.

Go to the configuration sections and then “General Settings”:

It’s very simple, just enable it and choose the VS version you’ll be using for debugging.

Let’s debug!

So, as you can see the process is easy and very straightforward, there are several other ways for remote debugging a webApp, but the one I’m explaining here is the easiest one.

Your setup is already done, you can now add a breakpoint to the code you want to debug, (make sure you’re in the same code version that you built in debug mode in the previous step, otherwise you’ll get the message saying the breakpoint cannot be attached).

So, that’s it! You should now be able to troubleshoot and fix your issue on an Azure webApp.

Thanks for your reading!

Sitecore server side performance tweaks, tips and quick wins!

Performance is one of the key points when it comes to websites, please find here some quick wins I’ve learnt while dealing with server side performance on my projects.

Sitecore Caching

Yes, this is an obvious one, but is also the first place to start. So, first thing is to make sure you have a proper caching strategy.

I’m not deeping into sitecore cache configuration, there is a lot in the internet and also official documentation:

Custom Caching

Great! But, don’t cache that much! Take care about cache sizes!

Caching is a good strategy to improve the website performance and CPU usage, but be careful! you might be ending on moving the issue form CPU to Memory. This is something I noticed a lot on some Sitecore implementations. Also, if the cache size is too big, and the dictionary contains a huge number ok keys, then the processes to resolve the key or to clean up the caches will also impact on the performance.

There are a lot of posts on google explaining how to extend Sitecore with your custom cache.

DisableCacheSizeLimits?

Sitecore provides this setting that removes the cache limitation giving the full memory size from the server to each cache. This is not a good option, even though Sitecore recommends to disable cache limits when running with enough RAM (>16GB), like an P3V2 plan in Azure, I encountered some issues, like the one mentioned before, or also some bad HTML caching configuration that eats a lot of memory ending up on a server recycle due to high memory consumption as the caches are never scavenged and grow uncontrollably.

The cache key indexing

Cache key indexing can significantly reduce the time it takes to perform operations on a large cache, and is particularly useful in large solutions where items are frequently renamed, moved, copied, and deleted. Cache key indexing is available for the following caches:

  • Access Result cache
  • Item cache
  • Item paths cache
  • Paths cache

To enable those, we can apply this config patch:

<configuration
    xmlns:patch="http://www.sitecore.net/xmlconfig/" 
    xmlns:set="http://www.sitecore.net/xmlconfig/set/">
    <sitecore>
        <settings>
            <setting name="Caching.CacheKeyIndexingEnabled.AccessResultCache" set:value="true"/>
            <setting name="Caching.CacheKeyIndexingEnabled.ItemCache" set:value="true"/>
            <setting name="Caching.CacheKeyIndexingEnabled.ItemPathsCache" set:value="true"/>
            <setting name="Caching.CacheKeyIndexingEnabled.PathsCache" set:value="true"/>
        </settings>
    </sitecore>
</configuration>

The AccessResults Cache

This particular cache is really interesting when it comes to performance analysis. I had cases where this cache was increasing exponentially during the load tests, no matter how much memory you assign to it, it will reach the limit at some point. Also, the log is flooded by messages saying this cache is getting flushed.

But, what is this cache for? Each and every time a user requests an item, it resolves the security rights and then stores it in this cache. So, first question is, has our website an account feature? Do we have restricted items? If the answer is no, we can gain 20/30% performance by disabling security access on the web DB:

<database id="web">
    <securityEnabled>false</securityEnabled>
    ...
</database>

Check logs

An obvious but important thing to mention is to check logs. Before deep dive into your code and to find any potential bottleneck, make sure that your application is running smoothly, and the logs are empty, no errors nor exceptions. If you have any, first thing to do is to fix any issue in your code.

Disable WebDAV

We can gain some performance disabling WebDAV from our CD servers and, if we are not making use of it, from the CM as well. Sitecore recommends disabling WebDAV on the production content delivery servers to reduce the number of log files being created. Also, Sitecore recommends disabling WebDAV on the content management servers if the WebDAV functionality is not being used.

Indexing

This is also a quick win, seems to be another obvious thing, but I still finding it a lot. Normally the indexes are maintained on the CM, this means the server that has the responsibility of updating those is the CM, but sometimes the index configuration is not proper and we’re using CD resources to update the indexes. Disable it if you are not doing it in purpose for any specific reason.

You just need to patch the update strategy on the CD servers:

<index id="sitecore_index" type="Sitecore.ContentSearch.SolrProvider.SolrSearchIndex, 
 Sitecore.ContentSearch.SolrProvider">
 <param desc="name">$(id)</param>
 <param desc="core">$(id)</param>
 <param desc="propertyStore"
 ref="contentSearch/indexConfigurations/databasePropertyStore"
 param1="$(id)" />
 <strategies hint="list:AddStrategy">
 <strategy ref="contentSearch/indexConfigurations/indexUpdateStrategies/manual" />
 </strategies>

Disable Performance Counters

Performance counters are useful when troubleshooting performance, but also generate some overhead, so disable those until you those:

<setting name="Counters.Enabled" value="false" />

Disable Bucket Debug

If bucket debugging is enabled, every query executed against an item bucket is written to the search log file. Sitecore recommends disabling bucket debug logging on content delivery servers to reduce log file size, mitigate pollution of the log files, and help minimize any overhead
required to write and maintain log files.

<setting name="BucketConfiguration.EnableBucketDebug" value="false" />

Disable Memory Monitor

Memory Monitor creates frequent CPU spikes and should only be enabled when troubleshooting memory related issues.

Sitecore recommends disabling the Memory Monitor in production environments, and only enabling it for troubleshooting memory related issues.

<hooks>
 <hook type="Sitecore.Diagnostics.HealthMonitorHook, Sitecore.Kernel" />
 <!--<hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
 <param desc="Threshold">800MB</param>
 <param desc="Check interval">00:00:05</param>
 <param desc="Minimum time between log entries">00:01:00</param>
 <ClearCaches>false</ClearCaches>
 <GarbageCollect>false</GarbageCollect>
 <AdjustLoadFactor>false</AdjustLoadFactor>
 </hook>-->
 </hooks>

The next steps

If after applying all the above your solution still not performing as expected, then is time to go deeper and analyze what’s going on.

I’d suggest to start by running some performance or load tests while profiling your app with Visual Studio or dotTrace, you can have a quick look at this post if you’re running your local instance in Docker containers. Find here a quick script to stress your app with JMeter.

This will point you to the code bottlenecks, this task can take hours but will definitely help you to improve your solution.

Sitecore provides some useful tools that helps the developers on this topic, like the pipelines profiling: https://[yoursite]/sitecore/admin/pipelines.aspx and the “debug mode”.

Those are just some quick actions than can help you on improving performance. Of course there are a lot more to do, troubleshooting performance is a complex, time consuming (sometimes tedious) but funny thing to do. I hope this helps you a bit, let’s make your website blazing fast!

Troubleshooting performance on your containerized Sitecore instances with dotTrace, dotMemory and PerfView

In the following videos I’m showing how to use dotTrace to take a profiling session and how to take a memory dump to analyze and troubleshoot performance issues of your application running in Docker containers.

In my previous post you can find a quick way to get your Sitecore Demo up and running, have a look!

Profile Sitecore running in Docker containers

Getting a memory dump from a container

I hope this helps you on your performance troubleshooting when running Docker containers!

Using Redis as Sitecore custom cache

In this post I’ll share how to use Azure Redis Cache as Sitecore custom cache provider.

Azure Cache for Redis is a fully managed, distributed, in-memory cache that enables high-performance and scalable architectures. You can use it to create cloud or hybrid deployments that handle millions of requests per second at sub-millisecond latency, all with the configuration, security and availability benefits of a managed service. More info here.

The first step is to create the Redis cache in Azure, for this we log in to the Azure Portal and then add a new resource, search for “Azure Cache for Redis” and choose a plan, for this demo I selected a “Basic C1” plan, we can scale it later if needed.

Azure Redis Cache is now deployed and ready to connect to.

The next step is to get the connection string data and add a new entry “redis.sessions” into the connectionstrings.config file:

Now our app is connected to the Redis cache. Let’s now have a look at a custom cache implementation.

We start by creating a cache provider:

[Service(typeof(IRedisCacheProvider), Lifetime = Lifetime.Singleton)]
public class RedisCacheProvider : IRedisCacheProvider
{
    private static readonly Lazy<ConnectionMultiplexer> LazyConnection = new Lazy<ConnectionMultiplexer>(() =>
    {
        var connectionString = ConfigurationManager.ConnectionStrings["redis.sessions"].ConnectionString;
        var options = ConfigurationOptions.Parse(connectionString);

        options.AllowAdmin = true;
        options.SyncTimeout = 60000;
        options.ConnectRetry = 5;

        return ConnectionMultiplexer.Connect(options);
    });

    public static ConnectionMultiplexer Connection => LazyConnection.Value;

    private readonly IDatabase _redisCache;

    public RedisCacheProvider()
    {
        _redisCache = Connection.GetDatabase();
    }

    public IDatabase GetRedisCache()
    {
        return _redisCache;
    }

    public IServer GetServer()
    {
        return Connection.GetServer(Connection.GetEndPoints().FirstOrDefault());
    }
}

Now we need to a create a cache manager, that class will contain all the methods to call the cache and to communicate with Redis:

[Service(typeof(ICacheManager), Lifetime = Lifetime.Singleton)]
public class CacheManager : ICacheManager
{
    private readonly IDatabase _redisCache;
    private readonly IServer _redisServer;

    public CacheManager(IRedisCacheProvider redisCacheProvider)
    {
        _redisCache = redisCacheProvider.GetRedisCache();
        _redisServer = redisCacheProvider.GetServer();
    }

    private static readonly Dictionary<string, object> CacheKeyDictionary = new Dictionary<string, object>();

    public object Get(string key)
    {
        return Get(key, string.Empty);
    }

    public object Get(string key, string site)
    {
        var siteName = string.IsNullOrEmpty(site) ? Context.Site?.Name : site;
        var cacheKey = $"{siteName}{Context.Database?.Name}{Context.Language}{key}";
        var res = _redisCache.StringGet(cacheKey);

        return !string.IsNullOrEmpty(res) ? JsonConvert.DeserializeObject(res) : res;
    }

    public void Set(string key, object value)
    {
        Set(key, value, string.Empty);
    }

    public void Set(string key, object value, string site)
    {
        var siteName = string.IsNullOrEmpty(site) ? Context.Site?.Name : site;
        var cacheKey = $"{siteName}{Context.Database?.Name}{Context.Language}{key}";

        _redisCache.StringSet(cacheKey, JsonConvert.SerializeObject(value));
    }

    public IList<string> GetAllKeys()
    {
        return _redisServer.Keys().Select(k => k.ToString()).ToList();
    }

    public void Remove(string key)
    {
        _redisCache.KeyDelete(key);
    }

    public void ClearCache(object sender, EventArgs args)
    {
        Log.Info($"RedisCache Cache Clearer.", this);

        _redisServer.FlushAllDatabases();

        Log.Info("RedisCache Cache Clearer done.", (object)this);
    }

    public TObj GetCachedObject<TObj>(string cacheKey, Func<TObj> creator) where TObj : class
    {
        return GetCachedObject(cacheKey, creator, string.Empty);
    }

    public TObj GetCachedObject<TObj>(string cacheKey, Func<TObj> creator, string site) where TObj : class
    {
        if (string.IsNullOrEmpty(site))
        {
            site = Context.Site.Name;
        }

        var obj = Get(cacheKey, site) as TObj;

        if (obj == null)
        {
            // get the lock object
            var lockObject = GetCacheLockObject(cacheKey, site);

            try
            {
                lock (lockObject)
                {
                    obj = creator.Invoke();

                    Set(cacheKey, obj);
                }
            }
            finally
            {
                RemoveCacheLockObject(cacheKey, site);
            }
        }

        return obj;
    }

    private object GetCacheLockObject(string cacheKey, string site)
    {
        cacheKey += site;

        lock (CacheKeyDictionary)
        {
            if (!CacheKeyDictionary.ContainsKey(cacheKey))
            {
                CacheKeyDictionary.Add(cacheKey, new object());
            }

            return CacheKeyDictionary[cacheKey];
        }
    }

    private void RemoveCacheLockObject(string cacheKey, string site)
    {
        cacheKey += site;

        lock (CacheKeyDictionary)
        {
            if (CacheKeyDictionary.ContainsKey(cacheKey))
            {
                CacheKeyDictionary.Remove(cacheKey);
            }
        }
    }
}

It’s important to keep in mind that this is a distributed cache, meaning that all Sitecore instances connected to the same cache are sharing it, for example, if we’ve a setup with one CM instance and two CDs, all of those will be sharing the same cache, while in memory cache is specific to the instance. That’s why I’m adding the site name, database and language to the cache key.

Almost done, but now we have to think about one of the most important things when working with caches, when and how to invalidate those.

We can just call the ClearCache() on the publish:end and publish:end:remote events, but I wanted to make it a bit flexible, as the cache is shared across instances is better to keep control on that rather than just flushing everything on each publish action.

I decided to go with a custom event handler approach. Check the config patch, I’m introducing the customCache:rebuild and customCache:rebuild:remote events:

<!--For more information on using transformations see the web.config examples at http://go.microsoft.com/fwlink/?LinkId=214134. -->
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set" xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore>
    <pipelines>
      <initialize>
        <processor type="Foundation.RedisCache.Pipelines.Initialize, Foundation.RedisCache" method="InitializeFromPipeline" />
      </initialize>
    </pipelines>
    <commands>
      <command name="rediscache:cleancache" type="Foundation.RedisCache.Commands.CleanCacheCommand, Foundation.RedisCache" />
    </commands>
    <events xdt:Transform="Insert">
      <event name="customCache:rebuild">
        <handler type="Foundation.RedisCache.Events.EventHandlers.CacheRebuildEventHandler, Foundation.RedisCache" method="OnCustomCacheRebuild" />
      </event>
      <event name="customCache:rebuild:remote">
        <handler type="Foundation.RedisCache.Events.EventHandlers.CacheRebuildEventHandler, Foundation.RedisCache" method="OnCustomCacheRebuild" />
      </event>
    </events>
  </sitecore>
</configuration>

The initialize pipeline:

public class Initialize
{
    /// <summary>
    /// Initializes event subscription
    /// </summary>
    /// <param name="args">Args</param>
    public virtual void InitializeFromPipeline(PipelineArgs args)
    {
        var action = new Action<CacheRebuildEvent>(RaiseRemoteEvent);

        Sitecore.Eventing.EventManager.Subscribe<CacheRebuildEvent>(action);
    }

    /// <summary>
    /// Raises remote event
    /// </summary>
    /// <param name="cacheRebuildEvent"></param>
    private void RaiseRemoteEvent(CacheRebuildEvent cacheRebuildEvent)
    {
        var eventArgs = new object[] { new CacheRebuildEventArgs(cacheRebuildEvent) };

        Sitecore.Events.Event.RaiseEvent(Constants.CustomCacheRebuildEventNameRemote, eventArgs);
    }
}

I’ve also decided to create a simple command that we can just call from the Sitecore ribbon in order to flush this cache manually, this can help in case something get wrong and to avoid the need of manually flushing the redis cache from Azure.

[Serializable]
public class CleanCacheCommand : Sitecore.Shell.Framework.Commands.Command
{
    public override void Execute(Sitecore.Shell.Framework.Commands.CommandContext context)
    {
        var raiser = new CacheRebuildEventRaiser();
        var ev = new CacheRebuildEvent { CacheKey = Constants.ClearAll };

        raiser.RaiseEvent(ev);

        SheerResponse.Alert("Redis Cache flushed");
    }
}

That’s very much it! Let’s see this in action now!

So, to make use of this caching foundation, we just need to inject the ICacheManager and use the GetCachedObject method:

var cacheKey = $"RedisCacheTest-{path}";

            return _cacheManager.GetCachedObject(cacheKey, () =>
            {
                var slowMe = DateTime.Now + TimeSpan.FromSeconds(5);

                while (DateTime.Now < slowMe)
                {
                    //This is just an expensive operation...
                }

                return "/some/url";
            });

Please note that at the end the cache key will be generated by: {Site Name}{Database Name}{Language Name}{RedisCacheTest}-{path}.

Let’s check now the Redis Cache Console in Azure, we can run the command SCAN 0 COUNT 1000 MATCH * to get all keys from the cache:

As you can see the “RedisCacheTest” is there!

Let me take the opportunity to introduce the Redis Cache Visual Code extension, find the details here.

The extension provided a quick and easy way to browse the Redis cache contents,

I hope you find this interesting!

You can find the full code in Github.

Lighthouse Demo is now available in the Sitecore Container Registry, let’s try it!

The Lighthouse Demo joins the list of Docker images available in the Sitecore container registry (SCR). Let’s compose those images and have a look at this Sitecore 10 + SXA showcase!

This post assumes you’re familiar with Docker, you’ve the latest Docker desktop version installed on your Win 10 (1809 or higher) and you have a valid Sitecore license.

Spinning up a Sitecore environment has never been easier thanks to Docker containers.

Please refer to Github for more details about the Lighthouse demo, and find the detailed instructions here.

Let’s go!

Before starting, you just need to clone the repository locally:

  1. Open PowerShell with admin rights and navigate to your repository clone folder: cd C:\Projects\Sitecore.Demo.Platform
  2. Create certificates and initialize the environment file: .\init.ps1 -InitEnv -LicenseXmlPath C:\license\license.xml -AdminPassword b. (You can change the admin password and the license.xml file path to match your needs).
  3. Pull the latest demo Docker images: docker-compose pull
  4. Stop the IIS service: iisreset /stop
  5. Start the demo containers: docker-compose up -d
  6. Check the progress of the initialization by viewing the init container’s logs: docker-compose logs -f init

Troubleshooting errors

  1. If you get the following error

“ERROR: for traefik Cannot start service traefik: failed to create endpoint sitecore-xp0_traefik_1 on network nat: failed during hnsCallRawResponse: hnsCall failed in Win32: The process cannot access the file because it is being used by another process. (0x20)”

This normally means a port that is needed is already in use by another service. Make sure any of those ports are in use: 443, 8079, 8081, 8984, and 14330. Have a look here for more details.

In my case I had the port 8079 in use by Java:

Just by stopping the service fixed the issue. (Also make sure you stopped IIS as I mentioned in the first step).

If the issue still, and any of the needed ports are in use, you can also try this:

Stop-Service docker
Stop-service hns
Start-service hns
Start-Service docker
docker network prune

That’s it!

  • When you’re done with the demo, just stop it! docker-compose stop

I hope you find it interesting and enjoy as I did, a real quick and easy way to get a showcase Sitecore instance running locally in few minutes!