Sitecore smart translation tool with SPE and Azure Cognitive Services (AI)

In my previous posts about images cropping, I’ve used Azure Cognitive Services (Vision) for managing media cropping in a smart way. Now, I’m sharing another usage of Azure Cognitive Services (Language) for building a Powershell tool that makes possible to translate your Sitecore content in a quick and easy way.

Handling item versioning and translation from the Sitecore content editor is a kinda tedious work for editors, especially when it comes to manually creating localized content for your site.

The idea of the PSE tool is to make the editor’s life easier, so in several clicks can achieve the language version creation of the item (including subitems and datasources) and also populate the items with translated content!

Azure Translator – An AI service for real-time text translation

Translator is a cloud-based machine translation service you can use to translate text in near real-time through a simple REST API call. The service uses modern neural machine translation technology and offers statistical machine translation technology. Custom Translator is an extension of Translator, which allows you to build neural translation systems. The customized translation system can be used to translate text with Translator or Microsoft Speech Services. For more info please refer to the official documentation.

About the tool

As I mentioned before, this tool is based on SPE, so it’s easy to integrate on your Sitecore instance. I’ll share the full implementation details but also the code and packages. The service API layer has been implemented on .NET.

The context menu script
Demo

Creating the Azure service

Before proceeding with the implementation, let’s see how to create the Translator service in Azure. The steps are very straightforward as usual when creating such resources.

  • Login to Azure portal (https://portal.azure.com/) and click on create new resource.
  • Search for Translator and finally click on the create button.
Azure Translator Resource
  • Fill the required options and choose a plan. For testing purposes there is a free plan!.
  • Free plan limits: 2M chars of any combination of standard translation and custom training free per month.
  • More details about the available plans here.
Azure Translator Options
  • That’s it! You have your translator service created, now just take a look at the keys and endopint section, you will need it for updating in your config file:
Keys and Endopint

Service implementation (C#)

TranslatorService.cs

This is the service that communicates with the Azure API, it’s quite basic and straightforward, you can also find examples and documentation in the official sites.

using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Sitecore.Cognitive.Translator.PSE.Caching;
using Sitecore.Cognitive.Translator.PSE.Models;
using Sitecore.Configuration;

namespace Sitecore.Cognitive.Translator.PSE.Services
{
    public class TranslatorService : ITranslatorService
    {
        private readonly string _cognitiveServicesKey = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey", "");
        private readonly string _cognitiveServicesUrl = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl", "");
        private readonly string _cognitiveServicesZone = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone", "");

        public async Task<TranslationResult[]> GetTranslatation(string textToTranslate, string fromLang, string targetLanguage, string textType)
        {
            return await CacheManager.GetCachedObject(textToTranslate + fromLang + targetLanguage + textType, async () =>
            {
                var route = $"/translate?api-version=3.0&to={targetLanguage}&suggestedFrom=en";

                if (!string.IsNullOrEmpty(fromLang))
                {
                    route += $"&from={fromLang}";
                }

                if (!string.IsNullOrEmpty(textType) && textType.Equals("Rich Text"))
                {
                    route += "&textType=html";
                }

                var requestUri = _cognitiveServicesUrl + route;
                var translationResult = await TranslateText(requestUri, textToTranslate);

                return translationResult;
            });
        }

        async Task<TranslationResult[]> TranslateText(string requestUri, string inputText)
        {
            var body = new object[] { new { Text = inputText } };
            var requestBody = JsonConvert.SerializeObject(body);

            using (var client = new HttpClient())
            using (var request = new HttpRequestMessage())
            {
                request.Method = HttpMethod.Post;
                request.RequestUri = new Uri(requestUri);
                request.Content = new StringContent(requestBody, Encoding.UTF8, "application/json");
                request.Headers.Add("Ocp-Apim-Subscription-Key", _cognitiveServicesKey);
                request.Headers.Add("Ocp-Apim-Subscription-Region", _cognitiveServicesZone);

                var response = await client.SendAsync(request).ConfigureAwait(false);
                var result = await response.Content.ReadAsStringAsync();
                var deserializedOutput = JsonConvert.DeserializeObject<TranslationResult[]>(result);

                return deserializedOutput;
            }
        }
    }
}

The code is simple, I’m just adding a caching layer on top to avoid repeated calls to the API.

You can check the full parameters list in the official documentation, but let me just explain the ones I used:

  • api-version (required): Version of the API requested by the client. Value must be 3.0.
  • to (required): Specifies the language of the output text. The target language must be one of the supported languages included in the translation scope.
  • from (optional): Specifies the language of the input text. Find which languages are available to translate from by looking up supported languages using the translation scope. If the from parameter is not specified, automatic language detection is applied to determine the source language.
  • textType (optional): Defines whether the text being translated is plain text or HTML text. Any HTML needs to be a well-formed, complete element. Possible values are: plain (default) or html. In this case, I’m passing the HTML when is translating from a Rich Text field.

We need also to create the models where the data is parsed into (TranslationResult), I’m not adding the code here to make it simple, but you can check the source code for full details.

TranslationExtensions.cs

using System.Linq;
using System.Threading.Tasks;
using Sitecore.Cognitive.Translator.PSE.Services;
using Microsoft.Extensions.DependencyInjection;
using Sitecore.DependencyInjection;

namespace Sitecore.Cognitive.Translator.PSE.Extensions
{
    public class TranslationExtensions
    {
        private readonly ITranslatorService _translatorService;

        public TranslationExtensions(ITranslatorService translatorServices)
        {
            _translatorService = translatorServices;
        }

        public TranslationExtensions()
        {
            _translatorService = ServiceLocator.ServiceProvider.GetService<ITranslatorService>();
        }

        public async Task<string> TranslateText(string input, string fromLang, string destLang, string textType)
        {
            var res = await _translatorService.GetTranslatation(input, fromLang, destLang, textType);

            if (res != null && res.Any() && res[0].Translations.Any())
            {
                return res[0].Translations[0].Text;
            }

            return string.Empty;
        }
    }
}

Sitecore.Cognitive.Translator.PSE.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey" value="{YOUR_APP_KEY}" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl" value="https://api.cognitive.microsofttranslator.com/" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone" value="{YOUR_APP_ZONE}" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.CacheSize" value="10MB" />
    </settings>
    <services>
      <configurator type="Sitecore.Cognitive.Translator.PSE.DI.RegisterContainer, Sitecore.Cognitive.Translator.PSE" />
    </services>
    <events>
      <event name="publish:end:remote">
        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
      </event>
      <event name="customCache:rebuild:remote">
        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
      </event>
    </events>
  </sitecore>
</configuration>

Powershell Scripts

We need basically one main script to be added in the context menu (Add Language Version and Translate) and then few functions that has been written in this way to make it more readable and modular.

Add Language Version and Translate

Import-Function GetLanguages
Import-Function GetItems
Import-Function ConfirmationMessage
Import-Function Translate
Import-Function GetUserOptions
Import-Function GetUserFieldsToTranslate
Import-Function ConfirmationMessage

# Global variables
$location = get-location
$currentLanguage = [Sitecore.Context]::Language.Name
$langOptions = @{}
$destinationLanguages = @{}
$options = @{}

# Variables from user input - Custom Object
$userOptions = [PSCustomObject]@{
    'FromLanguage'   = $currentLanguage
    'ToLanguages' = @()
    'IncludeSubitems' = $false
    'IncludeDatasources' = $false
    'IfExists' = "Skip"
    'FieldsToTranslate' = @()
}

# Get language options
GetLanguages $langOptions $destinationLanguages

# Ask user for options
$result = GetUserOptions $currentLanguage $langOptions $destinationLanguages $userOptions
if($result -ne "ok") {
    Write-Host "Canceling"
    Exit
}

# Get all items 
$items = @()
$items = GetItems $location $userOptions.IncludeSubitems $userOptions.IncludeDatasources

# Ask user for fields to translate
$dialogResult = GetUserFieldsToTranslate $items $options $userOptions
if($dialogResult -ne "OK") {
    Write-Host "Canceling"
    Exit
}

# Ask user for confirmation
$proceed = ConfirmationMessage $items.Count $options $userOptions
if ($proceed -ne 'yes') {
    Write-Host "Canceling"
    Exit
}

# Call the translator service 
Translate $items $userOptions

GetLanguages

function GetLanguages {
    [CmdletBinding()]
    param($langOptions, $destinationOptions)
	
	$user = Get-User -Current
	$languages = Get-ChildItem "master:\sitecore\system\Languages"
    $currentLanguage = [Sitecore.Context]::Language.Name
	
	# Get list of languages with writting rights and remove the origin language
    foreach ($lang in $languages) {
        $langOptions[$lang.Name] = $lang.Name    
        if (Test-ItemAcl -Identity $user -Path $lang.Paths.Path -AccessRight language:write) {
            $destinationOptions[$lang.Name] = $lang.Name
        }
    }
    
    $destinationOptions.Remove($currentLanguage)
}

GetUserOptions

function GetUserOptions {
    [CmdletBinding()]
    param($currentLanguage, $langOptions, $destinationLanguages, [PSCustomObject]$userOptions)
     
    # Version overwritting options
    $ifExistsOpts = @{};
    $ifExistsOpts["Append"]    = "Append";
    $ifExistsOpts["Skip"]      = "Skip";
    $ifExistsOpts["Overwrite"] = "OverwriteLatest";

    $result = Read-Variable -Parameters `
        @{ Name = "fLang"; Value=$currentLanguage; Title="From Language"; Options=$langOptions; },
        @{ Name = "tLang"; Title="Destination Languages"; Options=$destinationLanguages; Editor="checklist"; },
        @{ Name = "iSubitems"; Value=$false; Title="Include Subitems"; Columns = 4;},
        @{ Name = "iDatasources"; Value=$false; Title="Include Datasources"; Columns = 4 },
        @{ Name = "iExist"; Value="Skip"; Title="If Language Version Exists"; Options=$ifExistsOpts; Tooltip="Append: Create new language version and translate content.<br>" `
                  + "Skip: skip it if the target has a language version.<br>Overwrite Latest: overwrite latest language version with translated content."; } `
        -Description "Select a the from and target languages with options on how to perform the translation" `
        -Title "Add Language and Translate" -Width 650 -Height 660 -OkButtonName "Proceed" -CancelButtonName "Cancel" -ShowHints
    
    $userOptions.FromLanguage = $fLang
    $userOptions.ToLanguages += $tLang
    $userOptions.IncludeSubitems = $iSubitems
    $userOptions.IncludeDatasources = $iDatasources
    $userOptions.IfExists = $iExist
    
    return $result
}

GetItems

function GetItems {
    [CmdletBinding()]
    param($location, $includeSubitems, $includeDatasources)
 
    Import-Function GetItemDatasources
    
    $items = @()
    $items += Get-Item $location
 
    # add subitems
    if ($includeSubitems) {
        $items += Get-ChildItem $location -Recurse
    }
     
    # add datasources
    if ($includeDatasources) {
        Foreach($item in $items) {
            $items += GetItemDatasources($item)
        }
    }
     
    # Remove any duplicates, based on ID
    $items = $items | Sort-Object -Property 'ID' -Unique
    
    return $items
}

GetFields

function GetFields {
    [CmdletBinding()]
    param($items, $options)

    Import-Function GetTemplatesFields
    
    Foreach($item in $items) {
        $fields += GetTemplatesFields($item)
    }
    
    # Remove any duplicates, based on ID
    $fields = $fields | Sort-Object -Property 'Name' -Unique
    
    # build the hashtable to show as checklist options
    ForEach ($field in $fields) {
    	$options.add($field.Name, $field.ID.ToString())
    }
    
    return $fields 
}

GetItemDatasources

function GetItemDatasources {
    [CmdletBinding()]
    param([Item]$Item)
 
    return Get-Rendering -Item $item -FinalLayout -Device (Get-LayoutDevice -Default) |
        Where-Object { -not [string]::IsNullOrEmpty($_.Datasource)} |
        ForEach-Object { Get-Item "$($item.Database):" -ID $_.Datasource }
}

GetTemplatesFields

function GetTemplatesFields {
    [CmdletBinding()]
    param([Item]$Item)
	
	$standardTemplate = Get-Item -Path "master:" -ID ([Sitecore.TemplateIDs]::StandardTemplate.ToString())
	$standardTemplateTemplateItem = [Sitecore.Data.Items.TemplateItem]$standardTemplate
	$standardFields = $standardTemplateTemplateItem.OwnFields + $standardTemplateTemplateItem.Fields | Select-Object -ExpandProperty key -Unique
	$itemTemplateTemplateItem = Get-ItemTemplate -Item $Item
	$itemTemplateFields = $itemTemplateTemplateItem.OwnFields + $itemTemplateTemplateItem.Fields
	$filterFields = $itemTemplateFields | Where-Object { $standardFields -notcontains $_.Name } | Sort-Object
	
	return $filterFields
}

GetUserFieldsToTranslate

function GetUserFieldsToTranslate {
    [CmdletBinding()]
    param($items, $options, [PSCustomObject]$userOptions)
    Import-Function GetFields
    
    # Get all fields from items
    $fields = @()
    $fields = GetFields $items $options
    
    # Promt the user for selecting the fields for translation
    $dialogParams = @{
        Title = "Fields selector"
        Description = "Select the fields you want to translate"
        OkButtonName = "OK"
        CancelButtonName = "Cancel"
        ShowHints = $true
        Width = 600
        Height = 800
        Parameters = @(
            @{
                Name = "fieldsIdToTranslate"
                Title = "Checklist Selector"
                Editor = "check"
                Options = $options
                Tooltip = "Select one or more fields"
            }
        )
    }
    
    $dialogResult = Read-Variable @dialogParams
    $userOptions.FieldsToTranslate = $fieldsIdToTranslate
    
    return $dialogResult
}

ConfirmationMessage

function ConfirmationMessage {
    [CmdletBinding()]
    param($itemsCount, $options, [PSCustomObject]$userOptions)
    
    $fieldsToUpdate = ""
    $opt = @()
    
    ForEach($ft in $userOptions.FieldsToTranslate) {
        $opt = $options.GetEnumerator() | ? { $_.Value -eq $ft }
        $fieldsToUpdate += "$($opt.Key), "
    }
    
    $fieldsToUpdate = $fieldsToUpdate.Substring(0,$fieldsToUpdate.Length-2)
     
    $message = "Updating <span style='font-weight: bold'>$itemsCount item(s)</span>!<br>"
    $message += "<br><table>"
    $message += "<tr><td style='width: 300px'>Origin Language:</td><td style='width: 450px'>$($userOptions.FromLanguage)</td></tr>"
    $message += "<tr><td style='width: 300px'>Destination Languages:</td><td style='width: 450px'>$($userOptions.ToLanguages)</td></tr>"
    $message += "<tr><td style='width: 300px'>Include Subitems:</td><td style='width: 450px'>$($userOptions.IncludeSubitems)</td></tr>"
    $message += "<tr><td style='width: 300px'>Include Datasources:</td><td style='width: 450px'>$($userOptions.IncludeDatasources)</td></tr>"
    $message += "<tr><td style='width: 300px'>Copy Method:</td><td style='width: 450px'>$($userOptions.IfExists)</td></tr>"
    $message += "<tr><td style='width: 300px'>Fields to Translate:</td><td style='width: 450px'>$($fieldsToUpdate)</td></tr>"
    $message += "</td></tr></table>"
     
    return Show-Confirm -Title $message
}

Translate

function Translate {
    [CmdletBinding()]
    param($items, [PSCustomObject]$userOptions)
    
    Write-Host "Proceeding with execution..."
    
    # Call the translator service
    $translatorService = New-Object Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions
    
    $items | ForEach-Object {
    	$currentItem = $_
    	foreach($lang in $userOptions.ToLanguages) {
    		Add-ItemLanguage $_ -Language $userOptions.FromLanguage -TargetLanguage $lang -IfExist $userOptions.IfExists
    		
    		Write-Host "Item : '$($currentItem.Name)' created in language '$lang'"
    		
    		Get-ItemField -Item $_ -Language $lang -ReturnType Field -Name "*" | ForEach-Object{ 
    		    # Only look within Single-line and Rich Text fields that has been choosen in the dialog box
                if(($_.Type -eq "Single-Line Text" -or $_.Type -eq "Rich Text" -or $_.Type -eq "Multiline Text") -and $userOptions.FieldsToTranslate.Contains($_.ID.ToString())) {
                    if (-not ([string]::IsNullOrEmpty($_))) {
                        # Get the item in the target created language
                        $langItem = Get-Item -Path "master:" -ID $currentItem.ID -Language $lang
        				
        				# Get the translated content from the service
        				$translated = $translatorService.TranslateText($currentItem[$_.Name], $userOptions.FromLanguage, $lang, $_.Type)
        				
        				# edit the item with the translated content
        				$langItem.Editing.BeginEdit()
        				$langItem[$_.Name] = $translated.Result
        				$langItem.Editing.EndEdit()
        				
        				Write-Host "Field : '$_' translated from '$($userOptions.FromLanguage)'" $currentItem[$_.Name] " to : '$lang'" $translated.Result
                    }
                }
    	    }
    	}	 
    }
}

In the Translate function, I’m doing the call to the API (Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions).

That’s very much it, now is time to test it! If everything went well, you will be able to add language versions to your items with also translated content from Azure Cognitive Translation.

Let’s see this in action!

For the purpose of this demo, I’ve created a simple content tree with 3 levels, the items has some content in english (plain and HTML) and I’ll be using the tool to create the Spanish-Argentina and French-France versions + translated content.

1- Click on the Home item and choose the Add Language Version and Translate option from the scripts section.

2- Choose the options, in this case I want to translate from the default ‘en‘ language to both ‘es-AR‘ and ‘fr-FR‘. Also I want to include the subitems, but as for this test the items doesn’t have a presentation nor datasources, I’m keeping this disabled. No versions in the target language exist for those items, so I’m keeping the “Skip” option.

3- Click on proceed and choose the fields you want to translate:

I’m selecting all fields, as you can check in the SPE code, I’m removing the standard fields from the items to be translated, normally you don’t want that and it will overpopulate the fields list.

4- Click OK, double check the data entered and click the OK button for making the magic to happen:

5- Click on the View script results link to check the output logs:

6- Check that the items have been created in the desired languages and the contents are already translated. Review them, publish and have a cup of coffee :).

fr-FR items version:

es-AR items version:

Voila! After few clicks you have your content items created in the language version with the content translated, I hope you like it us much as I do.

Find the source code in GitHub, download the Sitecore package here or get the asset image from Docker Hub.

Thanks for reading!

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision – Part III.

In my previous post I’ve shared the custom image field implementation that makes use of the Azure Computer Vision service in order to crop and generate the thumbnails using AI. Please before proceed with this reading, make sure you already went through the previous posts: Part I and Part II.

Now, I’ll be sharing the last, but not least part of this topic, how to make it working in the front-end side, the media request flow and so on.

Image request flow

Image request flow
The image request flow

So, the request flow is described in the following graph, basically follows the normal Sitecore flow but with the introduction of the Azure Computer Vision and Image Sharp to generate the proper cropping version of the image.

AICroppingProcessor

This custom processor will be overriding the Sitecore OOTB ThumbnailProcessor. It’s basically a copy from the original code with a customization to check the “SmartCropping” parameter from the image request.

using Sitecore.Diagnostics;
using Sitecore.Resources.Media;
using System;
using Microsoft.Extensions.DependencyInjection;
using System.IO;
using System.Linq;
using Sitecore.Computer.Vision.CroppingImageField.Services;
using Sitecore.DependencyInjection;

namespace Sitecore.Computer.Vision.CroppingImageField.Processors
{
    public class AICroppingProcessor
    {
        private static readonly string[] AllowedExtensions = { "bmp", "jpeg", "jpg", "png", "gif" };

        private readonly ICroppingService _croppingService;

        public AICroppingProcessor(ICroppingService croppingService)
        {
            _croppingService = croppingService;
        }

        public AICroppingProcessor()
        {
            _croppingService = ServiceLocator.ServiceProvider.GetService<ICroppingService>();
        }

        public void Process(GetMediaStreamPipelineArgs args)
        {
            Assert.ArgumentNotNull(args, "args");

            var outputStream = args.OutputStream;

            if (outputStream == null)
            {
                return;
            }

            if (!AllowedExtensions.Any(i => i.Equals(args.MediaData.Extension, StringComparison.InvariantCultureIgnoreCase)))
            {
                return;
            }

            var smartCrop = args.Options.CustomOptions[Constants.QueryStringKeys.SmartCropping];

            if (!string.IsNullOrEmpty(smartCrop) && bool.Parse(smartCrop))
            {
                Stream outputStrm;

                outputStrm = Stream.Synchronized(_croppingService.GetCroppedImage(args.Options.Width, args.Options.Height, outputStream.MediaItem));
                args.OutputStream = new MediaStream(outputStrm, args.MediaData.Extension, outputStream.MediaItem);
            }
            else if (args.Options.Thumbnail)
            {
                var transformationOptions = args.Options.GetTransformationOptions();
                var thumbnailStream = args.MediaData.GetThumbnailStream(transformationOptions);

                if (thumbnailStream != null)
                {
                    args.OutputStream = thumbnailStream;
                }
            }
        }
    }
}

We need also to customize the MediaRequest to also take the “SmartCropping” parameter into account:

using Sitecore.Configuration;
using Sitecore.Diagnostics;
using Sitecore.Resources.Media;

using System.Web;

namespace Sitecore.Computer.Vision.CroppingImageField.Requests
{
    using System.Collections.Specialized;

    public class AICroppingMediaRequest : MediaRequest
    {
        private HttpRequest _innerRequest;
        private MediaUrlOptions _mediaQueryString;
        private MediaUri _mediaUri;
        private MediaOptions _options;

        protected override MediaOptions GetOptions()
        {
            var queryString = this.InnerRequest.QueryString;

            if (queryString == null || queryString.Count == 0)
            {
                _options = new MediaOptions();
            }
            else
            {
                SetMediaOptionsFromMediaQueryString(queryString);

                if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
                {
                    SetCustomOptionsFromQueryString(queryString);
                }
            }

            if (!this.IsRawUrlSafe)
            {
                if (Settings.Media.RequestProtection.LoggingEnabled)
                {
                    string urlReferrer = this.GetUrlReferrer();

                    Log.SingleError(string.Format("MediaRequestProtection: An invalid/missing hash value was encountered. " +
                        "The expected hash value: {0}. Media URL: {1}, Referring URL: {2}",
                        HashingUtils.GetAssetUrlHash(this.InnerRequest.RawUrl), this.InnerRequest.RawUrl,
                        string.IsNullOrEmpty(urlReferrer) ? "(empty)" : urlReferrer), this);
                }

                _options = new MediaOptions();
            }

            return _options;
        }

        private void SetCustomOptionsFromQueryString(NameValueCollection queryString)
        {
            this.ProcessCustomParameters(_options);

            if (!string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping))
                    && !_options.CustomOptions.ContainsKey(Constants.QueryStringKeys.SmartCropping)
                    && !string.IsNullOrEmpty(queryString.Get(Constants.QueryStringKeys.SmartCropping)))
            {
                _options.CustomOptions.Add(Constants.QueryStringKeys.SmartCropping, queryString.Get(Constants.QueryStringKeys.SmartCropping));
            }
        }

        private void SetMediaOptionsFromMediaQueryString(NameValueCollection queryString)
        {
            MediaUrlOptions mediaQueryString = this.GetMediaQueryString();

            _options = new MediaOptions()
            {
                AllowStretch = mediaQueryString.AllowStretch,
                BackgroundColor = mediaQueryString.BackgroundColor,
                IgnoreAspectRatio = mediaQueryString.IgnoreAspectRatio,
                Scale = mediaQueryString.Scale,
                Width = mediaQueryString.Width,
                Height = mediaQueryString.Height,
                MaxWidth = mediaQueryString.MaxWidth,
                MaxHeight = mediaQueryString.MaxHeight,
                Thumbnail = mediaQueryString.Thumbnail,
                UseDefaultIcon = mediaQueryString.UseDefaultIcon
            };

            if (mediaQueryString.DisableMediaCache)
            {
                _options.UseMediaCache = false;
            }

            foreach (string allKey in queryString.AllKeys)
            {
                if (allKey != null && queryString[allKey] != null)
                {
                    _options.CustomOptions[allKey] = queryString[allKey];
                }
            }
        }

        public override MediaRequest Clone()
        {
            Assert.IsTrue((base.GetType() == typeof(AICroppingMediaRequest)), "The Clone() method must be overridden to support prototyping.");

            return new AICroppingMediaRequest
            {
                _innerRequest = this._innerRequest,
                _mediaUri = this._mediaUri,
                _options = this._options,
                _mediaQueryString = this._mediaQueryString
            };
        }
    }
}

This code is very straightforward, it will basically check if the “SmartCropping=true” parameter exists in the media request, and then executes the custom code to crop the image.

The “Get Thumbnails” method limitations

As we can see in the official documentation, there are some limitations on the thumbnail generator method.

  • Image file size must be less than 4MB.
  • Image dimensions should be greater than 50 x 50.
  • Width of the thumbnail must be between 1 and 1024.
  • Height of the thumbnail must be between 1 and 1024.

The most important one is that the width and height cannot exceed the 1024px, this is problematic as sometimes we need to crop on a bigger ratio.

So, in order to make it more flexible, I’m doing the cropping using the Graphics library but getting the focus point coordinates from the “Get Area Of Interest” API method:

using Sitecore.Data.Items;
using Microsoft.Extensions.DependencyInjection;
using System.IO;
using Sitecore.DependencyInjection;
using Sitecore.Resources.Media;
using System.Drawing;
using System.Drawing.Imaging;
using System.Drawing.Drawing2D;

namespace Sitecore.Computer.Vision.CroppingImageField.Services
{
    public class CroppingService : ICroppingService
    {
        private readonly ICognitiveServices _cognitiveServices;

        public CroppingService(ICognitiveServices cognitiveServices)
        {
            _cognitiveServices = cognitiveServices;
        }

        public CroppingService()
        {
            _cognitiveServices = ServiceLocator.ServiceProvider.GetService<ICognitiveServices>();
        }

        public Stream GetCroppedImage(int width, int height, MediaItem mediaItem)
        {
            using (var streamReader = new MemoryStream())
            {
                var mediaStrm = mediaItem.GetMediaStream();

                mediaStrm.CopyTo(streamReader);
                mediaStrm.Position = 0;

                var img = Image.FromStream(mediaStrm);

                // The cropping size shouldn't be higher than the original image
                if (width > img.Width || height > img.Height)
                {
                    Sitecore.Diagnostics.Log.Warn($"Media file is smaller than the requested crop size. " +
                        $"This can result on a low quality result. Please upload a proper image: " +
                        $"Min Height:{height}, Min Width:{width}. File: {mediaItem.DisplayName}, Path{mediaItem.MediaPath}", this);
                }

                // if the cropping size exceeds the cognitive services limits, get the focus point and crop 
                if (width > 1025 || height > 1024)
                {

                    var area = _cognitiveServices.GetAreaOfImportance(streamReader.ToArray());
                    var cropImage = CropImage(img, area.areaOfInterest.X, area.areaOfInterest.Y, width, height);

                    return cropImage;
                }

                var thumbnailResult = _cognitiveServices.GetThumbnail(streamReader.ToArray(), width, height);

                return new MemoryStream(thumbnailResult);
            }
        }

        public string GenerateThumbnailUrl(int width, int height, MediaItem mediaItem)
        {
            var streamReader = MediaManager.GetMedia(mediaItem).GetStream();
            {
                using (var memStream = new MemoryStream())
                {
                    streamReader.Stream.CopyTo(memStream);

                    var thumbnail = _cognitiveServices.GetThumbnail(memStream.ToArray(), width, height);
                    var imreBase64Data = System.Convert.ToBase64String(thumbnail);

                    return $"data:image/png;base64,{imreBase64Data}";
                }
            }
        }

        private Stream CropImage(Image source, int x, int y, int width, int height)
        {
            var bmp = new Bitmap(width, height);
            var outputStrm = new MemoryStream();

            using (var gr = Graphics.FromImage(bmp))
            {
                gr.InterpolationMode = InterpolationMode.HighQualityBicubic;
                using (var wrapMode = new ImageAttributes())
                {
                    wrapMode.SetWrapMode(WrapMode.TileFlipXY);
                    gr.DrawImage(source, new Rectangle(0, 0, bmp.Width, bmp.Height), x, y, width, height, GraphicsUnit.Pixel, wrapMode);
                }
            }

            bmp.Save(outputStrm, source.RawFormat);

            return outputStrm;
        }
    }
}

Let’s see this in action!

After picking your picture in the AI Cropping Image field, it gets already cropped and you can see the different thumbnails. You can choose or change the thumbnails by updating the child items here: /sitecore/system/Settings/Foundation/Vision/Thumbnails.

Also note that you get an auto generated Alt text “Diego Maradona holding a ball” and a list of tags.

AI Cropping Image Field
AI Cropping Image Field

The results

This is how the different cropped images will look like in the front end. Depending on your front end implementation, you will define different cropping sizes per breakpoints.

In this following implementation, I’m setting the image as a background and using the option to render the image URL as follows:

<div class="heroBanner__backgroundWrapper">
    <div v-animate-on-inview="{class: 'animateScaleOut', delay: 10}"
         v-animate-on-scroll="{class: 'animateOverlay'}"
         class="heroBanner__background @Model.HeroClass" role="img" aria-label="@Model.GlassModel.ProductHeroImage.Alt"
         v-background="{
                        '0': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 600, Height = 600, OnlyUrl = true})',
                        '360': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 900, Height = 900, OnlyUrl = true})',
                        '720': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 1667, Height = 750, OnlyUrl = true})',
                        '1280': '@Html.Sitecore().AICroppingImageField("AI Image", Model.GlassModel.Item, new AdvancedImageParameters {Width = 2000, Height = 900, OnlyUrl = true})'
                        }">
    </div>
Tablet Version
Tablet
Desktop Version
Desktop
Mobile Version
Mobile

Sitecore Media Cache

As I mentioned before, the cropped images are also stored in the media cache, as we can confirm by checking the media cache folder

The Sitecore media cache

Other usages and helpers

Sitecore HTML helper

You can use the @Sitecore.Html helper to render an image tag, as usual, or to generate just the URL of the image (src).

Code
@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new AdvancedImageParameters { Width = 600, Height = 600, AutoAltText = true })
Result
<img alt="a close up of a person wearing glasses"
 src="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=600&h=600&smartCropping=true&hash=C2E215FE2CF74D4C8142E35619ABB8DE">

Note: Have a look at the AdvancedImageParameters:

  • OnlyUrl: If true it will just render the image URL (for being used as src in the img tag).
  • AutoAltText: If true, the alt text will be replaced by the one generated from Azure IA.
  • Width and Height: int values, to specify the cropping size.
  • Widths and Sizes: If set, it will generate a srcset image with for the different breakpoints.
  • SizesTag and SrcSetTag: Those are mandatories if when using the previous settings.
Code
@Html.Sitecore().AICroppingImageField("AI Image", Model.Item, new 
AdvancedImageParameters {Widths = "170,233,340,466", Sizes = "50vw,(min-width: 
999px) 25vw,(min-width: 1200px) 15vw", SizesTag = "data-sizes", SrcSetTag = "data-
srcset", AutoAltText = true })
Result
<img alt="a close up of a person wearing glasses" data-sizes="50vw,(min-width: 
999px) 25vw,(min-width: 1200px) 15vw" data-
srcset="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=170&hash=1D04C1F551E9606AB2EEB3C712255651 
170w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=233&hash=DD2844D340246D3CF8AEBB63CE4E9397 
233w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=340&hash=3B773ACB5136214979A0009E24F25F02 
340w,https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=466&hash=424F7615FBECFED21F48DA0AE1FE7A5B 466w" 
src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==">

GlassMapper extension

At last, an extension method has been added in order to get the media URL from the image field.

Code
<img src="@Model.AiImage.GetImageUrl(600, 600)" />
Result
<img src="https://vision.test.cm/-/media/project/vision/homepage/iatestimage.png?
w=600&h=600&smartCropping=true&hash=C2E215FE2CF74D4C8142E35619ABB8DE">

Sitecore Package and code

Please find the whole implementation in my GitHub repo, also feel free to contribute 🙂

You can also download the Sitecore package from here. Note: It has been tested on Sitecore 8.2, 9.x and 10.x.

You can also get the Docker asset image from Docker Hub!

docker pull miguelminoldo/sitecore.computer.vision

That’s it! I hope you find it interesting and useful! Any feedback is always welcome!

Sitecore smart image cropping, tags and alt text with AI: Azure Computer Vision – Part II.

In my previous post I’ve shared a quick overview on the Azure Computer Vision API service and it’s implementation. If you didn’t read it yet, please do before proceeding to this reading!

With the basics and the CognitiveServices in place, let’s move forward and create a custom image field that uses this service to handle the image cropping, tagging and alt text description, all with AI.

I’ll be sharing the whole implementation in GitHub later and also a package plugin, but let’s get into the implementation details first.

Custom Image Field

The first step is to create the custom field, for doing that, go to the core DB and duplicate the /sitecore/system/Field types/Simple Types/Image field item. Let’s call it “AICroppedImage“.

Keep everything as it is except the assembly and class fields

AICroppedImage Class

For the implementation, we just decompiled the code from Sitecore.Kernel (Sitecore.Shell.Applications.ContentEditor.Image) and made all our needed customizations.

using Sitecore.Configuration;
using Sitecore.Data.Items;
using Sitecore.DependencyInjection;
using Sitecore.Diagnostics;
using Sitecore.Globalization;
using Sitecore.Resources.Media;
using Sitecore.Shell.Applications.ContentEditor;
using Sitecore.Web.UI.Sheer;
using System;
using System.IO;
using System.Text;
using System.Web;
using System.Web.UI;
using Microsoft.Extensions.DependencyInjection;
using System.Linq;
using Sitecore.Computer.Vision.CroppingImageField.Models.ImagesDetails;
using Sitecore.Computer.Vision.CroppingImageField.Services;
namespace Sitecore.Computer.Vision.CroppingImageField.Fields
{
    public class AICroppedImage : Image
    {
        private readonly string ThumbnailsId = Settings.GetSetting("Sitecore.Computer.Vision.CroppingImageField.AICroppingField.ThumbnailsFolderId");
        private readonly ICognitiveServices _cognitiveServices;
        private readonly ICroppingService _croppingService;
       
        public AICroppedImage(ICognitiveServices cognitiveServices, ICroppingService croppingService) : base()
        {
            _cognitiveServices = cognitiveServices;
            _croppingService = croppingService;
        }
        public AICroppedImage() : base()
        {
            _cognitiveServices = ServiceLocator.ServiceProvider.GetService<ICognitiveServices>();
            _croppingService = ServiceLocator.ServiceProvider.GetService<ICroppingService>();
        }
        protected override void DoRender(HtmlTextWriter output)
        {
            Assert.ArgumentNotNull((object)output, nameof(output));
            Item mediaItem = this.GetMediaItem();
            string src;
            this.GetSrc(out src);
            string str1 = " src=\"" + src + "\"";
            string str2 = " id=\"" + this.ID + "_image\"";
            string str3 = " alt=\"" + (mediaItem != null ? HttpUtility.HtmlEncode(mediaItem["Alt"]) : string.Empty) + "\"";
            this.Attributes["placeholder"] = Translate.Text(this.Placeholder);
            string str = this.Password ? " type=\"password\"" : (this.Hidden ? " type=\"hidden\"" : "");
            this.SetWidthAndHeightStyle();
            output.Write("<input" + this.ControlAttributes + str + ">");
            this.RenderChildren(output);
            output.Write("<div id=\"" + this.ID + "_pane\" class=\"scContentControlImagePane\">");
            string clientEvent = Sitecore.Context.ClientPage.GetClientEvent(this.ID + ".Browse");
            output.Write("<div class=\"scContentControlImageImage\" onclick=\"" + clientEvent + "\">");
            output.Write("<iframe" + str2 + str1 + str3 + " frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" width=\"100%\" height=\"128\" " +
                "allowtransparency=\"allowtransparency\"></iframe>");
            output.Write("<div id=\"" + this.ID + "_thumbnails\">");
            output.Write(GetThumbnails());
            output.Write("</div>");
            output.Write("</div>");
            output.Write("<div>");
            output.Write("<div id=\"" + this.ID + "_details\" class=\"scContentControlImageDetails\">");
            string details = this.GetDetails();
            output.Write(details);
            output.Write("</div>");
            output.Write("</div>");
        }
        protected override void DoChange(Message message)
        {
            Assert.ArgumentNotNull((object)message, nameof(message));
            base.DoChange(message);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
            if (string.IsNullOrEmpty(this.Value))
            {
                this.ClearImage();
            }
            SheerResponse.SetReturnValue(true);
        }
        protected new void BrowseImage(ClientPipelineArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            base.BrowseImage(args);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
        }
        protected new void ShowProperties(ClientPipelineArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            base.ShowProperties(args);
            if (Sitecore.Context.ClientPage.Modified)
            {
                this.Update();
            }
        }
        public override void HandleMessage(Message message)
        {
            Assert.ArgumentNotNull((object)message, nameof(message));
            base.HandleMessage(message);
           
            string name = message.Name;
            if (name == "contentimage:clear")
            {
                this.ClearImage();
            }
            else if (name == "contentimage:refresh")
            {
                this.Update();
            }
        }
        private void ClearImage()
        {
            if (this.Disabled)
            {
                return;
            }
            if (this.Value.Length > 0)
            {
                this.SetModified();
            }
            this.XmlValue = new XmlValue(string.Empty, "image");
            this.Value = string.Empty;
            this.Update();
        }
        protected new void Update()
        {
            string src;
            this.GetSrc(out src);
            SheerResponse.SetAttribute(this.ID + "_image", "src", src);
            SheerResponse.SetInnerHtml(this.ID + "_thumbnails", this.GetThumbnails());
            SheerResponse.SetInnerHtml(this.ID + "_details", this.GetDetails());
            SheerResponse.Eval("scContent.startValidators()");
        }
        private string GetDetails()
        {
            var empty = string.Empty;
            MediaItem mediaItem = this.GetMediaItem();
            if (mediaItem != null)
            {
                var innerItem = mediaItem.InnerItem;
                var stringBuilder = new StringBuilder();
                var xmlValue = this.XmlValue;
                stringBuilder.Append("<div>");
                var item = innerItem["Dimensions"];
                var str = HttpUtility.HtmlEncode(xmlValue.GetAttribute("width"));
                var str1 = HttpUtility.HtmlEncode(xmlValue.GetAttribute("height"));
                ImageDetails imageDetails;
                using (var streamReader = new MemoryStream())
                {
                    var mediaStrm = mediaItem.GetMediaStream();
                    mediaStrm.CopyTo(streamReader);
                    imageDetails = _cognitiveServices.AnalyzeImage(streamReader.ToArray());
                }
                if (!string.IsNullOrEmpty(str) || !string.IsNullOrEmpty(str1))
                {
                    var objArray = new object[] { str, str1, item };
                    stringBuilder.Append(Translate.Text("Dimensions: {0} x {1} (Original: {2})", objArray));
                }
                else
                {
                    var objArray1 = new object[] { item };
                    stringBuilder.Append(Translate.Text("Dimensions: {0}", objArray1));
                }
                stringBuilder.Append("</div>");
                stringBuilder.Append("<div style=\"padding:2px 0px 0px 0px; text-align=left; \">");
                var str2 = HttpUtility.HtmlEncode(innerItem["Alt"]);
                var str3 = imageDetails.Description.Captions.FirstOrDefault()?.Text;
                if (!string.IsNullOrEmpty(str3) && !string.IsNullOrEmpty(str2))
                {
                    var objArray2 = new object[] { str3, str2 };
                    stringBuilder.Append(Translate.Text("AI Alternate Text: \"{0}\" (Default Alternate Text: \"{1}\")", objArray2));
                }
                else if (!string.IsNullOrEmpty(str3))
                {
                    var objArray3 = new object[] { str3 };
                    stringBuilder.Append(Translate.Text("AI Alternate Text: \"{0}\"", objArray3));
                }
                else
                {
                    var objArray4 = new object[] { str2 };
                    stringBuilder.Append(Translate.Text("Default Alternate Text: \"{0}\"", objArray4));
                }
                stringBuilder.Append("</br>");
                var objArray5 = new object[] { str3 };
                stringBuilder.Append(Translate.Text("Tags: \"{0}\"", string.Join(",", imageDetails.Description.Tags), objArray5));
                stringBuilder.Append("</div>");
                empty = stringBuilder.ToString();
            }
            if (empty.Length == 0)
            {
                empty = Translate.Text("This media item has no details.");
            }
            return empty;
        }
        private Item GetMediaItem()
        {
            var attribute = this.XmlValue.GetAttribute("mediaid");
            if (attribute.Length <= 0)
            {
                return null;
            }
            Language language = Language.Parse(this.ItemLanguage);
            return Sitecore.Client.ContentDatabase.GetItem(attribute, language);
        }
        private MediaItem GetSrc(out string src)
        {
            src = string.Empty;
            MediaItem mediaItem = (MediaItem)this.GetMediaItem();
            if (mediaItem == null)
            {
                return null;
            }
            var thumbnailOptions = MediaUrlOptions.GetThumbnailOptions(mediaItem);
            int result;
            if (!int.TryParse(mediaItem.InnerItem["Height"], out result))
            {
                result = 128;
            }
            thumbnailOptions.Height = Math.Min(128, result);
            thumbnailOptions.MaxWidth = 640;
            thumbnailOptions.UseDefaultIcon = true;
            src = MediaManager.GetMediaUrl(mediaItem, thumbnailOptions);
            return mediaItem;
        }
        private string GetThumbnails()
        {
            var html = new StringBuilder();
            var src = string.Empty;
            var mediaItem = this.GetSrc(out src);
            if (mediaItem == null)
            {
                return string.Empty;
            }
            html.Append("<ul id=" + this.ID + "_frame\" style=\"display: -ms-flexbox;display: flex;-ms-flex-direction: row;flex-direction: row;-ms-flex-wrap: wrap;flex-wrap: wrap;\">");
            var thumbnailFolderItem = Sitecore.Client.ContentDatabase.GetItem(new Sitecore.Data.ID(ThumbnailsId));
            if (thumbnailFolderItem != null && thumbnailFolderItem.HasChildren)
            {
                foreach (Item item in thumbnailFolderItem.Children)
                {
                    GetThumbnailHtml(item, html, mediaItem);
                }
            }
           
            html.Append("</ul>");
            return html.ToString();
        }
        private void GetThumbnailHtml(Item item, StringBuilder html, MediaItem mediaItem)
        {
            if (item.Fields["Size"]?.Value != null)
            {
                var values = item.Fields["Size"].Value.Split('x');
                var width = values[0];
                var height = values[1];
                int w, h;
                if (int.TryParse(width, out w) && Int32.TryParse(height, out h) && w > 0 && h > 0)
                {
                    var imageSrc = _croppingService.GenerateThumbnailUrl(w, h, mediaItem);
                    html.Append(string.Format("<li id=\"Frame_{0}_{1}\" style=\"width: {2}px; height: {3}px; position: relative; overflow: hidden; display: inline-block;border: solid 3px #fff;margin: 5px 5px 5px 0;\">" +
                        "<img style=\"position: relative;position: absolute;left: 0;top: 0;margin: 0;display: block;width: auto; height: auto;min-width: 100%; min-height: 100%;max-height: none; max-width: none;\" " +
                        "src=\"{4}\"><img /><span style=\"position: absolute;" +
                        "top: 0;left: 0;padding: 2px 3px;background-color: #fff;opacity: 0.8;\">{5}</span></li>", this.ID, item.ID.ToShortID(), w, h, imageSrc, item.DisplayName));
                }
            }
        }
    }
}

We’re basically modifying the way Sitecore renders the field with some small customizations, basically to add the thumbnails generated by the Azure Cognitive service and also the Alt and Tags texts.

Ok, so that’s very much it, let’s deploy our code and see how it looks in the Sitecore Content Editor. The only thing you need to do next, is create a template and make use of the newly created “AI Cropped Image” field.

Et Voila! The image field is now rendering a few thumbnails that gives you an idea of the final results when rendering the image in the front-end. As you can see, it gives also some tags and a description (“Diego Maradona holding a ball”) used as alt text, everything coming from the Azure AI service, awesome!

Make the field rendered to work as an OOTB Sitecore image field

Next step, is to make sure we can still using the Sitecore helpers for rendering this field. For making this possible, we want to customize the Sitecore.Pipelines.RenderField.GetImageFieldValue processor. Same as before, we decompile the OOTB code from Sitecore.Kernel and we make our updates there. Then just patch the config like that:

<pipelines>
  <renderField>
    <processor patch:after="processor[@type='Sitecore.Pipelines.RenderField.GetImageFieldValue, Sitecore.Kernel']"
                   type="Foundation.Vision.Pipelines.RenderAICroppingImageField, Foundation.Vision">
    </processor>
  </renderField>
</pipelines>

Here, we just need to add the newly created field type (AI Cropped Image) as a valid image field type by overriding the IsImage() method.

using Sitecore.Diagnostics;
using Sitecore.Pipelines.RenderField;
namespace Sitecore.Computer.Vision.CroppingImageField.Pipelines
{
    public class RenderAICroppingImageField : GetImageFieldValue
    {
        public override void Process(RenderFieldArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            if (!this.IsImage(args))
            {
                return;
            }
            var renderer = this.CreateRenderer();
            this.ConfigureRenderer(args, renderer);
            this.SetRenderFieldResult(renderer.Render(), args);
        }
        protected override bool IsImage(RenderFieldArgs args)
        {
            return args.FieldTypeKey == "AI Cropped Image";
        }
    }
}

Make it working with GlassMapper

Now, we can do some quick updates to GlassMapper as well so we can benefit from the glass helpers. Let’s add a custom field mapper, again after decompiling Glass.Mapper.Sc.DataMappers.SitecoreFieldImageMapper, we can just extend it to work in the same way with the newly introduced AI Cropping Image field.

using Glass.Mapper.Sc;
using Glass.Mapper.Sc.Configuration;
using Glass.Mapper.Sc.DataMappers;
using Sitecore.Data;
using Sitecore.Data.Fields;
using Sitecore.Data.Items;
using System;
using Sitecore.Computer.Vision.CroppingImageField.Fields;
namespace Sitecore.Computer.Vision.CroppingImageField.Mappers
{
    public class AICroppedImageFieldMapper : AbstractSitecoreFieldMapper
    {
        public AICroppedImageFieldMapper(): base(typeof(AICroppedImage))
        {
        }
        public override object GetField(Field field, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var img = new AICroppedImage();
            var sitecoreImage = new AICroppedImageField(field);
            SitecoreFieldImageMapper.MapToImage(img, sitecoreImage);
            return img;
        }
        public override void SetField(Field field, object value, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var img = value as AICroppedImage;
            if (field == null || img == null)
            {
                return;
            }
            var item = field.Item;
            var sitecoreImage = new AICroppedImageField(field);
            SitecoreFieldImageMapper.MapToField(sitecoreImage, img, item);
        }
        public override string SetFieldValue(object value, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            throw new NotImplementedException();
        }
        public override object GetFieldValue(string fieldValue, SitecoreFieldConfiguration config, SitecoreDataMappingContext context)
        {
            var item = context.Service.Database.GetItem(new ID(fieldValue));
            if (item == null)
            {
                return null;
            }
            var imageItem = new MediaItem(item);
            var image = new AICroppedImage();
            SitecoreFieldImageMapper.MapToImage(image, imageItem);
            return image;
        }
    }
}

We need also to create our custom field that inherits from Glass.Mapper.Sc.Fields.Image

using Glass.Mapper.Sc.Fields;
namespace Sitecore.Computer.Vision.CroppingImageField.Mappers
{
    public class AICroppedImage : Image
    {
    }
}

Last step is to add the mapper to the create resolver from the GlassMapperSCCustom.cs

public static  class GlassMapperScCustom
{
    public static IDependencyResolver CreateResolver(){
        var config = new Glass.Mapper.Sc.Config();
        var dependencyResolver = new DependencyResolver(config);
        // add any changes to the standard resolver here
        dependencyResolver.DataMapperFactory.First(() => new AICroppedImageFieldMapper());
        dependencyResolver.Finalise();
			
	return dependencyResolver;
    }
}

Custom Caching

In order to reduce the calls to the service, an extra layer of caching has been implemented. This cache, as any other Sitecore cache gets flushed after a publishing and the size can be easily configured through it’s configuration.

ComputerVision[CroppedImages] custom cache
...
   <setting name="Sitecore.Computer.Vision.CroppingImageField.AICroppingField.CognitiveServices.CacheSize" value="100MB" />
    </settings>
    <events>
      <event name="publish:end:remote">
        <handler type="Sitecore.Computer.Vision.CroppingImageField.Caching.CacheManager, Sitecore.Computer.Vision.CroppingImageField" method="ClearCache" />
      </event>
      <event name="customCache:rebuild:remote">
        <handler type="Sitecore.Computer.Vision.CroppingImageField.Caching.CacheManager, Sitecore.Computer.Vision.CroppingImageField" method="ClearCache" />
      </event>
    </events>
...

In my next post, I’ll be sharing the front end implementation, the full media request flow and the customizations needed to make it working in your site. Stay tuned!