Sitecore smart translation tool with SPE and Azure Cognitive Services (AI)

/ Miguel Minoldo

In my previous posts about images cropping, I’ve used Azure Cognitive Services (Vision) for managing media cropping in a smart way. Now, I’m sharing another usage of Azure Cognitive Services (Language) for building a Powershell tool that makes possible to translate your Sitecore content in a quick and easy way.

Handling item versioning and translation from the Sitecore content editor is a kinda tedious work for editors, especially when it comes to manually creating localized content for your site.

The idea of the PSE tool is to make the editor’s life easier, so in several clicks can achieve the language version creation of the item (including subitems and datasources) and also populate the items with translated content!

Azure Translator – An AI service for real-time text translation

Translator is a cloud-based machine translation service you can use to translate text in near real-time through a simple REST API call. The service uses modern neural machine translation technology and offers statistical machine translation technology. Custom Translator is an extension of Translator, which allows you to build neural translation systems. The customized translation system can be used to translate text with Translator or Microsoft Speech Services. For more info please refer to the official documentation.

About the tool

As I mentioned before, this tool is based on SPE, so it’s easy to integrate on your Sitecore instance. I’ll share the full implementation details but also the code and packages. The service API layer has been implemented on .NET.

The context menu script
Demo

Creating the Azure service

Before proceeding with the implementation, let’s see how to create the Translator service in Azure. The steps are very straightforward as usual when creating such resources.

  • Login to Azure portal (https://portal.azure.com/) and click on create new resource.
  • Search for Translator and finally click on the create button.
Azure Translator Resource
  • Fill the required options and choose a plan. For testing purposes there is a free plan!.
  • Free plan limits: 2M chars of any combination of standard translation and custom training free per month.
  • More details about the available plans here.
Azure Translator Options
  • That’s it! You have your translator service created, now just take a look at the keys and endopint section, you will need it for updating in your config file:
Keys and Endopint

Service implementation (C#)

TranslatorService.cs

This is the service that communicates with the Azure API, it’s quite basic and straightforward, you can also find examples and documentation in the official sites.

using System;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Sitecore.Cognitive.Translator.PSE.Caching;
using Sitecore.Cognitive.Translator.PSE.Models;
using Sitecore.Configuration;

namespace Sitecore.Cognitive.Translator.PSE.Services
{
    public class TranslatorService : ITranslatorService
    {
        private readonly string _cognitiveServicesKey = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey", "");
        private readonly string _cognitiveServicesUrl = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl", "");
        private readonly string _cognitiveServicesZone = Settings.GetSetting($"Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone", "");

        public async Task<TranslationResult[]> GetTranslatation(string textToTranslate, string fromLang, string targetLanguage, string textType)
        {
            return await CacheManager.GetCachedObject(textToTranslate + fromLang + targetLanguage + textType, async () =>
            {
                var route = $"/translate?api-version=3.0&to={targetLanguage}&suggestedFrom=en";

                if (!string.IsNullOrEmpty(fromLang))
                {
                    route += $"&from={fromLang}";
                }

                if (!string.IsNullOrEmpty(textType) && textType.Equals("Rich Text"))
                {
                    route += "&textType=html";
                }

                var requestUri = _cognitiveServicesUrl + route;
                var translationResult = await TranslateText(requestUri, textToTranslate);

                return translationResult;
            });
        }

        async Task<TranslationResult[]> TranslateText(string requestUri, string inputText)
        {
            var body = new object[] { new { Text = inputText } };
            var requestBody = JsonConvert.SerializeObject(body);

            using (var client = new HttpClient())
            using (var request = new HttpRequestMessage())
            {
                request.Method = HttpMethod.Post;
                request.RequestUri = new Uri(requestUri);
                request.Content = new StringContent(requestBody, Encoding.UTF8, "application/json");
                request.Headers.Add("Ocp-Apim-Subscription-Key", _cognitiveServicesKey);
                request.Headers.Add("Ocp-Apim-Subscription-Region", _cognitiveServicesZone);

                var response = await client.SendAsync(request).ConfigureAwait(false);
                var result = await response.Content.ReadAsStringAsync();
                var deserializedOutput = JsonConvert.DeserializeObject<TranslationResult[]>(result);

                return deserializedOutput;
            }
        }
    }
}

The code is simple, I’m just adding a caching layer on top to avoid repeated calls to the API.

You can check the full parameters list in the official documentation, but let me just explain the ones I used:

  • api-version (required): Version of the API requested by the client. Value must be 3.0.
  • to (required): Specifies the language of the output text. The target language must be one of the supported languages included in the translation scope.
  • from (optional): Specifies the language of the input text. Find which languages are available to translate from by looking up supported languages using the translation scope. If the from parameter is not specified, automatic language detection is applied to determine the source language.
  • textType (optional): Defines whether the text being translated is plain text or HTML text. Any HTML needs to be a well-formed, complete element. Possible values are: plain (default) or html. In this case, I’m passing the HTML when is translating from a Rich Text field.

We need also to create the models where the data is parsed into (TranslationResult), I’m not adding the code here to make it simple, but you can check the source code for full details.

TranslationExtensions.cs

using System.Linq;
using System.Threading.Tasks;
using Sitecore.Cognitive.Translator.PSE.Services;
using Microsoft.Extensions.DependencyInjection;
using Sitecore.DependencyInjection;

namespace Sitecore.Cognitive.Translator.PSE.Extensions
{
    public class TranslationExtensions
    {
        private readonly ITranslatorService _translatorService;

        public TranslationExtensions(ITranslatorService translatorServices)
        {
            _translatorService = translatorServices;
        }

        public TranslationExtensions()
        {
            _translatorService = ServiceLocator.ServiceProvider.GetService<ITranslatorService>();
        }

        public async Task<string> TranslateText(string input, string fromLang, string destLang, string textType)
        {
            var res = await _translatorService.GetTranslatation(input, fromLang, destLang, textType);

            if (res != null && res.Any() && res[0].Translations.Any())
            {
                return res[0].Translations[0].Text;
            }

            return string.Empty;
        }
    }
}

Sitecore.Cognitive.Translator.PSE.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <settings>
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiKey" value="{YOUR_APP_KEY}" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiUrl" value="https://api.cognitive.microsofttranslator.com/" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.ApiZone" value="{YOUR_APP_ZONE}" />
      <setting name="Sitecore.Cognitive.Translator.PSE.TranslateService.CacheSize" value="10MB" />
    </settings>
    <services>
      <configurator type="Sitecore.Cognitive.Translator.PSE.DI.RegisterContainer, Sitecore.Cognitive.Translator.PSE" />
    </services>
    <events>
      <event name="publish:end:remote">
        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
      </event>
      <event name="customCache:rebuild:remote">
        <handler type="Sitecore.Cognitive.Translator.PSE.Caching.CacheManager, Sitecore.Cognitive.Translator.PSE" method="ClearCache" />
      </event>
    </events>
  </sitecore>
</configuration>

Powershell Scripts

We need basically one main script to be added in the context menu (Add Language Version and Translate) and then few functions that has been written in this way to make it more readable and modular.

Add Language Version and Translate

Import-Function GetLanguages
Import-Function GetItems
Import-Function ConfirmationMessage
Import-Function Translate
Import-Function GetUserOptions
Import-Function GetUserFieldsToTranslate
Import-Function ConfirmationMessage

# Global variables
$location = get-location
$currentLanguage = [Sitecore.Context]::Language.Name
$langOptions = @{}
$destinationLanguages = @{}
$options = @{}

# Variables from user input - Custom Object
$userOptions = [PSCustomObject]@{
    'FromLanguage'   = $currentLanguage
    'ToLanguages' = @()
    'IncludeSubitems' = $false
    'IncludeDatasources' = $false
    'IfExists' = "Skip"
    'FieldsToTranslate' = @()
}

# Get language options
GetLanguages $langOptions $destinationLanguages

# Ask user for options
$result = GetUserOptions $currentLanguage $langOptions $destinationLanguages $userOptions
if($result -ne "ok") {
    Write-Host "Canceling"
    Exit
}

# Get all items 
$items = @()
$items = GetItems $location $userOptions.IncludeSubitems $userOptions.IncludeDatasources

# Ask user for fields to translate
$dialogResult = GetUserFieldsToTranslate $items $options $userOptions
if($dialogResult -ne "OK") {
    Write-Host "Canceling"
    Exit
}

# Ask user for confirmation
$proceed = ConfirmationMessage $items.Count $options $userOptions
if ($proceed -ne 'yes') {
    Write-Host "Canceling"
    Exit
}

# Call the translator service 
Translate $items $userOptions

GetLanguages

function GetLanguages {
    [CmdletBinding()]
    param($langOptions, $destinationOptions)
	
	$user = Get-User -Current
	$languages = Get-ChildItem "master:\sitecore\system\Languages"
    $currentLanguage = [Sitecore.Context]::Language.Name
	
	# Get list of languages with writting rights and remove the origin language
    foreach ($lang in $languages) {
        $langOptions[$lang.Name] = $lang.Name    
        if (Test-ItemAcl -Identity $user -Path $lang.Paths.Path -AccessRight language:write) {
            $destinationOptions[$lang.Name] = $lang.Name
        }
    }
    
    $destinationOptions.Remove($currentLanguage)
}

GetUserOptions

function GetUserOptions {
    [CmdletBinding()]
    param($currentLanguage, $langOptions, $destinationLanguages, [PSCustomObject]$userOptions)
     
    # Version overwritting options
    $ifExistsOpts = @{};
    $ifExistsOpts["Append"]    = "Append";
    $ifExistsOpts["Skip"]      = "Skip";
    $ifExistsOpts["Overwrite"] = "OverwriteLatest";

    $result = Read-Variable -Parameters `
        @{ Name = "fLang"; Value=$currentLanguage; Title="From Language"; Options=$langOptions; },
        @{ Name = "tLang"; Title="Destination Languages"; Options=$destinationLanguages; Editor="checklist"; },
        @{ Name = "iSubitems"; Value=$false; Title="Include Subitems"; Columns = 4;},
        @{ Name = "iDatasources"; Value=$false; Title="Include Datasources"; Columns = 4 },
        @{ Name = "iExist"; Value="Skip"; Title="If Language Version Exists"; Options=$ifExistsOpts; Tooltip="Append: Create new language version and translate content.<br>" `
                  + "Skip: skip it if the target has a language version.<br>Overwrite Latest: overwrite latest language version with translated content."; } `
        -Description "Select a the from and target languages with options on how to perform the translation" `
        -Title "Add Language and Translate" -Width 650 -Height 660 -OkButtonName "Proceed" -CancelButtonName "Cancel" -ShowHints
    
    $userOptions.FromLanguage = $fLang
    $userOptions.ToLanguages += $tLang
    $userOptions.IncludeSubitems = $iSubitems
    $userOptions.IncludeDatasources = $iDatasources
    $userOptions.IfExists = $iExist
    
    return $result
}

GetItems

function GetItems {
    [CmdletBinding()]
    param($location, $includeSubitems, $includeDatasources)
 
    Import-Function GetItemDatasources
    
    $items = @()
    $items += Get-Item $location
 
    # add subitems
    if ($includeSubitems) {
        $items += Get-ChildItem $location -Recurse
    }
     
    # add datasources
    if ($includeDatasources) {
        Foreach($item in $items) {
            $items += GetItemDatasources($item)
        }
    }
     
    # Remove any duplicates, based on ID
    $items = $items | Sort-Object -Property 'ID' -Unique
    
    return $items
}

GetFields

function GetFields {
    [CmdletBinding()]
    param($items, $options)

    Import-Function GetTemplatesFields
    
    Foreach($item in $items) {
        $fields += GetTemplatesFields($item)
    }
    
    # Remove any duplicates, based on ID
    $fields = $fields | Sort-Object -Property 'Name' -Unique
    
    # build the hashtable to show as checklist options
    ForEach ($field in $fields) {
    	$options.add($field.Name, $field.ID.ToString())
    }
    
    return $fields 
}

GetItemDatasources

function GetItemDatasources {
    [CmdletBinding()]
    param([Item]$Item)
 
    return Get-Rendering -Item $item -FinalLayout -Device (Get-LayoutDevice -Default) |
        Where-Object { -not [string]::IsNullOrEmpty($_.Datasource)} |
        ForEach-Object { Get-Item "$($item.Database):" -ID $_.Datasource }
}

GetTemplatesFields

function GetTemplatesFields {
    [CmdletBinding()]
    param([Item]$Item)
	
	$standardTemplate = Get-Item -Path "master:" -ID ([Sitecore.TemplateIDs]::StandardTemplate.ToString())
	$standardTemplateTemplateItem = [Sitecore.Data.Items.TemplateItem]$standardTemplate
	$standardFields = $standardTemplateTemplateItem.OwnFields + $standardTemplateTemplateItem.Fields | Select-Object -ExpandProperty key -Unique
	$itemTemplateTemplateItem = Get-ItemTemplate -Item $Item
	$itemTemplateFields = $itemTemplateTemplateItem.OwnFields + $itemTemplateTemplateItem.Fields
	$filterFields = $itemTemplateFields | Where-Object { $standardFields -notcontains $_.Name } | Sort-Object
	
	return $filterFields
}

GetUserFieldsToTranslate

function GetUserFieldsToTranslate {
    [CmdletBinding()]
    param($items, $options, [PSCustomObject]$userOptions)
    Import-Function GetFields
    
    # Get all fields from items
    $fields = @()
    $fields = GetFields $items $options
    
    # Promt the user for selecting the fields for translation
    $dialogParams = @{
        Title = "Fields selector"
        Description = "Select the fields you want to translate"
        OkButtonName = "OK"
        CancelButtonName = "Cancel"
        ShowHints = $true
        Width = 600
        Height = 800
        Parameters = @(
            @{
                Name = "fieldsIdToTranslate"
                Title = "Checklist Selector"
                Editor = "check"
                Options = $options
                Tooltip = "Select one or more fields"
            }
        )
    }
    
    $dialogResult = Read-Variable @dialogParams
    $userOptions.FieldsToTranslate = $fieldsIdToTranslate
    
    return $dialogResult
}

ConfirmationMessage

function ConfirmationMessage {
    [CmdletBinding()]
    param($itemsCount, $options, [PSCustomObject]$userOptions)
    
    $fieldsToUpdate = ""
    $opt = @()
    
    ForEach($ft in $userOptions.FieldsToTranslate) {
        $opt = $options.GetEnumerator() | ? { $_.Value -eq $ft }
        $fieldsToUpdate += "$($opt.Key), "
    }
    
    $fieldsToUpdate = $fieldsToUpdate.Substring(0,$fieldsToUpdate.Length-2)
     
    $message = "Updating <span style='font-weight: bold'>$itemsCount item(s)</span>!<br>"
    $message += "<br><table>"
    $message += "<tr><td style='width: 300px'>Origin Language:</td><td style='width: 450px'>$($userOptions.FromLanguage)</td></tr>"
    $message += "<tr><td style='width: 300px'>Destination Languages:</td><td style='width: 450px'>$($userOptions.ToLanguages)</td></tr>"
    $message += "<tr><td style='width: 300px'>Include Subitems:</td><td style='width: 450px'>$($userOptions.IncludeSubitems)</td></tr>"
    $message += "<tr><td style='width: 300px'>Include Datasources:</td><td style='width: 450px'>$($userOptions.IncludeDatasources)</td></tr>"
    $message += "<tr><td style='width: 300px'>Copy Method:</td><td style='width: 450px'>$($userOptions.IfExists)</td></tr>"
    $message += "<tr><td style='width: 300px'>Fields to Translate:</td><td style='width: 450px'>$($fieldsToUpdate)</td></tr>"
    $message += "</td></tr></table>"
     
    return Show-Confirm -Title $message
}

Translate

function Translate {
    [CmdletBinding()]
    param($items, [PSCustomObject]$userOptions)
    
    Write-Host "Proceeding with execution..."
    
    # Call the translator service
    $translatorService = New-Object Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions
    
    $items | ForEach-Object {
    	$currentItem = $_
    	foreach($lang in $userOptions.ToLanguages) {
    		Add-ItemLanguage $_ -Language $userOptions.FromLanguage -TargetLanguage $lang -IfExist $userOptions.IfExists
    		
    		Write-Host "Item : '$($currentItem.Name)' created in language '$lang'"
    		
    		Get-ItemField -Item $_ -Language $lang -ReturnType Field -Name "*" | ForEach-Object{ 
    		    # Only look within Single-line and Rich Text fields that has been choosen in the dialog box
                if(($_.Type -eq "Single-Line Text" -or $_.Type -eq "Rich Text" -or $_.Type -eq "Multiline Text") -and $userOptions.FieldsToTranslate.Contains($_.ID.ToString())) {
                    if (-not ([string]::IsNullOrEmpty($_))) {
                        # Get the item in the target created language
                        $langItem = Get-Item -Path "master:" -ID $currentItem.ID -Language $lang
        				
        				# Get the translated content from the service
        				$translated = $translatorService.TranslateText($currentItem[$_.Name], $userOptions.FromLanguage, $lang, $_.Type)
        				
        				# edit the item with the translated content
        				$langItem.Editing.BeginEdit()
        				$langItem[$_.Name] = $translated.Result
        				$langItem.Editing.EndEdit()
        				
        				Write-Host "Field : '$_' translated from '$($userOptions.FromLanguage)'" $currentItem[$_.Name] " to : '$lang'" $translated.Result
                    }
                }
    	    }
    	}	 
    }
}

In the Translate function, I’m doing the call to the API (Sitecore.Cognitive.Translator.PSE.Extensions.TranslationExtensions).

That’s very much it, now is time to test it! If everything went well, you will be able to add language versions to your items with also translated content from Azure Cognitive Translation.

Let’s see this in action!

For the purpose of this demo, I’ve created a simple content tree with 3 levels, the items has some content in english (plain and HTML) and I’ll be using the tool to create the Spanish-Argentina and French-France versions + translated content.

1- Click on the Home item and choose the Add Language Version and Translate option from the scripts section.

2- Choose the options, in this case I want to translate from the default ‘en‘ language to both ‘es-AR‘ and ‘fr-FR‘. Also I want to include the subitems, but as for this test the items doesn’t have a presentation nor datasources, I’m keeping this disabled. No versions in the target language exist for those items, so I’m keeping the “Skip” option.

3- Click on proceed and choose the fields you want to translate:

I’m selecting all fields, as you can check in the SPE code, I’m removing the standard fields from the items to be translated, normally you don’t want that and it will overpopulate the fields list.

4- Click OK, double check the data entered and click the OK button for making the magic to happen:

5- Click on the View script results link to check the output logs:

6- Check that the items have been created in the desired languages and the contents are already translated. Review them, publish and have a cup of coffee :).

fr-FR items version:

es-AR items version:

Voila! After few clicks you have your content items created in the language version with the content translated, I hope you like it us much as I do.

Find the source code in GitHub, download the Sitecore package here or get the asset image from Docker Hub.

Thanks for reading!

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s