Since 7.5, Sitecore introduced Media Request Protection that is essentially a hash added to some URL’s. Some people find this annoying, but I think this was a good move. If someone would just hammer your site by downloading media images with random width and height parameters, it would cause a lot of image resizing and consequently eat up all CPU.

The idea behind the Media Request Protection is to avoid these kind of denial of service attacks, by having a hash parameter on the URL. On an incoming media request, Sitecore calculates the hash of the request query string parameters and compares it with the given hash. If they are equal, Sitecore performs the resizing options (or whatever parameters are provided). If they are not equal, Sitecore will just send the original file as-is.

Error messages in the log like this one indicates there is something wrong with the media request protection:

ERROR MediaRequestProtection: An invalid/missing hash value was encountered. The expected hash value: ~40 hex digits. Media URL: /-/media/...., Referring URL: ....

You should monitor your log files for this error. Your site might look correct, but your visitors might be downloading non-resized images adding page weight. The most common reason for this error is an incorrect created media URL. If you create the URL by yourself, ensure to call HashingUtils.ProtectAssetUrl(mediaUrl).

One reason for this to occur, is a known bug in Sitecore Rich Text Editor. When you link a document, such as a PDF, in a RTE field, Sitecore adds the language parameter to the URL. Essentially, the rendered URL becomes something like this: /-/media/....document.pdf?la=en-GB. The “la” parameter triggers hash evaluation, despite the document not being a manipulable image. A workaround for this can be to remove the language parameter from the protected parameter list, like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:x="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <mediaLibrary>
      <requestProtection>
        <protectedMediaQueryParameters>
          <parameter description="language code" name="la">
            <patch:delete />
          </parameter>
        </protectedMediaQueryParameters>
      </requestProtection>
    </mediaLibrary>
  </sitecore>
</configuration>

If you add new functionality to your media processing pipelines that uses its own parameters, such as cropping, image format transformation etc, ensure to add those parameters accordingly to this <protectedMediaQueryParameters> list as well.

Moving forward
So let’s say you’ve fixed all hashing errors, you may see these errors in the log file anyway. Search engines etc have probably picked up the faulty URL. The protection only prevents hammering on the server, so an incorrect hash serves the original file with a 200 OK. To make this go away, we can add a noindex header for those URL’s. We can do this by adding the HTTP header “X-Robots-Tag: noindex“.

You can override the MediaRequestHandler and send the header when the hash is missing or incorrect, like this:

protected virtual void SendRobotsTag(MediaOptions options, HttpContext context)
{
	if (!IsRawUrlSafe)
	{
		var headers = context.Response.Headers;
		headers["X-Robots-Tag"] = "noindex";
	}
}

// Concept copied from MediaRequest
private bool IsRawUrlSafe
{
	get
	{
		if (!Settings.Media.RequestProtection.Enabled)
			return true;

		var rawUrl = HttpContext.Current.Request.RawUrl;
		return !ProtectRequest(Context.Site) ||
				HashingUtils.IsSafeUrl(rawUrl) ||
				HashingUtils.IsUrlHashValid(rawUrl);
	}
}

// Copied from MediaRequest
private bool ProtectRequest(SiteContext site)
{
	if (!Settings.Media.RequestProtection.Enabled)
		return false;
	if (site == null)
		return true;
	return !MediaManager.Config.RequestProtection.SitesToIgnore.Contains(site.Name.ToLowerInvariant());
}

Then add the call to SendRobotsTag after the SendMediaHeaders calls in the DoProcessRequest method.

A cleaner way of doing this would be to override the MediaRequest. Despite having nice virtual methods, it is still quite hard to override this class without a lot of code duplication because its Clone method prevents this.

I faced a problem where I need to control the Sitecore html cache from within a controller action. There are probably many scenarios where this is needed. In my particular case, the output from the controller is cacheable in the Sitecore html cache, but my controller action is also dependent on a third party system. The communication with this system could sometimes fail. Essentially it means that if the controller fails, I don’t want the html output to be cached either. If it is cached, as it is by default, my error message will stay there until the html cache is cleared, regardless if the third party system is online or not.

So the scenario is quite simple, but it turned out to be quite complex to solve. I found a rather hackish way to do it, by hooking into the <mvc.renderRendering< pipeline, like this:

public class ControllerCacheDisabler : RenderRenderingProcessor
{
	public static void DisableCache()
	{
		var id = Sitecore.Mvc.Presentation.RenderingContext.Current?.Rendering?.UniqueId ?? Guid.Empty;
		var ctx = HttpContext.Current;
		if (id != Guid.Empty && ctx != null)
		{
			ctx.Items[$"HtmlOutputCacheDisabler_{id}"] = "true";
		}
	}

	public override void Process(RenderRenderingArgs args)
	{
		Assert.ArgumentNotNull(args, "args");

		if (HttpContext.Current == null || !args.Cacheable || string.IsNullOrWhiteSpace(args.CacheKey))
		{
			return;
		}

		var id = args.Rendering.UniqueId;
		if (!string.IsNullOrWhiteSpace(HttpContext.Current.Items[$"HtmlOutputCacheDisabler_{id}"] as string))
		{
			args.Cacheable = false;
		}
	}
}

From within my Sitecore controller action, I can call the static ControllerCacheDisabler.DisableCache() method and it’ll register that this particular rendering shouldn’t be cached. Then I add this processor just before the AddRecordedHtmlToCache processor like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:x="http://www.sitecore.net/xmlconfig/">
	<sitecore>
		<pipelines>
			<mvc.renderRendering>
				<processor
					patch:before="processor[@type='Sitecore.Mvc.Pipelines.Response.RenderRendering.AddRecordedHtmlToCache, Sitecore.Mvc']" 
					type="Your.Namespace.ControllerCacheDisabler, Your.Assembly" />
			</mvc.renderRendering>
		</pipelines>
	</sitecore>
</configuration>

Essentially this means that a cached output of the controller will be rendered. If it’s not cached, it’ll normally execute. If successful, it’ll be cached and if unsuccessful it’ll not be added to the cache.

Perhaps I’ve missed something obvious. This solution feels like over complicating things. Dealing with errors and exceptions from Sitecore MVC Controllers and have the result non-cached doesn’t seem to be a very uncommon scenario. Perhaps there is a way to get the RenderRenderingArgs object from a controller, so I can just set the Cacheable property to false directly in the controller.

Please leave a comment if you know a better solution to this.

Thank you Sitecore for awarding me “Technology Most Valuable Professional” (MVP) again! Fifth year in a row!

The Sitecore MVP Award celebrates the most active Sitcore community members from around the world who provide valuable online and offline expertise that enriches the community experience and makes a difference.

My contribution to Sitecore and the community over the last year have, besides this blog, been continuous dialog with Sitecore staff on how to improve the author experience in the product and over forty confirmed bug reports.

As a lead architect and DevSecOp of this huge website project, www.volvoce.com, I expect a lot of blog posts during 2017 about our findings and how to create and maintain great websites.

Some time ago, Martin Davies started a thread on the Sitecore MVP community forum about where to store page content. It caught a lot of attention and many replies. Since that’s a closed forum, I thought I’d might as well write a public post on my view on this topic, but also put into a wider perspective. It almost ended up in a recipe on how to build Sitecore websites. Nothing in this post is new really, so I’ve kept each topic as brief as possible.

This post is only about the basics of Sitecore as a CMS. Sitecore can do so much more. But I’ve seen so many, in my opinion, implementation failures, where the customer can’t get passed the content management hell and never starts walking up the experience maturity ladder. The implementing partners plays a very important role here ensuring the content authoring environment doesn’t become messy, time consuming or non-intuitive.

Few projects are equal, so what I describe here is not a silver bullet in any way. But it’s an approach that works for us at Stendahls and we’re constantly refining it and adjusting it for each customer needs. The starting point here is building website that in a good way supports multi-language, mulit-brand, multi-market website (or sites) where there are multiple authors. Content governance is important and there are limited resources among authors.

Content structure

Regarding organising content, one has to decide if one shared tree is used for all languages or if they could diverge. We typically create one tree per market, where some markets may use more than one language, such as Switzerland, Canada and so on, having two or more languages. On such markets, the client typically wants the same content and structure just translated into two or more languages, compared to other markets.

We typically define a site root item that only acts as a container for the site content. A Home item is created as child of the website container, and pages are created as descendants of the Home item. Thereby we can have other site related items as siblings of the Home item outside its web root.

Creating a master site is usually a good idea that we can clone into each market. The master site we do central translation into multiple languages, but we don’t publish it live. Thereby we can efficiently manage translations and let the content inherit to market sites where local adaptations are made.

When cloning the master site into a market site, we keep only the target language versions. When governing this kind of structure, switching between authoring language is a pain unless it’s addressed.

Creating pages

Pages should be organised in a clear item structure reflecting the URL structure. Site navigation, menus etc is something else. They are usually very similar, but don’t build you item structure based on site navigation alone. The item structure is also important when it comes to security configuration so that inheritance can be used as much as possible. These factors are also important when it comes to items living outside the site root or items that doesn’t have a rendering.

Pages is typically built from components, i.e a set of renderings with datasources pointing to other items containing its content. Usually those are bound only to the current page, but some may be shared between pages and some even shared between sites. Some people embraces a fully atomic design where everything is divided into small pieces. We find there is a risk of breaking it up into too small pieces, so sometimes it’s better to have to have larger building blocks with rendering parameters (or compatible renderings) to gain speed in the Experience Editor, make it easier for editors and ensuring personalisation works as expected. Rendering parameters should of course be templated and when using compatible renderings, one should ensure that the compatible rendering links goes both ways.

We store page local content items in a folder hierarchy as descendants of the page, site shared content items in a folder hierarchy as sibling to the site home, and globally shared items as siblings to the site roots. When defining datasource locations on renderings, the pipe character can be used to define multiple locations, so that authors can easily use local page contents as default, but also easily select the right location for site shared and global components as well.

We create the folder hierarchy automatically, as needed, during content creation. We create a custom set of folder templates with a clearly diverging appearance from items representing pages. Those folders have configured insert options etc on their standard values and are marked as hidden items. Content authors usually don’t have to see those. Even though they are hidden, they are displayed in the datasource selection dialog boxes. Content folders are also marked as protected (__Read Only) in standard values, since authors should not be able to accidentally move or edit those items. Datasource locations are of course created using Sitecore relative axes queries making them universal between sites. The folder hierarchy is typically two levels deep, where the first is just a “Content” folder and the second defines the content type based on template. This is because the experience editor displays all items based on the datasource location query regardless if their type, but only compatible types are selectable. By having this two level structure, only selectable items are shown in the UI.

Content lifecycle

An area that’s sometime confusing for authors is knowing when (s)he is editing content of a local component or shared component that affects multiple pages. We’ve targeted this by highlighting shared components in the UI with a warning sign telling the author that when editing the selected content, the content will be changed on other pages as well. We also did a list of all those pages, so that it can easily be identified if a content change is supposed to go to all the affect pages or not. From Sitecore 8.2 this feature is built-in into the Experience Editor.

One of the advantages of having local page components as descendants of the page, is that security is inherited from the page and they follow along when renaming, moving, copying and cloning a page. Keeping track of all existing URL’s and automatically create 301 redirects as pages are moved or renamed, also helps authors to keep the site structure clean and friendly without compromising SEO.

As the sites evolve, components are replaced and eventually there may be a lot of unused page component items. We typically don’t want to remove those automatically because there could be valuable content in them. But we indicate those orphan items so that they can be (bulk)archived when a content governor decides to do so.

One should spend some time to get this right from the beginning. Correcting it afterwards is a real pain. Don’t make it a hassle for authors to work with the experience editor.

User friendly components

With minimal training, authors should be able to work fast in the experience editor, so a lot of focus should go into structuring layouts and renderings. It should be intuitive to create and work with pages in the experience editor. One have to invest time to configure everything to get this right. Pages needs insert options/insert rules, placeholder settings has to be configured correctly and so on.

Both page templates and renderings should have friendly naming, be filtered according to what fits the location, have a representative screen shots or at least a decent icon.

Sometimes there is a need to have container items that doesn’t represent a page. In the experience editor, those items has to have a rendering to make it possible to create sub items using the experience editor insert page command. We typically create a separate layout for those that isn’t available on the published site. We give those more of a “Sitecore layout” rather than the website layout, so that it is clear to the authors that those pages are not part of the site.

The component code also needs some love and care for the Experience Editor to work properly. There are some caveats to handle where components are new or empty on the page. For example, empty sections should usually be collapsed on preview/published pages, but when in editing mode they have to be visible so that authors can write content.

Cloning tool

An annoying thing with Sitecore is its lack of rewriting links when copying/cloning items. In essence, when you clone a set of pages, their internal links needs to be rewritten so that they become internal in the copy/clone branch. This is a big topic, on its own, so I’m only gonna touch on a few areas. I guess all Sitecore partners already have their own cloning tool, but here are some of the features I like with ours:

We use a custom notification provider where we can configure that some notifications should be automatically accepted/rejected. This is really useful on fields where you wan’t values to be automatically inherited, or children in specific branches to be synced etc. We made this pluggable and configurable on a per template and per field level, so that it can be customized to suit the site information architecture.

We’ve also added a lot of functions to handle local changes to a field. This is particularly useful for field types such as rich text and the Rendering fields. We can highlight pages with gutters and fields with suggestion validators where the local content is modified compared to the clone master. This requires some field parsing to avoid false positives, since rewritten links in a clone is not null and thereby not inherited. We also ensured that a reset field operation doesn’t results in incorrect links.

It’s typically a good idea to add the Rendering and Final Rendering fields to the ItemCloning.NonInheritedFields setting.

Gutters and validators

Example of surplus language, orphan item and local adaption gutters

Gutters and validators are really helpful for both content authors and people governing content. Keep in mind though that validators shouldn’t be intrusive and information overload may result in authors ignoring validator warnings etc.

Sitecore comes with a few built-in validators, but we rarely use any of these. They are too hard coded, not very universal and some of them are buggy as well. We create parameter driven validators where we can for example define on the item what severity it should have and so on.

Languages, links etc can also be a pain in the clone hell, so gutters and validators can help us here as well. We sometimes decorate our website definitions with information on what languages are expected on the site and so on. With this we can more easily find items with missing language versions or surplus languages, items having unexpected links that perhaps links to another site due to clone/copy issues etc. Tools for jumping between languages are really helpful too when working centrally with multiple language versions.

On a page level, we also validate related rendering items, so that a page is validated as a whole instead of item-by-item. It’s more intuitive for authors to validate and approve a whole page instead of the default piece-by-piece approach. This is fixed in Sitecore 8.2.

Workflow and publishing

Workflow can be a pain and there’s no universal solution to it. One has to start with investigating the clients internal processes. No system can solve non-existing or non-functional processes. Identify what the process looks like. When doing this, my experience is that customers, naturally, mix workflows with security rights, roles, publish restrictions and versioning. Mapping this correctly and have a common understanding is well worth the effort.

When pages are built from multiple components, it can be hard for authors to manage the workflow of all page local components. Those items still needs to be in a workflow in order to have proper versioning. We try to have those in the same workflow as the page itself. By adding actions to the workflow commands, we can automatically bring the rendering datasource items to the next workflow state, thereby approving a whole page as it is seen by the author in edit/preview mode. This makes it much more intuitive for authors and saves a lot of panic phone calls from desperate authors that published half-broken pages.

I hope Sitecore in a near future will implement work packages, so that this can be handled more smoothly. In the meantime we’ll have to live with the item-by-item workflow. A lot of this has changed already in 8.2, but there more things to do in this area.

To further improve the workflow process, we modify the getValue and publishing pipelines in such way that the author sees what’s actually going to be published. A problem with the default behaviour are the different kinds of item value resolving, such as standard values, cloning and language fallback. For example; if a field value is null and it fallbacks to a clone parent value and that item version is not approved, publishable or for any other reason can’t be published, Sitecore will by default show one value in preview and another after publish. By tapping into these pipelines, we can improve the user experience of both the content editor and experience editor.

We typically flatten the web database(s) at publish time. This also avoids headaches when local authors doesn’t stand a chance to approve changes performed by master site editors. This also has a good side effect, by reducing time consuming fallback resolving at run-time. Strategies on clearing caches is also important as more sites and authors are added to the application. Clearing all html caches just because one page on one site is published isn’t really effective.

One need to consider what impact publishing has on the site performance. On larger sites, it’s probably not a good idea to publish items automatically as they reaches the final workflow step. It usually better to train authors that approving content means it is ready for publish and anyone can hit the publish button at any time. A scheduled publishing engine is also a must have, since the default behavior of Sitecore isn’t really logical to end users.

We also like to make item locking as seamless as possible to the authors. They shouldn’t really have to care about that. So we let Sitecore auto-lock the items in the usual way and then use an unlocking tool to ensure that items are unlocked some time after editing sessions have ended.

Sitecore have awarded Volvo Construction Equipment, with partner Stendahls, winner of the Sitecore Experience Award 2016, Marketing Agility category. The Marketing Agility award recognizes marketing teams that have made significant, measurable gains in productivity and marketing ROI through the Sitecore platform. This year’s winners outlined clear before and after scenarios for team output and content publishing times as well as any associated organizational or team advantages, as a result of time/resource savings.

I’ve spent most of my working time during 2015 and 2016 on this huge project, where we built the whole marketing platform and rolled out 123 market and dealer websites, representing Volvo’s business in more than 140 countries on 30+ languages. The solution contains about 370k items and is managed by more than 150 editors worldwide, all in the same Sitecore solution.

A lot of my time went into creating a streamlined development process, build and maintain a fast and stable hosting platform and build a very efficient authoring environment. Many of my blog posts during the last year have sprung out of this project.

Volvo CE Press release

Some say target="_blank" is one of most underestimated vulnerabilities on the web. When you make a link to an external into a new tab or window, that site gets access to your site. If it’s a design flaw in browsers or something else is debatable, but luckily there is a simple fix for it by adding rel="noopener" to external links.

Update: The latest update of Sitecore 8.2 already have this implemented, so if you’re on the current version, you’re already covered and don’t need to bother about this. However, you need to be aware of the potential vulnerability to ensure any links not rendered by Sitecore fields are made correct as well.

How it works
When you have a link on a website that opens a new tab or window, like <a href="external.url" target="_blank">, the linked page gets access to the source page in the original tab through the window.opener property. This means an attacker could change the visible content, or even the url, of the source page. This could be used for phishing attacks etc. Here is an example of this in action.

To prevent this, we can add the rel="noopener" attribute to links, like <a href="external.url" target="_blank" rel="noopener">. This will makes the window.opener property null on the target site. If you’re opening windows using javascript, you’ll have to set the opener property to null manually, such as var win=window.open(); win.opener=null; win.location = url;.

Even if you’re linking legit website, you can’t be sure the target sites aren’t hacked and could leverage from this vulnerability. Besides, there are rarely a required need for external sites to access the DOM of your site, so always adding the noopener option rarely causes any trouble.

Solving it in Sitecore
Besides any static links, we can let Sitecore add the rel="noopener" attribute to all content author created links. We essentially have two kind of links: The Sitecore field types “General Link” and “General Link with Search”, and links within Rich Text fields. The General Link (with Search) field types can be fixed at render time and the content for Rich Text fields can be updated at save time.

We can extend the <renderField> pipeline by adding the the attribute to the render field parameters when rendering links that are external.

For rich text fields, it would be quite time consuming to do this runtime, so instead we can hook into the item:saving event of all rich text fields. By parsing the rich text with HtmlAgilityPack we can easily load all links and update the link attributes accordingly.

Sample code for solving this is available here: https://github.com/mikaelnet/Sitecore.RelNoOpener

Let me know what you think about this solution.

Sitecore has a quite advanced access right management system. However, I’ve found a few quite common requirements that, as far as I know, isn’t supported out of the box. One is to allow content authors to remove individual item versions without allowing them to remove the entire item. This is especially useful for multi language sites. Another requirement is to allow authors to delete items they have created themselves, but no other items.

I’ve seen people work around these kind of issues by playing around in the core database, modifying ribbon buttons etc. Personally I don’t like that approach. That would just hide the button and if the user could initiate the command in any other way, Sitecore will gladly perform the delete action. It’s easy to forget a command action in a context menu or something like that.

Instead I created a very small Sitecore module to solve these issues. All the source is available on Github.

Ability to remove item versions

While looking into this, I reflected parts of the Sitecore security model and found that this is already partly implemented. Looking into the Sitecore.Security.AccessControl.AccessRight class, we’ll see that there is already a hard coded item:removeversion access right. Looking at the common Item class, we already have the method item.Access.CanRemoveVersion(). Looking further into the QueryState() method of the DeleteVersion command, I found that it also evaluates using these access rights method.

So far so good. However, the ItemAuthorizationHelper.GetAccess() that evaluates this, is hard coded to explicitly require item delete access when evaluating the item:removeversion access right. So by replacing the two default AuthorizationProviders (SqlServerAuthorizationProvider and BucketAuthorizationProvider), with new ones that uses an overridden version of the GetAccess() method, we can easily change the behavior of this.

To be able to configure remove version access rights without delete item access rights, we can just add a “Remove version” right to the list of existing rights, like this:

<accessRights>
  <rights>
    <add patch:after="add[@name='item:delete']"
         name="item:removeversion"
         comment="Remove version right for items."
         title="Remove version"
         modifiesData="true" />
  </rights>
</accessRights>

With this, the access rights will appear in all UI’s, like this:

Ability to remove own items

Allowing authors to delete their own items can be implemented in a similar way. I just added the logic for it in the same overridden ItemAuthorizationHelper.GetAccess() method. When the method evaluates access rights, its results is “Allow”, “Deny” or “NotSet”. “NotSet” means Deny in practice, but this is very useful in this scenario. If Item Delete access is explicitly set to “Allow” or “Deny”, that setting will rule. But when it evaluates to “NotSet”, I’ll look at the item creator and see if it matches the current user.

One caveat though is that the CreatedBy field is a versioned field, so I can’t just look at that one. The current author could be the creator of the current language version, but might not have created the item in the first place. So I load all the language versions of the items and use the very first version and compares the current author with the initial creator.

As I mentioned previously, I prefer implementing the functionality close to the core, like this. That way all existing UI’s follows as expected. In the screen shot below, I’ve created two items as the user “test” and allowed the “test” user to remove item versions. On one of the items, I’ve explicitly set the deny delete item access.

Let me know what you think about this. Do you find this to be a common requirement too? Is this a good way of solving it? Should it perhaps go into the product?

A colleague of mine spent several hours finding a very nasty problem in one of our Sitecore environments. Thomas did a great job on this, so all creds him. I thought we should share this finding with the community as well.

The problem we were facing was that our Content Delivery servers stopped processing remote events, such as html cache clearing, index updates etc. This seems to be a quite common problem caused by incorrect server configuration and is easy to find when the error is consistent. Our problem was a bit different, but I’ll get to that later on.

A common configuration error is getting the instance names incorrect. The InstanceName setting is empty by default, meaning that it will combine the server name and the IIS website name. This is fine for most scenarios, but your publishing instance, typically the Content Management server, needs a unique name that all other servers can refer to. I typically create two config files, like this:

A InstanceName.config file goes onto the publishing instance only:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <settings>
      <setting name="InstanceName" set:value="CM" />
    </settings>
  </sitecore>
</configuration>

A PublishingInstance.config file goes onto all the servers:

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <settings>
      <setting name="Publishing.PublishingInstance" set:value="CM" />
    </settings>
  </sitecore>
</configuration>

Another common problem we’ve seen is when restoring a production database into dev/test/qa environments. What can happen is that the EQStamp properties comes out of sync or the EventQueue is too big etc. An easy fix for this is to just remove the EQStamp rows from the Properties tables.

Now to our odd problem that at least I haven’t heard of before. We had all the servers correctly configured and the remote event would run for a few minutes and then it just silently stopped. Nothing in the log files, even in debug mode. An application pool recycle would get the event queue going again for a while and then it would stop again.

It turned out to be caused by some really bad code in Sitecore.Kernel. Looking at the Sitecore.Search.IndexLocker constructor. It contains this piece of code:

if (!this.lockInstance.Obtain(int.MaxValue))
    throw new Exception("Could not obtain lock " + this.lockInstance);

Guess what, that exception will never the thrown if the code can’t obtain a lock. The int.MaxValue is used as timeout in milliseconds, so it’s effectively almost 25 days…

The lock it was trying to obtain turned out to be a Lucene write lock (despite we’re using Solr), and there turned out to be an old lock file in the Data/indexes/__system folder. That’s an old legacy index used for internal search and seems to be finally gone in the latest version.

So what does this have to do with remoting? Well, there is this Sitecore.Services.Heartbeat class that loops over all registered agents. It turns out this isn’t very robust. The entire heartbeat work loop stops when it gets stucked on one of the agents, like what happened above. That effectively stopped them all.

So, if you’re facing this issue too, stop the website, remove any existing *.lock files in the Data/indexes subfolders, and restart the website again.

By default, Sitecore stores media files as blobs in the database. This is usually good, but if you have a large volume of files, this can become too heavy to handle. So I wrote a Sitecore plugin where you can seamlessly store all the binaries in the cloud.

The advantage of moving this to a cloud storage are

• Bottomless storage. No need to resize database volumes
• Reduced stress on SQL Server
• No need to publish the binaries themselves results in faster publish
• Faster and simpler database backups
• Easier to migrate databases between environment
• Less need for large disk space on dev/test/stage machines
• Shorter response times (in some scenarios)

The way I implemented this was by overriding the SqlServerDataProvider in Sitecore, so it would store binaries using a separate blob storage provider, instead of directly in the database. I made such provider for Amazon S3, but you could use Azure or any cloud storage provider you like.

The way it works is that when attaching a binary to a blob field, it’ll upload it to the cloud storage and keep blob reference in the Sitecore database as-is, with just a null value, and keep a cached version on local disk to reduce response times. This could potentially make response times even faster than loading the binaries from the database.

The good thing with this is that the cloud binaries doesn’t have to be published, because they’ll always get a new unique id if a binary is changed. Secondly, the binary isn’t accessible unless the binary reference row is published to the web database, so this won’t be a security issue either.

The implementation also includes migration of binaries from and to the cloud, so you can easily migrate existing data to the cloud, or move it back into the Sitecore database. The implementation also works in a mixed scenario, were only some blobs are in the cloud.

Feel free to grab the code from Github:
https://github.com/mikaelnet/sitecore-blob-storage

Thank you Sitecore for awarding me “Technology Most Valuable Professional” (MVP) again! Six years in a row!

The Sitecore MVP Award celebrates the most active Sitcore community members from around the world who provide valuable online and offline expertise that enriches the community experience and makes a difference.

My contribution to Sitecore and the community over the last year have, besides this blog, have been a continuous dialog with Sitecore staff on how to improve the product and I’ve filed around thirty confirmed bugs. I’ve also held a few talks on Sitecore User Group Gothenburg (SUGGOT) and a few modules are shared on GitHub.

During an upgrade project to Sitecore 9, I got some insights worth sharing. Some findings in this post applies to multiple Sitecore versions and some are specific to Sitecore 9. I’ve been using SolrCloud 6.6, but some of it applies to other versions as well. It be came a long, yet very abbreviated, post covering many areas.

In this post:

• Solr Managed schemas in Sitecore 9
• Tune and extend Solr managed schema to fit your needs
• How to fix Sitecore config for correct Solr indexing and stemming
• How to make switching index work with Solr Cloud
• How to reduce index sizes and gain speed using opt-in
• How to make opt-in work with Sitecore (bug workaround)
• Why (myfield == Guid.Empty) won’t give you the result you’re expecting

Working with managed schemas

From Sitecore 9, the “Generate the Solr Schema.xml file” command in the Control Panel is gone. Instead there’s a new “Populate Solr Managed Schema” command. It’ll use the REST interface of Solr to update the schema file. Solr can share collection configurations. This essentially means that you can have multiple collections (formerly known as cores) sharing the same configuration and this is usually the case. Here’s a first finding: It can conflict with the Populate Solr Managed Schema command, as it’ll list all you indexes and let you update them all at once. This may result in a raise condition, as Sitecore will send multiple commands to update the schema, for multiple indexes, that are sharing the same schema. To work around this, you should instead check only one of the indexes and uncheck all the others and click update. Solr will make sure all the others are the same anyway. If you don’t use shared Solr config like ZooKeeper, you obviously need to run this for all indexes, and in that case you can run them all at once.

Generate Solr Schema.xml in the Control Panel up to Sitecore 8.2

Populate Solr Managed Schema in the Control Panel of Sitecore 9

When using shared configuration in Solr, you should populate the schema for only one index. Otherwise they may collide.

So, what will Sitecore do with the Solr managed schema when this process is execute? It’ll add and update the required fields to the schema to make sure Sitecore items can be indexed properly. You’ll probably find that you want to make additional changes to schema. Previously we could just change the schema.xml file to whatever we want. You can still do this the old way by setting the <schemaFactory> class to ClassicIndexSchemaFactory in the solrconfig.xml file in your Solr instances and manage your schema.xml manually. However, once you get the hang of the new way of working with managed schemas, you’ll probably find it better. Especially when working with multiple environments and Solr clusters. You’ll find it much easier to just let Sitecore update everything from the Control Panel, instead of playing around with schema.xml files and ensure they are up to date, distributed and loaded properly into all Solr instances.

So how do we tune the managed schema to what we want? Well, you could just manually call the Solr API, but it would make more sense to have the application specific configuration together with our Sitecore application code. That’ll make a solution less fragmented and easier to keep all environments in sync. Sitecore does this in the <contentSearch.PopulateSolrSchema> pipeline. You can either replace the existing Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields processor, given that you provide the necessary functionality, or add your own processor. I found it easier to replace the existing one as I found a couple of errors in the built in one that I needed to address anyway. Some of the errors are covered in this post, so I’ll describe the replace scenario here.

You can patch the default processor like this:

CustomPopulateSolrSchema.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore search:require="Solr">
    <pipelines>
      <contentSearch.PopulateSolrSchema>
        <processor type="Your.Namespace.PopulateSolrSchemaFields, Your.Assembly"
                   patch:instead="processor[@type='Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields, Sitecore.ContentSearch.SolrProvider']"/>
      </contentSearch.PopulateSolrSchema>
    </pipelines>
  </sitecore>
</configuration>

Then you’ll need to implement the new PopulateSolrSchemaFields class and make an implementation of the ISchemaPopulateHelper interface:

PopulateSolrSchemaFields.cs

namespace Your.Namespace
{
    public class PopulateSolrSchemaFields : Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.PopulateFields
    {
        protected override Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.ISchemaPopulateHelper GetHelper(SolrNet.Schema.SolrSchema schema)
        {
            Assert.ArgumentNotNull(schema, "schema");
            return new YourSchemaPopulateHelper(schema);
        }
    }
	
	public class YourSchemaPopulateHelper : Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.ISchemaPopulateHelper
	{
		private readonly SolrSchema _solrSchema;

        public YourSchemaPopulateHelper(SolrSchema solrSchema)
        {
            Assert.ArgumentNotNull(solrSchema, "solrSchema");
            _solrSchema = solrSchema;
        }
		
		// Implementation goes here. 
		// Look at and copy stub code from Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema
	}
}

You can reflect the default implementation Sitecore.ContentSearch.SolrProvider.Pipelines.PopulateSolrSchema.SchemaPopulateHelper in Sitecore.ContentSearch.SolrProvider.dll to get an idea of how it works. You’ll probably find that the reflected code for GetAddFields() and GetReplaceTextGeneralFieldType() isn’t very readable. I refactored this slightly so it became much more readable and managable without changing the logic. Perhaps the original code is readable too, but was just messed up by the compiler.

Looking at bit deeper into GetAddFields(), I found a couple of errors in Sitecores implementation, that is now also comfirmed bugs as by Sitecore. There may be differences between versions and releases of Sitecore, but I’ve found the following mapping errors in many 8.x and in all three 9.x releases. So check these in your configuration:

• "*_t_en" is incorrectly mapped to "text_general". This should be mapped to "text_en"
• "*_t_cz" is incorrectly mapped to "text_cz". This is initially a bug in Solr referring to Czech. The language code for Czech is "cs". So the correct mapping should be "*_t_cs" to "text_cz", given that you leave the name of the default Solr text stemming configuration for Czech as is.
• "*_t_no" is mapped to "text_no". “Norwegian” is typically referred to as “Nynorsk” (nn) or “Bokmål” (nb). To make this work better, two additional mappings are needed here. Unless you have specific/custom stemming rules in Solr, you can map both "*_t_nn" and "*_t_nb" to "text_no".
• An array of DateTime is mapped as "{0}_dtm" in Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config, but there is no mapping provided for "*_dtm", so a multi-value date field needs to be added.

You’ll also notice that the format for updating managed schemas is different from the schema itself. The class generates xml snippets that are sent as update commands to Solr. A very annoying thing is that incorrect update commands are just silently accepted. You’ll get a “success”-message in Sitecore regardless if your update snippets are correct and stored in Solr or not. So make sure you read the managed schema file in the Solr UI to verify that the changes have actually been stored.

Sitecore may show a success message even when the managed schema update are not applied in Solr.

A good example of the differences between the schema.xml format and the xml structure for updating it, can be found in the GetReplaceTextGeneralFieldType() method. This is also a method worth replacing. Looking closer to its content, it configures text_general to use the default stop words and synonyms filter. Those contains English text in a default setup. This doesn’t really make much sense. I believe text_general should be used for languages where there is no stemming support in Solr. So I suggest you remove the stopwords and potentially the synonyms filter.

Look into the managed schema stored in your Solr instance/cluster to ensure your config changes has been applied properly.

You may also find solr.WordDelimiterGraphFilterFactory very useful for stemming things like product names, trade marks, phone numbers, SKU numbers etc. I believe an equivalent filter was configured as default in Solr 5.3, but in 6.6 it’s not. I found myself working through all the Solr analyzers for all the language definitions to have good search results. Time consuming, but well worth it.

Here’s a sample of Solr schema configuration for a Swedish stemmed text field type, leveraging from the WordDelimiterGraphFilterFactory:

<fieldType name="text_sv" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.StandardTokenizerFactory"/>
    <filter class="solr.StopFilterFactory" format="snowball" words="lang/stopwords_sv.txt" ignoreCase="true"/>
    <filter class="solr.WordDelimiterGraphFilterFactory" catenateNumbers="1" generateNumberParts="1" splitOnCaseChange="1" generateWordParts="1" splitOnNumerics="1" catenateAll="0" catenateWords="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.SnowballPorterFilterFactory" language="Swedish"/>
  </analyzer>
</fieldType>

To achieve the configuration above, by letting Sitecore update the managed schema, you actually need to produce the XML snippet below in an implementation of ISchemaPopulateHelper interface that Sitecore sends to the Solr API.

<replace-field-type>
  <name>text_sv</name>
  <class>solr.TextField</class>
  <positionIncrementGap>100</positionIncrementGap>
  <analyzer>
    <tokenizer>
      <class>solr.StandardTokenizerFactory</class>
    </tokenizer>
    <filters>
      <class>solr.StopFilterFactory</class>
      <ignoreCase>true</ignoreCase>
      <words>lang/stopwords_sv.txt</words>
      <format>snowball</format>
    </filters>
    <filters>
      <class>solr.WordDelimiterGraphFilterFactory</class>
      <catenateNumbers>1</catenateNumbers>
      <generateNumberParts>1</generateNumberParts>
      <splitOnCaseChange>1</splitOnCaseChange>
      <generateWordParts>1</generateWordParts>
      <catenateAll>0</catenateAll>
      <catenateWords>1</catenateWords>
      <splitOnNumerics>1</splitOnNumerics>
    </filters>
    <filters>
      <class>solr.LowerCaseFilterFactory</class>
    </filters>
    <filters>
      <class>solr.SnowballPorterFilterFactory</class>
      <language>Swedish</language>
    </filters>
  </analyzer>
</replace-field-type>

As I mentioned above, the compiled and reflected code of this isn’t very readable, so I created a small fluent pattern to generate this configuration. I’m happy to share that code, but I assume Sitecore will have a preferred way of working with this in a near future release. @Sitecore: Please reach out to me if you’re interested in incorporating my solution into the product.

SolrCloud

Solr includes the ability to set up a cluster of Solr servers that combines fault tolerance and high availability. It’s now called SolrCloud, and has the capabilities to provide distributed indexing and search, with central configuration, automatic load balancing and fail-over. SolrCloud is not a cloud service. It’s just a version of Solr. You can run the same SolrCloud application and index index configuration on a single instance on your local dev machine, as well as on your large scale, sharded and clustered, production instances. Worth noticing is that Solr “cores” are called “collections” in SolrCloud. A “core” in SolrCloud is just a shard or replica of a collection.

To make Sitecore work well with Solr, you should use switching indexes. Otherwise your indexes will be blank during rebuilds! To make this work with SolrCloud, you’ll need to use the Sitecore.ContentSearch.SolrProvider.SwitchOnRebuildSolrCloudSearchIndex index type. The idea is that you have two Solr collections for every index together with two aliases. The main alias points to one of the indexes and a rebuild alias points to the other index. So after index rebuild, Sitecore will swap the two aliases. This means a fully functional index is always available even when rebuilding the index.

The constructor of SwitchOnRebuildSolrCloudSearchIndex uses a few more arguments to make this work. When this is defined in the config files, the constructor arguments are just listed as a <param> list. This makes it a bit tricky to patch properly, since it’s somewhat tricky to ensure the order of the <param> list from Sitecore patch files. What’s worse is that if you get those arguments wrong, Sitecore just won’t start. You can’t even use the /sitecore/admin/ShowConfig.aspx page. Doh!

Sitecore 9 also introduced a new bug regarding switching indexes. Currently you must set the ContentSearch.Solr.EnforceAliasCreation setting to true. Otherwise, Sitecore won’t update the Solr index aliases properly after a rebuild. But ensure not to do this on your Content Delivery servers. Otherwise you’ll end up with multiple servers switching the aliases ending up in undefined states.

Another common issue with Solr is that commits of large updates takes longer time than the default timeout, causing exceptions on the Sitecore client side. We can simply increase the timeout by changing the ContentSearch.Solr.ConnectionTimeout setting. Note that the timeout is specified in milliseconds, so 600000 will give a 10 minute timeout. The final commit after a full index rebuild can easily take a few minutes.

Resently I became aware of another setting, ContentSearch.SearchMaxResults, that you really should set. This setting tells Sitecore how many rows should be returned from Solr, unless you specify a .Take(n) Linq expression in your queries. By default, the value of this field is empty, causing Sitecore to ask for int.Max, i.e. 2³¹-1 = 2,147,483,647, rows. This is not good practice, as the resulting rows will be read into memory. You should set this to a more reasonable value, such as 1000 or similar.

If you’re using SolrCloud 6.6 in a clustered and sharded environment you might also run into the following strange behavior, unless you reduce SearchMaxResults. It turns out that Solr is a bit buggy when rows gets close to int.Max. When a call to Solr has the rows parameter set to int.Max (2³¹-1), I got a strange "NegativeArraySizeException" from Solr. If I send int.Max+1 (2³¹), I get a more sensible NumberFormat exception from Solr and that’s completely fine. If I send int.Max-1 (2³¹-2) I get an IllegalArgumentException, with the message “maxSize must be <= 2147483630" (2³¹-17). If I send that specified max number, I get a java heap OutOfMemoryError… Running SolrCloud on a single server instance, nothing of this is reproducible. Very strange… and you may not get much valuable data about this on the Sitecore side… So just set ContentSearch.SearchMaxResults and you’re good to go.

If you, like me, prefer reading code, the above can be summarized in a config template like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" 
               xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore search:require="Solr">
    <sc.variable name="solrIndexPrefix" value="my_sc9_prefix" />
    <settings>
      <setting name="ContentSearch.Solr.ConnectionTimeout" set:value="600000" />
      <setting name="ContentSearch.Solr.EnforceAliasCreation" set:value="true" role:require="ContentManagement or Standalone" />
      <setting name="ContentSearch.SearchMaxResults" set:value="1000" />
    </settings>
    <contentSearch>
      <configuration>
        <indexes>
          <index id="sitecore_master_index" type="Sitecore.ContentSearch.SolrProvider.SwitchOnRebuildSolrCloudSearchIndex, Sitecore.ContentSearch.SolrProvider" solrPrefix="$(solrIndexPrefix)"
                 role:require="ContentManagement or Standalone">
            <param desc="name">$(id)</param>
            <param desc="mainalias">$(solrPrefix)_$(id)</param>
            <param desc="rebuildalias">$(solrPrefix)_$(id)_Rebuild</param>
            <param desc="collection">$(solrPrefix)_$(id)_1</param>
            <param desc="rebuildcollection">$(solrPrefix)_$(id)_2</param>
            <param ref="contentSearch/solrOperationsFactory" desc="solrOperationsFactory" />
            <param ref="contentSearch/indexConfigurations/databasePropertyStore" desc="propertyStore" param1="$(id)" />
            ...
            ...
          </index>
        </indexes>
      </configuration>
    </contentSearch>
  </sitecore>
</configuration>

Solr Opt-in

By default, Sitecore indexes all fields. This means that Sitecore will send the content of all fields to Solr for indexing, expect the ones that you explicitly exclude from indexing (Opt-out). This may be ok for small solutions, but for large solutions, this will cause indexes to become very large and indexing to be CPU heavy and time consuming.

If you don’t make queries on a field or use the stored result, you don’t need it in the index. So why index it then? It’s just a waste of resources. Note that free text search isn’t affected by this, as this is typically made on computed fields, such as _content or a more suitable computed field you make yourself.

The <indexAllFields&gt element controls this in the <indexConfiguration> section. When setting it to false, Sitecore will only index the fields you explicitly include for indexing (Opt-in). I’d recommend you set this to false when starting new projects, since changing this to false later on could potentially require code refactoring and a lot of testing.

Sitecore has confirmed a bug in AbstractDocumentBuilder<T> when using opt-in. There is an if-statement in the AddItemFields() method, that only loads all field values when using opt-out. So when using opt-in, fields with null values won’t be properly indexed. This means that values coming from standard values, clones or language fallback won’t be indexed. This seems to apply to most versions of Sitecore, but luckily it’s a very simple patch:

SolrDocumentBuilder.cs

public class SolrDocumentBuilder : Sitecore.ContentSearch.SolrProvider.SolrDocumentBuilder
{
	public SolrDocumentBuilder(IIndexable indexable, IProviderUpdateContext context)
	  : base(indexable, context)
	{
	}

	protected override void AddItemFields()
	{
		Indexable.LoadAllFields();
		base.AddItemFields();
	}
}

SolrDocumentBuilder.patch.config

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/" xmlns:role="http://www.sitecore.net/xmlconfig/role/" xmlns:search="http://www.sitecore.net/xmlconfig/search/">
  <sitecore role:require="Standalone OR ContentManagement OR ContentDelivery OR Processing OR Reporting" search:require="Solr">
    <contentSearch>
      <indexConfigurations>
        <defaultSolrIndexConfiguration type="Sitecore.ContentSearch.SolrProvider.SolrIndexConfiguration, Sitecore.ContentSearch.SolrProvider">
          <documentBuilderType>Your.Namespace.SolrDocumentBuilder, Your.Assembly</documentBuilderType>
        </defaultSolrIndexConfiguration>
      </indexConfigurations>
    </contentSearch>
  </sitecore>
</configuration>

Going for an opt-out solution does give you some more work, so is there a real gain of the opt-out approach? I recently converted one of our existing solutions from opt-out to opt-in principle. The size of the Solr indexes was reduced from almost 70GB to 3.5GB. Index time was reduced from 4 hours to 30 minutes on a 16 core/64GB RAM server. I think those figures speaks for themselves and your queries will be faster too.

Tricky ContentSearch Linq to Solr bug

Recently I noticed a Sitecore ContentSearch query that didn’t return the result I was expecting. It took a while to really get a hold of what was really going on. I can’t fully disclose what kind of queries I’m performing in that project, so I’ve translated this into a more generic scenario that I also hope makes it easier to understand the issue.

Let’s say you have a site with a lot of pages and you want to generate a sitemap.xml page. You can do this by querying all your indexable pages that has a layout and so on and just make an xml output of it. Now, let’s say you have some deprecated pages, duplicate content or whatever that you don’t want to be indexed. A way of solving that could be to have a canonical link to an alternative page containing the equivalent content. That can simply be stored in a DropLink Sitecore field and we render it as a <canonical /> html tag.

Obviously we don’t want to include those pages having a canonical link pointing to an alternative page in the sitemap xml. So we simply construct our Sitecore ContentSearch query like this, right?

var pages = searchContext.GetQueryable<SitemapDocument>()
    .Filter(f => f.IsLatestVersion && // More filters here obviously to get only pages with layouts etc...
                 f.CanonicalLink == Guid.Empty)
    .GetResults()
    .Select(r => r.Document)
...
...
public class SitemapDocument 
{
  [IndexField("_latestversion")]
  public bool IsLatestVersion { get; set; }

  [IndexField("canonicallink")]
  public Guid CanonicalLink { get; set; }  
  ...
  ...
}

Well, it turns out that the above doesn’t work! Why?

Well, Guid isn’t a nullable type, so Guid.Empty is actually {00000000-0000-0000-0000-000000000000} and is indexed as 00000000000000000000000000000000. I’ll abbreviate this as 000… from now on. So, when Sitecore indexes a blank Guid field, it’ll index it as 000… When searching a field == Guid.Empty, like in the sample code above, Sitecore will make a Solr query looking like this:

?q=*.*&fq=(_latestversion:(True) AND canonicallink_s:(000....)...

This means that if an item doesn’t have the Canonical link field, it won’t be returned.

I’ve tried all sorts of ways to solve this. I’ve tried making the property a nullable Guid, making it a string field and so on. So have Sitecore support and this is now a confirmed bug (though I’m not sure the root cause was properly provided to product department).

I think that an empty guid should never be stored as 000… in the index. If there is no guid, there should be no data in the index either, i.e. null. Likewise, if 000…. is actually stored in a guid field it should also be indexed as null. At query time, I think a field == Guid.Empty statement should be serialized to -field_s:*. This would make the query behavior consistent and somewhat reduce the index size as well.

The best way I’ve found to workaround this for now, is to exclude (or not include if you’re following my recommendations above) the canonical field from the index entirely and replace it with a computed field that’ll always return the link Guid, or Guid.Empty regardless if the field is empty or not present on the item being indexed. Never return null. That way the query will always return correct data.

Note that the inverted query works just fine. A query like field != Guid.Empty does give the expected result.

Closing

Okay, this was a very long first post of my 6th year as a Sitecore MVP. Hope you find it valuable and please share your thoughts on this. I’d love to hear your experiences in this area. Have you all ran into these issues too, or is it just me pushing some boundaries again?

Btw, have you noticed the Index Manager dialog result message, “Finished in X seconds”? The value isn’t accurate when building multiple indexes. Do you see what’s wrong? It’s a confirmed bug too, but I doubt it’ll ever be corrected, and I’m perfectly fine with that. It’s a cosmetic bug well below minor level.

This is partly a follow-up post of my previous post on Workign with Solr and Sitecore Content Search in Sitecore 9. In that post I raised a few issues that needs to be dealt with, and I’ve found some more. Most of what’s in this post I’ve found on Sitecore 9.0 update-1 and/or Sitecore 8.2 update-5, and it seems like most of the things applies to many more versions too. So I’ve focused on how you can verify if your solution needs patching too.

This is essentially just a brief list of some issues I’ve found over the last few weeks, while working against the clock towards the releases of two large Sitecore projects. Big thank you to all my great colleagues that have put an enormous effort into getting things to work.

Indexes are not updated on language fallback

When using the syncMaster strategy, typically on the sitecore_master_index, it may not update properly when using language fallback. Here’s how to test this on a fresh instance:

• Have enableFieldLanguageFallback set to true for indexes and the sites “shell” and “website”
• Configure some languages and have their language fallback point to the “en” English language.
• Have a template with at least one versioned field that has Enable language fallback set to true.
• Have an item based on the template above and ensure it has a few language versions.
• Let the versioned field fallback to the “en” English master language. All or some language versions should now have inherited/language fallback values
• Re-build the sitecore_master_index and verify that the item has been properly indexed.
• Change the value of the versioned field on the “en” English language and save the item
• Look again in the sitecore_master_index and check if the fallback values have been updated

If the fallback versions hasn’t been updated, you’re also affected by the bug. At the time of writing this, there is no patch to this as far as I’m aware of. You’ll have to contact Sitecore support and ask for a fix. I’ve seen this error in multiple versions.

Indexes are not updated when using Publish Service when editing shared fields

When using the onPublishEndAsync strategy together with the Sitecore Publish Service, your web indexes may not update properly. Here’s how to test this on a fresh instance:

• Have a template with a shared field
• Have an item based on that template, with multiple language versions
• Publish the item and rebuild the web index, to ensure a correct state
• Change the value of the shared field and publish the item
• Look in the sitecore_web_index and check if the shared value has been updated

If the shared value has been updated only on the version that you actually edited and published, and the index was not updated for all other language versions of the item, you’re affected by this bug. At the time of writing this, there is a patch for this. Personally I don’t like the way this has been solved, but at least it works. You’ll have to contact Sitecore support and ask for the fix, as I’ve been asked not to distribute the patch. This bug seems to exist in all current versions of the Publish Service.

Indexes are not updated when using Publish Service when editing unversioned fields

Update: This might have been a side effect of another problem I had, so this might not be a bug after all. However, the patch above is still relevant.

This bug is very similar to the one above. Here’s how to test this on a fresh instance:

• Have a template with an unversioned field
• Have an item based on that template, with multiple versions on the same language
• Ensure the latest version is not publishable, like having it in Draft state or similar
• Publish the item and rebuild the web index, to ensure a correct state
• Change the value of the unversioned field on the latest, non-publishable version of the item
• Publish the item again. At this stage, there is no “change” to the published item itself, but the unversioned field causes a content change.
• Look in the sitecore_web_index and check if the unversioned value has been updated

If the unversioned value have not been updated, you’re affected by this bug. At the time of writing this, the patch above seems to solve this error as well. You’ll have to contact Sitecore support and ask for the fix, as I’ve asked not to distribute the patch. This bug seems to exist in all current versions of the Publish Service.

Publish Service may not publish all versions

When moving an item, the Publish Service may not publish the item properly. Here’s how you can check this:

• Create two items as children of a Home item. I’ll refer to them as A and B. Ensure both items have at least two language versions.
• Create a child item to A. I’ll refer to this item as C.
• Select the Home item and choose Publish -> Publish Item. Check “include subitems” and publish. Ensure the languages are checked
• Verify that the web database has been properly updated
• Move the child item C from A to B.
• Select the Home item and perform the same publish operation again as above
• Look in the web database and verify the existence of language versions of the C item.

If the C item doesn’t have all the expected languages you’re affected by this bug. At the time of writing this, the very latest version of the Publish Service has fixed this, so you’ll have to make an upgrade. Many thanks to John for finding this!

The search.log file may be flooded with warnings

Your search.log may be flooded with warnings during an index rebuild. The warning messages looks like this:

WARN Processing Field Name : __validator_bar_validation_rules Search field name in Solr with Template Resolver is returning no entry.

This is a confirmed bug, no 195567, but Sitecore says the warning message “is expected”. There is currently no fix for this, but as a workaround you can add a filter to your log4net config in order to suppress the messages. Many thanks to Thomas for finding this!

Index rebuild based on index data

During an index rebuild, you may see search.log flooded with queries like this:

INFO Solr Query - ?q=_name:(noindex)&rows=0&fq=_templatename:("Template field")&fq=_indexname:(sitecore_master_index)&wt=xml

Yes, Sitecore is querying the index that is currently being re-built… These log entries may be for several fields for every item version, so it’s easy to run out of disk space. That’s how I found it with a 10GB+ log file. This is a confirmed bug and at the time of writing this there is no fix for it. As a workaround you can specify all the field names and their types in the fieldNames section, like this:

<fieldNames hint="raw:AddFieldByFieldName">
<field fieldName="your_field_name" returnType="text" />

I don’t know what happens if you have two different templates having a field each sharing the same name but with different types. I don’t know the actual cause of this, but to me it seems like Sitecore doesn’t look at the configured <fieldType> mapping.

Lucene config in the Solr files

If you’re using Solr, you may want to take a look at your config file Sitecore.Marketing.Definitions.MarketingAssets.Repositories.Solr.IndexConfiguration.config. It may have incorrect Lucene references, though it doesn’t seem to do too much harm. You can simply change it to follow the Solr config pattern.

Clone errors in the publish log

When using clones together with the Publish Service, you may get the following error in the publishing.log (the log for the Publish Service – not the one in Sitecore Data/logs):

[Warning] Clone Source Item a40fff41-e146-45f0-a225-0508da3a6477 does not exist on source database. Clones will still be published.

The error itself is incorrect. I’ve ensured that the source item do exist and is in a publishable state. At the time of writing this, I haven’t found the cause of this nor what affect it has on the published content. It might be connected with the next error, but it doesn’t seem like that right now.

Published clones may lose data

When using clones together with the Publish Service, you may find missing field values in the published web items. Here’s at least one scenario where this can happen:

• Have a “master” item.
• Clone the master item into a “clone-item”. Leave the original values.
• Clone the clone-item into a “clone-of-clone-item”. Leave the original values.
• Publish the three items using the Publish Service.
• Look at the clone-of-clone-item in the web database.

If you’re missing field values in the web database, you’re affected by this bug. At the time of writing this, I’m not aware of any solution to this. Your only option is really to go with the old publisher. If you let the old publisher correct the data and then enable the Publish Service again, it will remove the field values again. I also need to spend some more time to see if this is the only scenario where data gets lost, but it takes time on a solutions with 450k+ items…

This error is really serious, since content is actually lost on the published site.

Wrap up

So these are just some of the issues I’ve been fighting over the last few weeks. Some of them are hard to spot, so you may be affected by some of those issues without knowing it. Maybe this can help you smash some bugs in your solutions too.

Sitecore: When will you start a bug bounty program?

Language Fallback is a powerful feature in Sitecore. It’s been around for years as a module and since Sitecore 8.1 it is part of the core product. In short, it allow values to be inherited between item language versions. This allows you to show default content when translation is missing. You may have dialects of a languages, such as US English vs British English, and you can use Language Fallback to avoid translating content that is the same for the two dialects etc.

TL;DR
Increase Caching.SmallCacheSize to about 10MB if you’re using Language Fallback in Sitecore.

Figuring out what value to actually render in a field becomes a bit more complex, and Sitecore uses a few additional caches to speed this up. However, I’ve found that one of them being incorrect configured by default. If you’re finding one of the following warning messages in your Sitecore log looking like this, you’re probably affected by this:


... WARN master[isLanguageFallbackValid] cache is cleared by ...
... WARN web[isLanguageFallbackValid] cache is cleared by ...


Background
Sitecore has a set of default named cache sizes:

Those cache sizes are sometimes referenced when a cache doesn’t have a specific size setting. The Language Fallback has a cache containing the validity of a fallback field. Essentially it just stores a boolean, “1” or “0”, for each entry in this cache. There is a cache for each database, so the name of the caches is “dbname[isLanguageFallbackValid]”, and its size is hard coded to Small cache size in DatabaseCaches.Initialize().

The issue
One would think that this wouldn’t be a problem. If we’re caching just a true/false value, wouldn’t 100KB be enough? Well, it turns out that behind the scenes, this isn’t performed in a very efficient way. The cached value is dependent on multiple parameters, making the cache key quite complex and large in size. In the Sitecore implementation, it uses a specific IsLanguageFallbackValidCacheKey object to describe the key. It contains the Item ID, the Field ID, the Language and, for some strange reason, the Database. (The cache is per database anyway).

In addition, the Item ID and Field ID is converted into strings. A Guid is 16 bytes in size. A Sitecore ID.ToString() becomes 38 characters, and a character in Windows is two bytes, so this 16 byte Guid suddenly becomes almost 80 bytes. So there is a lot of room for cache key size reduction here. The cache itself also of a “Indexed” cache type, so that gives a lot of overhead as well. As lists, hash tables etc grows, they also pre-allocate space for future objects.

The sum of all this is that an entry in this IsLanguageFallbackValidCache is almost a kilobyte each. So essentially only about 100 entries or so will fit in the cache. You probably have something in the magnitude of 100,000 field values in your solution if you find Language Fallback useful.

I took a snapshot of a running instance with increased cache size and analyzed it with dotMemory. As you can see, just 29,250 cached booleans occupies 57MB:

The solution
At the time of writing this, the only workaround is to increase the SmallCacheSize to a size that suits your solution. I found that 50-100MB was a good size for mine. You set the parameter in a config file, like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <settings>
      <setting name="Caching.SmallCacheSize" set:value="100MB" />
    </settings>
  </sitecore>
</configuration>

Look at the size of the cache in /sitecore/admin/cache.aspx and look for cache clear warnings in your logs to find a good balance that suits your solution.

A potential downside is that all other caches that use the “SmallCacheSize” is also changed. I’ve filed a wish to Sitecore to have a unique cache size settings for each cache and just use the pre-defined small/medium/large cache sizes as fallback.

When writing this, I have an open ticket on this too, and I assume Sitecore will come to the same conclusion that the cache key objects needs to be rewritten to reduce its memory foot print and the default size should be set to something like “MediumCacheSize”.

While investigating stability issues, I’ve found a few things that may need addressing.

Sitecore updates indexes in batches. This is good in general, but it turned out it may be very memory hungry. There are essentially two config parameters you can control the batch size with:

<setting name="ContentSearch.ParallelIndexing.Enabled" value="true" />
<setting name="ContentSearch.IndexUpdate.BatchSize" value="300" />

The default config above, essentially means Sitecore will start multiple threads processing 300 indexable objects each. This might not be an issue at all, but when combined with a multi-language setup, media indexing and crazy authors, this may become a real problem.

Let’s say you have 30+ languages in a solution. When uploading a media item, Sitecore creates a version on every language. (Yes, you can disable this, but you need a version on each language in order to have localized meta data, such as alt-texts on images etc.) When Sitecore indexes this, all these 30+ versions will be loaded as individual “Indexable” items into the indexing batch. On a media item, such as a PDF, the binary data will be extracted and placed into a BatchUpdateContext queue that’ll be sent to Solr (or whatever search provider you use). While in the batch, they are all loaded into RAM as individual copies waiting to be handled by Solr.

I found a document uploaded by an author that became over 20 million characters when extracted into the _content computed field. Given that each char is two bytes, this 40MB string times the number of language versions occupied almost 1GB of RAM while being in the indexing queue.

We did have a lot of indexing issues, but I found this by accident while analyzing a memory dump. I looked at what media items where in the database and found that if I’m unlucky, two threads with 300 indexables can easily consume all my 32GB RAM and crash the process…

Quick solution to this was to just reduce the BatchSize, so that fewer large objects would go into SolrBatchUpdateContext.

Long term solution is to replace the entire _content computed fields with something more optimized. I rarely find any good use of this field anyway, since it doesn’t perform proper language stemming anyway. Nor does it make sense to load the shared binary data over and over again for each language when indexing it. Extracting useful content into a separate shared, hidden, text field at upload is much more efficient.

After having tons of problems and several filed tickets on the initial release of Sitecore Publish Service 3.1, I was happy to find that Sitecore have addressed many of the problems of the previous versions. This update contains 12 fixes and I found my customer support ticket number listed six times.

Unfortunately the update didn’t solve these issues properly, so while I’m waiting for new patches I thought I’d share a UI fix that wasn’t included in the release. When working with multiple languages, the language list isn’t very user friendly in the Publish Service UI. It’s essentially just becomes a small letterbox with unsorted languages and a large area for displaying the targets.

This is the layout provided as default when having multiple languages:

This can be fixed by changing the SPEAK layout. Patching SPEAK UI’s isn’t very practical. I hope Sitecore will get rid of this as soon as possible. Anyway, here’s what needs to be done:

In the core database, find the item /sitecore/client/Applications/Publishing/DialogWindowFullRePublish, ID {36798F82-AA2E-47DC-879A-FB5B3661470D}. In the __Renderings field, find the ScrollablePanelLanguages rendering (ID {49E22055-BF50-472B-9B8C-040EA68951D7}). Raise the Height parameter from 110 to 360. Then find the ScrollablePanelTargets rendering (ID {49E22055-BF50-472B-9B8C-040EA68951D7}). Lower the Height parameter from 110 to 60.

Then go find the item /sitecore/client/Applications/Publishing/PublishDialog/PageSettings/TabControlPublish/PublishTab, ID {1FE1550C-A1E2-486B-8CDE-2965D4CF379B} and do a similar change. (Why reuse code when you can duplicate it…?) In the __Renderings field, find the ScrollablePanelLanguages rendering (ID {3B5A865D-E3B1-49A6-B603-4E5349FB6878}). Raise the Height parameter from 160 to 400. Then find the ScrollablePanelTargets rendering (ID {D879F50A-669D-4012-B0E1-5990E3297DA1}). Lower the Height parameter from 160 to 60.

Here’s the result of this change:

The language list is still unsorted though. I don’t know why Sitecore find it so hard to fix these things. It’s wrong almost everywhere, and every language listing seems to be a separate implementation.

A Sitecore solution is typically behind one or several reverse proxies, such as load balancers, content delivery networks etc. From a Content Delivery server perspective, the remote address, i.e. “the visible client IP” is the closes proxy instead of the IP of the connecting client. To solve this, the chain of proxies adds a http header with the IP address it’s communicating with. This header is typically called X-Forwarded-For or X-Real-IP.

Below is an example of such setup. Each proxy adds the IP they’re receiving the connection from:

Note: A corporate proxy typically don’t disclose their internal IP, so the a.a.a.a record in this example is not very common, but having it here makes it easier to follow.

From the example picture above, the Nginx Reverse Proxy closest to the Content Delivery Server would be seen with the RemoteAddress variable. It’ll add the load balancer IP at the end of the X-Forwarded-For header (d.d.d.d). The load balancer adds the CDN endpoint (c.c.c.c) and the CDN edge adds the client IP or the proxy it’s communicating with (b.b.b.b). The a.a.a.a IP is typically not seen, but could be there if a visitor is accessing the site via a proxy.

In Sitecore we use the XForwardedFor processor in the createVisit pipeline to resolve the public IP address for the client. This processor has an optional HeaderIpIndex in order to specify the number of proxies you have chained before facing the Internet. So in this setup, the HeaderIpIndex should be set to 2, i.e. the third IP from the end in X-Forwarded-For IP list.

Now, there’s currently a bug in the XForwardedFor class provided by Sitecore in at least version 9.0 update 1 and update 2. It probably applies to more versions. The processor counts the IP array from the start instead of the end. This means it “works” when there’s no proxy between the client and your internet facing proxy. The client may also spoof any IP, as the client can pass along any X-Forwarded-For header to the request.

Fortunately it’s easy to fix this by just replacing the processor. The GetIpFromHeader method just needs a few small changes:

protected virtual string GetIpFromHeader(string header)
{
    Assert.ArgumentNotNull(header, nameof(header));
    var ipAddresses = header.Split(',');
    int headerIpIndex = HeaderIpIndex;
    // "ipAddresses.Length - headerIpIndex" as index instead of just "headerIpIndex"
    string ipString = headerIpIndex < ipAddresses.Length ? ipAddresses[ipAddresses.Length - headerIpIndex - 1] : ipAddresses.LastOrDefault();
    if (string.IsNullOrWhiteSpace(ipString))
        return null;
    ...

As Grant Killian points out in his post on this, some reverse proxies adds port number to the IP address. If so, the port number needs to be filtered out. It’s also worth noticing that the list of IP addresses may also contain IPv6 addresses. Those use colon (:) instead of dots (.) as group separators. In addition, IPv6 addresses can also be shortened by omitting groups with zeros. Se we can’t really strip a port number in a reliable way for IPv6.

Continuing the above method, could look something like this:

private static readonly Regex IPv4WithPort = new Regex(@"^([\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}\.[\d]{1,3}):([\d]+)$", RegexOptions.Compiled);
private static readonly Regex IPv6WithPort = new Regex(@"^\[([0-9a-fA-F:]+)\]:([\d]+)$", RegexOptions.Compiled);
    ...
    ipString = ipString.Trim();
    if (ipString.IndexOf(':') >= 0) // Quick test for performance
    {
        // Is this in the format 0.0.0.0:0 ?
        var match = IPv4WithPort.Match(ipString);
        if (match.Success)
        {
            return match.Groups[1].Value;
        }
        // Is this in the format [2001:db8:85a3::8a2e:370:7334]:0
        // As IPv6 formats may be shortened, so must be escaped (RFC 3986, sec 3.2.2)
        match = IPv6WithPort.Match(ipString);
        if (match.Success)
        {
            return match.Groups[1].Value;
        }
    }
    return ipString;
}

The Sitecore Publish Service vastly improves the publish performance in Sitecore. For me it was really hard to get it working properly and I’ve blogged about some of the issues before. I received a lot of good help from Sitecore Support and now it seems like I’ve got into a quite stable state.

However, there is a backside of the Publish Service that may affect the editing performance. Publish Service doesn’t use the PublishQueue table for knowing what to publish. Instead it has an event mechanism for detecting what needs to be published. As an item is saved, Sitecore emits events to the Publish Service so that it knows what pages should be put into the publish manifest.

Note: The solution in this post may not suit every project. Address this only if you’re experiencing the performance decade described and make sure you test everything well. Make sure you fully understand this approach before dropping it into your project.

As part of the Publish Service package, a item:saved event handler is added to do some post processing. When a unversioned field is changed on an item, the event handler loops over all versions of that language and updates the __Revision field. When a shared field is changed on an item, the event handler loops over all versions on all languages and updates the __Revision field. Thereby the Publish Service gets a notification that the content of the item has been changed.

Here’s where the problem started for me. As a shared field is edited, all languages and versions of that item is saved, as just described. This caused performance issues in a large solution I work with. With over 40 languages under version control, an item can easily have 100 versions or more in total. Saving 100 item versions, trigger 100 item:saving events and 100 item:saved events and so on takes time. Many seconds it turned out.

So I tried to optimize this, by reducing the number of fired events so that the Publish Service would be notified without all the heavy load from the rest of the save events. Essentially my approach is to update the __Revision field and save it silently (i.e. without firing the save events) and manually emit the item changed notification to the Publish Service. This reduces the amount of work needed to be performed since:

• The Save action of the item versioned being edited, is already triggering a refresh on the index on all language versions. No need to do it again
• The Links database doesn’t need to be updated for every other language version, as only the __Revision field is changed on those.
• The Globalization ItemEventHandler isn’t needed, as no content is changed.
• PlaceholderCacheEventHandler shouldn’t be affected by a change in the __Revision field.
• Update of item archiving and reminders aren’t affected by the revision field.

Note: This solution may only apply to Publish Service 3.0.1. I’ve heard Sitecore is working on another approach to address this. That solution may be under NDA, so I’ll just leave it at that with no further comments.

As mentioned, we do need to emit the notifications to the Publish Service and for that we need to grab an IItemOperationEmitter. To get hold of that one, we need to change how this is registered in the ServicesConfigurator. So we register a replacement of the service configurator, like this:

<services>
  <configurator type="SitecorePatches.Service.CustomPublishingServiceConfigurator, SitecorePatches"
                 patch:instead="*[@type='Sitecore.Publishing.Service.PublishingServiceConfigurator, Sitecore.Publishing.Service']"/>
</services>

The new service configurator is slightly different from the original:

public class CustomPublishingServiceConfigurator : IServicesConfigurator
{
    public virtual void Configure(IServiceCollection serviceCollection)
    {
        var existingPublishServiceDescriptor = serviceCollection.FirstOrDefault(x => x.ImplementationType == typeof(BasePublishManager));
        if (existingPublishServiceDescriptor != null)
            serviceCollection.Remove(existingPublishServiceDescriptor);

        // Add the IItemOperationEmitter into the service locator, so that we can use it later
        var operationsEmitter = ServiceDescriptor.Singleton<IItemOperationEmitter>(serviceProvider =>
        {
            var requiredService = serviceProvider.GetRequiredService<BaseFactory>();
            return (IItemOperationEmitter) requiredService.CreateObject("publishing.service/operationEmitter", true);
        });
        serviceCollection.Add(operationsEmitter);

        var publishServiceDescriptor = ServiceDescriptor.Singleton(sp => {
            var requiredService = sp.GetRequiredService<BaseFactory>();
            // Use the same instance as registered above.
            var emitter = sp.GetRequiredService<IItemOperationEmitter>();
            return (BasePublishManager) ActivatorUtilities.CreateInstance<Sitecore.Publishing.Service.PublishManager>(sp, 
                requiredService.CreateObject(string.Format(CultureInfo.InvariantCulture, "{0}/publishingJobProviderFactory", "publishing.service"), true), 
                requiredService.CreateObject(string.Format(CultureInfo.InvariantCulture, "{0}/publishJobQueueService", "publishing.service"), true), 
                requiredService.CreateObject(string.Format(CultureInfo.InvariantCulture, "{0}/publisherOpsService", "publishing.service"), true),
                emitter);
        });
        serviceCollection.Add(publishServiceDescriptor);
    }
}

We then modify the UpdateItemVariantRevisions event like this:

public void UpdateItemVariantRevisions(object sender, EventArgs args) 
{
    // .... (cut out unchanged code)

    using (new SecurityDisabler())
    {
        // Grab the emitter instance we registered in the DI container above
        var emitter = ServiceLocator.ServiceProvider.GetService<IItemOperationEmitter>();
        var itemPath = savedItem.Paths.GetPath(ItemPathType.ItemID);
        foreach (Item item in itemsToUpdate)
        {
            item.Editing.BeginEdit();
            var revisionGuid = Guid.NewGuid();
            item.Fields[FieldIDs.Revision].SetValue(revisionGuid.ToString(), true);
            item.Editing.EndEdit(false, true);  // Save silently

            // Manually send the notifications to Publish Service:
            var restrictions = new ItemOperationRestrictions(savedItem.Publishing.PublishDate, savedItem.Publishing.UnpublishDate, item.Publishing.ValidFrom, item.Publishing.ValidTo);
            var variantUri = new ItemVariantLocator(item.Database.Name, item.ID.Guid, item.Language.Name, item.Version.Number);
            var operationData = new ItemOperationData(DateTime.UtcNow, restrictions, variantUri, revisionGuid,
                itemPath, Sitecore.Framework.Publishing.PublisherOperations.PublisherOperationType.VariantSaved);
            emitter?.PostOperation(operationData);
        }
    }
}

By using this approach, I got the time for saving an item from several seconds down to well under one second.

Some people find it a bit tricky to write Sitecore config files. It can sometimes be a bit tricky or time consuming to get the element structure correct. Ever found yourself debugging an issue where it turned out the config file wasn’t applied properly due to an element structure mistake?

The XPath Tools plugin, by Uli Weltersbach, for Visual Studio is a great help for creating those config patch files. Here’s a way to create those in a fast and simple way:

Start with a Sitecore config boiler plate:

Then grab the expanded Sitecore config file through Sitecore Rocks:

Navigate to the section where you want to apply your config change and select “Copy XML structure”. In this example I’ll add a computed field to the default Solr index section:

Then paste it into your config template:

From here on it’s a simple task to just remove the surplus attributes you don’t need, such as database="SQLServer" xmlns:patch… and so on:

So XPath tool is a small but very helpful tool that makes life just a little bit easier.

You can get the XPath tool from within Visual Studio or download it here, or grab it from GitHub.

I’ve written a few posts on Sitecore Content Search and Solr already, but there seems to be an infinite amount of things to discover and learn in this area. Previously I’ve pointed out the importance of configuring the Solr index correctly and the benefit of picking the fields to index, i.e. not indexing all fields as default (<indexAllFields>false</indexAllFields>). This will vastly improve the performance of Content Search operations and reduce the index size in large solutions.

Recently I’ve been investigating a performance issue with one of our Sitecore solutions. This one is running Sitecore 9 with quite a lot of data in it. It’s been performing quite well, but as the client were loading more data into it, it got a lot slower. Our metrics also showed the response time (P95) in the data center that got quite high. It measured around 500 ms instead of the normal 100 ms.

The cause of this wasn’t obvious and most things looked good. Database queries were fast, caches worked as expected, CPU load was somewhat high, but not overloaded. I also looked at our Solr queries and the query time reported by Solr were typically below 25 ms. It wasn’t until I looked at network traffic between the servers, I discovered that a lot of data were sent between Solr and IIS. IIS received about ten times the data as it was sending/receiving to/from any other system.

So I started looking at the actual queries being performed and I knew some queries returns many records. But I was surprised when I found queries returning payloads of 500kB or more. This creates latency and CPU load when parsing it!

So I started digging into our code and at first glance everything looked quite good. The code followed a regular pattern, like:

public class MyModel {
  [IndexField("_group")]
  [TypeConverter(typeof(IndexFieldIDValueConverter))]
  public ID ItemId { get; set; }
  
  [IndexField("_language")]
  public string LanguageName { get; set; }
  // more needed fields ....
}

var result = searchContext.GetQueryable<MyModel>()
  .Filter(filters)
  .GetResults();
foreach (var myModel in result.Hits) {
  // do stuff
}

I soon realized that Sitecore puts fl=*,score (Field List) in the Solr query string. You can see this in your Search.log. This means all stored fields are returned in the result. But in most cases we were only interesting in one or a few fields, like the ones specified in our model MyModel. So I looked at finding a way to fetch only the required fields. writing the queries in a way where only the needed fields were returned. It turned out that the LinqToSolrIndex were indeed managing a field set and builds the Solr fl parameter out of that.

So it turns out we can use the Select feature before GetResults to control the fields list. However, it complicates the code a bit, because it means that MyModel in the example above won’t be returned, but it is used when picking the fields and for constructing the query. The code pattern below explains it a bit more. Note: You don’t have to use different models and the query model doesn’t have to inherit the result model. I’ve written it like this to clarify the relation. It’ll probably become easier to understand when writing this in Visual Studio and you see what Intellisense gives you.

public class MyResultModel {
  [IndexField("_group")]
  [TypeConverter(typeof(IndexFieldIDValueConverter))]
  public ID ItemId { get; set; }
  // more fields requested
} 

public class MyQueryModel : MyResultModel   {
  [IndexField("_language")]
  public string LanguageName { get; set; }
  // more fields needed for filtering.... 
}

var result = searchContext.GetQueryable<MyQueryModel>()
  .Filter(filters)
  .Select(x => new { x.ItemId, /* more fields requested */ })
  .GetResults()
  .Select(x => new MyResultModel {
    ItemId = x.Document.ItemId,
    /* mapping of more requested fields */
  });
foreach (var myResultModel in result) {
  // do stuff
}

Essentially, you can use Select to control what fields are returned, but you’ll also get a dynamic type returned that you probably want to convert back into a typed model.

So, was it worth doing those code changes in the project. Well, the answer was YES! I went through all Content Search queries that could return more than ten rows, i.e. queries that didn’t have a low .Take(n) value.

As I mentioned previously, we had already optimized the index storage, so we were already storing only the field we’re actually using in the application. Still I found that every record was quite heavy. In the serialized XML used in the network transport between Solr and Sitecore, every record were around 3630 bytes. When picking only the fields needed, we got down to around 270 bytes each. The performance within the data center improved a lot. The response time figure looked like this before/after deploying this code change:

I was blown away how much difference this did to the solution as a whole. And the Solr queries are still not perfect. Looking at the search log, there are still a few additional fields returned from Solr that I don’t need. It turnes out that Sitecore adds the fields _uniqueid, _datasource and score regardless of what I select. So my 270 bytes/record could go as low as 70 bytes/record if those were removed too, but that would probably be sub optimizing…

I don’t know why Sitecore always adds the first two, and the last one, score, turned out to be a bug. The GetResults method has an optional GetResultsOptions enum parameter. It may be Default or GetScores. However this parameter doesn’t do anything. Sitecore will always ask for score, unless you do something like ToArray instead of GetResults. That could be an option to reduce it a bit more, but then you won’t get the TotalSearchResults or Facets properties.

So my recommendation is to look through your Search.log and filter out the rows where fl=*,score and rows=a large number. Consider optimizing those with a limited field list or number of returned rows.

Thank you Sitecore for awarding me “Most Valuable Professional” (MVP) again! Seven years in row!

The Sitecore MVP Award celebrates the most active Sitecore community members from around the world who provide valuable online and offline expertise that enriches the community and makes a difference

My contribution to Sitecore and the community over the last year have, besides the nine posts on this blog, have been mostly focused on improving the product by having a dialog with various Sitecore staff. During 2018 I filed over 50 confirmed bugs, mostly related to Sitecore Publish Service and Content Search and a handful of accepted product enhancements.