SetMyProfilePicture fails on cross farm MySites host

Well, this one was a doozy.  Here’s the short version:  If you have a multi-farm SharePoint topology in which the MySites host web application is in a different farm than the Content web application (as depicted in MS diagram here), attempting to use the PeopleManager.SetMyProfilePicture API method will fail with a very enlightening “System.IO.FileNotFoundException”.

Here’s the topology we were working with:

Services Farm: contains User Profile Service and MySites host web application

Content Farm: contains collaboration web application

Provider Hosted App: makes API calls back to Content farm context.

The use case is as follows:

1. User browses to content farm url.
2. User clicks on provider hosted app launch icon.
3. Page in provider hosted app displays upload control to select new image for user profile.  Code in provider hosted app then takes that image and makes API call back to host web API url. (A good summary of such a callback can be found at http://www.vrdmn.com/2013/11/set-userprofile-picture-using-net.html)
4. Error is returned from API call.

The specific ULS log entry is recorded on the content farm web server, indicating that the upload attempt failed.
05/02/2016 17:19:54.87        w3wp.exe (0x15E4)        0x3138        SharePoint Portal Server        User Profiles        aj37q        Unexpected        PeopleManager.SetMyProfilePicture:Exception:System.IO.FileNotFoundException: The Web application at https://mysites.contoso.com/ could not be found. Verify that you have typed the URL correctly. If the URL should be serving existing content, the system administrator may need to add a new request URL mapping to the intended application.     at Microsoft.SharePoint.SPSite.LookupSiteInfo(SPFarm farm, Boolean contextSite, Boolean swapSchemeForPathBasedSites, Uri& requestUri, Boolean& lookupRequiredContext, Guid& applicationId, Guid& contentDatabaseId, Guid& siteId, Guid& siteSubscriptionId, SPUrlZone& zone, String& serverRelativeUrl, Boolean& hostHeaderIsSiteName, Boolean& appWebRequest, String& appHostHeaderRedirectDomain, String& appSiteDomainPrefix, String& subscriptionName, String& appSiteDomainId, Uri& primaryUri)     at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, Boolean swapSchemeForPathBasedSites, SPUserToken userToken)     at Microsoft.SharePoint.SPSite..ctor(String requestUrl)     at Microsoft.Office.Server.UserProfiles.UserProfileGlobal.GetOrCreatePictureFolder(String mySiteHostUrl, ProfileType profileType, Boolean createIfNotFound, Boolean forFeedAttachment)     at Microsoft.Office.Server.UserProfiles.PeopleManager.SetUsersPicture(Stream picture, Stream largePicture, UserProfile up, Hashtable properties, Boolean allowUnsafeUpdates)     at Microsoft.Office.Server.UserProfiles.PeopleManager.SetUsersPicture(Stream picture, UserProfile up, Hashtable properties, Boolean allowUnsafeUpdates)     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.Office.Server.UserProfiles.PeopleManager.SetMyProfilePicture(Stream picture)        ac43789d-8d0c-102a-5be8-9d2f8473fda7

In similar scenarios, we had to ensure that all the Server-to-Server (S2S) OAuth authentication channels were in the same realm and successfully using matched TrustedSecurityTokenIssuers.  But in this case, it was apparent that this was not an authentication issue.  So we had to dig deeper.

Using ILSpy I opened up the Microsoft.Office.Server.UserProfiles.dll found in the SharePoint hive.

Here’s the code for the SetUserPicture method.  You’ll see that it first needs to verify that the upload destination is found.  For user profile images, this location is the User Photos document library on the root site collection of the MySites host.

The UserProfileGlobal.GetOrCreatePictureFolder method takes the MySites Url as its first parameter.  We knew our Service Connection to the published User Profile Service application from the services farm worked because it was able to correctly connect and get the MySiteHostUrl value.

Next I jumped in to the method used to verify the upload folder.

Here we see that the first step is to attempt to open a SPSite object with the MySites url – which in our topology is in a different Farm.  Thus, the site cannot be found, and the upload sequence aborts.

Here’s what it looks like when we follow the same steps in PowerShell on the Content Farm web server.

The fix was to recreate the MySites host web application in the Content Farm, so that the lookup succeeded.  

Microsoft appears to have conflicting standards regarding the location of the Mysites Host site.  In the Plan for My Sites in SharePoint Server 2013 article, there is this paragraph under the Planning for geographically distributed deployments heading: