Improving REST API performance with JSON Light

Copyright from: www.vrdmn.com

2817429​ Minimal and no metadata are now enabled as supported JSON formats. 

This really caught my attention as it was something I was waiting for. When we make a call with the REST API, there is a lot of additional data which we get back in the response. This additional data consists of the metadata and the Hyper Media Controls along with the required JSON data.  To know more about this, please see this excellent post describing REST Maturity Models: http://martinfowler.com/articles/richardsonMaturityModel.html

This makes the payload size of the REST response too big. If you are developing a REST heavy application, then each extra byte of data that travels over the wire to your client adds up and ends up hampering performance.

I did some digging around and found out that my SharePoint Online Tenant was already supporting these JSON formats! All I had to do was to modify the Accept header of my REST call.

Note: As mentioned before, for these formats to work on an On-Premises SharePoint 2013, you will need to install SP1.

I ran some tests using the Chrome Developer Tools and Postman REST Client to see the effect on the payload size.

All tests were done to simply fetch the current web details with the following api call:

https://siteurl.sharepoint.com/sites/test/_api/web

1) Accept : "application/json;odata=verbose"

According to Microsoft Guidance published before, to get the JSON data back from the REST call, we need to specify the Accept header as"application/json;odata=verbose" But I think that guidance might be a little outdated now. Here is the result of the REST Call made with the above header:

Payload
Size: 2.0 KB
Content Size: 5.1 KB

    (Click on Image to Enlarge)

And here is the JSON we get back. You can see that there is a lot of metadata along with the Hypermedia Controls. 

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354

{ "d": { "__metadata": { "id": "https://siteurl.sharepoint.com/sites/test/_api/Web", "uri": "https://siteurl.sharepoint.com/sites/test/_api/Web", "type": "SP.Web" }, "FirstUniqueAncestorSecurableObject": { "__deferred": { "uri": "https://siteurl.sharepoint.com/sites/test/_api/Web/FirstUniqueAncestorSecurableObject" } }, "RoleAssignments": { "__deferred": { "uri": "https://siteurl.sharepoint.com/sites/test/_api/Web/RoleAssignments" } }, "AllProperties": { "__deferred": { "uri": "https://siteurl.sharepoint.com/sites/test/_api/Web/AllProperties" } }, "AppTiles": { "__deferred": { "uri": "https://siteurl.sharepoint.com/sites/test/_api/Web/AppTiles" } }, //*******Lots of Hypermedia Controls removed for formatting************ "AllowRssFeeds": true, "AlternateCssUrl": "", "AppInstanceId": "00000000-0000-0000-0000-000000000000", "Configuration": 0, "Created": "2014-05-27T21:02:57", "CustomMasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "Description": "", "DocumentLibraryCalloutOfficeWebAppPreviewersDisabled": false, "EnableMinimalDownload": false, "Id": "fd6009e8-2965-4e50-9564-06a10d13add1", "Language": 1033, "LastItemModifiedDate": "2014-06-04T09:28:01Z", "MasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "QuickLaunchEnabled": true, "RecycleBinEnabled": true, "ServerRelativeUrl": "/sites/test", "SiteLogoUrl": null, "SyndicationEnabled": true, "Title": "Test Bench", "TreeViewEnabled": false, "UIVersion": 15, "UIVersionConfigurationEnabled": false, "Url": "https://siteurl.sharepoint.com/sites/test", "WebTemplate": "STS" } }

view rawrestodataverbose.js hosted with ❤ by GitHub

2) Accept : "application/json;odata=minimalmetadata" (OR Accept : "application/json")

Now when optimizing for performance, we do not need all the metadata and all the Hyper Media Controls as we are only concerned with the JSON data. We can then use this header to optimize them. It will return the required JSON with very minimal metadata attached to it. This is also the default option if you only specify "application/json" in your Accept header without specifying the odata parameter.

Payload
Size: 1.5 KB
Content Size: 1.0 KB

   (Click on Image to Enlarge)

And this is the JSON returned. You can see that the Hyper Media Controls are no longer returned and only some of the metadata is returned.

123456789101112131415161718192021222324252627282930

{ "odata.metadata": "https://siteurl.sharepoint.com/sites/test/_api/$metadata#SP.ApiData.Webs/@Element", "odata.type": "SP.Web", "odata.id": "https://siteurl.sharepoint.com/sites/test/_api/Web", "odata.editLink": "Web", "AllowRssFeeds": true, "AlternateCssUrl": "", "AppInstanceId": "00000000-0000-0000-0000-000000000000", "Configuration": 0, "Created": "2014-05-27T21:02:57", "CustomMasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "Description": "", "DocumentLibraryCalloutOfficeWebAppPreviewersDisabled": false, "EnableMinimalDownload": false, "Id": "fd6009e8-2965-4e50-9564-06a10d13add1", "Language": 1033, "LastItemModifiedDate": "2014-06-04T09:28:01Z", "MasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "QuickLaunchEnabled": true, "RecycleBinEnabled": true, "ServerRelativeUrl": "/sites/test", "SiteLogoUrl": null, "SyndicationEnabled": true, "Title": "Test Bench", "TreeViewEnabled": false, "UIVersion": 15, "UIVersionConfigurationEnabled": false, "Url": "https://siteurl.sharepoint.com/sites/test", "WebTemplate": "STS" }

view rawrestodataminimal.js hosted with ❤ by GitHub

3) Accept : "application/json;odata=nometadata"

This is the option for the extreme "optimizers" who want no metadata or Hypermedia Controls attached to the response and are only concerned with the JSON

Payload
Size: 1.4 KB
Content Size: 800 Bytes

   (Click on Image to Enlarge)

And this is the JSON returned:

1234567891011121314151617181920212223242526

{ "AllowRssFeeds": true, "AlternateCssUrl": "", "AppInstanceId": "00000000-0000-0000-0000-000000000000", "Configuration": 0, "Created": "2014-05-27T21:02:57", "CustomMasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "Description": "", "DocumentLibraryCalloutOfficeWebAppPreviewersDisabled": false, "EnableMinimalDownload": false, "Id": "fd6009e8-2965-4e50-9564-06a10d13add1", "Language": 1033, "LastItemModifiedDate": "2014-06-04T09:28:01Z", "MasterUrl": "/sites/test/_catalogs/masterpage/seattle.master", "QuickLaunchEnabled": true, "RecycleBinEnabled": true, "ServerRelativeUrl": "/sites/test", "SiteLogoUrl": null, "SyndicationEnabled": true, "Title": "Test Bench", "TreeViewEnabled": false, "UIVersion": 15, "UIVersionConfigurationEnabled": false, "Url": "https://siteurl.sharepoint.com/sites/test", "WebTemplate": "STS" }

view rawrestodatanometadata.js hosted with ❤ by GitHub

So as you can see, if you want to reduce the payload size from the REST response, you can use the different JSON formats depending on the degree of optimization you want.

Hope you found this information useful! 

Parsing XML with jQuery using XPath or CSS path or CSS Selectors

Copyright from: http://www.kumarchetan.com

This is just a continuation to my previous post. I missed a very simple thing in the post. jQuery allows you to use CSS path, commonly known as CSS selectors, which is somewhat similar to XPath. For uninformed, XPath is a simpler way to query nodes in an XML document. You can even use filters with these selectors. Visit this doc page on jQuery doc website for more. Using the same code base I made following changes to access items node from my XML feed file.

//The xpath way
$(xml).find( "channel>item" ).each(
    function(){
        var item = $(this),
        title =  item.find('title').text(),
        link =  item.find('link').text(),
        itemid =  item.attr('id');//get an attribute
        console.log(title);
        console.log(link);
        console.log('The attribute value "' + itemid + '".');
    }
);

I have update the code on github page. How can you

Parsing XML with jQuery Part 1

Copyright from: www.kumarchetan.com

I recently worked on a small Android App using PhoneGap & jQuery. The app did a very small task fetch the XML feed from my blog and display it. Simple. This meant parsing XML using jQuery. Most of the examples on web used this particular example in one form or another. Not all XML document are like record sets. For example a blog feed will have following structure

 
 http://www.kumarchetan.com/blog
    
        
        http://www.kumarchetan.com/blog/2012/01/14/my-first-android-app-or-how-to-develop-an-android-app-using-phonegap-and-jquery/
    
    
        
        http://www.kumarchetan.com/blog/2011/12/11/how-not-to-do-a-ci-controller-tips-to-develop-codeigniter-controllers/
    

The interesting part in this XML is that tags or nodes with similar names exist through out the document at different depth. This XML has attributes attached to its nodes. This is how a typical XML may look like. How do you actually parse this using jQuery? jQuery has find(). From the documentation

Given a jQuery object that represents a set of DOM elements, the .find() method allows us to search through the descendants of these elements in the DOM tree and construct a new jQuery object from the matching elements.

We can convert this XML into a jQuery object and then use find() to actually parse it. Here is a code snippet

var xml = '  

SharePoint Online: Sideloading of apps is not enabled on this site.

Copyright from: www.superedge.net

If you are deploying a solution to SharePoint Online, at some point you will likely come across the following error: Error occurred in deployment step 'Install app for SharePoint': Sideloading of apps is not enabled on this site.

So...In short, you will not be able to deploy any solution from Visual Studio to SharePoint Online/Office 365 without enabling Sideloading first. I will show you how to fix this and then explain later, so you don’t waste any time looking for the answer. After all, I believe you came here to find the fix first

2-Steps Solution:
1) Download and install SharePoint Online Management Shell 
2) Execute the script below. (Note that the Sideloading Feature ID is harcoded in SharePoint, as are many other OOTB features)

#CODE STARTS HERE
$programFiles = [environment]::getfolderpath("programfiles")
add-type -Path $programFiles'\SharePoint Online Management Shell\Microsoft.Online.SharePoint.PowerShell\Microsoft.SharePoint.Client.dll'
Write-Host 'Ready to enable Sideloading'
$siteurl = Read-Host 'Site Url'
$username = Read-Host "User Name"
$password = Read-Host -AsSecureString 'Password'
 
$outfilepath = $siteurl -replace ':', '_' -replace '/', '_'
 
try
{
    [Microsoft.SharePoint.Client.ClientContext]$cc = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)
    [Microsoft.SharePoint.Client.SharePointOnlineCredentials]$spocreds = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)
    $cc.Credentials = $spocreds
    $site = $cc.Site;

    $sideLoadingGuid = new-object System.Guid "AE3A1339-61F5-4f8f-81A7-ABD2DA956A7D"
    $site.Features.Add($sideLoadingGuid, $true, [Microsoft.SharePoint.Client.FeatureDefinitionScope]::None);
     
    $cc.ExecuteQuery();
     
    Write-Host -ForegroundColor Green 'SideLoading feature enabled on site' $siteurl
    #Activate the Developer Site feature
}
catch
{ 
    Write-Host -ForegroundColor Red 'Error encountered when trying to enable SideLoading feature' $siteurl, ':' $Error[0].ToString();
}

#CODE ENDS HERE

In your SPO Management Shell enter the url of the site you want to deploy the solution, the administrator username and the password. The powershell will take care of the rest.

Now off you go. You’re ready to deploy solutions from your Visual Studio straight into Office 365. You can also get a more elaborate code in the MSDN Code Solution Repository.

How to copy files between sites using JavaScript REST in Office365 / SharePoint 2013

I’m currently playing with a POC for an App, and wanted to try to do the App as a SharePoint hosted one, only using JavaScript and REST.

The starting point was to call _vti_bin/ExcelRest.asmx on the host web from my app web, but this end-point does neither support CORS nor JSONP, so it can’t be used directly. My next thought was; Ok, let’s copy the file from the host web over to my app web, then call ExcelRest locally. Easier said than done!

While the final solution seems easy enough, the research, trial and error have taken me about 3 days. I’m now sharing this with you so you can spend your valuable time increasing the international GDP instead.

Note: If you want to copy files between two libraries on the same level, then you can use the copyTo method. http://server/site/_api/web/folders/GetByUrl('/site/srclib')/Files/getbyurl('madcow.xlsx')/copyTo(strNewUrl = '/site/targetlib/madcow.xlsx,bOverWrite = true)

Problem

Copy a file from a document library in one site to a document library in a different site using JavaScript and REST.
The code samples have URL’s using the App web proxy, but it’s easily modifiable for non-app work as well.

Step 1 – Reading the file

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

var hostweburl = decodeURIComponent(getParameterByName('SPHostUrl')); var appweburl = decodeURIComponent(getParameterByName('SPAppWebUrl'));   var fileContentUrl = "_api/SP.AppContextSite(@target)/web/GetFileByServerRelativeUrl('/site/library/madcow.xlsx')/$value?@target='" + hostweburl + "'";   var executor = new SP.RequestExecutor(appweburl); var info = {     url: fileContentUrl,     method: "GET",     binaryStringResponseBody: true,     success: function (data) {         //binary data available in data.body         var result = data.body;     },     error: function (err) {         alert(JSON.stringify(err));     } }; executor.executeAsync(info);

The important parameter here is setting binaryStringResponseBody to true. Without this parameter the response is being decoded as UTF-8 and the result in the success callback is garbled data, which leads to a corrupt file on save.

The  binaryStringResponseBody parameter is not documented anywhere, but I stumbled upon binaryStringRequestbody in an msdn article which was used when uploading a file, and I figured it was worth a shot. Opening SP.RequestExecutor.debug.js I indeed found this parameter.

Step 2 – Patching SP.RequestExecutor.debug.js

Adding binaryStringResponseBody will upon return of the call cause a script error as seen in the figure below.




The method in question is reading over the response byte-by-byte from an Uint8Array, building a correctly encoded string. The issue is that it tries to concatenate to a variable named ret, which is not defined. The defined variable is named $v_0, and here we have a real bug in the script. The bug is there both in Office365 and SharePoint 2013 on-premise.

Luckily for us patching JavaScript is super easy. You merely override the methods involved somewhere in your own code before it’s being called. In the below sample it’s being called once the SP.RequestExecutor.js library has been loaded. The method named BinaryDecode is the one with the error, but you have to override more methods as the originator called is internalProcessXMLHttpRequestOnreadystatechange, and it cascades to calling other internal functions which can be renamed at random as the method names are autogenerated. (This happened for me today and I had to change just overrinding the first function).

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85

$.getScript(scriptbase + "SP.RequestExecutor.js", function(){ SP.RequestExecutorInternalSharedUtility.BinaryDecode = function SP_RequestExecutorInternalSharedUtility$BinaryDecode(data) {    var ret = '';      if (data) {       var byteArray = new Uint8Array(data);         for (var i = 0; i < data.byteLength; i++) {          ret = ret + String.fromCharCode(byteArray[i]);       }    }    ;    return ret; };   SP.RequestExecutorUtility.IsDefined = function SP_RequestExecutorUtility$$1(data) {    var nullValue = null;      return data === nullValue || typeof data === 'undefined' || !data.length; };   SP.RequestExecutor.ParseHeaders = function SP_RequestExecutor$ParseHeaders(headers) {    if (SP.RequestExecutorUtility.IsDefined(headers)) {       return null;    }    var result = {};    var reSplit = new RegExp('\r?\n');    var headerArray = headers.split(reSplit);      for (var i = 0; i < headerArray.length; i++) {       var currentHeader = headerArray[i];         if (!SP.RequestExecutorUtility.IsDefined(currentHeader)) {          var splitPos = currentHeader.indexOf(':');            if (splitPos > 0) {             var key = currentHeader.substr(0, splitPos);             var value = currentHeader.substr(splitPos + 1);               key = SP.RequestExecutorNative.trim(key);             value = SP.RequestExecutorNative.trim(value);             result[key.toUpperCase()] = value;          }       }    }    return result; };   SP.RequestExecutor.internalProcessXMLHttpRequestOnreadystatechange = function SP_RequestExecutor$internalProcessXMLHttpRequestOnreadystatechange(xhr, requestInfo, timeoutId) {    if (xhr.readyState === 4) {       if (timeoutId) {          window.clearTimeout(timeoutId);       }       xhr.onreadystatechange = SP.RequestExecutorNative.emptyCallback;       var responseInfo = new SP.ResponseInfo();         responseInfo.state = requestInfo.state;       responseInfo.responseAvailable = true;       if (requestInfo.binaryStringResponseBody) {          responseInfo.body = SP.RequestExecutorInternalSharedUtility.BinaryDecode(xhr.response);       }       else {          responseInfo.body = xhr.responseText;       }       responseInfo.statusCode = xhr.status;       responseInfo.statusText = xhr.statusText;       responseInfo.contentType = xhr.getResponseHeader('content-type');       responseInfo.allResponseHeaders = xhr.getAllResponseHeaders();       responseInfo.headers = SP.RequestExecutor.ParseHeaders(responseInfo.allResponseHeaders);       if (xhr.status >= 200 && xhr.status < 300 || xhr.status === 1223) {          if (requestInfo.success) {             requestInfo.success(responseInfo);          }       }       else {          var error = SP.RequestExecutorErrors.httpError;          var statusText = xhr.statusText;            if (requestInfo.error) {             requestInfo.error(responseInfo, error, statusText);          }       }    } }; });

Step 3 – Uploading the file

The next step is to save the file in a library on my app web. The crucial part again is to make sure the data is being treated as binary, this time withbinaryStringRequestBody set to true. Make a note of the digest variable as well. On a page inheriting the SP masterpage you can get this value with $("#__REQUESTDIGEST").val(). If not then you have to execute a separate call to _api/contextinfo. The code for that is at the bottom of this post.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

var appweburl = decodeURIComponent(getParameterByName('SPAppWebUrl')); var executor = new SP.RequestExecutor(appweburl); var info = {     url: "_api/web/GetFolderByServerRelativeUrl('/appWebtargetFolder')/Files/Add(url='madcow.xlsx', overwrite=true)",     method: "POST",     headers: {         "Accept": "application/json; odata=verbose",         "X-RequestDigest": digest     },     contentType: "application/json;odata=verbose",     binaryStringRequestBody: true,     body: data.body,     success: function(data) {          alert("Success! Your file was uploaded to SharePoint.");     },     error: function (err) {         alert("Oooooops... it looks like something went wrong uploading your file.");     } }; executor.executeAsync(info);

Journey

I started out using jQuery.ajax for my REST calls, but I did not manage to get the encoding right no matter how many posts I read on this. I read through a lot on the following links which led me to the final solution:

• http://social.msdn.microsoft.com/Forums/sharepoint/en-US/02bfbcdc-73c8-4fa5-8967-cfd903a0d72e/javascript-client-object-model-openbinary-method – which got me started on reading files, but the encoding was wrong


• http://www.shillier.com/archive/2013/03/26/uploading-files-in-sharepoint-2013-using-csom-and-rest.aspx – which has good code and samples on uploading files with CSOM and REST, also mentioning that CSOM has a limit on files being 1.5mb, which is why I went for REST.


• http://msdn.microsoft.com/en-us/library/jj164022.aspx – General REST information


• http://blogs.msdn.com/b/uksharepoint/archive/2013/04/20/uploading-files-using-the-rest-api-and-client-side-techniques.aspx – Sample code for uploading using binaryStringRequestBody

Get the digest value

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

$.ajax({     url: "_api/contextinfo",     type: "POST",     contentType: "application/x-www-url-encoded",     dataType: "json",     headers: {         "Accept": "application/json; odata=verbose",     },     success: function (data) {         if (data.d) {             var digest = data.d.GetContextWebInformation.FormDigestValue;         }     },     error: function (err) {         alert(JSON.stringify(err));     } });