Background Info

The reason I’m writing this article is because I recently designed an solution that utilises a RESTful API Service. This solution communicates with the RESTful service to GET and POST data using their given API. Here is a quick link to the wiki if you want to learn more about the REST architecture.

REST API

REST APIs are pretty cool when you think about it. There are already lots of sites out there provide an API for developers to communicate with to access their services. Below is a quick list to some well known sites and their API documentation:

• Twitter API
• LinkedIn API
• Google Maps API
• Flickr API
• Dropbox API

With APIs you can integrate any data service exposed over the web and have it seemlessly integrated into your very own application!

REST API – Requests

Accessing the API is pretty easy. Especially if you just want to grab data from the RESTful service. Communication with a REST API is done using basic HTTP GET, POST, PUT and DELETE requests. And for this reason, a REST API can be access via your web browser for all GET requests.

:: Here’s a Twitter example – http://search.twitter.com/search.json?q=clouds
Navigating to the above URL, you are basically executing a Twitter search on the word ‘clouds’. Twitter then provides you a response to your HTTP GET request in the form of JSON text. JSON and XML are two most common REST response types.

::Here’s a Google Maps example - http://maps.googleapis.com/maps/api/directions/xml?origin=sydney city&destination=melbourne&sensor=false
Navigating to the above URL, you are querying the Google Map directions API for directions to Melbourne from Sydney! Here the response is provided as XML

Steps to consume a REST service

Here is a the list of topics that will be covered, and by the end of this article, hopefully you have enough information to integrate this solution into your own applications!

1. Setting up the WCF REST Starter Kit for Visual Studio
2. Building your HttpClient
3. Initiate HTTP GET requests
4. Generate .Net classes from the REST Response (XML)
5. Transform the response into your .Net classes
6. Sample POST
7. Sample PUT
8. Sample DELETE

1. Setting up the WCF REST Starter Kit for Visual Studio

First up you will need some Microsoft libraries… Goto the Microsoft WCF REST website and download the latest WCF REST Starter Kit. Install the kit.  Once done, you will have 2 libraries that you will need to reference, these DLLs are located in the folder C:\Program Files\Microsoft WCF REST\WCF REST Starter Kit Preview 2\Assemblies\

These are:

• Microsoft.Http and
• Microsoft.Http.Extensions

Add these as library references to your project.

2. Building your HttpClient

Next up, lets create a static method to return a httpClient. You will be using the httpClient object to create and process all your API calls to the REST Service. Create a class called HttpClientManager and add the following ‘using’ statement for the two libraries you just referenced.

using Microsoft.Http;
using System.Xml.Serialization;

This class will have one static method that returns a HttpClient object all set up and ready to go. There are 3 main parts to this function, and depending on your environment and infrastructure you can choose to include or skip certain sections. The 3 sections are:

1. Setting the base url for the httpClient
2. Setting the Proxy configurations
3. Setting the Authentication headers for the REST service

1. Setting the base url for the httpClient

Each API library will have a standard base url for which all other API calls are ‘nested’ under this url. For example, our Google Map directions API has a base URL of:  http://maps.googleapis.com/maps/api/

2. Setting the Proxy configurations

If your corporate network contains a firewall/proxy. You may need to configure your httpClient to use the correct Proxy Credentials.  You can optionally skip this step if no proxy is required.

3. Setting the Authentication headers for the REST service

Most API services out there will require some sort of authentication. This will most likely be a token or some sort, which may or may not include MD5 hashes, other encryption, salts and/or other secure data transfers in order for the remote service to recognise who you really are. You can optionally skip this step if no authentication is required by the API service.

Below is the snippet that covers point one and two above.

public class HttpClientManager
{

    public static Microsoft.Http.HttpClient GetHttpClient()
    {
        string clientURL = "http://maps.googleapis.com/maps/api/";
        Microsoft.Http.HttpClient httpClient = new Microsoft.Http.HttpClient(clientURL);
        httpClient.DefaultHeaders.ContentType = "application/xml";

        // PROXY - If required
        System.Net.WebProxy p = new System.Net.WebProxy(proxy, int.Parse(proxyPort));
        p.Credentials = new System.Net.NetworkCredential(userName, password, domain);
        httpClient.TransportSettings.Proxy = p;

        return httpClient;
    }
}

If Authentication is required, you will need to determine what data to pass with the HTTP Request; and how. In the following code, we add the ‘authentication string’ as part of the HTTP Request Header called ‘Authorization’.

        // PROXY - If required
        // .....

        // AUTHENTICATION
        string authString = "xxxxxxxx";
        httpClient.DefaultHeaders["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(authString));

        return httpClient;

Keep in mind, the API service you use will outline what data needs to be passed; the format they expect, and any additional processing that is required from the authentication string.

That’s it.  Now you have a re-usable static method to call whenever you need a new HttpClient object.

Next up, you will be using the method above to grab a HttpClient object, and then writing a few additional lines of code to perform the various API calls for GET, POST, PUT and DELETE requests!

3. Initiate HTTP GET requests

Lets jump right into the GET request.  Your first API GET request will programmatically call Google Maps’ Directions API and perform a search. The structure of your method will be of the following:

• Grab a new HttpClient
• Determine the URL of the API call – and add this to the HttpClient’s Base URL address
• Determine any Query String Parameters that need to be passed through – add this to the API path
• Perform the API GET request
• Read the Response

Below is the simplest GET request you can have. It adds to the Base URL – the API location ‘directions/xml’. And then appends the search parameters as query strings.  The Base URL + the API location + the Parameter string makes up the full URL to Google’s Direction Search API.

To initiate the request, we invoke the Get() method on the httpClient object.
This returns a HttpResponseMessage object for us to interpret, and handle the response data as necessary. Here we are just storing the returned XML response into a local string variable.

    public static class GoogleDirectionsManager
    {

        public static void GetDirections(string origin, string destination)
        {
            HttpClient httpClient = HttpClientManager.GetHttpClient();

            // I'm sure you can think of a better way of managing these parameters!
            string url = "directions/xml";
            string param = "?origin=" + origin + "&destination=" + destination + "&sensor=false";

            HttpResponseMessage resp = httpClient.Get(url + param);
            string respString = resp.Content.ReadAsString();
        }
    }

To finish off our GetDirections() method, we are missing one elegant part… the Entity (or POCO) transformation.

The above method returns the response as XML. If your application wants to manipulate the XML directly, then thats all you require. However going one step further I will show how you can easily transfrom the XML into .Net Entity Classes. Not only does this provide extra readability, you can pass these entity objects to other layers of your application in a nice and clean manner.

3. Generate .Net classes from the REST Response (XML)

• First up: Run your application, and grab the XML response as a string. Copy this into Notepad.  And save the file. Lets call it “Direction.xml”.
• Open up Visual Studio’s command prompt (this is not the stand windows cmd promt). You will find the shortcut in your Start Menu -> Microsoft Visual Studio -> Visual Studio Tools -> Visual Studio Command Prompt menu
• Goto the directory that you saved the “Direction.xml” file into
• Run the following two commands in order, one after the other:
  • xsd Direction.xml
  • xsd Direction.xsd /c
• The first command creates a XSD file from the XML file
• The second command creates a CS class file from the XSD file

Once you have done that, you will find a “Directions.CS” file – Copy/Paste the contents of the CS file into your application.  That’s it!!  You have generated a ‘Direction’ C# class from your XML response!

5. Transform the response into your .Net classes

To do this, instead of reading the HttpResponseMessage as a string, we need to use the ReadAsXmlSerializable() method, and then define the type of object we want to transform the response text into. To achieve this, replace the above ReadAsString() method with a call to ReadAsXmlSerializable() as shown below.  Declare the Class type we generated in step.4 — and thats it.

The ReadAsXMLSerializable() method will do all the work, and at the end, you will have a fully functional Direction Object with all properties and nested classes in one!!

    public static class GoogleDirectionsManager
    {
        public static void GetDirections(string origin, string destination)
        {
            HttpClient httpClient = HttpClientManager.GetHttpClient();

            // I'm sure you can think of a better way of managing these parameters!
            string url = "directions/xml";
            string param = "?origin=" + origin + "&destination=" + destination + "&sensor=false";

            HttpResponseMessage resp = httpClient.Get(url + param);

            Entities.Direction d = resp.Content.ReadAsXmlSerializable<Entities.Direction>();
        }
    }

6. Sample POST

That’s all the hard part done!  The other Http Request methods are pretty straight foward, all you need to do is invoke the corresponding httpClient method of either Post(), Put() or Delete().

A Http POST request is all about ‘creating’ new data via the REST Service. Where a Http PUT request is all about ‘updating’ existing data. And of course Http DELETE removes existing data.

Http POST  and PUT will require ONE additional step. And that is to include the data you want to create or update with the API call. In the examples below, I will refer to a generic Ticketing system instead of Google Map.  POST will allow our system to POST a Ticket via XML to the API – in order to create a new ticket in their system.  Likewise a PUT will allow our system to UPDATE a Ticket (also via XML) to the API – in order to update an existing record.

Sending data via a POST/PUT command requires the data to be included in the Http Request as part of its Http Content. Also known as the Http Body.  Just like sending an email, there is a subject section and a body section!

Below is an example of how you will POST a new Ticket with 2 simple properties (Email and Comment) to the API located in the sub-path (tickets.xml). :

note: The structure of the content is purely dependant on the REST Service. You will need to consult the API to know what structure and data to pass through.

        public static void CreateTicket(Entities.Ticket ticket)
        {
            HttpClient httpClient = HttpClientManager.GetHttpClient();

            // I'm sure you can think of a better way of generating the XML! 
            HttpContent content = Microsoft.Http.HttpContent.Create("<ticket><email>test@test.com</email><comment>my comments are here!</comment></ticket>"
                                                            , "application/xml");

            string url = "tickets.xml";
            HttpResponseMessage resp = httpClient.Post(url, content);

            string respString = resp.Content.ReadAsString();
        }

7. Sample PUT

Http PUT is similar to a Http POST.  The differences would be the API path to navigate to, and the XML data that you send across the wire. Below we update a Ticket, sending over the email and comment we wish to update the record with.  Everything else… remains the same as above!

        public static void UpdateTicket(Entities.Ticket ticket)
        {
            HttpClient httpClient = HttpClientManager.GetHttpClient();

            // I'm sure you can think of a better way of generating the XML! 
            HttpContent content = Microsoft.Http.HttpContent.Create("<ticket><email>test@test.com</email><comment>UPDATED comments!</comment></ticket>"
                                                            , "application/xml");

            string url = "tickets/" + ticket.ID + ".xml";
            HttpResponseMessage resp = httpClient.Put(url, content);

            string respString = resp.Content.ReadAsString();
        }

8. Sample DELETE
A Http DELETE is alot simpler than a Http POST and PUT.  All we need to do is grab the API path to navigate to (which coincidentally will most likely contain the record ID) and execute the httpClient DELETE method.

Below is a snippet that shows you how this is done.

        public static void DeleteTicket(Entities.Ticket ticket)
        {
            HttpClient httpClient = HttpClientManager.GetHttpClient();

            string url = "tickets/" + ticket.ID + ".xml";
            HttpResponseMessage resp = httpClient.Delete(url);

            string respString = resp.Content.ReadAsString();
        }

Conclusion

Well there you have it!!  You now have all the knowledge you need to create you own apps and integrate them with 3rd party APIs out there. It may be alot of information to consume, but once you complete the GET request, you will see how everything clicks in!  Then its smooth sailing from there onwards

Hope this article helped! Have fun!

Prerequisites

• How to enable SQL FileStream Types
• Using SQL FileStream Types in your Databases
• MaxRequestLength – Packet sizes: Size isn’t everything
• RIA Services: File uploading
  

Here I will go into detail on how to code your domain service class to receive data packets sent across the wire from the File Stream Uploader generic handler class.

Attachment Service

Ok first up – The Domain Service.  I called it ‘Attachment Service’.  This example will include two basic functions: Reading streamed data from a SQL FileStream type, and Uploading streamed data to a SQL FileStream type.

The service is pretty basic.. just pipe your data packet parameters from the client, into the Manager class. Here’s the code:

   [EnableClientAccess()]
    public class AttachmentService : LinqToEntitiesDomainService<MyEntities>
    {
        private Manager.IAttachmentManager attachmentManager;

        public AttachmentService()
        {
            this.attachmentManager = new Manager.AttachmentManager();
        }

        [Invoke(HasSideEffects = false)]
        public byte[] ReadStreamed(Guid fileID, int offset, int lengthToRead)
        {
            return attachmentManager.ReadStreamed(fileID, offset, lengthToRead);
        }

        [Invoke(HasSideEffects = false)]
        public Guid UploadStreamed(Guid attachmentID, string newFileName, string originalFileName, byte[] fileData, int position)
        {
            return attachmentManager.UploadStreamed(attachmentID, newFileName, originalFileName, fileData, position);
        }

    }

Manager Class:

Our manager class is also pretty straight forward. There is nothing tricky that is needed here, so we again simply pipe our input parameters into the Repository class.

    public class AttachmentManager : IAttachmentManager
    {
        private Repository.IAttachmentRepository attachmentRep { get; set; }

        public AttachmentManager()
        {
            this.attachmentRep = new Repository.AttachmentRepository();
        }

        public byte[] ReadStreamed(Guid fileID, int offset, int lengthToRead)
        {
            AttachmentRepository rep = new AttachmentRepository();
            return rep.ReadStreamed(fileID, offset, lengthToRead);
        }

        public Guid UploadStreamed(Guid attachmentID, string newFileName, string originalFileName, byte[] fileData, int position)
        {
            AttachmentRepository rep = new AttachmentRepository();
            return rep.UploadStreamed(attachmentID, newFileName, originalFileName, fileData, position);
        }

    }

The Repository:

This is where the fun starts!  Our repository will contain the bulk of the code; and the majority of it is the actual communication with the Data tier.  Our class will contain 3 methods instead of just 2.  The extra method is called ‘InsertAttachmentStub’. This method will be used to create the initial attachment ‘stub’ in our database table.  As you will see below, the stored procedure for this insertion will simply create the record with the AttachmentID, and set the necessary file names and other data fields.

Note: Our AttachmentID being a GUID was generated in the code beforehand, and passed into the Generic Handler.

Note: This stub method is only needed for the very first data packet. Subsequent data packets will re-use the same Attachment record, and update the data component of the FileStream type accordingly.

Without further or do, here’s the signature for our 3 functions:

    public class AttachmentRepository
    {

        public Entities.AttachmentPM InsertAttachmentStub(System.Guid attachmentID, string newFileName, string originalFileName)
        { ... }

        public byte[] ReadStreamed(Guid fileID, int offset, int lengthToRead)
        { ... }

        public Guid UploadStreamed(Guid attachmentID, string newFileName, string originalFileName, byte[] fileData, int position)
        { ... }
    }

Inserting the stub:

As mentioned above, pretty simple Repository function here.  Just pipe all the basic details for your Attachment record into the InsertAttachmentStub Stored Proc:

        public Entities.AttachmentPM InsertAttachmentStub(System.Guid attachmentID, string newFileName, string originalFileName)
        {
            var list = DataAccessUtility.ExecuteStoredProcedure("uspAttachment_InsertAttachmentStub"
                                        , new SqlParameter("ID", attachmentID)
                                        , new SqlParameter("NewFileName", newFileName)
                                        , new SqlParameter("OriginalFileName", originalFileName)
                                        , new SqlParameter("UpdatedDateTime", System.DateTime.Now)
                                        , new SqlParameter("UpdatedBy", System.Data.SqlDbType.Int) { Value = 0 }
                                     );
            return list.FirstOrDefault();
        }

Here’s the SQL proc for InsertAttachmentStub. Once we complete this insert, we will have a database record to store to re-use when uploading the next packets of data!

CREATE PROCEDURE [dbo].[uspAttachment_InsertAttachmentStub]
	@ID						AS UNIQUEIDENTIFIER
	, @NewFileName			AS NVARCHAR(255)
	, @OriginalFileName		AS NVARCHAR(255)
	, @UpdatedDateTime		DATETIME2(7)
	, @UpdatedBy			AS INT
AS
BEGIN

	INSERT INTO Attachment (ID, NewFileName, OriginalFileName, FileData, UpdatedDateTime, UpdatedBy)
	OUTPUT inserted.FileData.PathName()		AS LogicalFilePath
			, inserted.ID					AS ID
	VALUES (@ID, @NewFileName, @OriginalFileName, 0x, @UpdatedDateTime, @UpdatedBy)
END

Reading SQL Filestream data:

Lets begin with the easier of the two methods.. the reading of data.

Our input parameters will identify which AttachmentID we are interested in. It also tells us which byte offset to start our reading from; and how many bytes we want to read and return back to the calling method.

So here’s the reading method, and it performs the following steps:

• Create a Transaction Context so that SQL knows who you are when reading the FileStream data
• Make an additional call to grab the File attachment record.  Include in this select the FilePath that we will use later on.
• Using the Transaction Context AND the FilePath of the file we are interested in reading, create and open your connection to the SQLFileStream.
• Once the SQLFileStream is opened, we can now read our required data packet.
• Read the data from the SQLFileStream object into a MemoryStream object
• Return the data back to the caller

        public byte[] ReadStreamed(Guid fileID, int offset, int lengthToRead)
        {
            System.IO.MemoryStream ms = new System.IO.MemoryStream();

            using (var transScope = new System.Transactions.TransactionScope())
            {

                    byte[] txContext = (byte[])DataAccessUtility.ExecuteScalar(System.Data.CommandType.Text, "SELECT GET_FILESTREAM_TRANSACTION_CONTEXT()");

                    var list = DataAccessUtility.ExecuteStoredProcedure("uspAttachment_GetAttachmentByID"
                                         , new SqlParameter("ID", fileID)
                                         );
                    string filePath = list.FirstOrDefault().LogicalFilePath;

                    System.Data.SqlTypes.SqlFileStream sqlFS = new System.Data.SqlTypes.SqlFileStream(filePath, txContext, System.IO.FileAccess.Read);

                    byte[] buffer = new byte[lengthToRead];

                    sqlFS.Position = offset;
                    sqlFS.Read(buffer, 0, lengthToRead);
                    ms.Write(buffer, 0, lengthToRead);
                    ms.Flush();

                    ms.Close();

                    sqlFS.Close();

                transScope.Complete();
            }
            return ms.ToArray();

        }

Uploading to a SQL FileStream:

Uploading is simply the opposite of what you have accomplished above.

To upload a data packet we perform the following steps:

• Check if this is the first packet uploaded
• If yes – call the InsertAttachmentStub method to generate an AttachmentID
• If not – grab the AttachmentID (and its corresponding FilePath data) from the database
• Create a Transaction Context so that SQL knows who you are.
• Using the Transaction Context AND the FilePath of the file we are interested in writing to, create and open your connection to the SQLFileStream.
• Once the SQLFileStream is opened, we can now write our required data packet.
• Grab the data packet from our input parameters, and write into the SQLFileStream object
• Once written, we can close the SQLFileStream object safely and return from this function.

        public Guid UploadStreamed(Guid attachmentID, string newFileName, string originalFileName, byte[] fileData, int position)
        {
            Entities.AttachmentPM attachment;

            using (var transScope = new System.Transactions.TransactionScope())
            {
                    // If Position = 0, then we are inserting the first data packet.
                    if (position == 0)
                    {
                        attachment = this.InsertAttachmentStub(attachmentID, newFileName, originalFileName);
                    }
                    else    // Else Attachment record already exists. Just retrieve the details
                    {
                        var list = DataAccessUtility.ExecuteStoredProcedure("uspAttachment_GetAttachmentByID"
                                        , new SqlParameter("ID", attachmentID)
                                        );
                        attachment = list.FirstOrDefault();
                    }
                    string filePath = attachment.LogicalFilePath;

                    byte[] txContext = (byte[])DataAccessUtility.ExecuteScalar(System.Data.CommandType.Text, "SELECT GET_FILESTREAM_TRANSACTION_CONTEXT()");

                    using (System.Data.SqlTypes.SqlFileStream sqlFS = new System.Data.SqlTypes.SqlFileStream(filePath, txContext, System.IO.FileAccess.ReadWrite))
                    {
                        byte[] buffer = new byte[fileData.Length];

                        sqlFS.Position = sqlFS.Length;
                        sqlFS.Write(fileData, 0, fileData.Length);
                    }

                transScope.Complete();
            }
            return attachment.ID;
        }

Conclusion:

Well that’s pretty much all there is to it! 

If you have any question, please let me know!  Hope this article helped!

Cheers.

These entities can then be grouped together to form organisational units. Organisational Units are created to tie common entities together; whether it be a geographical grouping or a structure grouping. Each entity is comprised of many attributes. These attributes provide additional information about the entity. For example a Person entity will have their Login, Firstname, Surname and Department etc as attributes.

Developing a software solution for a corporate environment will most likely entail Active Directory access for various reason. In this article I will explain how you can create your own Active Directory querying utility to suit your requirements…

System.DirectoryServices Namespace

The first thing you’d want to do is include a reference to the System.DirectoryServices namespace. This is the system library that contains all the objects needed to query your Active Directory service.

Making the connection

Once that is done; its time to connect to your Directory Service using LDAP.
LDAP stands for Lightweight Directory Access Protocol. It is one of many access protocols available to communicate with Acitve Directory.

Knowledge of the following .Net classes will be helpful in understanding how communication between your .Net application and the Active Directory service is achieved:

• DirectoryEntry – The main connection to your Active Directory server
• DirectorySearcher – The object passed into our LDAP query to filter for the required entity objects
• SearchResultCollection – A collection that stores multiple AD entities if more than one is found
• SearchResult – A single AD entity with its corresponding attributes

First up, create a connection to your Active Directory service. The ADServer should be something similar to “LDAP://my-server.com.au/dc=com,dc=au“.

            DirectoryEntry directoryEntry = new DirectoryEntry(this.ADServer.Text
                                                            , this.ADUsername.Text
                                                            , this.ADPassword.Text);

Create a new DirectorySearcher object to query the AD server with. For now, just a new object is sufficient. Later on in this article I will explain how to select which attributes to obtain from the AD server. Without explicitly listing your attributes the DirectorySearcher will simply return all attributes for all entities matching your filter criteria.

DirectorySearcher searcher = new DirectorySearcher(directoryEntry);

Filtering the data

The next step is to write our filter criteria for our query. This is simply the DirectorySearcher.Filter property of your DirectorySearcher object. The filter you will use is going to tell the AD Server to only query the “user” entities, that lie within the “person” category, where the sAMAccountName attribute equals to our input value. We also specify AD to search all sub-trees of the hierarchical structure.

            searcher.Filter = "(&(objectClass=user)(objectCategory=person)(samaccountname=" + this.UsernameTextBox.Text + "))";
            searcher.SearchScope = System.DirectoryServices.SearchScope.Subtree;

That’s pretty much it!! The final is step is to execute the search..

Searching

Below is a snippet of the code that traverses all the entities in the ResultCollection. Spitting out all the attributes in the Entity object to a local String variable. Then displaying the output as textbox text.

            SearchResultCollection resultCollection = searcher.FindAll();

            foreach (SearchResult result in resultCollection)
            {
                string str = "";
                foreach (string propertyName in result.Properties.PropertyNames)
                {
                    try
                    {
                        str += propertyName + ":" + (string)(result.Properties[propertyName][0]) + System.Environment.NewLine;
                    }
                    catch
                    {
                    }

                }

                this.OutputTextBox.Text += str;
            }

To access a specific attribute from an entity; refer to the ‘Properties‘ property object, which is an array of values indexed by the attribute name. The value of the attribute is the first item in the array returned.

I have listed some examples below:

result.Properties["GivenName"][0]
result.Properties["Sn"][0]
result.Properties["Mail"][0]

Try out the code above. You may notice that it takes a few seconds to return just one entity. That is because in our DirectorySearcher object we didn’t limit the data we really wanted. So our query returned all AD attributes for us. You will most certainly only want to grab the data that is necessary for your situation. This will dramatically increase its performance.

The above code is useful if you want to check out all the available attributes within Active Directory.

Filtering the attributes

Let’s say we are only interested in the two attributes: sAMAccountName and Mail
Before we execute our query with the AD Server, we can optimise the search to only return these two attributes with the following lines of code:

            DirectorySearcher searcher = new DirectorySearcher(directoryEntry);
            searcher.PropertiesToLoad.Add("sAMAccountname");
            searcher.PropertiesToLoad.Add("Mail);

            searcher.Filter = "(&(objectClass=user)(objectCategory=person)(samaccountname=" + this.UsernameTextBox.Text + "))";
            searcher.SearchScope = System.DirectoryServices.SearchScope.Subtree;

With this new code, our Output string will only contain the two attributes we requested for.
Also take note of the time the query takes – it should be alot quicker than before!

So along comes WCF RIA Services and its validation framework. In this article I’ll provide details on how WCF RIA Services approachs validation; and how it has been designed to resolve issues of its predecessors..

There are two main categories of Validation within RIA Services:

1. Client side validation
2. Server side validation

Client side Validation

So we all know the benefits of Client-side validation.. Instant data validation without the need for any service or database calls. Because it’s extremely call-efficient it provides great performance gains in the system. But with one limitation – the client side must have all the necessary data for it to complete its validation logic. Which kind of makes sense when you think about it.. without having all its parameters to validate with it won’t be able to complete successfully.

Server side Validation

This is where it gets interesting. If your validation requires data not available on the Client, then server side validation is the definitely way to go. You can achieve server side validation by exposing Invoke methods in your Domain Service. The Client can explicitly make the domain service call on-demand. This fits into the traditional way of validation; and in many cases works absolutely fine. Of course, with RIA services you can go one step further.. using Custom Validation.

There are two styles of Validation available:

1. Declarative validation
2. Custom validation

Declarative Validation

Declarative Validation is the simplest of the two forms of Validation. It provides simple and easy to maintain validation logic triggered on the Client. This is achieved by using Metadata Annotations with your Entity object Properties that require the validation. These annotations are part of the System.ComponentModel.DataAnnotations namespace and are derivative classes of ValidationAttribute. There’s only a small list of these but they should suffice for simple validation logic:

• StringLength
• Required
• Range
• RegularExpression

Below is an example of how to use declarative validation in an Entity class called Person.

using System.ComponentModel.DataAnnotations;

    public class Person
    {
        [Key]
        public int ID { get; set; }

        [Required]
        [StringLength(50, ErrorMessage="Forename cannot exceed 50 characters.")]
        public string Forename { get; set; }

        [Required]
        [StringLength(50, ErrorMessage="Surname cannot exceed 50 characters.")]
        public string Surname { get; set; }

        [Range(1, 150, ErrorMessage="Age must be between 1 and 150.")]
        public int Age { get; set; }

        public int StreetNumber { get; set; }
        public string StreetName { get; set; }

        public int PostCode { get; set; }
    }

Custom Validation

Custom Validation can be used for both Client and Server side validation. Depending on your logic you will need to split your Client and Server side validation code into different classes. The basic principle should be:

• If the logic is simple enough to be covered by the Declarative Validation method – then use the available Declarative Validation attributes.
• However, if the logic is slightly more complicated + all validation parameters are available on the Client – then follow the Client Side Custom Validation method
• Lastly, if the logic is slightly more complicated + not all validation parameters are available on the Client – then follow the Server Side Custom Validation method.

Custom Validation – Client side

With Client side Custom Validation you will need to be defined in a *.shared file to ensure the code is generated on the Client. Along with this, your class needs to be declared as public static partial, and your validation methods declared as public static returning a ValidationResult object.

As an example, say you wanted to ensure that the StreetNumber property has a positive non-zero value. You can probably use the RANGE attribute to ensure it has a minimum value of 1.  But there really isn’t a maximum figure that you can put in. So by using declaring Custom Validation for the StreetNumber, you can simply check that it is not less than zero.

Below is the validation class that will be generated for the Client:

• File must be in the format of a shared file, eg: PersonClientValidation.shared.cs
• Class declared as public static partial class …
• Validation method declared as public static ValidationResult …
• Return ValidationResult.Success if validation passes
• Return a new ValidationResult object if validation fails

using System.ComponentModel.DataAnnotations;

    public static partial class PersonClientValidation
    {
        public static ValidationResult StreetNumberPositive(int streetNumber, ValidationContext validationContext)
        {
            if (streetNumber< 0)
            {
                return new ValidationResult("Street Number must be positive and non-zero.");
            }
            return ValidationResult.Success;
        }
    }

Once that is done, we’re ready to add the CustomValidation attribute to the StreetNumber property, and point it to the StreetNumberPositive(…) method. Just put this attribute above the Property, and that’s it! When the user enters a StreetNumber < 0 the application will automatically throw the validation error and display the message to the user.

using System.ComponentModel.DataAnnotations;

    public class Person
    {
        [Key]
        public int ID { get; set; }

        //... other properties hidden ... //

        [CustomValidation(typeof(PersonClientValidation), "StreetNumberPositive")]
        public int StreetNumber { get; set; }

        public string StreetName { get; set; }
    }

Custom Validation – Server side

With Server side Custom Validation the opportunities are limitless. Because the validation is processed on the Server you can utilise all your mid-tier code for data retrieval or additional processing while performing the required validation logic. One main reason we need to split the Server side Custom Validation from the Client side Custom Validation code is because the Server side logic may contain references to mid-tier code that is inaccessible by the Client. If we placed it in the same *.shared.cs class there will be a compilation error when creating mid-tier objects and service classes.

So create another class that we will use for the Server side validation of the Person class:

• This is just a mid-tier class, so the naming format is just a basic class file. ie don’t use *.shared.cs
• Class declared as public static class…
• Validation method declared as public static ValidationResult (obj propertyValue, ValidationContext ctx)
• Return ValidationResult.Success if validation passes
• Return a new ValidationResult object if validation fails

using System.ComponentModel.DataAnnotations;

    public static class PersonServerValidation
    {
        public static ValidationResult PostCodeExists(int postCode, ValidationContext validationContext)
        {
            // Here we call a Manager class that will return our boolean value
            // Any additional calls to class methods, or other services can be done here.

            AddressManager mgr = new AddressManager();
            bool exists = mgr.DoesPostCodeExist(postCode);

            if (!exists)
            {
                return new ValidationResult("The postcode entered is invalid.
                                            , new[] { validationContext.MemberName });
            }

            return ValidationResult.Success;
        }
    }

Finally, we can now update our PostCode property with the CustomValidation Attrubute. When RIA Services determine that validation of this property is needed, it will execute the Validation Method we declared above.

using System.ComponentModel.DataAnnotations;

    public class Person
    {
        //... other properties hidden ... //

        [CustomValidation(typeof(PersonServerValidation), "PostCodeExists")]
        public int PostCode { get; set; }
    }

So far we have been dealing with individual Entity Properties and its underlying validation processes. Alot of the times Property validation may be dependant on other Properties. For example, say that you wanted to validate the StreetName is a valid street in the Postcode specified; and that the street number is within the number of houses on that street. This is where we go one step beyond Property Validation to utilising Entity Validation.

Entity Validation

Entity Validation is pretty straight forward. It simply refers to the validation that takes place on an Entity Object as a whole, as opposed to validating individual Properties of that object. Entity Validation works in conjunction with Property Validation. Meaning that the Property Validations that you have defined in your class is still true and valid, and will automatically validate when required. Once all Property validation passes, the framework will then perform Entity Validation if one or more exist.

The process of defining a Validation Method for use with Entity Validation is similar to that described earlier. The only difference is that rather than having a parameter with the same type as the Property itself, we require a parameter of the Entity object instead. A sample Entity level Validation method is provided below:

using System.ComponentModel.DataAnnotations;

    public static class PersonServerValidation
    {
        public static ValidationResult StreetNameValid(Person person, ValidationContext validationContext)
        {
            AddressManager mgr = new AddressManager();
            bool validStreet = mgr.IsStreetValid(person.StreetName, person.PostCode);

            if (!validStreet)
            {
                return new ValidationResult("The street entered is not valid for with that postcode."
                                            , new[] { validationContext.MemberName });
            }

            return ValidationResult.Success;
        }

        public static ValidationResult StreetNumberValid(Person person, ValidationContext validationContext)
        {
            AddressManager mgr = new AddressManager();
            bool validStreetnumber = mgr.IsStreetNumberValid(person.StreetNumber
                                                            , person.StreetName
                                                            , person.PostCode);
            if (!validStreetnumber )
            {
                return new ValidationResult("The street number entered is not valid for that street
                                            , new[] { validationContext.MemberName });
            }

            return ValidationResult.Success;
        }
    }

The final step is to declare the Entity Validation for the Person class and link it to the corresponding Validation methods above:

using System.ComponentModel.DataAnnotations;

    [CustomValidation(typeof(PersonServerValidation), "StreetNameValid")]
    [CustomValidation(typeof(PersonServerValidation), "StreetNumberValid")]
    public class Person
    {
        //... other properties hidden ... //
    }

Validation via Invoke Methods

As briefly mentioned previously in this article, you can as a last resort create an Invoke method that will contain the logic you require for your validation. Using invoke methods for validation is often useful when you need to validate across entities, or if your design calls for out-of-the-ordinary situations that don’t tie into the scenarios listed above.

Conclusion

Well there you have it – a explanation of the validation framework available for you to use with the WCF RIA Services framework. As you can see, this validation framework works very well if you want to declare and keep all your validation logic within the mid-tier code that has been developed by your mid-tier developers. This removes the need for your UI developers to concern themselves with any of the business rules and validation logic that comes with it.

Traditional ASP.Net would have seen the duplication of your validation logic to the UI-tier making code maintenance extremely troublesome.

I hope this article has given you enough information to implement your own validation logic moving forward!

Introduction

All your databases have a SYS schema; and within this schema is a view called OBJECTS.

It is in this OBJECTS view where all your database elements are stored and referenced; including Table names, Primary/Foreign Key constraints, Procedures, Functions, View.. everything!

Basic Query

So here’s the basic query…

This will list every element associated to the Database you run the query under:

SELECT *
FROM SYS.OBJECTS O

Filtering by Element Type

To narrow down your search, you can filter the above query using the [type] column that is returned from the OBJECTS view. Below is a short list of the different type codes available and what they refer to:

• P = Stored Procedure
• U = User Table
• V = View
• TT = Type Table
• FN = Scalar Function
• IF = Inline Table Function
• PK = Primary Key Constraint
• F = Foreign Key Constraint

SELECT *
FROM SYS.OBJECTS O
WHERE [type] = 'P'

Searching the Element Text

The final step is to locate the data content of your database elements to complete your search. Here we need the help of a second View called SYSCOMMENTS.  This view returns a ‘text’ component to your database elements. Specifically for Stored Procedures, Function and View this component holds the code counterpart for those elements.

To proceed, join your OBJECTS view with the SYSCOMMENTS view on the object_id and id columns respectively as shown below:

SELECT *
FROM SYS.OBJECTS O
INNER JOIN SYS.syscomments C on C.id = O.object_id
WHERE O.[type] = 'P'
	AND C.text LIKE '%xyz%'

Conclusion

There you go. So whenever you need to search your database for strings, you know how it’s done!!

Hope this was helpful, make use of it well!!

The simplest representation of hierarchies is in the form of an upside down ‘Tree‘.  The ‘root‘ of the tree is where it all begins, the ‘root’ expands into multiple ‘tree nodes‘, these ‘tree nodes’ may contain other tree nodes; and at the end of each ‘tree node’ is a ‘leaf node‘..  for more information on data tree structures please refer to this article: Tree (data structure)

In software development, no doubt you would have come across tree structures, and specifically hierarchical data structures, in your career. One pitfall that you would have experienced is that computing the entire tree structure on the fly for any system is a costly process. As the saying goes ‘..the apple never falls far from the tree’, and thus even computing the structure of a single tree node can be just as costly.

This is where a special data type in SQL comes to play: the hierarchyid type!  In this article I will explain what the herierachyid type is, and provide examples on how you can use this in your database design for a clean, succinct data structure, and how you will benefit from the massive performance gains by doing so..

The Basics

Tree Structures

A well defined hierarchical structure can be achieved in many different ways within SQL server; ranging from simple self-referencing tables, to more complex one-to-one, or one-to-many relation tables. No matter what your design is you cannot move away from having an EntityID column and a ParentEntityID column (or a similar field referencing its parent entity).  These two fields are the fundamental properties that define your hierarchical structure… think of them as your link between one ‘tree branch’ to the next ‘tree branch’. You can traverse up the tree and ultimately reach the ‘root’ entity (remember the tree is upside down!). Or you can traverse down the tree and ultimately reach the ‘leaf nodes’.

Time consuming computations

Computing the structure on-the-fly is pretty demanding, even if its processing within SQL Server. The idea is that if you have a random tree node, and wish to retrieve all descendants (ie every other tree node and leaf node below the current selected node) then the system needs to traverse quite possibly a large number of nodes.. processing each node recursively as it goes, until it ultimately reaches nodes without any child nodes (aka the leaf node).

In practice, if you consider an organisation structure with Managers (represented by tree nodes) and Employees (represented by leaf nodes).. then this is analogous to retrieving all Employees under a certain Manager..  say the CIO and all employees reporting underneath him/her.

The same logic applies if you wish to go up the tree to find all the ancestors of a given leaf node. With the organisation example above, this is like grabbing all Upper Managers for an Employee.

Whatever your requirement is, there is bound to be a need for tree traversal. Often is the case that if you have a poorly designed database structure then your system will be plagued by poor performance.

The Alternative Solution

One way of achieving great performance in such a scenario is by the use of SQL hierarchyid types. hierarchyid types help alleviate the computation stress on your server by pre-computing the relationship amongst records at the time of insertion.

What is stored in the hierarchyid column is the binary representation of the full entity path from its root entity. This is near-identical to the Unix File system and how they represent file paths:

• ‘/’ – denotes the root entity path
• ‘/Files’ – denotes the path to the Files folder
• ‘/Files/test.log’ – denotes the path to a file called test.log inside the Files folder.. and so on

Instead of using alphanumeric values for pathing, SQL Server hierarchyids are represented by numeric values:

• ‘/’ – denotes the root entity path
• ‘/1/’ – denotes the first child added to the structure
• ‘/2/’ – denotes the second child added to the structure
• ‘/2/1/’ – denotes the first sub child added underneath the second child of the root entity

SQL Server automatically calculates the logical path of the entity being inserted. It does this by inspecting the hierarchyid of the Parent entity for which the new entity is to be attached to.

Hierarchyid Methods

SQL Server has built in functions that help you navigate and retrieve data tied to hierarchyid fields.

• GetAncestor (int n) – Gets the n’th ancestor of the entity
• IsDescendantOf (hierarhcyid h) – Returns true if the current entity is a descendant of ‘h’
• ToString () – Returns the logical path of the entity in string format
• GetDescendant (child1, child2) – Returns a child hierarchyid of the current entity. The output varies based on the value of the two input parameters.

These are some of the more common methods that you will be using. The usage syntax is pretty straight forward. All you need to do is simply call the method on a hierarchyid variable or field.

For example:

DECLARE @Manager hierarchyid
SELECT @Manager = CAST('/3/1' AS hierarchyid)

-- Returns all Employees that is a descendant of @Manager
SELECT *
FROM Employee E
WHERE E.Hierarchy.IsDescendantOf(@Manager) = 1

Practical Example

Consider the following database schema:

Before we proceed to insert sample data, lets visit the INSERT logic for this table.

Insert Logic

The general idea is as follows:

1. Check if this is the first ‘root’ entity to be inserted.
2. If true: the hierarchyid = to the root path of ‘/’.
3. If false: we need to grab the Manager record and use its hierarchyid to determine the next hierarchyid for its next child.
4. Once we determine the new child hierarchyid, we can proceed with the insert.

The complex part is determining the next hierarchyid to insert for this new child record.

• GetAncestor(1) = @ManagerHierarchyID: This formula simply filters the records and returns all the children of the record identified with hierarchyid = @ManagerHierarchyID
• NewHierarchyID = MAX(Hierarchy): Returns the last child record in the list
• GetDescendant(@NewHierarchyID, NULL): Generates the next hierarchyid AFTER that of @NewHierarchyID. In other words, whatever the path is defined as the last child record, generate the subsequent hierarchyid. So if @NewHierarchyID = ‘/1/2/’, this method call will return ’1/3/’

Insert

CREATE PROC [dbo].[uspEmployee_Insert]
	@EmployeeID		VARCHAR(5)
	, @Firstname	VARCHAR(50)
	, @Surname		VARCHAR(50)
	, @PositionTitle	VARCHAR(50)
	, @ManagerID		VARCHAR(5)
AS
BEGIN

	DECLARE @ManagerHierarchyID hierarchyid
	DECLARE @NewHierarchyID hierarchyid

	IF @ManagerID IS NULL
	BEGIN
		SET @NewHierarchyID = hierarchyid::GetRoot()
	END
	ELSE
	BEGIN

		-- [1] Identify the Manager Record
		-- [2] Find the LAST child attached to the Manager Record (we need this when calling the GetDescendant method)
		-- [3] Generate the next hierarchyid that is AFTER the LAST child retrieved above
		-- [4] Insert the new Employee with the newly generated hierarchyid

		-- [1]
		SELECT @ManagerHierarchyID = E.Hierarchy
		FROM Employee E
		WHERE E.EmployeeID = @ManagerID

		-- [2]
		SELECT @NewHierarchyID = MAX(Hierarchy)
		FROM Employee E
		WHERE E.Hierarchy.GetAncestor(1) = @ManagerHierarchyID

		-- [3]
		SELECT @NewHierarchyID = @ManagerHierarchyID.GetDescendant(@NewHierarchyID, NULL)
	END

	-- [4]
	INSERT INTO Employee
	SELECT @EmployeeID
			, @Firstname
			, @Surname
			, @PositionTitle
			, @NewHierarchyID

END

With this completed, we can now create a hierarchical structure for our table!

Data Insert

The follow SQL will generate a small organisation structure as depicted below:

EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0001',	@Firstname = N'David',		@Surname = N'Smith',		@PositionTitle = N'CEO',			@ManagerID = NULL
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0002',	@Firstname = N'Peter',		@Surname = N'Levy',			@PositionTitle = N'COO',			@ManagerID = 'E0001'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0003',	@Firstname = N'Trevor',		@Surname = N'Bailey',		@PositionTitle = N'GTS Manager',	@ManagerID = 'E0001'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0004',	@Firstname = N'Kurt',		@Surname = N'Douglas',		@PositionTitle = N'Sales Director',	@ManagerID = 'E0001'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0005',	@Firstname = N'Ken',		@Surname = N'Kumar',		@PositionTitle = N'CIO',			@ManagerID = 'E0002'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0006',	@Firstname = N'Samantha',	@Surname = N'Dwellings',	@PositionTitle = N'HR General Manager',	@ManagerID = 'E0002'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0007',	@Firstname = N'Verity',		@Surname = N'McDonald',		@PositionTitle = N'IT Manager',		@ManagerID = 'E0005'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0008',	@Firstname = N'Douglas',	@Surname = N'Morgan',		@PositionTitle = N'IT Developer',	@ManagerID = 'E0007'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0009',	@Firstname = N'Paul',		@Surname = N'Lynch',		@PositionTitle = N'BI Developer',	@ManagerID = 'E0007'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0010',	@Firstname = N'Jenny',		@Surname = N'Davison',		@PositionTitle = N'DB Developer',	@ManagerID = 'E0007'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0011',	@Firstname = N'Monica',		@Surname = N'Adams',		@PositionTitle = N'HR Manager',		@ManagerID = 'E0006'
EXEC [dbo].[uspEmployee_Insert] @EmployeeID = N'E0012',	@Firstname = N'Bree',		@Surname = N'Lucas',		@PositionTitle = N'HR Advisor',		@ManagerID = 'E0011'

You should now have the following records in your Employee table:

Retrieval Logic

This is where the fun starts… In the examples below, I will use Employee E0006 (HR General Manager) as a point of reference.

All Child records

Firstly, to retrieve all the Children for an Employee we use the logic: 

• item.GetAncestor(1) = @SourceHierarchyID

The GetAncestor() method returns all records that is an ancestor (ie parent) of the current item. The parameter defines how far up the hierarchy you wish to go. So the above logic returns records ‘one level’ above the current item which is our queried E0006 Employee.

	DECLARE @SourceHierarchyID AS hierarchyid

	SELECT @SourceHierarchyID = E.Hierarchy
	FROM Employee E
	WHERE E.EmployeeID = 'E0006'

	SELECT 'E0006'				AS EmployeeID
			, E.EmployeeID		AS ChildEmployeeID
			, E.Firstname		AS ChildFirstname
			, E.Surname			AS ChildSurname
			, E.PositionTitle	AS ChildPositionTitle
			, E.Hierarchy		AS ChildHierarchy
			, E.Hierarchy.GetLevel()	AS Level
	FROM Employee E
	WHERE E.Hierarchy.GetAncestor(1) = @SourceHierarchyID
	ORDER BY E.Hierarchy.GetLevel() ASC

Wrapping this into a completed function:

CREATE FUNCTION [dbo].[udfEmployee_GetAllChildrenByEmployeeID]
(
	@SourceEmployeeID AS VARCHAR(5)
)
RETURNS @tbl TABLE (EmployeeID				VARCHAR(5)
					, ChildEmployeeID		VARCHAR(5)
					, ChildFirstname		VARCHAR(50)
					, ChildSurname			VARCHAR(50)
					, ChildPositionTitle	VARCHAR(50)
					, ChildHierarchy		hierarchyid
					, ChildLevel			INT
					)

AS
BEGIN

	DECLARE @SourceHierarchyID AS hierarchyid

	SELECT @SourceHierarchyID = E.Hierarchy
	FROM Employee E
	WHERE E.EmployeeID = @SourceEmployeeID

	INSERT INTO @tbl
	SELECT @SourceEmployeeID	AS EmployeeID
			, E.EmployeeID		AS ChildEmployeeID
			, E.Firstname		AS ChildFirstname
			, E.Surname			AS ChildSurname
			, E.PositionTitle	AS ChildPositionTitle
			, E.Hierarchy		AS ChildHierarchy
			, E.Hierarchy.GetLevel()	AS Level
	FROM Employee E
	WHERE E.Hierarchy.GetAncestor(1) = @SourceHierarchyID
	ORDER BY E.Hierarchy.GetLevel() ASC

	RETURN

END

To call this new function, pass it the EmployeeID of the reference entity you want to start with; and it should return all child records as requested:

SELECT *
FROM dbo.udfEmployee_GetAllChildrenByEmployeeID ('E0006') t

All Child records: output

All Descendant records

Another common function is the retrieval of all child records, done recursively until the bottom of the tree structure is reached. This is referred to as all the descendants of an entity.

note: IsDescendantOf (@SourceHierarchyID) will return true if the parameter object is a sub child, or the parameter object itself. In otherwords, an entity is a descendant of itself…!

CREATE FUNCTION [dbo].[udfEmployee_GetAllDescendantsByEmployeeID]
(
	@SourceEmployeeID AS VARCHAR(5)
)
RETURNS @tbl TABLE (EmployeeID				VARCHAR(5)
					, ChildEmployeeID		VARCHAR(5)
					, ChildFirstname		VARCHAR(50)
					, ChildSurname			VARCHAR(50)
					, ChildPositionTitle	VARCHAR(50)
					, ChildHierarchy		hierarchyid
					, ChildLevel			INT
					)

AS
BEGIN

	DECLARE @SourceHierarchyID AS hierarchyid

	SELECT @SourceHierarchyID = E.Hierarchy
	FROM Employee E
	WHERE E.EmployeeID = @SourceEmployeeID

	INSERT INTO @tbl
	SELECT @SourceEmployeeID	AS EmployeeID
			, E.EmployeeID		AS ChildEmployeeID
			, E.Firstname		AS ChildFirstname
			, E.Surname			AS ChildSurname
			, E.PositionTitle	AS ChildPositionTitle
			, E.Hierarchy		AS ChildHierarchy
			, E.Hierarchy.GetLevel()	AS Level
	FROM Employee E
	WHERE E.Hierarchy.IsDescendantOf(@SourceHierarchyID) = 1
	ORDER BY E.Hierarchy.GetLevel() ASC

	RETURN

END

SELECT *
FROM dbo.udfEmployee_GetAllDescendantsByEmployeeID ('E0006') t

All Descendant records: output

All Ancestor records

This is the opposite of All Descendants and simply goes up the tree until the root entity is returned.

CREATE FUNCTION [dbo].[udfEmployee_GetAllAncestorsByEmployeeID]
(
	@SourceEmployeeID AS VARCHAR(5)
)
RETURNS @tbl TABLE (EmployeeID				VARCHAR(5)
					, ChildEmployeeID		VARCHAR(5)
					, ChildFirstname		VARCHAR(50)
					, ChildSurname			VARCHAR(50)
					, ChildPositionTitle	VARCHAR(50)
					, ChildHierarchy		hierarchyid
					, ChildLevel			INT
					)

AS
BEGIN

	DECLARE @SourceHierarchyID AS hierarchyid

	SELECT @SourceHierarchyID = E.Hierarchy
	FROM Employee E
	WHERE E.EmployeeID = @SourceEmployeeID

	INSERT INTO @tbl
	SELECT @SourceEmployeeID	AS EmployeeID
			, E.EmployeeID		AS ChildEmployeeID
			, E.Firstname		AS ChildFirstname
			, E.Surname			AS ChildSurname
			, E.PositionTitle	AS ChildPositionTitle
			, E.Hierarchy		AS ChildHierarchy
			, E.Hierarchy.GetLevel()	AS Level
	FROM Employee E
	WHERE @SourceHierarchyID.IsDescendantOf(E.Hierarchy) = 1
	ORDER BY E.Hierarchy.GetLevel() DESC

	RETURN

END

All Ancestor records: output

All Sibling records

A Sibling is considered all entities with the same parent entitiy. So in this example, all siblins of E0006 has the same Manager of E0006 which is E0002.

CREATE FUNCTION [dbo].[udfEmployee_GetAllSiblingsByEmployeeID]
(
	@SourceEmployeeID AS VARCHAR(5)
)
RETURNS @tbl TABLE (EmployeeID				VARCHAR(5)
					, ChildEmployeeID		VARCHAR(5)
					, ChildFirstname		VARCHAR(50)
					, ChildSurname			VARCHAR(50)
					, ChildPositionTitle	VARCHAR(50)
					, ChildHierarchy		hierarchyid
					, ChildLevel			INT
					)

AS
BEGIN

	DECLARE @ManagerHierarchyID AS hierarchyid

	SELECT @ManagerHierarchyID = E.Hierarchy.GetAncestor(1)
	FROM Employee E
	WHERE E.EmployeeID = @SourceEmployeeID

	INSERT INTO @tbl
	SELECT @SourceEmployeeID	AS EmployeeID
			, E.EmployeeID		AS ChildEmployeeID
			, E.Firstname		AS ChildFirstname
			, E.Surname			AS ChildSurname
			, E.PositionTitle	AS ChildPositionTitle
			, E.Hierarchy		AS ChildHierarchy
			, E.Hierarchy.GetLevel()	AS Level
	FROM Employee E
	WHERE @ManagerHierarchyID = E.Hierarchy.GetAncestor(1)
	ORDER BY E.Hierarchy.GetLevel() DESC

	RETURN

END

All Sibling records: output

Update / Move Logic

Finally, one last thing to cover… is the ability to move an entity and its entire sub-hierarchy into a different branch of the tree. E0006 (HR General Manager) currently reports to E0002 (COO). If we wanted to move E0006 to report under say E0005 (CIO), we need to perform the following logic:

1. Grab the Manager record and use its hierarchyid to determine the next hierarchyid for its next child.
2. Once we determine the new child hierarchyid, we use GetReparentedValue(@oldH, @newH) to restructure the current entity and its entire sub-tree.

The complete logic to achieve this update is as follows:

ALTER PROCEDURE [dbo].[uspEmployee_Move](
	@EmployeeID			AS VARCHAR(5)
	, @NewManagerID		AS VARCHAR(5)
)
AS
BEGIN

	DECLARE @oldH hierarchyid
	DECLARE @newH hierarchyid
	DECLARE @ManagerHierarchyID hierarchyid

	SELECT @oldH = E.Hierarchy
	FROM Employee E
	WHERE E.EmployeeID = @EmployeeID

	SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
	BEGIN TRANSACTION

		-- [1] Identify the Manager Record
		-- [2] Find the LAST child attached to the Manager Record (we need this when calling the GetDescendant method)
		-- [3] Generate the next hierarchyid that is AFTER the LAST child retrieved above
		-- [4] Insert the new Employee with the newly generated hierarchyid

		-- [1]
		SELECT @ManagerHierarchyID = Hierarchy
		FROM Employee e
		WHERE E.EmployeeID = @NewManagerID

		-- [2]
		SELECT @newH = MAX(Hierarchy)
		FROM Employee E
		WHERE E.Hierarchy.GetAncestor(1) = @ManagerHierarchyID

		-- [3]
		SELECT @newH = @ManagerHierarchyID.GetDescendant(@newH, NULL)

		UPDATE Employee
		SET Hierarchy = Hierarchy.GetReparentedValue(@oldH, @newH)
		WHERE Hierarchy.IsDescendantOf(@oldH) = 1

	COMMIT
END

Once we have this Stored Procedure created, we can now safely test it out. As above, lets move the entire sub-tree of E0006 to report to E0005.  Below you will find the snippet that does this, along with the reversal back to the original structure.

-- Move E0006 (HR General Manager) to report under E0005 (CIO)
-- Note the recursive native of the update will also change the hierarhcy for descendants of E0006
EXEC	[dbo].[uspEmployee_Move]
		@EmployeeID = N'E0006',
		@NewManagerID = N'E0005'

-- Move back E0006 (HR General Manager) to report under E0002 (COO)
EXEC	[dbo].[uspEmployee_Move]
		@EmployeeID = N'E0006',
		@NewManagerID = N'E0002'

Update / Move: Before the change

Update / Move: After the change

Conclusion

If you’ve come this far, well done!  Hope this article has given you insight into the SQL Server hierarchyid types. Along with that, you should now be in a better position to decide whether hierarchyid types is well suited for your system, or whether you prefer traditional data relationships to define your data structure.

If you have any questions, let me know.

Pagination can be seen everywhere; from Blogs and Forums, to News sites. All these use some form of pagination or another. Without pagination Forums would be showing thousands of topics at once instead of showing 20 topics per page; Blogs would be showing hundreds of blog posts per page instead of 10 posts per page. News sites… well.. they would take a rather long time to load if they displayed all of articles at once!

There are several forms of Pagination available; and no doubt you would have come across one or more already:

• The first is what I refer to as Preemptive Loading
• The second is On Demand Loading

In this article I will explain each of these in detail. I will explain the short comings of Preemptive Loading, and the benefits of On Demand Loading; and also explain how On Demand Loading can be achieved via SQL Server…

Different Types of Pagination

Preemptive Loading:

The First, and easiest type of Pagination is Client side pagination. This is when all the data that ‘may be’ required to be displayed is transferred from the Server to the Client all one go; and then pagination is achieved on the Client side through local components (such as data grids, data pagers, or custom logic). I call it Preemptive Loading. The key point in preemptive loading is that all entities available to the paged component are loaded even before the data is visible to the user. Is this good or bad ?

There’s no distinctive answer because some times, preloading is amazing from a usability point of view. But preloading can also mean trouble when loading massive amounts of unnecessary data.

Preemptive Loading – When it’s Good:

Take an Image Gallery application for example. You click on a thumbnail to preview the full sized image; navigation options allow you to view Previous and Next images in the gallery. This is a great example where preloading the next image is great! When the user clicks next, the image is already preloaded for viewing immediately; and this would then trigger another preload for the next image in the list and so forth. From a usability point of view, the images appear right away instead of waiting for the system to transfer the file ‘after’ the user has clicked ‘next’.

Preemptive Loading – When it’s Bad:

Hypothetically, say that you have a Blog that stores 1mb of data for each page; and that the system has been built to preload a total of 5 pages (refer to the image below). A total of 5mb is transferred on initial load. What would happen if the user only wanted to view the 1st page. The system would have processed and transferred a total of 5mb when only 1mb of data was really needed by the user. Multiply this with 100 concurrent users, and you will being to see the impact this will have on the overall system performance.

Premptive Loading: Total = 5 pages. 1mb per page.

On Demand Loading:

The second type of Pagination is what I call On Demand Loading; and this is the opposite of Preemptive Loading. On Demand Loading refers to a ‘delayed’ loading of the data, and when loads it only loads enough data that has been requested by the user.

Take for example a Forum that holds a total of 5000 topics. The Forum has been Paginated to show the first 1000 topics (lets say 1000 topics = 1mb for example). With Preemptive Loading, 5 pages of 1000 topics each would have been transferred at once; a total of 5mb. On the other hand, with On Demand Loading, a mere 1 page with 1000 topics are transferred; saving the system 4mb of unnecessary bandwidth! In contrast to the above diagram, the diagram below depicts On Demand Loading with 1mb of data transfer per page, per request.

Examples like this is exactly where you should prefer a system with On Demand Loading over Preemptive Loading!

On Demand Loading: Total = 1 page. 1mb per page.

Is it really necessary ?

For small to medium sized applications On Demand Loading might not be necessary; and it could be more trouble than what it is worth. However On Demand Loading in the essence of pagination is critical to Enterprise development as a means of achieving high efficiency and performance output in the system.

So how do I get it done ?

Here I will show you how you can achieve On Demand Loading using SQL Server.

A) Start by creating a Product table in your database. You can use the following script to do so…:

CREATE TABLE [dbo].[Product](
	[ID] [int] IDENTITY(1,1) NOT NULL,
	[Name] [nvarchar](50) NOT NULL,
	[Price] [decimal](18, 2) NOT NULL,
	[CurrentProduct] [bit] NOT NULL,
	[Quantity] [int] NOT NULL,
	[ExpiryDate] [datetime] NOT NULL,
	[Supplier] [nvarchar](50) NULL,
	[Ranking] [int] NOT NULL,
 CONSTRAINT [PK_Product] PRIMARY KEY CLUSTERED
(
	[ID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

B) Script out a few hundred records and insert them into the Product table.

C) Once you’ve done that create a stored procedure with the following signature:

CREATE PROCEDURE [dbo].[uspProduct_GetAllProducts]
	@SearchString				AS VARCHAR(50)
	, @SortColumn				AS VARCHAR(100)
	, @SortAscending			AS BIT
	, @PageIndex				AS INT
	, @PageSize					AS INT
	, @TotalCount				AS INT OUTPUT
AS
BEGIN
END

Parameters:

• @SearchString: A filtering parameter to limit our data (optional).
• @SortColumn: The name of the data column for which our data set will be ordered by.
• @SortAscending: A boolean value specifying whether to sort ascending or not.
• @PageIndex: The page index of the data that we want returned.
• @PageSize: The number of entities per page
• @TotalCount: Output parameter that will hold the total number entities that matched the query

The Design

Before I dive into the body, let me give you the overall design for this stored procedure.

The concept behind SQL Pagination is as follows:

1. Create temporary table to hold ALL the records matching your query. This table is to hold a column called RowNumber which will be unique per row.
2. Insert ALL records that match your query into the temporary table. For the RowNumber column use the SQL ‘OVER‘ clause to sort and order the records
3. Set the @TotalCount output parameter from the temporary table by using a simple SELECT COUNT(*)
4. Paginate the data in your temporary table by using the RowNumber, @PageIndex and @PageSize values to return all records GREATER THAN [first entity] AND LESS THEN [last entity]

Now that we have the concept at hand, lets check out the code..

D) Declare your temporary table as follows. Note the RowNumber field in this temporary table that will store the sort order of the records.

	DECLARE @tmp AS TABLE (
			RowNumber				INT
			, ProductID				INT
			, Name					NVARCHAR(50)
			, Price					DECIMAL(18,2)
			, CurrentProduct		BIT
			, Quantity				INT
			, ExpiryDate			DATETIME
			, Supplier				NVARCHAR(50)
			, Ranking				INT
			)

Below is the main query. This should be changed to your own custom logic to filter out the records prior to pagination. For this example, we simply filter all products that match our @SearchString parameter.

			-- The main SQL query that returns ALL the valid data
			SELECT *
			FROM Product P
			WHERE P.Name like '%' + ISNULL(@SearchString, '') + '%'

Sorting using the SQL ‘OVER’ clause:

The SQL Server ‘OVER’ clause allows you to order your record set and generate row rankings across an ORDER BY subquery. What this means is that you have an ORDER BY clause that will sort based on the input parameters @SortColumn and @SortAscending. Then based on this ordering, your record set (ie your main query records) are then ordered by the ROW_NUMBER function based on the ORDER BY subquery. The end result is that your temporary table will be populated with RowNumbers 1 to xxx in the sort order specified by your @Sort Parameters.

E) Below is a snippet on how to number our records based on the ‘ID’ column:

	INSERT INTO @tmp
	SELECT ROW_NUMBER() OVER
	( ORDER BY
		CASE		-- Numeric ASC
			WHEN @SortColumn = 'ID' AND @SortAscending = 1 THEN Q.ID
			ELSE NULL
		END ASC
		, CASE		-- Numeric DESC
			WHEN @SortColumn = 'ID' AND @SortAscending = 0 THEN Q.ID
			ELSE NULL
		END DESC
	) AS RowNumber
	, Q.ID
	, Q.Name
	, Q.Price
	, Q.CurrentProduct
	, Q.Quantity
	, Q.ExpiryDate
	, Q.Supplier
	, Q.Ranking
	FROM (
			-- The main SQL query that returns ALL the valid data
			SELECT *
			FROM Product P
			WHERE P.Name like '%' + ISNULL(@SearchString, '') + '%'
	) AS Q

F) Once you have all the records in your temporary table. You can now set the @TotalCount output parameter as follows:

	SET @TotalCount = ISNULL((SELECT COUNT(RowNumber) FROM @tmp), 0)

G) Finally, you can now safely paginate your temporary table data and return the paginated data as per the @PageIndex and @PageSize values. There’s some small logic here to grab the correct minimum RowNumber and maximum RowNumber values.. but other than that, it should be straight forward:

NOTE: Passing in @PageIndex = -1, @PageSize = -1 will return ALL records from your query un-paginated. This can be changed as you see fit in the logic below. But I have added this logic incase it was needed.

	-- The final SQL query that returns:
	--		The PAGINATED data only, Sorted as specified
	SELECT *
	FROM @tmp T
	WHERE T.RowNumber > CASE WHEN ISNULL(@PageIndex, 0) * ISNULL(@PageSize, 0) <= 0 THEN 0
							ELSE ISNULL((@PageIndex)*@PageSize, 0)
						END
		AND T.RowNumber <= CASE WHEN ISNULL(@PageIndex+1, 0) * ISNULL(@PageSize, 0) <= 0 THEN @TotalCount
								ELSE ISNULL((@PageIndex+1)*@PageSize, 100)
							END

Testing it all out

Once that’s done, that’s it! You can now run your stored procedure to see the results

DECLARE	@TotalCount int

EXEC	[dbo].[uspProduct_GetAllProducts]
		@SearchString = N'a',
		@SortColumn = N'ID',
		@SortAscending = 1,
		@PageIndex = 0,
		@PageSize = 10,
		@TotalCount = @TotalCount OUTPUT

SELECT	@TotalCount as N'@TotalCount'

Summary

Hopefully everything worked out fine! Congratulations, you have created a procedure that returns paginated data with Sorting integrated into its logic.

NOTE: The Sorting can be extended to all the fields you require. In the snippet below I have included the entire Stored Procedure logic that allows sorting on all columns of the Product table.

CREATE PROCEDURE [dbo].[uspProduct_GetAllProducts]
	@SearchString				AS VARCHAR(50)
	, @SortColumn				AS VARCHAR(100)
	, @SortAscending			AS BIT
	, @PageIndex				AS INT
	, @PageSize					AS INT
	, @TotalCount				AS INT OUTPUT
AS
BEGIN
	DECLARE @tmp AS TABLE (
			RowNumber				INT
			, ProductID				INT
			, Name					NVARCHAR(50)
			, Price					DECIMAL(18,2)
			, CurrentProduct		BIT
			, Quantity				INT
			, ExpiryDate			DATETIME
			, Supplier				NVARCHAR(50)
			, Ranking				INT
			)
	INSERT INTO @tmp
	SELECT ROW_NUMBER() OVER
	( ORDER BY
		CASE		-- Numeric ASC
			WHEN @SortColumn = 'ID' AND @SortAscending = 1 THEN Q.ID
			WHEN @SortColumn = 'Quantity' AND @SortAscending = 1 THEN Q.Quantity
			WHEN @SortColumn = 'Price' AND @SortAscending = 1 THEN Q.Price
			ELSE NULL
		END ASC
		, CASE		-- Numeric DESC
			WHEN @SortColumn = 'ID' AND @SortAscending = 0 THEN Q.ID
			WHEN @SortColumn = 'Quantity' AND @SortAscending = 0 THEN Q.Quantity
			WHEN @SortColumn = 'Price' AND @SortAscending = 0 THEN Q.Price
			ELSE NULL
		END DESC
		, CASE		-- Text	ASC
			WHEN @SortColumn = 'Name' AND @SortAscending = 1 THEN Q.Name
			WHEN @SortColumn = 'Supplier' AND @SortAscending = 1 THEN Q.Supplier
			ELSE NULL
		END ASC
		, CASE		-- Text DESC
			WHEN @SortColumn = 'Name' AND @SortAscending = 0 THEN Q.Name
			WHEN @SortColumn = 'Supplier' AND @SortAscending = 0 THEN Q.Supplier
			ELSE NULL
		END DESC
		, CASE		-- Date ASC
			WHEN @SortColumn = 'ExpiryDate' AND @SortAscending = 1 THEN Q.ExpiryDate
			ELSE NULL
		END ASC
		, CASE		-- Date DESC
			WHEN @SortColumn = 'ExpiryDate' AND @SortAscending = 0 THEN Q.ExpiryDate
			ELSE NULL
		END DESC
	) AS RowNumber
	, Q.ID
	, Q.Name
	, Q.Price
	, Q.CurrentProduct
	, Q.Quantity
	, Q.ExpiryDate
	, Q.Supplier
	, Q.Ranking
	FROM (
			-- The main SQL query that returns ALL the valid data
			SELECT *
			FROM Product P
			WHERE P.Name like '%' + ISNULL(@SearchString, '') + '%'
	) AS Q

	SET @TotalCount = ISNULL((SELECT COUNT(RowNumber) FROM @tmp), 0)
	-- The final SQL query that returns:
	--		The PAGINATED data only, Sorted as specified
	SELECT *
	FROM @tmp T
	WHERE T.RowNumber > CASE WHEN ISNULL(@PageIndex, 0) * ISNULL(@PageSize, 0) <= 0 THEN 0
							ELSE ISNULL((@PageIndex)*@PageSize, 0)
						END
		AND T.RowNumber <= CASE WHEN ISNULL(@PageIndex+1, 0) * ISNULL(@PageSize, 0) <= 0 THEN @TotalCount
								ELSE ISNULL((@PageIndex+1)*@PageSize, 100)
							END
END

Conclusion

Hope this article has helped explain the different types of Pagination. Both Preemptive Loading and On Demand Loading has its uses; and you will need to make a judgment call on which is best suited to your application design. Please take into consideration the performance impact that both types of Pagination could have in your environment.

Have fun!

In this article I will explain the architecture that combines a Silverlight Client Application to a RIA services solution. The solution will connect to a SQL Database back-end that uses SQL Server FileStream types for file storage.

Specifically I will explain how you can transfer files in a Silverlight + RIA Services architecture to and from a SQL Server Database using FileStream Types…

Prerequisites

• How to enable SQL FileStream Types
• Using SQL FileStream Types in your Databases
• MaxRequestLength – Packet sizes: Size isn’t everything

The Basics

To begin with let me explain some of the basic components:

• You have a Silverlight Client Application with XAML pages
• You have Domain Services exposed to the Client Application with servicing methods
• The mid-tier traverses through a communications layer to reach Repository Classes
• Repository Classes are used to connect to the Database to set and retrieve information

With this in mind, the diagram below represents a Request from the Client to the Database Server:

RIA Services – Client to Server Request:

Downloading

Before we consider File Uploading via RIA Services lets consider its easier counter-part:  File Downloading via RIA Services! Please take 5 minutes to read my article on File Buffering and Streaming because the concepts used here are exactly the same…

Below is the initial Download Request from the Client Application. The Request passes through a File Handler. This File Handler is the key component for Streaming data between two systems. The Request is pretty straight forward, the File Handler pipes the Request to the Domain Service – which in turn calls our Manager Class and Repository Classes.

The Repository Class is responsible for querying the SQL Database to get the File Stream data from the SQL Table.

Later in this article there is a code-snippet that explains how I achieved this.

File Download – Client to Server Request:

Once the Repository has a reference to the File entity. The Repository passes the Byte[] array data through all of the channels back via the Domain Service to the File Handler. The diagram below is a representation of this process:

note: The RED arrows indicate that Streaming is happening and shows the direction that the streamed data is moving.

File Download (Streamed) – Server to Client Response:

Uploading

Now lets discuss the File Upload counter-part. When I first designed the solution I ran into performance issues. To give you a better understanding I will describe my original design below:

• The XAML page loads a BackgroundWorker to upload packets of data to the File Handler.
• On the ProcessRequest() method of the File Handler I piped the data through the Domain Service
• Which in turn piped the data to the Repository
• The Repository opened a data stream to the Database and flushed the data into this steam.
• The Repository then closes the database connection.
• This repeats for as many upload Packets as necessary.

The performance hit that got me was that the BackgroundWorker’s processing method needed to wait until the File Handler completed its entire process; and this happens after all the steps above have finished – including open and closing the database connection and writing the contents to the FileStream table column.

So an alternate approch was devised…

File uploading via RIA Service is split into 2 parts:

1. Transferring the file from the Client Application to the Web Server via the File Handler
2. Transferring the file from the Web Server to the Database Server as a SQL FileStream Type

1. File Transfer (Client Application to Web Server)

This first process is identical to the steps explained in the article: File Buffering and Streaming

File Upload (Streamed) – Client to Web Server:

2. File Transfer (Web Server to DB Server)

This diagram is the representation on how the File Handler transfers data packets to the SQL Database via Domain Services:

File Upload (Streamed) – Web Server to DB Server:

We extend what we previously had in the File Handler ProcessRequest() method by adding extra functionality that executes when the last file packet is retrieved and appended to the Web Server file. Below is the entire ProcessRequest() method – you will notice an additional call is made on detection of the “completed” flag to a private method TransferFileToDB().

        public void ProcessRequest(HttpContext context)
        {
            if (context.Request.QueryString.AllKeys.Contains("attachmentid")
                    && context.Request.QueryString.AllKeys.Contains("filename")
                    && context.Request.QueryString.AllKeys.Contains("uniquefilename"))
            {
                // Write to the Upload Folder
                System.IO.FileInfo fi = new System.IO.FileInfo(@"C:\MyUploads"
                                                + context.Request.QueryString["uniquefilename"]);
                System.IO.FileStream fs;

                try
                {
                    if (fi.Exists)
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.Append);
                    }
                    else
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.OpenOrCreate
                                                          , System.IO.FileAccess.Write);
                    }

                    // Insert the data chunk into the file
                    byte[] b = new byte[context.Request.InputStream.Length];
                    context.Request.InputStream.Read(b, 0, b.Length);
                    fs.Write(b, 0, b.Length);
                    fs.Close();

                    // Check if this is the last data packet.
                    // If so, call TransferFileToDB() to upload to the DB
                    if (context.Request.QueryString.AllKeys.Contains("completed"))
                    {
                        TransferFileToDB(
                                        System.Guid.Parse(
                                              context.Request.QueryString["attachmentid"]
                                        )
                                        , fi.FullName
                                        , context.Request.QueryString["uniquefilename"]
                                        , context.Request.QueryString["filename"]);
                    }
                }
                catch (Exception ex)
                {
                    //Handle Error - Remove the temporary file etc
                }
            }
        }

The TransferFileToDB method is responsible for streamlining the local Web Server file contents to a Domain Service method that has been coded to accept data packets. So we continuously call the UploadStreamed() method of the Domain Service until we reach the end of the file.

When we transfer the final packet to the SQL Server we can safely clean up by deleting the temporary file on the Web Server – as we are now sure that the entire contents have been uploaded to the SQL Server Database.

But if any exception is thrown you need to clean up the partially uploaded file on the SQL Database. Plus remove the temporary file on the Web Server, and alert the User accordingly.

        private void TransferFileToDB(
                                            System.Guid attachmentID
                                            , string filePath
                                            , string newFileName
                                            , string originalFileName)
        {
            AttachmentService svc = new AttachmentService();
            System.IO.FileStream fs;

            // Get the file from the Upload Folder
            System.IO.FileInfo fi = new System.IO.FileInfo(filePath);

            if (fi.Exists)
            {
                try
                {
                    byte[] buffer;
                    int position = 0;
                    int packetSize = 10 * 1000 * 1024;

                    // Open the file for reading
                    fs = new System.IO.FileStream(filePath, System.IO.FileMode.Open);
                    long length = fs.Length;

                    // Stream the data packets to the Domain Service for storage
                    while (position < length)                     {                          if (position + packetSize > length)
                        {
                            buffer = new byte[length - position];
                        }
                        else
                        {
                            buffer = new byte[packetSize];
                        }

                        fs.Read(buffer, 0, buffer.Length);
                        svc.UploadStreamed(attachmentID
                                            , newFileName
                                            , originalFileName
                                            , buffer
                                            , position);
                        position += buffer.Length;
                    }
                    fs.Close();

                    // Once all data packets are uploaded, clean up the file.
                    fi.Delete();

                }
                catch (Exception ex)
                {
                    //Handle error here
                }
            }
        }

Next up…. the Domain Service implementation!

Enjoy!

In this article I will provide an example on how to stream data from a Silverlight Client to the Web Server using the Buffered and Steaming approach. A general knowledge of .Net Generic Handlers is ideal because this solution relies heavily on custom handlers to feed data to and from our Web server.

Prerequisites:

• MaxRequestLength – Packet sizes: Size isn’t everything
  

The Environment

Consider the environment below:

• We have a Client Application with a XAML page.
• The XAML page makes a request to the Handler which is responsible for transferring the file to the Web Server.

Bulk Insert

The Client

In the XAML page, we have an OnClick() event handler… This handler grabs the selected file using the OpenFileDialog() method and then instantiates a BackgroundWorker to process the file. The BackgroundWorker then calls a private function called ProcessFileUpload() which does all the necessary processing. Here the BackgroundWorker is used to make an asynchronous call to the processing function rather than blocking off the UI thread.

        private void browse_Click(object sender, System.Windows.RoutedEventArgs e)
        {
            OpenFileDialog d = new OpenFileDialog();
            d.Multiselect = false;
            d.Filter = "All Files|*.*";

            if ((bool)d.ShowDialog())
            {
                // Construct URI to the File upload Handler ASHX file
                System.UriBuilder ub = new System.UriBuilder(
                     System.Windows.Browser.HtmlPage.Document.DocumentUri.Scheme
                     , System.Windows.Browser.HtmlPage.Document.DocumentUri.Host
                     , System.Windows.Browser.HtmlPage.Document.DocumentUri.Port
                     , "FileStreamUploader.ashx");

                // Set any parameters the Handler will require.
                // Here we pass in the original + unique file name
                ub.Query = "filename=" + d.File.Name
                            + "&uniquefilename=" + System.DateTime.Now.ToString("yyyyMMdd_HHmmss_")
                            + d.File.Name;

                // Initiate a background worker so we don't hold up the UI of the Client App
                BackgroundWorker bw = new BackgroundWorker();

                bw.DoWork += (bw_sender, bw_e) =>
                {
                    // The function that does the background work
                    ProcessFileUpload(ub.Uri
                                        , (System.IO.FileInfo)bw_e.Argument
                                        , (System.ComponentModel.BackgroundWorker)bw_sender
                                        , bw_e);
                };

                bw.RunWorkerAsync(d.File);
            }
            else //cancel
            {

            }
        }

When the code enters into the ProcessFileUpload() function the following things need to happen:

• Open the file for reading.
• Grab its length.
• Set a Byte[] array variable to store the file contents.
• Open a connection to the File Handler
• Pass this Byte[] array into the File Handler’s data stream
• Close the stream

This is handled by the following code snippet:

        private void ProcessFileUpload(System.Uri uri
                                            , System.IO.FileInfo fileInfo
                                            , System.ComponentModel.BackgroundWorker worker
                                            , System.ComponentModel.DoWorkEventArgs e)
        {
            System.IO.Stream fs = fileInfo.OpenRead();

            long length = fs.Length;

            byte[] buffer;
            buffer = new byte[length];

            // Read the contents of the file into the buffer
            fs.Read(buffer, 0, buffer.Length);

            // Open a connection to the Request Stream to the file Handler
            // Then pipe the data across..
            System.Net.WebClient wc = new System.Net.WebClient();
            wc.OpenWriteCompleted += (wc_s, wc_e) =>
            {
                UploadPacketToHandler(buffer, wc_e.Result);
                wc_e.Result.Close();
            };

            wc.OpenWriteAsync(uri);
            fs.Close();
        }

        private void UploadPacketToHandler(byte[] fileData, System.IO.Stream outputStream)
        {
            // Pipe the packet into the Steam to the Server
            outputStream.Write(fileData, 0, fileData.Length);
            outputStream.Close();
        }

At the end of this process, what has happened is that the file contents have now been sent to the Web Server via the File Handler’s input stream.

Next, it is up to the File Handler logic to process this data whichever way it needs to..

The File Handler

        public void ProcessRequest(HttpContext context)
        {
            if (context.Request.QueryString.AllKeys.Contains("filename")
                    && context.Request.QueryString.AllKeys.Contains("uniquefilename"))
            {
                // Write to the Upload Folder
                System.IO.FileInfo fi = new System.IO.FileInfo(@"C:\MyUploads";
                                                + context.Request.QueryString["uniquefilename"]);
                System.IO.FileStream fs;

                try
                {
                    if (fi.Exists)
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.Append);
                    }
                    else
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.OpenOrCreate
                                                          , System.IO.FileAccess.Write);
                    }

                    // Insert the data into the file
                    byte[] b = new byte[context.Request.InputStream.Length];
                    context.Request.InputStream.Read(b, 0, b.Length);
                    fs.Write(b, 0, b.Length);
                    fs.Close();
                }
            }
        }

The entry point for the File Handler is the ProcessRequest() function. We grab the filename parameters from the QueryString and then open up a FileInfo object to the path of the uploaded file. Then we write the entire contents of the input stream into the FileInfo object, and close the stream. Pretty simple!

Conclusion:

That’s it! You have successfully implemented an upload function using a Silverlight Client and a Generic Handler.

But what do packet size and streaming have to do with this example you ask??

Well you’re right. With this example, the largest file you can upload without reaching the Request Limit is 4MB. Try uploading a 10MB file and you should see the HTTP Request object throwing an exception about your request being too large and reaching its maximum limit. This is because we used the default maxRequestLength value of 4MB, and without any streaming..

Next I will explain how you change this code to allow for ANY file size, while still preserving the 4MB maxRequestLength property.

Streaming

This is easily accomplished with a few lines of extra code to the snippets above.
This is the overview of the changes we need to make:

• The BackgoundWorker’s processing function needs to make several calls to the File Handler instead of one call.
• With each call to the File Handler, the BackgroundWorker’s processing functions will send ONE packet of data.
• This process repeats itself until the BackgroundWorker processing functions reach the end of the input file.
• On the File Handler side of things, the main ProcessRequest() function will need to APPEND data to an existing file for subsequent writes. So that the input data is written to the same file in 4MB packets.
• Once the File Handler has detected the end-of-file (from a parameter) the File Handler can then use the file for further process like sending it as an email attachment etc.

BackgroundWorker Modifications

Below is the first change we need to make to the BackgroundWorker processing functions:

        private void ProcessFileUpload(System.Uri uri
                                    , System.IO.FileInfo fileInfo
                                    , System.ComponentModel.BackgroundWorker worker
                                    , System.ComponentModel.DoWorkEventArgs e)
        {
            try
            {
                System.IO.Stream fs = fileInfo.OpenRead();

                long length = fs.Length;

                byte[] buffer;
                int position = 0;
                int packetSize = 5 * 1000 * 1024;

                while (position < length)                 {
                    if (position + packetSize > length)
                    {
                        buffer = new byte[length - position];

                        // If this is the last buffer for this file..
                        // .. set the completed flag for the handler
                        System.UriBuilder ub = new System.UriBuilder(uri);
                        if (ub.Query == string.Empty) { ub.Query = "completed=1"; }
                        else { ub.Query = ub.Query.TrimStart('?') + "&completed=1"; }
                        uri = ub.Uri;
                    }
                    else
                    {
                        buffer = new byte[packetSize];
                    }

                    fs.Read(buffer, 0, buffer.Length);
                    position += buffer.Length;

                    // Mutex to ensure packets are sent in order
                    System.Threading.AutoResetEvent mutex =
                                        new System.Threading.AutoResetEvent(false);

                    System.Net.WebClient wc = new System.Net.WebClient();
                    wc.OpenWriteCompleted += (wc_s, wc_e) =>
                    {
                        UploadPacketToHandler(buffer, wc_e.Result);
                    };
                    wc.WriteStreamClosed += (ws_s, ws_e) =>
                    {
                        mutex.Set();
                    };
                    wc.OpenWriteAsync(uri);
                    mutex.WaitOne();
                }

                fs.Close();
            }
            catch (Exception ex)
            {
                //Handle errors here
            }
        }

File Handler modifications

In the File Handler code, we need to CREATE the file for the very first packet.
Then on subsequent packets APPEND the packet to the end of the file.

Once the ‘completed’ parameter is detected the File Handler knows this is the final packet to be appended. After this we can continue processing the file as we need to…

        public void ProcessRequest(HttpContext context)
        {
            if (context.Request.QueryString.AllKeys.Contains("filename")
                    && context.Request.QueryString.AllKeys.Contains("uniquefilename"))
            {
                // Write to the Upload Folder
                System.IO.FileInfo fi = new System.IO.FileInfo(@"C:\MyUploads"
                                                + context.Request.QueryString["uniquefilename"]);
                System.IO.FileStream fs;

                try
                {
                    if (fi.Exists)
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.Append);
                    }
                    else
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.OpenOrCreate
                                                          , System.IO.FileAccess.Write);
                    }

                    // Insert the data into the file
                    byte[] b = new byte[context.Request.InputStream.Length];
                    context.Request.InputStream.Read(b, 0, b.Length);
                    fs.Write(b, 0, b.Length);
                    fs.Close();

                    // Check if this is the last data packet.
                    // If so do additional processing as needed...
                    if (context.Request.QueryString.AllKeys.Contains("completed"))
                    {
                        // Additional process as required by your application:
                        // Send as attachment
                        // or Send to Database etc
                    }
                }
            }
        }

Conclusion

That’s it!! You have now successfully implemented a buffered/streaming file uploading component.
Pretty easy huh?

If you have any questions, feel free to drop a message

Hope this article helped!

Increasing this value to say 40,960 will allow 10x the default file size to be posted to your web server. This will fix your immediate file size limitation to 40MB. But what it also means is that you have now opened your Web application to accept any request 10x the standard request length. The question you need to ask yourself is, is this a good thing or is this a bad thing.. ?

In this article I will explain how to go one step further and show how you can implement a buffered and streaming approach to file downloading and uploading…

Background Information

The quickest way to resolve file upload limitations in a ASP.Net Application is to increase the maxRequestLength attribute in the Web.Config.  Below is the snippet that is required to accomplish this:

<system.web>
  <httpRuntime maxRequestLength="40960"/>     <!-- 40MB limit, instead of 4MB limit -->
</system.web>

So is this good or bad.. ? In short, I believe this is a bad.

It’s a very quick solution, and if your application is relatively small to medium then this solution is quite possible and fairly robust. One important consideration is the side-effect this will have to your Web application. Instead of accepting a maximum web request size of 4mb, you have now allowed the server to be bombarded with up to 40mb of data per request.  You need to decide if this makes your server prone to attacks and security mishaps..

The solution I propose is to understand how packet sizes, buffering and streaming work. Once you understand how this works, I promise you will never go back!

Packet Sizes

The core concept behind streaming is the ability send many small packets of data as opposed to large packets of data across the wire.

Consider the following example:

• You have a total network bandwidth of 100megabits/sec.  Which is equivalent to 12.5 MB/sec.
  Lets use 12MB for simplicity
• You have 2 concurrent users.
• Each user wants to transfer a 100MB file

The image below is a visual representation of our 12MB/sec network split into 3 sections.
Each section can handle 4MB.

4MB Example:

With maxRequestLength = 4MB, our 2 Users each use up 4MB out of the available 12MB bandwidth.
BLUE and GREEN successfully transfers 100MB each, totalilng 200MB in 25 secs.

note: Our network still has 4MB (RED) of bandwidth remaining for other traffic

40MB Example:

Now consider what happens if we increase maxRequestLength to 40MB.

• BLUE has 100MB to transfer.
• Each packet is 40MB because of our maxRequestLength value set to 40MB
• A total of 2.5 packets need to be sent

BLUE uses up the entire 12MB bandwidth for a total of 100MB/12MB per sec = 8.33 secs.
So in 8.33 secs BLUE completes its data transfer.

• GREEN also has 100MB to transfer.
• Each packet is also 40MB
• A total of 2.5 packets need to be sent

GREEN needs to wait for BLUE to finish before GREEN can begin its data transfer.
GREEN also takes 8.33 secs to transfer 100MB.

The entire process takes 16.6 secs to transfer a total of 200mb.

• A 4MB environment takes 25 secs to send 200MB for 2 users.
• A 40MB environment takes 16.6 secs to send 200MB for 2 users.

The latter would seem the better solution as it completes sooner. But your Network Infrastructure is crippled for a total of 16.6 secs as no bandwidth is available for other traffic across the network.

So hopefully this example has given you the information needed to understand why smaller packets are preferred over larger packets.