FOLLOWING on from my article about Packet Sizes I hope it has given you insight on how smaller packets are much more ideal over larger packets of data. Not only is it safer for your Web server, it also gives you control how much traffic you want your applications to use in your network.

In this article I will provide an example on how to stream data from a Silverlight Client to the Web Server using the Buffered and Steaming approach. A general knowledge of .Net Generic Handlers is ideal because this solution relies heavily on custom handlers to feed data to and from our Web server.

Prerequisites:

The Environment

Consider the environment below:

  • We have a Client Application with a XAML page.
  • The XAML page makes a request to the Handler which is responsible for transferring the file to the Web Server.


Bulk Insert

The Client

In the XAML page, we have an OnClick() event handler… This handler grabs the selected file using the OpenFileDialog() method and then instantiates a BackgroundWorker to process the file. The BackgroundWorker then calls a private function called ProcessFileUpload() which does all the necessary processing. Here the BackgroundWorker is used to make an asynchronous call to the processing function rather than blocking off the UI thread.

        private void browse_Click(object sender, System.Windows.RoutedEventArgs e)
        {
            OpenFileDialog d = new OpenFileDialog();
            d.Multiselect = false;
            d.Filter = "All Files|*.*";

            if ((bool)d.ShowDialog())
            {
                // Construct URI to the File upload Handler ASHX file
                System.UriBuilder ub = new System.UriBuilder(
                     System.Windows.Browser.HtmlPage.Document.DocumentUri.Scheme
                     , System.Windows.Browser.HtmlPage.Document.DocumentUri.Host
                     , System.Windows.Browser.HtmlPage.Document.DocumentUri.Port
                     , "FileStreamUploader.ashx");

                // Set any parameters the Handler will require.
                // Here we pass in the original + unique file name
                ub.Query = "filename=" + d.File.Name
                            + "&uniquefilename=" + System.DateTime.Now.ToString("yyyyMMdd_HHmmss_")
                            + d.File.Name;

                // Initiate a background worker so we don't hold up the UI of the Client App
                BackgroundWorker bw = new BackgroundWorker();

                bw.DoWork += (bw_sender, bw_e) =>
                {
                    // The function that does the background work
                    ProcessFileUpload(ub.Uri
                                        , (System.IO.FileInfo)bw_e.Argument
                                        , (System.ComponentModel.BackgroundWorker)bw_sender
                                        , bw_e);
                };

                bw.RunWorkerAsync(d.File);
            }
            else //cancel
            {

            }
        }

When the code enters into the ProcessFileUpload() function the following things need to happen:

  • Open the file for reading.
  • Grab its length.
  • Set a Byte[] array variable to store the file contents.
  • Open a connection to the File Handler
  • Pass this Byte[] array into the File Handler’s data stream
  • Close the stream

This is handled by the following code snippet:

        private void ProcessFileUpload(System.Uri uri
                                            , System.IO.FileInfo fileInfo
                                            , System.ComponentModel.BackgroundWorker worker
                                            , System.ComponentModel.DoWorkEventArgs e)
        {
            System.IO.Stream fs = fileInfo.OpenRead();

            long length = fs.Length;

            byte[] buffer;
            buffer = new byte[length];

            // Read the contents of the file into the buffer
            fs.Read(buffer, 0, buffer.Length);

            // Open a connection to the Request Stream to the file Handler
            // Then pipe the data across..
            System.Net.WebClient wc = new System.Net.WebClient();
            wc.OpenWriteCompleted += (wc_s, wc_e) =>
            {
                UploadPacketToHandler(buffer, wc_e.Result);
                wc_e.Result.Close();
            };

            wc.OpenWriteAsync(uri);
            fs.Close();
        }
        private void UploadPacketToHandler(byte[] fileData, System.IO.Stream outputStream)
        {
            // Pipe the packet into the Steam to the Server
            outputStream.Write(fileData, 0, fileData.Length);
            outputStream.Close();
        }

At the end of this process, what has happened is that the file contents have now been sent to the Web Server via the File Handler’s input stream.

Next, it is up to the File Handler logic to process this data whichever way it needs to..

The File Handler

        public void ProcessRequest(HttpContext context)
        {
            if (context.Request.QueryString.AllKeys.Contains("filename")
                    && context.Request.QueryString.AllKeys.Contains("uniquefilename"))
            {
                // Write to the Upload Folder
                System.IO.FileInfo fi = new System.IO.FileInfo(@"C:\MyUploads";
                                                + context.Request.QueryString["uniquefilename"]);
                System.IO.FileStream fs;

                try
                {
                    if (fi.Exists)
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.Append);
                    }
                    else
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.OpenOrCreate
                                                          , System.IO.FileAccess.Write);
                    }

                    // Insert the data into the file
                    byte[] b = new byte[context.Request.InputStream.Length];
                    context.Request.InputStream.Read(b, 0, b.Length);
                    fs.Write(b, 0, b.Length);
                    fs.Close();
                }
            }
        }

The entry point for the File Handler is the ProcessRequest() function. We grab the filename parameters from the QueryString and then open up a FileInfo object to the path of the uploaded file. Then we write the entire contents of the input stream into the FileInfo object, and close the stream. Pretty simple! :-)

Conclusion:

That’s it! You have successfully implemented an upload function using a Silverlight Client and a Generic Handler.

But what do packet size and streaming have to do with this example you ask??

Well you’re right. With this example, the largest file you can upload without reaching the Request Limit is 4MB. Try uploading a 10MB file and you should see the HTTP Request object throwing an exception about your request being too large and reaching its maximum limit. This is because we used the default maxRequestLength value of 4MB, and without any streaming..

Next I will explain how you change this code to allow for ANY file size, while still preserving the 4MB maxRequestLength property.

Streaming

This is easily accomplished with a few lines of extra code to the snippets above.
This is the overview of the changes we need to make:

  • The BackgoundWorker’s processing function needs to make several calls to the File Handler instead of one call.
  • With each call to the File Handler, the BackgroundWorker’s processing functions will send ONE packet of data.
  • This process repeats itself until the BackgroundWorker processing functions reach the end of the input file.
  • On the File Handler side of things, the main ProcessRequest() function will need to APPEND data to an existing file for subsequent writes. So that the input data is written to the same file in 4MB packets.
  • Once the File Handler has detected the end-of-file (from a parameter) the File Handler can then use the file for further process like sending it as an email attachment etc.

BackgroundWorker Modifications

Below is the first change we need to make to the BackgroundWorker processing functions:

        private void ProcessFileUpload(System.Uri uri
                                    , System.IO.FileInfo fileInfo
                                    , System.ComponentModel.BackgroundWorker worker
                                    , System.ComponentModel.DoWorkEventArgs e)
        {
            try
            {
                System.IO.Stream fs = fileInfo.OpenRead();

                long length = fs.Length;

                byte[] buffer;
                int position = 0;
                int packetSize = 5 * 1000 * 1024;

                while (position < length)                 {
                    if (position + packetSize > length)
                    {
                        buffer = new byte[length - position];

                        // If this is the last buffer for this file..
                        // .. set the completed flag for the handler
                        System.UriBuilder ub = new System.UriBuilder(uri);
                        if (ub.Query == string.Empty) { ub.Query = "completed=1"; }
                        else { ub.Query = ub.Query.TrimStart('?') + "&completed=1"; }
                        uri = ub.Uri;
                    }
                    else
                    {
                        buffer = new byte[packetSize];
                    }

                    fs.Read(buffer, 0, buffer.Length);
                    position += buffer.Length;

                    // Mutex to ensure packets are sent in order
                    System.Threading.AutoResetEvent mutex =
                                        new System.Threading.AutoResetEvent(false);

                    System.Net.WebClient wc = new System.Net.WebClient();
                    wc.OpenWriteCompleted += (wc_s, wc_e) =>
                    {
                        UploadPacketToHandler(buffer, wc_e.Result);
                    };
                    wc.WriteStreamClosed += (ws_s, ws_e) =>
                    {
                        mutex.Set();
                    };
                    wc.OpenWriteAsync(uri);
                    mutex.WaitOne();
                }

                fs.Close();
            }
            catch (Exception ex)
            {
                //Handle errors here
            }
        }

File Handler modifications

In the File Handler code, we need to CREATE the file for the very first packet.
Then on subsequent packets APPEND the packet to the end of the file.

Once the ‘completed’ parameter is detected the File Handler knows this is the final packet to be appended. After this we can continue processing the file as we need to…

        public void ProcessRequest(HttpContext context)
        {
            if (context.Request.QueryString.AllKeys.Contains("filename")
                    && context.Request.QueryString.AllKeys.Contains("uniquefilename"))
            {
                // Write to the Upload Folder
                System.IO.FileInfo fi = new System.IO.FileInfo(@"C:\MyUploads"
                                                + context.Request.QueryString["uniquefilename"]);
                System.IO.FileStream fs;

                try
                {
                    if (fi.Exists)
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.Append);
                    }
                    else
                    {
                        fs = new System.IO.FileStream(fi.FullName
                                                          , System.IO.FileMode.OpenOrCreate
                                                          , System.IO.FileAccess.Write);
                    }

                    // Insert the data into the file
                    byte[] b = new byte[context.Request.InputStream.Length];
                    context.Request.InputStream.Read(b, 0, b.Length);
                    fs.Write(b, 0, b.Length);
                    fs.Close();

                    // Check if this is the last data packet.
                    // If so do additional processing as needed...
                    if (context.Request.QueryString.AllKeys.Contains("completed"))
                    {
                        // Additional process as required by your application:
                        // Send as attachment
                        // or Send to Database etc
                    }
                }
            }
        }

Conclusion

That’s it!! You have now successfully implemented a buffered/streaming file uploading component.
Pretty easy huh?

If you have any questions, feel free to drop a message :)

Hope this article helped!

:-)