1
Vote

File size limitations

description

Kudos on the samples for FileStream in Katmai!
 
Our organization will be serving up video files from 1.6 to 3 GBs in size. I'm currently running the source code in a virtual machine session (VMWare Fusion 1.0, Win Server 2008, Katmai). Everything works great, until I try to upload a 1.6GB file, and I run into the ever popular "OutOfMemoryException".
 
My question: Am I stupid for trying to run this in a virtual machine with only 2GB of mem, or is there something inherent in the system that is going to limit us from uploading files of this size to SQL Server?
 
Thanks!

comments

ReedMe wrote Jan 15, 2008 at 6:05 PM

Other than the obvious limitations of the virtual environment that you're in, dizzyguy, there should be no reason that you'd be limited. How exactly are you trying to load the filestream? With SNAC?

dizzyguy wrote Jan 15, 2008 at 10:26 PM

I've done everything that was laid out in the demo, except with one not-so-minor change evidently. I'm trying to use the <input type='file'> tag to load the file into SQL. It works with 50MB files, but I get a 404 error when trying to work with files in the GB+ range.

In this case, the INPUT is an ASPNET FileUpload component, and I'm loading the FileStream from the FileUpload's InputStream like so:

// Open up a new stream to write the file to the blob.
destFile = new FileStream(handle, FileAccess.Write);

// Open up a new stream to read the source media file.
// --- Not using sourceFile: FileStream sourceFile = new
// FileStream(String.Concat(filePath, fileName), FileMode.Open, FileAccess.Read);
// Use the FileUpload.PostedFile.InputStream
byte[] buffer = new byte[FileUpload1.PostedFile.InputStream.Length];
int bytesRead;
while ((bytesRead = FileUpload1.PostedFile.InputStream.Read(buffer, 0, buffer.Length)) > 0)
{
destFile.Write(buffer, 0, bytesRead);
}
destFile.Close();
// sourceFile.Close();

Maybe using the INPUT-FILE tag isn't the right option? But we do need to enable offices from around the organization to upload the data in a standard manner -- and the INPUT-FILE tag was our first choice.

Any thoughts on that?

dizzyguy wrote Jan 15, 2008 at 10:28 PM

One more thing worth noting...I moved the code to a server with plenty of resources, but it still failed to load the larger files. Makes me think this is something related to the INPUT tag and how IIS is processing this????

dizzyguy wrote Jan 16, 2008 at 2:40 AM

After further reading, I learned that this is an ASPNET worker process "problem". I will need to either buy a component or figure out another way to get users from disparate networks to upload 2GB+ files to our servers.

dizzyguy wrote Jan 16, 2008 at 9:01 PM

Sorry for all of the comments, but can you tell me if you think that a WCF service set up to receive streamed data (e.g. transferMode.Streamed) might solve the OOM Exception? My thought is that Silverlight+WCF/Streaming might be an answer for getting > 1GB files into a FILESTREAM on SQL 2008? This is probably a question for the WCF team, because as I understand it, the problem with ASPNET is that the data is saved in memory - instead of being written to the hard drive - which is the reason we get OOM exceptions. But does streaming break the data up enough to solve any memory issues? All rhetorical questions on my part, but if someone out there can help me put this all into perspective, I'd really appreciate it!

ZachSkylesOwens wrote Jan 18, 2008 at 2:09 AM

Sorry that it took me a couple days to see this… I tried this with videos that were a few hundred MB without any problem. I’m not sure what limitation you might be running into with the GB size.

We have a sample that uses a Custom IIS 7 Module written in C++ which has great performance for serving these up. We were able to tap into some lower level system resources. We are working on solidifying that code and will publish it in the next few weeks.

Just to clarify… Are you having this issue when serving the file with the ASPX service?

dizzyguy wrote Jan 18, 2008 at 2:21 PM

Yep, that's right, within an ASPX page. And it appears that ASPNET doesn't play well with large files. My next attempt is going to be to build a WCF service (binary over TCP). This is all pretty new to me, so this may be yet another wasted effort, but I'll be happy to keep you all informed if you like.

dizzyguy wrote Jan 18, 2008 at 5:39 PM

And yeah, definitely interested in learning more about your custom IIS 7 module...

ZachSkylesOwens wrote Jan 24, 2008 at 11:46 PM

Have you made any progress on this dizzyguy? Have you identified where the bottle neck is? You mentioned the input-file tag… The code in the sample doesn’t actually provide an interface for uploading video via a web page.

Are you experiencing the file size issue when trying to serve up the video? It reads and writes the file in 512K chunks which shouldn’t cause an issue.

I’m interested to hear your conclusion.