Memory Size Limits in Elements and Progressive Loading

·

If you stitch large images, you’ve likely seen the “Image will be Loaded Progressively” message when loading a stitched file which has a size greater then the computer’s available memory.

Why do stitched images have size limits, whereas stack files, or standard ND2 site acquired files don’t? The answer is file paging. Basically, as you view an image from a stack, only that view is opened into memory (usually with a few images ahead of and behind the current image. Therefore the total image size may be 2TB, but the opened image(s) are only sitting in a few hundred kb of active memory. As you play the image sequence, or allow the computer to process the image sequence, each new chunk of images is opened, viewed, and then released during playback. This technique avoids the limits of 32 bit PC’s.

The tough part is, that this is not possible with a large image. In order for the image to be used in processing, the entire filesize needs to be opened at once into the program. This means that if you’ve acquired a 4gb image, you’ll need 4gb of free (not total) memory to view that image. Actually more, as processing software needs room to add layers like a binary, regions and so on.

So, the bottom line these days is that it’s obvious you’ll need a lot of memory working with stitched files, therefore if you are using stitched files you should use a x64 OS, which will allow you memory size of 32Gb and up.

Opening Elements up to more memory:

By default, Elements will only use a predetermined amount of memory. Fortunately the engineers allowed the user to have control of what this memory size is! You can increase the maximum allowable memory size for your program, by using the following code in a macro, or at the command line:

SetMaxMemoryImageSize(1073741824);

The number in the () is the Byte value or one Gigabyte. My computer has 16 GB, but I prefer to limit the filesize as a) I can increase it if needed and b) I’ll avoid approaching the max size of my PC and crash, when running processing operations.

Keep in mind, setting this size beyond your available memory will likely cause Elements to crash! On the other hand, I’ve seen several cases where files were just a little bit larger than the default limit, and increasing the limit allowed the operator to open the large file and process it, without chopping it up!

To make life easier on those who don’t want to code anything, I’ve made a small macro to do this operation. The macro also converts MB into Bytes, so you don’t have to worry about that part.

 

-Austin

 

Comments

4 responses to “Memory Size Limits in Elements and Progressive Loading”

  1. Michael Garcia Avatar

    Austin, I would really appreciate you thoughts on this…thank you in advance:

    Ray(Technical Instruments) and I are using his Dell to acquire images on my 90i. We are using both NIS AR and BR. We are using a DS-Ri1 at full res. I am testing a range of image sizes, but I anticipate I will be using a 60x oil and capturing 10x10x50 frame z-stack images.

    I noticed that during image acquisition, cpu usage is never more that 25% usually in combination with 100% frequency/clock speed, but even then only in bursts. Then I observed that during the “shading correction” it operated at an average of 8-9% cpu and never went above 13% cpu usage. Following that, during the “image composing” it operated at a slightly higher usage rate, but only a few %.

    Note: These observations are while acquiring a 25×50 frame z-stack image just shy of 40GB. I specifically acquired an image larger than the ram memory just to see if it would handle it or crash. It handled it but it has taken 1.5 hrs to acquire, 1 hr to perform the shading correction and 2 hrs for image composition.

    My main question is why is it only using 25% cpu at any time. Is this software? Is elements just not designed to utilize multiple cores? Do we just need to make a settings change to allowable ram?

    I put a stop to my new computer build until I can figure out the best config for Elements. Do I look for the fastest clock speed in an intel i7 which has 4 cores and can be overclocked easily to 5ghz or do I go with a newer Xeon E5-2650 v2 which has slower clock speed but 8 cores. I would have thought that the E5 would be better, but not if the software isn’t written to utilize the additional cores…

    It seems Nikon should be on top of this for you guys out in the field. No one likes a mystery…and from what I have found on the web blogs, images this size and larger are reasonably common.

    1. Austin Avatar
      Austin

      Hi Michael,

      I don’t think Elements is optimized for multi-threaded operations as much as it could be. Haven’t tested this extensively but I do remember several scenarios where only one core was used…one of the problems with any program like elements is that the programmer(s) won’t usually write everything soup to nuts. Normally any function which can be called from an existing routine (i.e. shading correction as an example) would be done via existing libraries. When you add up the qork required to write such a function (*the number of functions in a program like Elements) the labor cost is insanely high, so programmers will rely on licensed or open source libraries to save tons of time. Now, each library could or could not be memory/processor optimized, and this could be what you saw in elements (i.e. the shading DLL isn’t memory or processor optimized). As far as I know there isn’t an option to enable multiple core processing in elements – so if you need that level of processing power I agree this might lead you to get more single-core performance as an end-run, but also TI should push this up the chain for you to find out what the nikon guys say – it could be a simple switch which isn’t well known? This was Win 8×64 I assume?

      Regarding your image sizes, it’s not an every day thing, but also not uncommon – IMHO any time stitches are being processed things get more difficult as the entire image has to be handled in memory vs. slicing it up and running smaller portions. (or rather one COULD slice up the image but it’s not normally done in software of this genre). So – large image stitches are actually the best way to redline microscopy software – basically all of the available programs have a higher likelihood of crashing/failing when running stitch processes!

      1. Michael Garcia Avatar
        Michael Garcia

        Austin,
        Indeed Ray is pushing this up the chain…I just figure the response may be short and late to the party. In the mean time, thank you for your reply. Its tempting to wax on about the obvious issues in such an expensive and widely used software that should have either been resolved or explained. Nikon should provide a comprehensive list of “best suited” computer configurations. If not for the general public, surely for the reps in the field. My guess is that most users are students and researchers who just accept this because they didn’t pay for it themselves and have little say in the matter anyway… sorry about the little rant. If Nikon doesn’t have setting change to suggest, I’m thinking my best option is lots of Ram, Solid State Drives, and the fastest clock speeds I can find.
        As an aside, I think it was you who talked about running Elements in a Virtual environment with the MacOSX. What are your thoughts on doing that with one of the new Power Macs? Except for the add ons for the DS-U3 connections and a serial connection for the stage, which may be possible, would you recommend running elements this way?
        Thank you again, Michael

        1. Austin Avatar
          Austin

          You hit the nail on the head re: elemements. No apology needed for the rant! the sales guys deal with the limitations of this and similar programs day in and day out, but I think the challenge of new features, evolving device support and limited resources make the choices of what to fix/address a lot tougher than they would seem. I’m not sure, but this is my gut instinct as the middle guy. Re OSX – you could bootcamp it, but running in a virtual environment will be tough with proprietary device drivers. Esp the security key they use! Ask me how i know…

Leave a Reply

Your email address will not be published. Required fields are marked *

Social Media Auto Publish Powered By : XYZScripts.com