Very large image montage

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".
Post Reply
agni451
Posts: 4
Joined: 2013-03-07T01:14:57-07:00
Authentication code: 6789

Very large image montage

Post by agni451 »

I need to compile 10,000 1920x1080 24-bit BMP files into a single 192,000x108,000 PNG (20.7Gigapixels) image with maximum compression. The command line I've tried is

Code: Select all

"C:\Program Files\ImageMagick-6.8.3-Q16\montage.exe" -quality 90 -define registry:temporary-path="D:\Temp" -define png:include-chunk=none -mode concatenate -tile 100x100 "G:\R-*.bmp" "D:\Temp\FINAL.png"
Unfortunately, this causes ImageMagick to require more than 300GB of hard drive swap space (I don't actually know how much it really needs, as I only have 300GB available and it keeps failing. How might I calculate this?) when the 10,000 BMPs (already uncompressed!) only take up 58GB. Why does it require so much, and is there any way to do what I need with less HD space?

PS: I've even tried using the commandline above with -tile 100x1 to create 100 full-width rows, then tried merging those. I've also tried using 4 separate full-width, 25-row BMPs in the hope that fewer but larger images might make it more efficient, but it doesn't.

Windows 7 Ultimate x64
32GB memory (MAGICK_AREA_LIMIT 13GB)
snibgo
Posts: 12159
Joined: 2010-01-23T23:01:33-07:00
Authentication code: 1151
Location: England, UK

Re: Very large image montage

Post by snibgo »

For starters, you could use 8-bit IM. That will halve the space needed.

I'm curious to know why someone wants such huge PNG files.
snibgo's IM pages: im.snibgo.com
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Very large image montage

Post by anthony »

As you are simply appending and concatenating you may have other options.

For example appending 10 images at a time into larger 'row' images, then vertically appending the row images to form the file very large image.

Really Massive Image Handling
http://www.imagemagick.org/Usage/files/#massive

The key to such processing is image streaming. That is reading the source image data from there source files as needed, and writing results as they are found so as to use minimal memory. Typically handling the image 1 line at a time, or in the case of JPEG 8 lines at a time.

The problem is that while I have seen information on splitting up massive images into smaller pieces using 'streaming' methods, I have not seen information on taking a set of tile images and using streaming methods to generate a massive image.

It should be possible, and I even have idea how a stream image processing could do this, but I don't know of commands that can do this. The man pages of PbmPlus commands for example are very bad with regards to how 'stream friendly' they are.

Anyone else have any insights on this?


ASIDE: Most IM commands read whole images into memory, or if running out of memory into raw (uncompressed) disk cache, whcih can take a very long time to process. The IM "stream" command is meant to process images (with a very limited operation set) using row-by-row 'streaming' methods.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
darbly
Posts: 1
Joined: 2013-04-27T15:00:52-07:00
Authentication code: 6789

Re: Very large image montage

Post by darbly »

I am also running into difficulties generating large montages (of scientific images) on a machine running 64-bit Ubuntu 12.04 with 192 GB RAM and ImageMagick 6.6.9-7 2012-08-17 Q16.

Each image plane (800+) has a directory containing an 80 x 80 array of 2048 x 2048 pixel tiles named c[01-80]r[01-80].tif.

Trying to montage a single plane within a directory with:
montage c*r*.tif -limit area 0 -limit memory 0 -limit map 0 -tile 1x80 -size 2048x163840 -geometry 2048x163840+0+0 -depth 8 miff:- | convert - +append montage_test.tif

the memory usage skyrockets exceeding available RAM in minutes (although the section should be only 27 GB).

Thanks in advance for any suggestions and apologies for my ignorance,
Wei-
bugbear
Posts: 33
Joined: 2010-10-21T03:06:35-07:00
Authentication code: 8675308

Re: Very large image montage

Post by bugbear »

anthony wrote:As you are simply appending and concatenating you may have other options.
...

The problem is that while I have seen information on splitting up massive images into smaller pieces using 'streaming' methods, I have not seen information on taking a set of tile images and using streaming methods to generate a massive image.

It should be possible, and I even have idea how a stream image processing could do this, but I don't know of commands that can do this. The man pages of PbmPlus commands for example are very bad with regards to how 'stream friendly' they are.

Anyone else have any insights on this?
I would look to netpbm for this:

http://netpbm.sourceforge.net/doc/pnmcat.html

It's essentially BUILT on streaming.

BugBear
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Very large image montage

Post by anthony »

Information on exactly what level it handles streaming is not provided.

Also while concatenation vertically is easy, just read and output each image, on at a time, though you do need some pre-knowledge of the final hight of the resulting image.

Horizontal concatenation (using streaming) is harder. You essentially need to open N images simultaneously to read one line from each image in that row to generate one line of output.

Basically the documentation for streaming purposes leaves a lot to be desired, leaving you with source code diving :-(


Now if you have an example... I'd like to see it.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
bugbear
Posts: 33
Joined: 2010-10-21T03:06:35-07:00
Authentication code: 8675308

Re: Very large image montage

Post by bugbear »

anthony wrote:Information on exactly what level it handles streaming is not provided
Sorry - it's so implicit, it's almost never mentioned. The main mode of working in netpbm, for sequential operations is to PIPE together (in Unix) multiple
commands. There's another implicit assumption. Since all the commands ONLY accept netpbm's own format(s), the first and last stage of a pipeline "tend to be" input and output format converters.
Horizontal concatenation (using streaming) is harder. You essentially need to open N images simultaneously to read one line from each image in that row to generate one line of output.
Yes, that's exactly what concat does. So you need an open FD and a line buffer for each input image. You would "obviously" want to concat your images together into rows first, and the concat the rows into your final image.

If you want this to all be "purely" streamed, you'd need to use Unix's named pipes, since (just) stdin/stdout isn''t enough when dealing with multiple images (e.g. concat, or compositing).

Failing this, you'd need intermediate temp files.

BugBear
User avatar
anthony
Posts: 8883
Joined: 2004-05-31T19:27:03-07:00
Authentication code: 8675308
Location: Brisbane, Australia

Re: Very large image montage

Post by anthony »

bugbear wrote:
Horizontal concatenation (using streaming) is harder. You essentially need to open N images simultaneously to read one line from each image in that row to generate one line of output.
Yes, that's exactly what concat does. So you need an open FD and a line buffer for each input image. You would "obviously" want to concat your images together into rows first, and the concat the rows into your final image.
Do you have an example of this. I see "pnmcat" and "pnmundice" but it is unclear how you would use them to achieve a minimal memory foot print while processing each 'tile' image.

Though like all NetPBM/PbmPlus commands the man page does not explain how much of the image it holds in memory at most. I assume from what you say it is only 1 row, with all images held open as a file descriptors (as it needed to read all image sizes to start with). If so the limits would be number of file descriptors available on the system (low on very old machines), and perhaps image width, though I suppose it does not actually need that.
If you want this to all be "purely" streamed, you'd need to use Unix's named pipes, since (just) stdin/stdout isn''t enough when dealing with multiple images (e.g. concat, or compositing).
Understood, though again that is not clear. Still if it is pnmcat, than huge tiled images should be possible.


My current example uses pamcomp to merge small parts of an image back into a larger image.
Anthony Thyssen -- Webmaster for ImageMagick Example Pages
https://imagemagick.org/Usage/
bugbear
Posts: 33
Joined: 2010-10-21T03:06:35-07:00
Authentication code: 8675308

Re: Very large image montage

Post by bugbear »

anthony wrote:
bugbear wrote:
Horizontal concatenation (using streaming) is harder. You essentially need to open N images simultaneously to read one line from each image in that row to generate one line of output.
Yes, that's exactly what concat does. So you need an open FD and a line buffer for each input image. You would "obviously" want to concat your images together into rows first, and the concat the rows into your final image.
Do you have an example of this. I see "pnmcat" and "pnmundice" but it is unclear how you would use them to achieve a minimal memory foot print while processing each 'tile' image.

Though like all NetPBM/PbmPlus commands the man page does not explain how much of the image it holds in memory at most. I assume from what you say it is only 1 row, with all images held open as a file descriptors (as it needed to read all image sizes to start with). If so the limits would be number of file descriptors available on the system (low on very old machines), and perhaps image width, though I suppose it does not actually need that.
If you want this to all be "purely" streamed, you'd need to use Unix's named pipes, since (just) stdin/stdout isn''t enough when dealing with multiple images (e.g. concat, or compositing).
Understood, though again that is not clear. Still if it is pnmcat, than huge tiled images should be possible.


My current example uses pamcomp to merge small parts of an image back into a larger image.
A quick rummage in the source of pamundice shows that it simply open the output as a normal (netpbm) stream, and for each row of images, opens each image as a normal (netpbm) stream, and simply reads each pixel line from each sub-stream directly inplace to the output line.

All quite obvious, and fairly minimal.

Some of the format converters in the netpbm suite do read the entire raster into memory (the interlaced ones have little choice), and some of the functions also do (coding image rotation without a whole raster is tricky).

It's "quite important" to use "-force" with pnmtopng, otherwise it does a massive multipass attempt to work out the "best" way to represent the input as PNG, using various palettes and stuff. If you "know" you've just got fullcolour RGB, "-force" does what you want.

BugBear
bugbear
Posts: 33
Joined: 2010-10-21T03:06:35-07:00
Authentication code: 8675308

Re: Very large image montage

Post by bugbear »

For reasons of my own, I required a memory efficient montager.

I coded it around netpbm, especially pnmcat and it uses named pipelines (Posix FIFO's).

It only accepts JPEG and generates a PAM stream to stdout; typical usage would be, where BR is a row separator.

tess.pl tile0_0.jpg tile0_1.jpg tile0_3.jpg BR tile_1_0.jpg tile_1_1.jpg tile_1_2.jpg | pnmtojpeg -quality=85 > all.jpg

There may be remaining bugs, but it has worked for my purpose.

It may serve others either "as is", or as an example of technique.

BugBear

Code: Select all

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use File::Basename;
use Carp;
use POSIX;

sub fullyQuoted {
    my ($s) = @_;
    $s =~ s/'/'\\''/g;
    return "'$s'";
}

my @tmpFileList;

# any path passed to this method is recorded,
# and an attempt to unlink() it made at END time
sub deleteFileAtEnd {
    my ($f) = @_;
    push @tmpFileList, $f;
    return $f;
}

# takes a base and optional extension; the process ID
# and extension (if defined) are concatanated to the base
sub tmpFileName {
    my ( $base, $extension ) = @_;
    if ( defined($extension) ) {
        return $base . $$ . $extension;
    }
    else {
        return $base . $$;
    }
}

# wrapper that returns a fileName as per tmpFileName
# and also marks the file for eventual cleanup
sub tmpFileNameDeleteAtEnd {
    return deleteFileAtEnd( tmpFileName(@_) );
}

END {
    my $f;
    foreach $f (@tmpFileList) {
        if ( defined($f) && -e $f ) {
            unlink $f or croak "couldn't delete file $f:$!";
        }
    }
}

sub _backtick {
    my ( $cmd, $handler ) = @_;
    my $ret = `$cmd`;
    if($? == -1) {
        &$handler(
            sprintf( 'back tick : --%s-- could not be called', $cmd ) );
    }
    my $exit = ( $? >> 8 );
    if ( $exit != 0 ) {
        &$handler(
            sprintf( 'back tick : --%s-- failed, exit = %d', $cmd, $exit ) );
    }
    return $ret;
}

# execute like the `` operator, but catch, and deal
# with failures, by croak-ing
sub backtickdie {
    my ($cmd) = @_;
    _backtick(
        $cmd,
        sub {
            my ($msg) = @_;
            croak $msg;
        }
    );
}

# create a command that perform a LR concat of JPG files to pnm stdout 
sub lr {
    my ($files, $row_index) = @_;
    my $col_index = 0;
    my @ff;
    foreach my $f (@$files) {
	my $fifo_name = tmpFileNameDeleteAtEnd(sprintf("/tmp/f_%03d_%03d", $row_index, $col_index), "");
	mkfifo($fifo_name, 0666);
	# spawn a converter
	my $src = fullyQuoted($f);
	system("jpegtopnm $src > $fifo_name &");
	push @ff, $fifo_name;
	$col_index++;
    }
    my $row_name = tmpFileNameDeleteAtEnd(sprintf("/tmp/f_%03d", $row_index), "");
    mkfifo($row_name, 0666);
    my $src_list = join " ", @ff;
    system("pnmcat -lr $src_list > $row_name &");
    return $row_name;
}

sub tb {
    my ($tb) = @_;
    my $row_index = 0;
    my @ff;
    foreach my $r (@$tb) {
	push @ff, lr($r, $row_index++);
    }
    my $name = tmpFileNameDeleteAtEnd("/tmp/ff", "");
    mkfifo($name, 0666);
    my $src_list = join " ", @ff;
    system("pnmcat -tb $src_list > $name &");
    return $name;
}

sub stack_and_pump {
    my ($tb) = @_;
    my $src;
    if(scalar(@$tb) == 1) {
	$src = lr($tb->[0], 0);
    } else {
	$src = tb($tb);
    }
    my $in;
    open $in, "<", $src or die "cannot open $src";
    my $buf;
    while (read $in, $buf, 32 * 1024) {
	print STDOUT $buf;
    }
    close($in);
}

sub arg_parse {
    my ($argv) = @_;
    my @TB;
    my @LR;
    foreach my $f (@$argv) {
	if($f =~ /^br$/i) {
	    push @TB, [@LR];
	    @LR = ();
	} else {
	    push @LR, $f;
	}
    }
    if(scalar(@LR) > 0) {
	push @TB, [@LR];
    }
    stack_and_pump(\@TB);
}

unless(scalar(@ARGV) > 1) {
    die "usage $0 <file1> <file2> ['BR'] <file3> ... <filen> <output>";
}

# sort of like undice, but more pipe-lined

arg_parse(\@ARGV);

bugbear
Posts: 33
Joined: 2010-10-21T03:06:35-07:00
Authentication code: 8675308

Re: Very large image montage

Post by bugbear »

I just ran this on a 4x3 grid of 2.5Mb JPEG, which make up a 9524x10405 pixel output file (28 Mb JPEG)

Last night, using the "obvious" "appends" to make rows (of tiff) then "append" the rows, this took around 20 minutes, and
thrashed the hard drive most cruelly.

Just now, with the perl script (above) it took 30 seconds.

(the machine is a 10 year old laptop, Celeron M 340 1.5Ghz, 768 Mb of RAM, 40Gb HD, under 4 desktop LXDE GUI in Lubuntu)

The process was CPU bound :-)

BugBear
Post Reply