"-set" or "%[fx...]" taking very long time

[snibgo]

On Windows 7, IM 6.6.2-3, create a source image:

Code: Select all

convert -size 3200x3200 xc:red red.png

The following takes a reasonable 1 minute to create the 10,000 files:

Code: Select all

convert ^
  red.png ^
  -crop 32x32 ^
  +repage +adjoin ^
  x.png

But if we add one line, in preparation for a row_column output name:

Code: Select all

convert ^
  red.png ^
  -crop 32x32 ^
  -set filename:tile "%%[fx:page.x/32]_%%[fx:page.y/32]" ^
  +repage +adjoin ^
  x.png

Now the time increases to about 1 minute per output file, for a total time of 7 days. (One dual-core CPU is saturated, the other is idle. There is negligible memory use or disk activity.)

Am I doing something stupid, or is there a bug in "-set" or "%[fx...]"?

[fmw42]

On my Mac OSX Tiger G4 single processor and IM 6.6.2-7 Q16 beta


convert -size 128x128 xc:red red.png

time convert \
red.png \
-crop 32x32 \
-set filename:tile "%[fx:page.x/32]_%[fx:page.y/32]" \
+repage +adjoin \
x.png

creates 16 tiles in:

real 0m0.666s
user 0m0.217s
sys 0m0.145s

[snibgo]

Thanks. Can you repeat the test with "-size 3200x3200"?

[snibgo]

[Responding to a post, now vanished, from magick explaining that this isn't a bug but %[fx: ] is evaluated for every pixel.]

Thanks, that makes sense.

In a few places, the documentation says, "For use with -format, the value-escape %[fx: ] is evaluated just once for each image in the current image sequence." I suggest this is expanded: "When used with -set, value-escape %[fx: ] is evaluated once for every pixel."

The "Cutting and Bordering examples" page http://www.imagemagick.org/Usage/crop/#crop_tile could also be modified.

As a possible future enhancement, a facility for doing calulations once per image might be useful.

[magick]

Your expression is evaluated over 70k times. Since FX expressions are interpreted, it can be quite slow. We added fxc: in ImageMagick 6.6.2-7 Beta, available by sometime tomorrow. Here an fxc: expression is evaluated just once and cached. The next call returns the cached result rather than reevaluating the expression. Your command is quite fast when using fxc:page.x/32 rather than fx:page.x/32.

[snibgo]

Thanks for the speedy response. I can't currently compile. Someone else may test it.

Thanks again.

[fmw42]

Thanks for the speedy response. I can't currently compile. Someone else may test it.

Thanks again.

Alan,

I can test it once the beta is available. I am downloading whatever is the current version of the beta now. We will see if it has the change.

Fred

[fmw42]

OK. It seems to be in the new beta.


time convert \
red.png \
-crop 32x32 \
-set filename:tile "%[fxc:page.x/32]_%[fxc:page.y/32]" \
+repage +adjoin \
x.png

create 1024 image in:


real 0m9.612s
user 0m4.265s
sys 0m3.908s


Fred

[snibgo]

Thanks Fred. Good stuff. It's still slow, but we are now talking minutes rather than days.

[fmw42]

Thanks Fred. Good stuff. It's still slow, but we are now talking minutes rather than days.

My test was 9 sec not 9 minutes.

Fred

[snibgo]

Yes, 9.6 seconds for 1024 files, so 94 seconds (1.5 minutes) for the 10,000 files we would get from a 3200x3200 source. This may not be significantly slower than we were getting without the "-set filemame fx" option

[anthony]

In a few places, the documentation says, "For use with -format, the value-escape %[fx: ] is evaluated just once for each image in the current image sequence." I suggest this is expanded: "When used with -set, value-escape %[fx: ] is evaluated once for every pixel

The %[fx:] should be just once only for each image with i,j set to 0 and just the red channel used. That is how it was defined.

However their are some fx options that may require some identify settings, that could need to process all the pixels.
Specifically getting the min, max, and mean values of a image. However if it is implemented right it should only need to do that once only, and ONLY if requested.

A %[pixel:] on the other hand is supposed to do just one value with all color channels.

Only the -fx is suposed to process every pixel, over the specified -channel range (though there was talk of allowing it to use -region to limit how many pixels should be looked at.

[anthony]

Your expression is evaluated over 70k times. Since FX expressions are interpreted, it can be quite slow. We added fxc: in ImageMagick 6.6.2-7 Beta, available by sometime tomorrow. Here an fxc: expression is evaluated just once and cached. The next call returns the cached result rather than reevaluating the expression. Your command is quite fast when using fxc:page.x/32 rather than fx:page.x/32.

As I mentioned the original defintion of %[fx:] percent escape when it was first created was for just one fx evaluation per image.
Why is it not doing this?

Only -fx was suposed to iterate over all values.

As such I suggest the new %[fxc:] should really be the default %[fx:].

[magick]

We had code that stopped fx: from checking the property cache. We assume for a reason but we changed the algorithm to check the cache first as you suggested. Let us know if it broken anything.