The t/filter PerlMagick tests still fail in 6.3.6-2

[mi]

Code: Select all

t/filter..........FAILED tests 24, 31                                        
        Failed 2/58 tests, 96.55% okay

This was going to be fixed in 6.3.5-10...

[magick]

We expect some fuzziness with GaussianBlur() and Modulate tests. Try

• perl t/filter.t

and let us know what error it returns. If its above 1.2 or so there is definitely a problem. If its small like 0.05 its just expected floating point sensitivity.

[mi]

I think, last time we discussed it, you mentioned making the check less sensitive in 6.3.5-10.

Please, fix the test in the upcoming 6.3.6-0... Thanks!

[magick]

We patched ImageMagick 6.3.6-0 Beta to be less sensitive when regression testing a gaussian blur or modulate operation. ImageMagick 6.2.6 is scheduled for release by September 30th.

[mi]

Ok, the failing tests are thus:

Code: Select all

GaussianBlur('0.0x1.5') ...
  mean-error=0,000409859084298681, maximum-error=0,137254901960784
not ok 24
...
Modulate(brightness=>110,saturation=>110,hue=>110) ...
  mean-error=0,00454840589262133, maximum-error=0,298039215686275
not ok 31

Please, confirm that there is no problem.

Please, also publish the patches for both this test and for the memory leak, which neccessitated the 6.3.6-0 announcement. I would like to update FreeBSD's port of ImageMagick.

I'd rather not wait for 6.3.6-0, as it is likely to have some new problems Thanks!

[magick]

Here is the PerlMagick sensitivity patch:

Code: Select all

diff ~/ImageMagick-6.3.5/PerlMagick//t/filter.t PerlMagick/t/filter.t
103c103
< testFilterCompare('input.miff',  q//, 'reference/filter/GaussianBlur.miff', 'GaussianBlur', q/'0.0x1.5'/, 0.02, 0.02);
---
> testFilterCompare('input.miff',  q//, 'reference/filter/GaussianBlur.miff', 'GaussianBlur', q/'0.0x1.5'/, 0.02, 0.2);
124c124
< testFilterCompare('input.miff',  q//, 'reference/filter/Modulate.miff', 'Modulate', q/brightness=>110,saturation=>110,hue=>110/, 0.05, 0.05);
---
> testFilterCompare('input.miff',  q//, 'reference/filter/Modulate.miff', 'Modulate', q/brightness=>110,saturation=>110,hue=>110/, 0.02, 0.4);

and the memory leak patch:

Code: Select all

diff ~/ImageMagick-6.3.5/magick/string.c magick/string.c
221d220
<   *destination=(char *) NULL;

[mi]

Thanks. But was not setting the same limits for average and for maximum somewhat redundant to begin with? Maximum is never smaller than average...

[mi]

A freshly compiled 6.3.6-2:

Code: Select all

perl t/filter.t
...
ok 23
GaussianBlur('0.0x1.5') ...
  mean-error=0,000403084336889061, maximum-error=0,137254901960784
not ok 24
Implode(0.5) ...
ok 25
Level('20%') ...
ok 26
Magnify() ...
ok 27
Set(fuzz=>3276) ...
MatteFloodfill(geometry=>"+25+45") ...
ok 28
MedianFilter() ...
ok 29
Minify() ...
ok 30
Modulate(brightness=>110,saturation=>110,hue=>110) ...
  mean-error=0,00462525641117487, maximum-error=0,298039215686275
not ok 31
...

What happened -- again?

[magick]

You can safely increase the sensitivity for the two tests that failed so that they pass. For GaussianBlur() we have

• testFilterCompare('input.miff', q//, 'reference/filter/GaussianBlur.miff', 'GaussianBlur', q/'0.0x1.5'/, 0.004, 0.4);

As we mentioned in the past we expect some error for floating point operations. As long as it does not exceed 1.2 we do not get concerned.

[mi]

As long as it does not exceed 1.2 we do not get concerned.

Would it be too much to ask, that the real maximum safe values (1.2 in this particular one) be listed as the actual maximum safe values in the test scripts?

The way to look at the tests for all but the developers (you), is binary -- they either pass or fail.

There is no "almost pass". If the new build fails its own tests, it is a failure -- either of the build or of the tests. It is a relief to learn, that in this case the tests are buggy. Please, fix the bugs.

Thank you!

[magick]

We can only set tolerances based on our own testing. As mentioned we do not have access to a FreeBSD systems. On the seven systems we ran the regressions tests on, the tolerances we set permitted the tests to pass on each of these machines (Fedora Core, Redhat Enterprise, Mac OS X, Windows, Solaris 10, and a few others). We must rely on ImageMagick users to provide us with tolerance limits for machines we do not have access too.

[mi]

We can only set tolerances based on our own testing. As mentioned we do not have access to a FreeBSD systems. On the seven systems we ran the regressions tests on, the tolerances we set permitted the tests to pass on each of these machines (Fedora Core, Redhat Enterprise, Mac OS X, Windows, Solaris 10, and a few others). We must rely on ImageMagick users to provide us with tolerance limits for machines we do not have access too.

Well, you did suggest, that up to 1.2 (of something) is Ok. I would think, that number is the real maximum tolerance. And by "maximum" I mean: "Don't bother us, if your system's deviation is smaller than that."

Why, then, put a smaller number -- and be bothered -- into the scripts?

A test failure means, something went wrong. Yet this entire thread is about you convincing me, nothing went wrong. Why, then, are the tests failing?

[magick]

We are satisfied with our approach to regression testing. We would rather keep the tolerances tight and loosen them if we get any reports of failure from our ImageMagick users.

[Hero]

-------------------------------------------------------------------------------
t/read.t 48 1 2.08% 42
t/write.t 33 1 3.03% 29
Failed 2/23 test scripts, 91.30% okay. 2/334 subtests failed, 99.40% okay.
*** Error code 2

Stop in /usr/ports/graphics/ImageMagick/work/ImageMagick-6.3.5/PerlMagick.
*** Error code 1

I'm on freebsd 6.2 also (new box),
are these errors the ones you are discussing above?

if so, how can I stop all t error testing?

thanks


Edit:
it seems like another problem, that it cannot read XWD files,
see this thread: viewtopic.php?f=1&t=9973&p=31133