Suhosin: canary mismatch on efree() - heap overflow detected

October 12th, 2008 | by Stefan Esser |

Users of Suhosin-Patch will sooner or later see messages like “canary mismatch on efree() - heap overflow detected” in their error log. When this happens they are often confused and don’t understand what it means.

The first questions they often ask themself are:

  • Did they trigger a bug in Suhosin?

In reality this means that when PHP internally tried to free some allocated memory the memory manager security features of Suhosin detected that the memory to be freed is somehow corrupted. This corruption occured somewhen between the allocation of the memory area and the attempt to free it. Suhosin detects this by writing canary values infront and after allocated memory areas. These canary values are random numbers that are stored in one of Suhosin’s internal variables. The check consists of comparing the internally stored random number with the values found infront and after the memory area. Those values are outside of anything PHP is supposed to write to and therefore a failed canary comparision can only mean two things.

  1. some memory corruption destroyed/overwrote the internally stored random values
  2. some memory corruption destroyed/overwrote the canaries next to the allocated memory

This means everytime Suhosin gives out an error there was a memory corruption. There are no false positives as some of the PHP developers claim again and again. This means that everytime you see this error there has been a memory corruption in PHP or one of the loaded extensions. This means there is a bug and some code in one of the loaded components that behaves wrongly.

Unfortunately the PHP developers refuse to support PHP users that decided to go with the more secure version of PHP, because the patch could influence how PHP works and be the cause of the bug. They require their users to reproduce the problem with a vanilla PHP with valgrind running.

The obvious problems with this are

  1. Running PHP in valgrind has a much greater influence on how PHP works than Suhosin-Patch has
  2. Using valgrind wrongly means the memory corruption bugs cannot be found
  3. Using valgrind correctly does not necessary reveal the bug, because it might be hidden by memory alignment or just because valgrind cannot detect that class of memory corruptions
  4. A memory corruption that Suhosin might detect everytime might only crash a vanilla (valgrind) PHP once every thousand invocations or might not result in a crash at all but in wrong computation results
  5. Some memory corruptions occur in memory areas that are unused due to alignment, they will not result in misbehaviour of vanilla PHP - However any memory corruption is a bug in PHP (or extensions) that should be fixed and a little harmless memory corruption today could turn into a remote exploit in the future when the code is changed (or when PHP is executed on another (new) architecture).
  6. A normal user will not be able to recompile PHP and reproduce with valgrind correctly
  7. A not reproduced memory corruption still exists but will not be fixed

That said the only option for users of the more secure PHP is at the moment to try to get support for their problem at their distribution’s PHP maintainers because the people at PHP.net will not be interested in helping them. However due to the reasons above the distribution maintainers might not be able to help you, because they cannot reproduce the bug (although Suhosin proved its existance).

If you know what you are doing then just use valgrind on the Suhosin patched PHP and simply claim that you reproduced the bug without Suhosin installed. This will also give you a higher probability of reproducing the problem.

Because of this I will think up some debugging functionality for Suhosin that will make it easier for users to tell the PHP developers where their bug is located. More information about how this will be implemented will be released during the next month.

  1. 8 Responses to “Suhosin: canary mismatch on efree() - heap overflow detected”

  2. By Jory on Oct 12, 2008 | Reply

    I don’t know how Suhosin works, but like you said, the canaries not matching up can mean corruption in the memory used to store them internally, not the memory PHP is using.
    So these entries - at least a part of them - could also be signs of a bug in Suhosin, right? (And I would call that a false positive - Suhosin thinks it detected memory corruption in PHP, but its own memory is corrupted instead.)

  3. By Greg Beaver on Oct 12, 2008 | Reply

    the last paragraph (a way to detect what is causing the problem) would be a godsend for me. I have poured over ext/phar for memory corruption issues, and found a few early on, but would love to have an even more certain way to be sure my code isn’t the cause of a canary mismatch.

  4. By Stefan Esser on Oct 12, 2008 | Reply

    Suhosin is a patch to PHP and therefore shares the same memory as PHP. A corruption of one of suhosin’s internal variables is therefore also a corruption of PHP’s own memory.

    Most of Suhosin’s code consists of reading and comparing values, which obviously cannot result in a memory corruption. The places where Suhosin actually writes to memory are very few, but often executed places. In addition to that the logic where the writes occur is very small. This also means that a memory corruption problem caused by a bug in Suhosin-Patch would occur very very often and therefore cause problems on nearly all installations.

    This obviously does not happen…

  5. By Stas on Oct 13, 2008 | Reply

    I think the problem that we face with canary error reports is that while it, with high probability, indicates a corruption problem, unlike valgrind trace - which often has useful backtraces and also origins for allocation and deallocation, where appropriate - canary errors only indicate something is wrong, but not what is wrong. I.e. by that report alone it is very hard to find out what was wrong and where. It would indeed be very useful if there would be some functionality that would allow further investigation of the cause of the bug.

  6. By AntonioCS on Oct 13, 2008 | Reply

    Why is it called “canary”??

  7. By Stefan Esser on Oct 13, 2008 | Reply

    As far as I know the term canary is used for this kind of protections because of the analogy to the birds (canaries) used to detect deadly concentrations of gas in the mining industry in the past.

    When the birds die this means the air is deadly and it is time to get out of the mine fast…

    If the canary is overwritten the memory is corrupt and it is time to stop the process soon…

  8. By Vas on Nov 15, 2009 | Reply

    Put suhosin.session.encrypt=off to php.ini

  1. 1 Trackback(s)

  2. Jan 22, 2009: PHP function question - openSUSE Forums

Post a Comment