base64_decode() unreachable heap corruption on 32-bit systems

Hello,

this is a bug by Guido Vranken from our bug bounty program. After analysis, we found that there are no codepaths that allow the attacker to specify such a big input size to base64_decode() hence this bug should not be exploitable. More checking should be done, and there might be more instances of this rounding pattern around our codebase.

Here follows the bug report as received:

int
base64_decode(char *dest, size_t destlen, const char *src, size_t srclen)
{
...
...
  if (destlen < (srclen*3)/4)
    return -1;
  if (destlen > SIZE_T_CEILING)
    return -1;

The problem here is that the multiplication (by 3) occurs before the division (by 4).

For source strings larger than 0xFFFFFFFF / 3 == 0x55555555, an overflow will occur within this calculation. If the result of the overflow-affected calculation is smaller than what destlen is, then this check will be passed and memory will be corrupted.