Subject: There appears to be a major memory leak in mod_ssl/OpenSSL

From: "Avery, Ken" <[email-masked]>
Date: Fri, 30 Jan 2004 11:02:06 -0600

I have been tracking this down for a couple of weeks and thought it was
in the code my company is developing and it appears that is not the
case. In order to eliminate our code from the mix and isolate the
problem here is what I did:
This was done on Windows and Linux: 
1. Download the latest Apache from www.apache.org. 
2. Download the latest OpenSSL from www.openssl.org. 
3. Build them both, with apache add the mod_ssl option and also for
Linux use the MPM worker module. 
4. Install and modify the ssl.conf file ServerName value. 
5. Run Apache (httpd) 
6a. Run the Performance monitor on Windows and look at Private Bytes for
the second Apache process. 
6b. On Linux run top -p pid(httpd1) -p pid(http2) ..... -p pid(httpN)
watching the size of the processes 
7. Set you browser to not cache requests and check for a new page every
time. 
8. Start fetching a page from https://localhost and keep refreshing the
page. 
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all
verified that it leaks like a sieve.
We were all trying to figure out why no one else has complained about
such a huge leak so we ran another test. We tried using the prefork MPM
and it turns out that worked fine. Based on the results it appears the
OS is cleaning up memory for the prefork module and the threaded model
never gets its memory freed. I have used a debugger on Windows and set
break points on the CRYPTO_malloc and CRYPTO_free functions and have
seen gobs of memory CRYPTO_malloc(ed) and not one time have I seen
CRYPTO_free called. I was not sure if having the OS cleanup memory was
part of the design (if indeed that is what is happening) or if there is
potentially a problem in the OpenSSL memory management code.
With all this said, I am by no means an expert on this code and could
really use some help understanding what is going on here?
Any and all help is appreciated, 
Ken

---

From: Joe Orton <[email-masked]>
Date: Fri, 30 Jan 2004 17:13:00 +0000

> Linux use the MPM worker module. 

Are you using 2.0.48?  Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
______________________________________________________________________
Apache Interface to OpenSSL (mod_ssl)                   www.modssl.org
User Support Mailing List                      [email-masked]
Automated List Manager                            [email-masked]

---

From: "Adrien Felon" <[email-masked]>
Date: Fri, 30 Jan 2004 18:23:15 +0100

There appears to be a major memory leak in mod_ssl/OpenSSLHello,

I encountered many memory leak trouble with OpenSSL. I used quite old
versions (from 0.9.6c), so I don't know if this is relevant or not for
you... Anyway I figured out that nobody seemed to ever call the
CRYPTO_thread_cleanup(). I just added a call to that function, and the
memory heap became clean. I hope the solution will be that simple in your
case.

The OpenSSL code did not look robust to me on that matter... But I think
that OpenSSL people are trying to (or did) improve the cleanup code.

Adrien

----- Original Message ----- 
From: Avery, Ken
To: [email-masked]
Sent: Friday, January 30, 2004 6:02 PM
Subject: There appears to be a major memory leak in mod_ssl/OpenSSL

I have been tracking this down for a couple of weeks and thought it was in
the code my company is developing and it appears that is not the case. In
order to eliminate our code from the mix and isolate the problem here is
what I did:
This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for Linux
use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for the
second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN) watching
the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all verified
that it leaks like a sieve.
We were all trying to figure out why no one else has complained about such a
huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.
With all this said, I am by no means an expert on this code and could really
use some help understanding what is going on here?
Any and all help is appreciated,
Ken

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.574 / Virus Database: 364 - Release Date: 29/01/2004

---

From: "John Hughes" <[email-masked]>
Date: Fri, 30 Jan 2004 17:35:04 -0000

There appears to be a major memory leak in mod_ssl/OpenSSLAre u using:

> On linux you really should be using a shared memory session cache - like
> SSLSessionCache shmcb:logs/ssl_gcache_data(512000)

> SSLSessionCacheTimeout 300

and not the dbm cache

I posted some email about this just before XMAS where I had found a "memory
leak" - and Mads Toftum suggested the use of shmcb.  I then ran tests for
nearly a week - without a hint of a memory leak

John
  -----Original Mess
   age-----
  From: [email-masked]
[mailto:[email-masked]]On Behalf Of Avery, Ken
  Sent: 30 January 2004 17:02
  To: [email-masked]
  Subject: There appears to be a major memory leak in mod_ssl/OpenSSL

  6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN)

  We were all trying to figure out why no one else has complained about such
a huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.