Subject: RE: mod-ssl on win32 (mod_ssl bug 569, apache crash)

From: Jay Burgess <[email-masked]>
Date: Thu, 17 Jan 2002 09:30:14 -0600

I could have written your email word for word!  I've been trying to keep 
Apache/mod_ssl running on WIN32 since early December, and still haven't 
figured out what's wrong. Josef's patch seemed to work for him, but even 
after applying it, my SSL-enabled Apache only runs a few minutes under 
heavy load before crashing.

As far as getting a debug version built, I've had lots of trouble with that 
as well.  It seems to build just fine in debug mode in Visual C++, but when 
I attempt to debug the EXE after a crash, I get no source code to trace.

Is there anyone out there successfully running an SSL-enabled version of 
Apache on WIN32?  If so, did you have to do anything special?  And either 
way, does someone have a write-up explaining how to put together a debug build?

Thanks.

Jay

-----Original Message-----
From: Enrico Demarin [mailto:[email-masked]]
Sent: Thursday, January 17, 2002 12:19 PM
To: [email-masked]
Subject: mod-ssl on win32 (mod_ssl bug 569, apache crash)

I know it's not supported but... seems like something is still not good in 
mod_ssl , as reported by Josef Goebel,
who posted a patch, there is something odd in mod_ssl for win32. I applied 
his patch to ap_ctx.c , that cures
the crash in most of the cases
in ap_ctx_get() adds:
if ((ctx==NULL) || (key==NULL))
{
ap_log_assert("ap_ctx_get_bug: ctx or key are NULL!",
__FILE__, __LINE__);
return NULL;
}
and i also added the same check in ap_ctx_set. The problem is that 
sometimes mod_ssl calls such functions with ctx set to null and key set to 
null, and crashes apache. I have had other crashes which i haven't been 
able to debug ( how do i properly build a debug version of apache-ssl on 
win32 ? when the visual c debug window pops up after a crash all i can see 
is assembly code ). As reported by Josef the "null" calls seem to be 
generated by ssl_io_suck_read() in ssl_engine_io.c,
but why does it happen ? is there some memory corruption going inside 
mod_ssl that causes the structures to be set to null ? Apache with 
aforementioned patch is "almost" stable but... i have the sensation that 
there is a problem somewhere else.
- Enrico

______________________________________________________________________
Apache Interface to OpenSSL (mod_ssl)                   www.modssl.org
User Support Mailing List                      [email-masked]
Automated List Manager                            [email-masked]

---

From: Son <[email-masked]>
Date: Thu, 17 Jan 2002 10:02:58 -0800 (PST)

i've actually made some hacks to get it to work; i was able to build a
debug version and found that there were problems in the ssl log
code.  as a temporary fix, i was able to disable that portion of the code,
so it crashes much much less (i rarely see it anymore).  however, as a
drawback, i no longer have access to an ssl log.

-son

---

From: [email-masked]
Date: Thu, 17 Jan 2002 14:43:00 -0800

Don't have heavy load yet myself, but afraid I will have same problem on one machine.  Here is my thinking from experience with other code:

Heavy load and Win32 indicates the problem has to do with concurrency by threading.

Reasoning:  Easy load doesn't cause much concurrency.  And under Linux, as far as I am reading in documentation, most installations run with a separate process for each connection.  So what is unique to heavy load under Win32 is concurrency by threading instead of by processes.

Hope that reminder saves someone time in figuring this out.  Maybe I'll get to it later.  Also would be interested in hints about how to debug modules under Win32.

- Leo

At 09:30 AM 1/17/02 -0600, you wrote:
>I could have written your email word for word!  I've been trying to keep Apache/mod_ssl running on WIN32 since early December, and still haven't figured out what's wrong. Josef's patch seemed to work for him, but even after applying it, my SSL-enabled Apache only runs a few minutes under heavy load before crashing.
>
>As far as getting a debug version built, I've had lots of trouble with that as well.  It seems to build just fine in debug mode in Visual C++, but when I attempt to debug the EXE after a crash, I get no source code to trace.
>
>Is there anyone out there successfully running an SSL-enabled version of Apache on WIN32?  If so, did you have to do anything special?  And either way, does someone have a write-up explaining how to put together a debug build?
>
>Thanks.
>
>Jay
>
>I know it's not supported but... seems like something is still not good in mod_ssl , as reported by Josef Goebel,
>who posted a patch, there is something odd in mod_ssl for win32. I applied his patch to ap_ctx.c , that cures
>and i also added the same check in ap_ctx_set. The problem is that sometimes mod_ssl calls such functions with ctx set to null and key set to null, and crashes apache. I have had other crashes which i haven't been able to debug ( how do i properly build a debug version of apache-ssl on win32 ? when the visual c debug window pops up after a crash all i can see is assembly code ). As reported by Josef the "null" calls seem to be generated by ssl_io_suck_read() in ssl_engine_io.c,
>but why does it happen ? is there some memory corruption going inside mod_ssl that causes the structures to be set to null ? Apache with aforementioned patch is "almost" stable but... i have the sensation that there is a problem somewhere else.
>- Enrico
>

---

From: "Enrico Demarin" <[email-masked]>
Date: Thu, 17 Jan 2002 18:13:44 -0800

Well I became aware of the problem when I started refreshing an ssl page
very fast from localhost... So I guess the load doesn't have to be
-that- high to reproduce the problem. I agree there must be some problem
in the threading code. I tried apache 2.0 with mod_ssl but I couldn't
get it to work on win32. It would be interesting if the people involved
in the porting of Apache on Win32 would take the time to write a sort of
"developer howto", (or if it exists already, mail the pointers to it :)
explaining like you said how to debug apache modules and the apache code
itself ( apache modules on win32 are .so objects while one would expect
DLLs for example). 

In my case we will probably solve the problem by (temporarely?)
switching to IIS, but it would be nice to have an alternative to it, and
Apache would be the best alternative to it in my opinion.

Thanks,
Enrico

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of [email-masked]
Sent: January 17, 2002 2:43 PM
To: [email-masked]
Subject: RE: mod-ssl on win32 (mod_ssl bug 569, apache crash)

Don't have heavy load yet myself, but afraid I will have same problem on
one machine.  Here is my thinking from experience with other code:

Heavy load and Win32 indicates the problem has to do with concurrency by
threading.

Reasoning:  Easy load doesn't cause much concurrency.  And under Linux,
as far as I am reading in documentation, most installations run with a
separate process for each connection.  So what is unique to heavy load
under Win32 is concurrency by threading instead of by processes.

Hope that reminder saves someone time in figuring this out.  Maybe I'll
get to it later.  Also would be interested in hints about how to debug
modules under Win32.

- Leo

At 09:30 AM 1/17/02 -0600, you wrote:
>I could have written your email word for word!  I've been trying to 
>keep Apache/mod_ssl running on WIN32 since early December, and still 
>haven't figured out what's wrong. Josef's patch seemed to work for him,

>but even after applying it, my SSL-enabled Apache only runs a few 
>minutes under heavy load before crashing.
>
>As far as getting a debug version built, I've had lots of trouble with 
>that as well.  It seems to build just fine in debug mode in Visual C++,

>but when I attempt to debug the EXE after a crash, I get no source code

>to trace.
>
>Is there anyone out there successfully running an SSL-enabled version 
>of Apache on WIN32?  If so, did you have to do anything special?  And 
>either way, does someone have a write-up explaining how to put together

>a debug build?
>I know it's not supported but... seems like something is still not good

>in mod_ssl , as reported by Josef Goebel, who posted a patch, there is 
>something odd in mod_ssl for win32. I applied his patch to ap_ctx.c , 
>that cures the crash in most of the cases in ap_ctx_get() adds: if 
>((ctx==NULL) || (key==NULL)) {
sometimes mod_ssl calls such functions with ctx set to null and key set
to null, and crashes apache. I have had other crashes which i haven't
been able to debug ( how do i properly build a debug version of
apache-ssl on win32 ? when the visual c debug window pops up after a
crash all i can see is assembly code ). As reported by Josef the "null"
calls seem to be generated by ssl_io_suck_read() in ssl_engine_io.c,

---

From: "Patrick Willart" <[email-masked]>
Date: Thu, 17 Jan 2002 15:40:02 -0800

I used this very good HOWTO document by Balázs Bárány.
http://tud.at/programm/apache-ssl-win32-howto.php3

Besides that I also had set KeepAlive to Off to work around a multithreading
problem.

Another tip is to compile OpenSSL with ms\do_ms instead of ms\do_masm. The
masm compilation generated code that worked fine on my development machine
but not on my production server due to code optimization.

Hope this helps,

Patrick

---

From: Leo Baschy <[email-masked]>
Date: Thu, 17 Jan 2002 16:04:35 -0800

If you indeed refresh then that would be on the same connection and threading would not be involved if it is a page by itself in one file (i.e. without images, without frames).  Important detail:  Are you refreshing a single resource (file)?  If not, then please test that simplified case.  Knowing that helps pinpoint the problem.  Is there different behavior if you allow keeping alive connection or forcing not to keep alive?  (Do that from the configuration file.)  Do you known whether it keeps alive?  (There must be some way to log that, maybe you have to recompile specifically for that, maybe there is a configuration option, I'm not sure.)

- Leo

---

From: "Enrico Demarin" <[email-masked]>
Date: Fri, 18 Jan 2002 21:32:16 -0800

Hi all,

1) I recompiled Openssl ( the "ms" option ) just to be sure.
2) I recompiled Apache 1.3.22 + mod_ssl 2.8.5 from scratch, no patches,
release build
3) I set KeepAlive Off in the httpd.conf

Result:

Havy refreshing doesn't cause an halt.

4)  I re-enabled KeepAlive On

Heavy refreshing causes apache to crash.

- Enrico

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Leo Baschy
Sent: January 17, 2002 4:05 PM
To: [email-masked]
Subject: RE: mod-ssl on win32 (mod_ssl bug 569, apache crash)

If you indeed refresh then that would be on the same connection and
threading would not be involved if it is a page by itself in one file
(i.e. without images, without frames).  Important detail:  Are you
refreshing a single resource (file)?  If not, then please test that
simplified case.  Knowing that helps pinpoint the problem.  Is there
different behavior if you allow keeping alive connection or forcing not
to keep alive?  (Do that from the configuration file.)  Do you known
whether it keeps alive?  (There must be some way to log that, maybe you
have to recompile specifically for that, maybe there is a configuration
option, I'm not sure.)

- Leo

At 06:13 PM 1/17/02 -0800, you wrote:
>Well I became aware of the problem when I started refreshing an ssl 
>page very fast from localhost... So I guess the load doesn't have to be
>-that- high to reproduce the problem. I agree there must be some 
>problem in the threading code. I tried apache 2.0 with mod_ssl but I 
>couldn't get it to work on win32. It would be interesting if the people

>involved in the porting of Apache on Win32 would take the time to write

>a sort of "developer howto", (or if it exists already, mail the 
>pointers to it :) explaining like you said how to debug apache modules 
>and the apache code itself ( apache modules on win32 are .so objects 
>while one would expect DLLs for example).
>
>switching to IIS, but it would be nice to have an alternative to it, 
>and Apache would be the best alternative to it in my opinion.
>
>Thanks,
>Enrico
>
>Don't have heavy load yet myself, but afraid I will have same problem 
>on one machine.  Here is my thinking from experience with other code:
>

>aforementioned patch is "almost" stable but... i have the sensation 
>that there is a problem somewhere else.
> >- Enrico
> >
>

---

From: [email-masked]
Date: Sat, 19 Jan 2002 00:08:30 -0800

Seems to be the known problem then.  You have your workaround in your own answer.  KeepAlive Off.  Of course you'd only want that inside your

<VirtualHost _default_:443>

and there actually is a recommended

     SetEnvIf User-Agent ".*MSIE.*" \
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0

which I'd put there instead of the general KeepAlive off.

I don't know whether that still will be required with Apache 2 version of mod_ssl.

While not perfect, definitely beats using IIS.

- Leo

---

From: "Enrico Demarin" <[email-masked]>
Date: Sat, 19 Jan 2002 11:33:57 -0800

Yes with keepalive off all the crashes disappear. I did like you said to
leave Keepalive On and put KeepAlive Off only inside the SSL Virtualhost
section, And no crashes as well. I don't know about just putting the
User-Agent directive as I had crashes even with Mozilla. Ofcourse it's a
workaround but yes, it works.

- Enrico

-----Original Message-----
Sent: January 19, 2002 12:09 AM

Seems to be the known problem then.  You have your workaround in your
own answer.  KeepAlive Off.  Of course you'd only want that inside your

I don't know whether that still will be required with Apache 2 version
of mod_ssl.

- Leo