Subject: Problem with 2.8.13 and Solaris 2.6

From: Jazz <[email-masked]>
Date: Thu, 20 Mar 2003 14:42:28 -0300 (ART)

Hy,

    I compiled the latest mod_ssl 2.8.13+OpenSSL
0.9.6i with apache 1.3.27 and I had some problems...
the http apache works fine, but the https server dies
when a request is received. error_log reports this:

[notice] child pid 19396 exit signal Bus Error (10)

The main process still running, but the child
processes doesn't seem to stay up. Anyone had a
problem like this? This problem is just with Solaris
2.6/Sparc or is happening in other platforms?

Thank you all for your attention.

Jazz

_______________________________________________________________________
Busca Yahoo!
O serviço de busca mais completo da Internet. O que você pensar o Yahoo! encontra.
http://br.busca.yahoo.com/
______________________________________________________________________
Apache Interface to OpenSSL (mod_ssl)                   www.modssl.org
User Support Mailing List                      [email-masked]
Automated List Manager                            [email-masked]

---

From: Artur Pydo <[email-masked]>
Date: Thu, 20 Mar 2003 20:29:06 +0100

Hi,

I can see the same segmentation fault :

FreeBSD 4.8-STABLE
Apache 1.3.27
Openssl 0.9.7a
Modssl 2.8.13
PHP 4.3.1 / PHP 4.3.2RC1 / PHP 4.3.2-snapshot

It happens both with static compilation and as DSO.

The backtrace seems pointing out an error in
ssl_var_lookup_ssl_cert().

This problem only appears with PHP compiled in and
asking for a .php document. I mean asking for a html
document works fine.

Backtrace (sorry for the formatting) :

Program received signal SIGSEGV, Segmentation fault.
0x283a6e9a in ssl_var_lookup_ssl_cert () from 
/usr/local/apache/libexec/libssl.so
(gdb) bt
#0  0x283a6e9a in ssl_var_lookup_ssl_cert () from 
/usr/local/apache/libexec/libssl.so
#1  0x283a6d49 in ssl_var_lookup_ssl () from 
/usr/local/apache/libexec/libssl.so
#2  0x283a6291 in ssl_var_lookup () from /usr/local/apache/libexec/libssl.so
#3  0x283a11c8 in ssl_hook_Fixup () from /usr/local/apache/libexec/libssl.so
#4  0x805472b in run_method (r=0x815d034, offset=29, run_all=1) at 
http_config.c:370
#5  0x805480a in ap_run_fixups (r=0x815d034) at http_config.c:397
#6  0x806a7cc in process_request_internal (r=0x815d034) at 
http_request.c:1303
#7  0x806a866 in ap_process_request (r=0x815d034) at http_request.c:1324
#8  0x80610eb in child_main (child_num_arg=0) at http_main.c:4689
#9  0x80612cd in make_child (s=0x80b0034, slot=0, now=1048177481) at 
http_main.c:4813
#10 0x8061446 in startup_children (number_to_start=5) at http_main.c:4895
#11 0x8061a74 in standalone_main (argc=5, argv=0xbfbffb04) at 
http_main.c:5203
#12 0x80622f0 in main (argc=5, argv=0xbfbffb04) at http_main.c:5566
#13 0x804f4b1 in _start ()

-- 

Best regards,

Artur Pydo.

---

From: "Ralf S. Engelschall" <[email-masked]>
Date: Fri, 21 Mar 2003 12:30:36 +0100

> #13 0x804f4b1 in _start ()

Hmmm... I've in-depth looked at the changes to ssl_engine_vars.c
and they all look correct:

Index: ssl_engine_vars.c
===================================================================
RCS file: /e/modssl/cvs/mod_ssl/pkg.apache/src/modules/ssl/ssl_engine_vars.c,v
retrieving revision 1.51
retrieving revision 1.53
diff -u -d -u -3 -r1.51 -r1.53
--- ssl_engine_vars.c	29 Jun 2002 07:42:51 -0000	1.51
+++ ssl_engine_vars.c	29 Oct 2002 13:00:46 -0000	1.53
@@ -314,12 +314,16 @@
         result = ssl_var_lookup_ssl_cert_verify(p, c);
     }
     else if (ssl != NULL && strlen(var) > 7 && strcEQn(var, "CLIENT_", 7)) {
-        if ((xs = SSL_get_peer_certificate(ssl)) != NULL)
+        if ((xs = SSL_get_peer_certificate(ssl)) != NULL) {
             result = ssl_var_lookup_ssl_cert(p, xs, var+7);
+            X509_free(xs);
+        }
     }
     else if (ssl != NULL && strlen(var) > 7 && strcEQn(var, "SERVER_", 7)) {
-        if ((xs = SSL_get_certificate(ssl)) != NULL)
+        if ((xs = SSL_get_certificate(ssl)) != NULL) {
             result = ssl_var_lookup_ssl_cert(p, xs, var+7);
+            X509_free(xs);
+        }
     }
     return result;
 }
@@ -352,7 +356,7 @@
         xsname = X509_get_subject_name(xs);
         cp = X509_NAME_oneline(xsname, NULL, 0);
         result = ap_pstrdup(p, cp);
-        free(cp);
+        OPENSSL_free(cp);
         resdup = FALSE;
     }
     else if (strlen(var) > 5 && strcEQn(var, "S_DN_", 5)) {
@@ -364,7 +368,7 @@
         xsname = X509_get_issuer_name(xs);
         cp = X509_NAME_oneline(xsname, NULL, 0);
         result = ap_pstrdup(p, cp);
-        free(cp);
+        OPENSSL_free(cp);
         resdup = FALSE;
     }
     else if (strlen(var) > 5 && strcEQn(var, "I_DN_", 5)) {
@@ -543,6 +547,10 @@
     else
         /* client verification failed */
         result = ap_psprintf(p, "FAILED:%s", verr);
+
+    if (xs != NULL)
+        X509_free(xs);
+
     return result;
 }

Additionally, I still cannot reproduce the problem myself. So, can you
help me here by using a breakpoint at ssl_var_lookup_ssl_cert() and the
single-stepping until the problem occurs? This would help us in really
locating the problem.
                                       Ralf S. Engelschall
                                       [email-masked]
                                       www.engelschall.com

---

From: Joe Orton <[email-masked]>
Date: Fri, 21 Mar 2003 11:37:10 +0000

>              result = ssl_var_lookup_ssl_cert(p, xs, var+7);

That isn't safe, SSL_get_certificate doesn't increase the refcount on
the certificate (unlike SSL_peer_get_certificate).

Regards,

joe

---

From: Artur Pydo <[email-masked]>
Date: Fri, 21 Mar 2003 13:33:02 +0100

Hi,

> single-stepping until the problem occurs? This would help us in really

I recompiled static Apache binary with -g3.
First backtrace :

(gdb) run -X -f /usr/local/apache/conf/httpd.conf.static -DSSL
Starting program: /usr/local/src/build/test/apache_1.3.27/src/./httpd -X 
-f /usr/local/apache/conf/httpd.conf.static -DSSL

Program received signal SIGSEGV, Segmentation fault.
0x80a0b76 in ssl_var_lookup_ssl_cert (p=0x82a500c, xs=0x833d280, 
var=0x8214035 "V_END") at ssl_engine_vars.c:353
353             result = ssl_var_lookup_ssl_cert_valid(p, 
X509_get_notAfter(xs));
(gdb) bt
#0  0x80a0b76 in ssl_var_lookup_ssl_cert (p=0x82a500c, xs=0x833d280, 
var=0x8214035 "V_END") at ssl_engine_vars.c:353
#1  0x80a0a4d in ssl_var_lookup_ssl (p=0x82a500c, c=0x8352014, 
var=0x821402e "SERVER_V_END") at ssl_engine_vars.c:324
#2  0x80a0049 in ssl_var_lookup (p=0x82a500c, s=0x82e567c, c=0x8352014, 
r=0x82a5034, var=0x821402a "SSL_SERVER_V_END")
     at ssl_engine_vars.c:191
#3  0x809b74b in ssl_hook_Fixup (r=0x82a5034) at ssl_engine_kernel.c:1336
#4  0x8162d3f in run_method (r=0x82a5034, offset=19, run_all=1) at 
http_config.c:370
#5  0x8162e1e in ap_run_fixups (r=0x82a5034) at http_config.c:397
#6  0x8177e7e in ap_sub_req_method_uri (method=0x824fa8a "GET", 
new_file=0x82ee754 "index.php", r=0x833e034) at http_request.c:855
#7  0x8177ebf in ap_sub_req_lookup_uri (new_file=0x82ee754 "index.php", 
r=0x833e034) at http_request.c:880
#8  0x808e3bc in handle_dir (r=0x833e034) at mod_dir.c:163
#9  0x81631f1 in ap_invoke_handler (r=0x833e034) at http_config.c:518
#10 0x8178e10 in process_request_internal (r=0x833e034) at 
http_request.c:1308
#11 0x8178e7a in ap_process_request (r=0x833e034) at http_request.c:1324
#12 0x816f6ff in child_main (child_num_arg=0) at http_main.c:4689
#13 0x816f8e1 in make_child (s=0x829f034, slot=0, now=1048249519) at 
http_main.c:4813
#14 0x816fa5a in startup_children (number_to_start=5) at http_main.c:4895
#15 0x8170088 in standalone_main (argc=5, argv=0xbfbffaf4) at 
http_main.c:5203
#16 0x8170904 in main (argc=5, argv=0xbfbffaf4) at http_main.c:5566
#17 0x807d109 in _start ()

I'm going on to see if i can bring you more specific trace.
Your suggestions are welcome i am backtracing for the first time. :)

-- 

Best regards,

Artur Pydo.

---

From: "Ralf S. Engelschall" <[email-masked]>
Date: Fri, 21 Mar 2003 13:47:24 +0100

On Fri, Mar 21, 2003, Ralf S. Engelschall wrote:

> [...]

Ok, can the people who are able to reproduce the segfault problem,
please apply the following patch, retry it and give feedback? I think
these two bugfixes should fix the problem now. If yes, I'll release
mod_ssl 2.8.14 with it. Thanks for your help.

Index: ssl_engine_kernel.c
===================================================================
RCS file: /e/modssl/cvs/mod_ssl/pkg.apache/src/modules/ssl/ssl_engine_kernel.c,v
retrieving revision 1.136
diff -u -d -r1.136 ssl_engine_kernel.c
--- ssl_engine_kernel.c	19 Nov 2002 13:57:01 -0000	1.136
+++ ssl_engine_kernel.c	21 Mar 2003 12:39:47 -0000
@@ -1048,13 +1048,15 @@
                         "Re-negotiation handshake failed: Client verification failed");
                 return FORBIDDEN;
             }
+            cert = SSL_get_peer_certificate(ssl);
             if (   dc->nVerifyClient == SSL_CVERIFY_REQUIRE
-                && (cert = SSL_get_peer_certificate(ssl)) == NULL) {
+                && cert == NULL) {
                 ssl_log(r->server, SSL_LOG_ERROR,
                         "Re-negotiation handshake failed: Client certificate missing");
-                X509_free(cert);
                 return FORBIDDEN;
             }
+            if (cert != NULL)
+                X509_free(cert);
         }
     }

Index: ssl_engine_vars.c
===================================================================
RCS file: /e/modssl/cvs/mod_ssl/pkg.apache/src/modules/ssl/ssl_engine_vars.c,v
retrieving revision 1.53
diff -u -d -r1.53 ssl_engine_vars.c
--- ssl_engine_vars.c	29 Oct 2002 13:00:46 -0000	1.53
+++ ssl_engine_vars.c	21 Mar 2003 12:40:12 -0000
@@ -322,7 +322,9 @@
     else if (ssl != NULL && strlen(var) > 7 && strcEQn(var, "SERVER_", 7)) {
         if ((xs = SSL_get_certificate(ssl)) != NULL) {
             result = ssl_var_lookup_ssl_cert(p, xs, var+7);
-            X509_free(xs);
+            /* SSL_get_certificate() as of OpenSSL 0.9.7a does not increment
+               the reference count the same way SSL_get_peer_certificate does,
+               so no need to X509_free(xs) the stuff here. */
         }
     }
     return result;

                                       Ralf S. Engelschall
                                       [email-masked]
                                       www.engelschall.com

---

From: "Ralf S. Engelschall" <[email-masked]>
Date: Fri, 21 Mar 2003 13:34:49 +0100

> the certificate (unlike SSL_peer_get_certificate).

Ops, great catch! Yes, you're right, I was not aware of this subtle
difference. Will be fixed.
                                       Ralf S. Engelschall
                                       [email-masked]
                                       www.engelschall.com

---

From: Burkhard Ulrich <[email-masked]>
Date: Fri, 21 Mar 2003 14:30:14 +0100 (CET)

Hi,

this works on linux 2.2.16 and linux 2.4.19

Thanks

Burkhard

---

From: Ed Kubaitis <[email-masked]>
Date: Fri, 21 Mar 2003 07:34:21 -0600

> mod_ssl 2.8.14 with it. Thanks for your help.

The patch fixed the problem for me (no php, RH 7.3.)

--
Ed Kubaitis - [email-masked]
CITES/STS - University of Illinois at Urbana-Champaign

> RCS file:
> /e/modssl/cvs/mod_ssl/pkg.apache/src/modules/ssl/ssl_engine_kernel.c,v
>                          "Re-negotiation handshake failed: Client
> verification failed");                  return FORBIDDEN;
>              }
>                          "Re-negotiation handshake failed: Client
> certificate missing"); -                X509_free(cert);
>                  return FORBIDDEN;
>              }
> +            if (cert != NULL)
> +                X509_free(cert);
>          }
>      }
>
> Index: ssl_engine_vars.c
> RCS file:
> /e/modssl/cvs/mod_ssl/pkg.apache/src/modules/ssl/ssl_engine_vars.c,v
> retrieving revision 1.53
>      else if (ssl != NULL && strlen(var) > 7 && strcEQn(var, "SERVER_",
> 7)) {          if ((xs = SSL_get_certificate(ssl)) != NULL) {
> +            /* SSL_get_certificate() as of OpenSSL 0.9.7a does not
> increment +               the reference count the same way
> SSL_get_peer_certificate does, +               so no need to
> X509_free(xs) the stuff here. */
>          }
>      }
>      return result;
>
>                                        Ralf S. Engelschall
>                                        [email-masked]
>                                        www.engelschall.com
>

---

From: Artur Pydo <[email-masked]>
Date: Fri, 21 Mar 2003 15:17:28 +0100

Hi,

> mod_ssl 2.8.14 with it. Thanks for your help.

That's ok with static and DSO apache build on :

FreeBSD 4.8-STABLE
Apache 1.3.27
Openssl 0.9.7a
Modssl 2.8.13 + provided patch
PHP 4.3.1 and PHP 4.3.2RC1

Thanks !

-- 

Best regards,

Artur Pydo.

---

From: Burkhard Ulrich <[email-masked]>
Date: Fri, 21 Mar 2003 15:18:30 +0100 (CET)

Hi again,

I also tested it sucessfully with linux 2.0.35, linux 2.2.19 and with 
linux 2.2.20

Greetings

Burkhard

> > mod_ssl 2.8.14 with it. Thanks for your help.

...

---

From: Jazz <[email-masked]>
Date: Fri, 21 Mar 2003 12:15:48 -0300 (ART)

Hi All,

    It is OK with:

    Solaris 2.6/Sparc
    Apache 1.3.27 (DSO)
    Php 4.2.3
    OpenSSL 0.9.6i
    Mod_SSL 2.8.14

    Nice weekend for everybody!

    JAZZ

---

From: "Erki Kriks" <[email-masked]>
Date: Mon, 24 Mar 2003 11:10:54 +0200

Hi!

I'm using Apache 3.2.1, tomcat and mod_ssl 2.8.11.
When i'm using HTTPS with GET method then everything OK.
But when i try HTTPS with POST method then i get error:
"mod_ssl: SSL Re-negotiation in conjunction with POST method not supported!"
Can anybody explain what's missing or i suppose something is missing in
tomcat web.xml?

Thanx,
Erki

---

From: Dimitri Rebrikov <[email-masked]>
Date: Tue, 25 Mar 2003 08:10:00 +0100

> Can anybody explain what's missing or i suppose something is missing in

Hi,
you probably compiled your mod_ssl with SSL_CONSERVATIVE turned on.
Here is a code-snip from ssl_engine_io.c with description about
issues with re-negotiation during the POST-Request.

Regards

------------------------------snip---------------------------------------

/*  _________________________________________________________________
**
**  I/O Request Body Sucking and Re-Injection
**  _________________________________________________________________
*/

#ifndef SSL_CONSERVATIVE

/*
  * Background:
  *
  * 1. When the client sends a HTTP/HTTPS request, Apache's core code
  * reads only the request line ("METHOD /path HTTP/x.y") and the
  * attached MIME headers ("Foo: bar") up to the terminating line ("CR
  * LF"). An attached request body (for instance the data of a POST
  * method) is _NOT_ read. Instead it is read by mod_cgi's content
  * handler and directly passed to the CGI script.
  *
  * 2. mod_ssl supports per-directory re-configuration of SSL parameters.
  * This is implemented by performing an SSL renegotiation of the
  * re-configured parameters after the request is read, but before the
  * response is sent. In more detail: the renegotiation happens after the
  * request line and MIME headers were read, but _before_ the attached
  * request body is read. The reason simply is that in the HTTP protocol
  * usually there is no acknowledgment step between the headers and the
  * body (there is the 100-continue feature and the chunking facility
  * only), so Apache has no API hook for this step.
  *
  * 3. the problem now occurs when the client sends a POST request for
  * URL /foo via HTTPS the server and the server has SSL parameters
  * re-configured on a per-URL basis for /foo. Then mod_ssl has to
  * perform an SSL renegotiation after the request was read and before
  * the response is sent. But the problem is the pending POST body data
  * in the receive buffer of SSL (which Apache still has not read - it's
  * pending until mod_cgi sucks it in). When mod_ssl now tries to perform
  * the renegotiation the pending data leads to an I/O error.
  *
  * Solution Idea:
  *
  * There are only two solutions: Either to simply state that POST
  * requests to URLs with SSL re-configurations are not allowed, or to
  * renegotiate really after the _complete_ request (i.e. including
  * the POST body) was read. Obviously the latter would be preferred,
  * but it cannot be done easily inside Apache, because as already
  * mentioned, there is no API step between the body reading and the body
  * processing. And even when we mod_ssl would hook directly into the
  * loop of mod_cgi, we wouldn't solve the problem for other handlers, of
  * course. So the only general solution is to suck in the pending data
  * of the request body from the OpenSSL BIO into the Apache BUFF. Then
  * the renegotiation can be done and after this step Apache can proceed
  * processing the request as before.
  *
  * Solution Implementation:
  *
  * We cannot simply suck in the data via an SSL_read-based loop because of
  * HTTP chunking. Instead we _have_ to use the Apache API for this step which
  * is aware of HTTP chunking. So the trick is to suck in the pending request
  * data via the Apache API (which uses Apache's BUFF code and in the
  * background mod_ssl's I/O glue code) and re-inject it later into the Apache
  * BUFF code again. This way the data flows twice through the Apache BUFF, of
  * course. But this way the solution doesn't depend on any Apache specifics
  * and is fully transparent to Apache modules.
  */