Monday, December 6, 2010

Apache Segmentation Fault because of OCI / Zend in a non-standard environment

We are running Apache in a jailed environment in the context of a user that has no home directory, so the OCI8 client that comes with Zend segmentation faults when it tries to write diagnostic files.

By using GDB I made an educated guess that OCI8 is trying to figure out the home directory and probably crashes because there is no corresponding home directory in my /etc/passwd file.

GDB output:

Core was generated by `/usr/sbin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b6a1a5e51be in skgpgetinfo () from /usr/lib64/libclntsh.so.11.1

#0  0x00002b6a1a5e51be in skgpgetinfo () from /usr/lib64/libclntsh.so.11.1
#1  0x00002b6a1a74d89b in dbgroud_oradiag_user_dir () from /usr/lib64/libclntsh.so.11.1
#2  0x00002b6a1a74d067 in dbgrgad_get_adrbase_directory () from /usr/lib64/libclntsh.so.11.1
#3  0x00002b6a1a76b81d in kpeDbgGetInitFileParms () from /usr/lib64/libclntsh.so.11.1
#4  0x00002b6a1a76b5fe in kpeDbgGetInitFileParmsAndInitDBGC () from /usr/lib64/libclntsh.so.11.1
#5  0x00002b6a1a187c6d in nlstddd_do_alter_diag () from /usr/lib64/libclntsh.so.11.1
#6  0x00002b6a1a18594d in nlstddt_do_alter_trace () from /usr/lib64/libclntsh.so.11.1
#7  0x00002b6a1bbf699e in nlstdggo () from /usr/lib64/libclntsh.so.11.1
#8  0x00002b6a1bbf66af in nlstdgg () from /usr/lib64/libclntsh.so.11.1
#9  0x00002b6a1a23b5fb in nigini2 () from /usr/lib64/libclntsh.so.11.1
#10 0x00002b6a1a76be3c in kpeDbgGetNPDGlobal () from /usr/lib64/libclntsh.so.11.1
#11 0x00002b6a1a76bd10 in kpeDbgTLSInit () from /usr/lib64/libclntsh.so.11.1
#12 0x00002b6a1bd1c331 in kpummTLSGET1 () from /usr/lib64/libclntsh.so.11.1
#13 0x00002b6a1a76bfd3 in kpeDbgProcessInit () from /usr/lib64/libclntsh.so.11.1
#14 0x00002b6a1a76a457 in kpummpin () from /usr/lib64/libclntsh.so.11.1
#15 0x00002b6a1a0dcd13 in kpuenvcr () from /usr/lib64/libclntsh.so.11.1
#16 0x00002b6a1acfbdc1 in OCIEnvNlsCreate () from /usr/lib64/libclntsh.so.11.1
#17 0x00002b6a19a12a3b in php_oci_do_connect_ex () from /usr/local/zend/lib/php_extensions/oci8.so
#18 0x00002b6a19a12fcf in php_oci_do_connect () from /usr/local/zend/lib/php_extensions/oci8.so
#19 0x00002b6a19a23953 in zif_oci_connect () from /usr/local/zend/lib/php_extensions/oci8.so
#20 0x00002b6a131cdd69 in ?? () from /usr/local/zend/lib/apache2/libphp5.so
#21 0x00002b6a131a458e in execute () from /usr/local/zend/lib/apache2/libphp5.so
#22 0x00002b6a13180e99 in zend_execute_scripts () from /usr/local/zend/lib/apache2/libphp5.so
#23 0x00002b6a1312f7f8 in php_execute_script () from /usr/local/zend/lib/apache2/libphp5.so
#24 0x00002b6a1320887d in ?? () from /usr/local/zend/lib/apache2/libphp5.so
#25 0x00002b6a084cda4a in ap_run_handler ()
#26 0x00002b6a084d0ed8 in ap_invoke_handler ()
#27 0x00002b6a084db938 in ap_process_request ()
#28 0x00002b6a084d8b70 in ?? ()
#29 0x00002b6a084d4cd2 in ap_run_process_connection ()
#30 0x00002b6a084df789 in ?? ()
#31 0x00002b6a084dfa1a in ?? ()
#32 0x00002b6a084dfad0 in ?? ()
#33 0x00002b6a084e07bb in ap_mpm_run ()
#34 0x00002b6a084bae48 in main ()

This was fixed by disabling diagnostics in OCI by setting $ORACLE_HOME env to /tmp and then put "DIAG_ADR_ENABLED=OFF" into $ORACLE_HOME/network/admin/sqlnet.ora file.

2 comments:

  1. Know what... that's almost the only source of information that I could get about this crash, while searching by "kpeDbgGetInitFileParmsAndInitDBGC" that I see in valgrind. I don't know how you got this piece of information, but thanks!

    ReplyDelete
  2. :D Just stumbled upon your comment when looking over this old blog. Thanks for letting me know, this was a pain to track down!

    ReplyDelete