Chapter 18 Extending MySQL

This section describes the different functions that you need to define when you create a simple UDF. Section 18.2.2, “Adding a New User-Defined Function”, describes the order in which MySQL calls these functions.

The main xxx() function should be declared as shown in this section. Note that the return type and parameters differ, depending on whether you declare the SQL function XXX() to return STRING, INTEGER, or REAL in the CREATE FUNCTION statement:

For STRING functions:

char *xxx(UDF_INIT *initid, UDF_ARGS *args,
          char *result, unsigned long *length,
          char *is_null, char *error);

For INTEGER functions:

long long xxx(UDF_INIT *initid, UDF_ARGS *args,
              char *is_null, char *error);

For REAL functions:

double xxx(UDF_INIT *initid, UDF_ARGS *args,
              char *is_null, char *error);

The initialization and deinitialization functions are declared like this:

my_bool xxx_init(UDF_INIT *initid, UDF_ARGS *args, char *message);

void xxx_deinit(UDF_INIT *initid);

The initid parameter is passed to all three functions. It points to a UDF_INIT structure that is used to communicate information between functions. The UDF_INIT structure members follow. The initialization function should fill in any members that it wishes to change. (To use the default for a member, leave it unchanged.)

This section describes the different functions that you need to define when you create an aggregate UDF. Section 18.2.2, “Adding a New User-Defined Function”, describes the order in which MySQL calls these functions.

The xxx() function for an aggregate UDF should be declared the same way as for a nonaggregate UDF. See Section 18.2.2.1, “UDF Calling Sequences for Simple Functions”.

For an aggregate UDF, MySQL calls the xxx() function after all rows in the group have been processed. You should normally never access its UDF_ARGS argument here but instead return a value based on your internal summary variables.

Return value handling in xxx() should be done the same way as for a nonaggregate UDF. See Section 18.2.2.4, “UDF Return Values and Error Handling”.

The xxx_reset() and xxx_add() functions handle their UDF_ARGS argument the same way as functions for nonaggregate UDFs. See Section 18.2.2.3, “UDF Argument Processing”.

The pointer arguments to is_null and error are the same for all calls to xxx_reset(), xxx_clear(), xxx_add() and xxx(). You can use this to remember that you got an error or whether the xxx() function should return NULL. You should not store a string into *error! error points to a single-byte variable, not to a string buffer.

*is_null is reset for each group (before calling xxx_clear()). *error is never reset.

If *is_null or *error are set when xxx() returns, MySQL returns NULL as the result for the group function.

The args parameter points to a UDF_ARGS structure that has the members listed here:

The initialization function should return 0 if no error occurred and 1 otherwise. If an error occurs, xxx_init() should store a null-terminated error message in the message parameter. The message is returned to the client. The message buffer is MYSQL_ERRMSG_SIZE characters long, but you should try to keep the message to less than 80 characters so that it fits the width of a standard terminal screen.

The return value of the main function xxx() is the function value, for long long and double functions. A string function should return a pointer to the result and set *length to the length (in bytes) of the return value. For example:

memcpy(result, "result string", 13);
*length = 13;

MySQL passes a buffer to the xxx() function using the result parameter. This buffer is sufficiently long to hold 255 characters, which as of MySQL 4.1 can be multi-byte characters. The xxx() function can store the result in this buffer if it fits, in which case the return value should be a pointer to the buffer. If the function stores the result in a different buffer, it should return a pointer to that buffer.

If your string function does not use the supplied buffer (for example, if it needs to return a string longer than 255 characters), you must allocate the space for your own buffer with malloc() in your xxx_init() function or your xxx() function and free it in your xxx_deinit() function. You can store the allocated memory in the ptr slot in the UDF_INIT structure for reuse by future xxx() calls. See Section 18.2.2.1, “UDF Calling Sequences for Simple Functions”.

To indicate a return value of NULL in the main function, set *is_null to 1:

*is_null = 1;

To indicate an error return in the main function, set *error to 1:

*error = 1;

If xxx() sets *error to 1 for any row, the function value is NULL for the current row and for any subsequent rows processed by the statement in which XXX() was invoked. (xxx() is not even called for subsequent rows.)

*error = 1;
*is_null = 1;

Files implementing UDFs must be compiled and installed on the host where the server runs. This process is described below for the example UDF file sql/udf_example.cc that is included in the MySQL source distribution.

If a UDF will be referred to in statements that will be replicated to slave servers, you must ensure that every slave also has the function available. Otherwise, replication will fail on the slaves when they attempt to invoke the function.

The immediately following instructions are for Unix. Instructions for Windows are given later in this section.

The udf_example.cc file contains the following functions:

A dynamically loadable file should be compiled as a sharable object file, using a command something like this:

shell> gcc -shared -o udf_example.so udf_example.cc

If you are using gcc, you should be able to create udf_example.so with a simpler command:

shell> make udf_example.so

You can easily determine the correct compiler options for your system by running this command in the sql directory of your MySQL source tree:

shell> make udf_example.o

You should run a compile command similar to the one that make displays, except that you should remove the -c option near the end of the line and add -o udf_example.so to the end of the line. (On some systems, you may need to leave the -c on the command.)

After you compile a shared object containing UDFs, you must install it and tell MySQL about it. Compiling a shared object from udf_example.cc produces a file named something like udf_example.so (the exact name may vary from platform to platform).

As of MySQL 4.1.25, copy the shared object to server's plugin directory and name it udf_example.so. This directory is given by the value of the plugin_dir system variable.

Prior to MySQL 4.1.25, or if the value of plugin_dir is empty, the shared object should be placed in a directory such as /usr/lib that is searched by your system's dynamic (runtime) linker, or you can add the directory in which you place the shared object to the linker configuration file (for example, /etc/ld.so.conf).

On many systems, you can also set the LD_LIBRARY or LD_LIBRARY_PATH environment variable to point at the directory where you have the files for your UDF. You should set the variable in mysql.server or mysqld_safe startup scripts and restart mysqld. You might do this if you want to place the object file in a directory accessible only to the server and not in a public directory. The dlopen manual page tells you which variable to use on your system.

The dynamic linker name is system-specific (for example, ld-elf.so.1 on FreeBSD, ld.so on Linux, or dyld on Mac OS X). Consult your system documentation for information about the linker name and how to configure it.

On some systems, the ldconfig program that configures the dynamic linker does not recognize a shared object unless its name begins with lib. In this case you should rename a file such as udf_example.so to libudf_example.so.

On Windows, you can compile user-defined functions by using the following procedure:

After the shared object file has been installed, notify mysqld about the new functions with the following statements. If object files have a suffix different from .so on your system, substitute the correct suffix throughout (for example, .dll on Windows).

mysql> CREATE FUNCTION metaphon RETURNS STRING SONAME 'udf_example.so';
mysql> CREATE FUNCTION myfunc_double RETURNS REAL SONAME 'udf_example.so';
mysql> CREATE FUNCTION myfunc_int RETURNS INTEGER SONAME 'udf_example.so';
mysql> CREATE FUNCTION sequence RETURNS INTEGER SONAME 'udf_example.so';
mysql> CREATE FUNCTION lookup RETURNS STRING SONAME 'udf_example.so';
mysql> CREATE FUNCTION reverse_lookup
    ->        RETURNS STRING SONAME 'udf_example.so';
mysql> CREATE AGGREGATE FUNCTION avgcost
    ->        RETURNS REAL SONAME 'udf_example.so';

To delete functions, use DROP FUNCTION:

mysql> DROP FUNCTION metaphon;
mysql> DROP FUNCTION myfunc_double;
mysql> DROP FUNCTION myfunc_int;
mysql> DROP FUNCTION sequence;
mysql> DROP FUNCTION lookup;
mysql> DROP FUNCTION reverse_lookup;
mysql> DROP FUNCTION avgcost;

The CREATE FUNCTION and DROP FUNCTION statements update the func system table in the mysql database. The function's name, type and shared library name are saved in the table. You must have the INSERT or DELETE privilege for the mysql database to create or drop functions, respectively.

You should not use CREATE FUNCTION to add a function that has previously been created. If you need to reinstall a function, you should remove it with DROP FUNCTION and then reinstall it with CREATE FUNCTION. You would need to do this, for example, if you recompile a new version of your function, so that mysqld gets the new version. Otherwise, the server continues to use the old version.

An active function is one that has been loaded with CREATE FUNCTION and not removed with DROP FUNCTION. All active functions are reloaded each time the server starts, unless you start mysqld with the --skip-grant-tables option. In this case, UDF initialization is skipped and UDFs are unavailable.

MySQL takes several measures to prevent misuse of user-defined functions.

UDF object files cannot be placed in arbitrary directories. They must be located in some system directory that the dynamic linker is configured to search. To enforce this restriction and prevent attempts at specifying path names outside of directories searched by the dynamic linker, MySQL checks the shared object file name specified in CREATE FUNCTION statements for path name delimiter characters. As of MySQL 4.0.24 and 4.1.10a, MySQL also checks for path name delimiters in file names stored in the mysql.func table when it loads functions. This prevents attempts at specifying illegitimate path names through direct manipulation of the mysql.func table. For information about UDFs and the runtime linker, see Section 18.2.2.5, “Compiling and Installing User-Defined Functions”.

To use CREATE FUNCTION or DROP FUNCTION, you must have the INSERT or DELETE privilege, respectively, for the mysql database. This is necessary because those statements add and delete rows from the mysql.func table.

UDFs should have at least one symbol defined in addition to the xxx symbol that corresponds to the main xxx() function. These auxiliary symbols correspond to the xxx_init(), xxx_deinit(), xxx_reset(), xxx_clear(), and xxx_add() functions. As of MySQL 4.0.24 and 4.1.10a, mysqld supports an --allow-suspicious-udfs option that controls whether UDFs that have only an xxx symbol can be loaded. By default, the option is off, to prevent attempts at loading functions from shared object files other than those containing legitimate UDFs. If you have older UDFs that contain only the xxx symbol and that cannot be recompiled to include an auxiliary symbol, it may be necessary to specify the --allow-suspicious-udfs option. Otherwise, you should avoid enabling this capability.

If you have some very specific problem, you can always try to debug MySQL. To do this you must configure MySQL with the --with-debug or the --with-debug=full option. You can check whether MySQL was compiled with debugging by doing: mysqld --help. If the --debug flag is listed with the options then you have debugging enabled. mysqladmin ver also lists the mysqld version as mysql ... --debug in this case.

If you are using gcc, the recommended configure line is:

CC=gcc CFLAGS="-O2" CXX=gcc CXXFLAGS="-O2 -felide-constructors \
   -fno-exceptions -fno-rtti" ./configure --prefix=/usr/local/mysql \
   --with-debug --with-extra-charsets=complex

This avoids problems with the libstdc++ library and with C++ exceptions (many compilers have problems with C++ exceptions in threaded code) and compile a MySQL version with support for all character sets.

If you suspect a memory overrun error, you can configure MySQL with --with-debug=full, which installs a memory allocation (SAFEMALLOC) checker. However, running with SAFEMALLOC is quite slow, so if you get performance problems you should start mysqld with the --skip-safemalloc option. This disables the memory overrun checks for each call to malloc() and free().

If mysqld stops crashing when you compile it with --with-debug, you probably have found a compiler bug or a timing bug within MySQL. In this case, you can try to add -g to the CFLAGS and CXXFLAGS variables above and not use --with-debug. If mysqld dies, you can at least attach to it with gdb or use gdb on the core file to find out what happened.

When you configure MySQL for debugging you automatically enable a lot of extra safety check functions that monitor the health of mysqld. If they find something “unexpected,” an entry is written to stderr, which mysqld_safe directs to the error log! This also means that if you are having some unexpected problems with MySQL and are using a source distribution, the first thing you should do is to configure MySQL for debugging! (The second thing is to send mail to a MySQL mailing list and ask for help. See Section 1.7.1, “MySQL Mailing Lists”. If you believe that you have found a bug, please use the instructions at Section 1.8, “How to Report Bugs or Problems”.

In the Windows MySQL distribution, mysqld.exe is by default compiled with support for trace files.

If the mysqld server doesn't start or if you can cause it to crash quickly, you can try to create a trace file to find the problem.

To do this, you must have a mysqld that has been compiled with debugging support. You can check this by executing mysqld -V. If the version number ends with -debug, it is compiled with support for trace files. (On Windows, the debugging server is named mysqld-debug rather than mysqld as of MySQL 4.1.)

Start the mysqld server with a trace log in /tmp/mysqld.trace on Unix or C:\mysqld.trace on Windows:

shell> mysqld --debug

On Windows, you should also use the --standalone flag to not start mysqld as a service. In a console window, use this command:

C:\> mysqld-debug --debug --standalone

After this, you can use the mysql.exe command-line tool in a second console window to reproduce the problem. You can stop the mysqld server with mysqladmin shutdown.

The trace file can become very large! To generate a smaller trace file, you can use debugging options something like this:

mysqld --debug=d,info,error,query,general,where:O,/tmp/mysqld.trace

This only prints information with the most interesting tags to the trace file.

If you make a bug report about this, please only send the lines from the trace file to the appropriate mailing list where something seems to go wrong! If you can't locate the wrong place, you can open a bug report and upload the trace file to the report, so that a MySQL developer can take a look at it. For instructions, see Section 1.8, “How to Report Bugs or Problems”.

The trace file is made with the DBUG package by Fred Fish. See Section 18.4.3, “The DBUG Package”.

On most systems you can also start mysqld from gdb to get more information if mysqld crashes.

With some older gdb versions on Linux you must use run --one-thread if you want to be able to debug mysqld threads. In this case, you can only have one thread active at a time. It is best to upgrade to gdb 5.1 because thread debugging works much better with this version!

NPTL threads (the new thread library on Linux) may cause problems while running mysqld under gdb. Some symptoms are:

In this case, you should set the following environment variable in the shell before starting gdb:

LD_ASSUME_KERNEL=2.4.1
export LD_ASSUME_KERNEL

When running mysqld under gdb, you should disable the stack trace with --skip-stack-trace to be able to catch segfaults within gdb.

In MySQL 4.0.14 and above you should use the --gdb option to mysqld. This installs an interrupt handler for SIGINT (needed to stop mysqld with ^C to set breakpoints) and disable stack tracing and core file handling.

It is very hard to debug MySQL under gdb if you do a lot of new connections the whole time as gdb doesn't free the memory for old threads. You can avoid this problem by starting mysqld with thread_cache_size set to a value equal to max_connections + 1. In most cases just using --thread_cache_size=5' helps a lot!

If you want to get a core dump on Linux if mysqld dies with a SIGSEGV signal, you can start mysqld with the --core-file option. This core file can be used to make a backtrace that may help you find out why mysqld died:

shell> gdb mysqld core
gdb>   backtrace full
gdb>   quit

See Section B.5.4.2, “What to Do If MySQL Keeps Crashing”.

If you are using gdb 4.17.x or above on Linux, you should install a .gdb file, with the following information, in your current directory:

set print sevenbit off
handle SIGUSR1 nostop noprint
handle SIGUSR2 nostop noprint
handle SIGWAITING nostop noprint
handle SIGLWP nostop noprint
handle SIGPIPE nostop
handle SIGALRM nostop
handle SIGHUP nostop
handle SIGTERM nostop noprint

If you have problems debugging threads with gdb, you should download gdb 5.x and try this instead. The new gdb version has very improved thread handling!

Here is an example how to debug mysqld:

shell> gdb /usr/local/libexec/mysqld
gdb> run
...
backtrace full # Do this when mysqld crashes

Include the above output in a bug report, which you can file using the instructions in Section 1.8, “How to Report Bugs or Problems”.

If mysqld hangs you can try to use some system tools like strace or /usr/proc/bin/pstack to examine where mysqld has hung.

strace /tmp/log libexec/mysqld

If you are using the Perl DBI interface, you can turn on debugging information by using the trace method or by setting the DBI_TRACE environment variable.

On some operating systems, the error log contains a stack trace if mysqld dies unexpectedly. You can use this to find out where (and maybe why) mysqld died. See Section 5.3.1, “The Error Log”. To get a stack trace, you must not compile mysqld with the -fomit-frame-pointer option to gcc. See Section 18.4.1.1, “Compiling MySQL for Debugging”.

A stack trace in the error log looks something like this:

mysqld got signal 11;
Attempting backtrace. You can use the following information
to find out where mysqld died. If you see no messages after
this, something went terribly wrong...

stack range sanity check, ok, backtrace follows
0x40077552
0x81281a0
0x8128f47
0x8127be0
0x8127995
0x8104947
0x80ff28f
0x810131b
0x80ee4bc
0x80c3c91
0x80c6b43
0x80c1fd9
0x80c1686

You can use the resolve_stack_dump utility to determine where mysqld died by using the following procedure:

Note that before starting mysqld with --log you should check all your tables with myisamchk. See Chapter 5, MySQL Server Administration.

If mysqld dies or hangs, you should start mysqld with --log. When mysqld dies again, you can examine the end of the log file for the query that killed mysqld.

If you are using --log without a file name, the log is stored in the database directory as host_name.log In most cases it is the last query in the log file that killed mysqld, but if possible you should verify this by restarting mysqld and executing the found query from the mysql command-line tools. If this works, you should also test all complicated queries that didn't complete.

You can also try the command EXPLAIN on all SELECT statements that takes a long time to ensure that mysqld is using indexes properly. See Section 12.7.2, “EXPLAIN Syntax”.

You can find the queries that take a long time to execute by starting mysqld with --log-slow-queries. See Section 5.3.5, “The Slow Query Log”.

If you find the text mysqld restarted in the error log file (normally named hostname.err) you probably have found a query that causes mysqld to fail. If this happens, you should check all your tables with myisamchk (see Chapter 5, MySQL Server Administration), and test the queries in the MySQL log files to see whether one fails. If you find such a query, try first upgrading to the newest MySQL version. If this doesn't help and you can't find anything in the mysql mail archive, you should report the bug to a MySQL mailing list. The mailing lists are described at http://lists.mysql.com/, which also has links to online list archives.

If you have started mysqld with --myisam-recover, MySQL automatically checks and tries to repair MyISAM tables if they are marked as 'not closed properly' or 'crashed'. If this happens, MySQL writes an entry in the hostname.err file 'Warning: Checking table ...' which is followed by Warning: Repairing table if the table needs to be repaired. If you get a lot of these errors, without mysqld having died unexpectedly just before, then something is wrong and needs to be investigated further. See Section 5.1.2, “Server Command Options”.

It is not a good sign if mysqld did die unexpectedly, but in this case, you should not investigate the Checking table... messages, but instead try to find out why mysqld died.

If you get corrupted tables or if mysqld always fails after some update commands, you can test whether this bug is reproducible by doing the following:

You can also use the script mysql_find_rows to just execute some of the update statements if you want to narrow down the problem.