Table of Contents
MyISAM
Storage EngineMERGE
Storage EngineMEMORY
(HEAP
) Storage EngineBDB
(BerkeleyDB
) Storage EngineEXAMPLE
Storage EngineFEDERATED
Storage EngineARCHIVE
Storage EngineCSV
Storage EngineBLACKHOLE
Storage EngineMySQL supports several storage engines that act as handlers for different table types. MySQL storage engines include both those that handle transaction-safe tables and those that handle non-transaction-safe tables:
MyISAM
manages non-transactional tables. It
provides high-speed storage and retrieval, as well as fulltext
searching capabilities. MyISAM
is supported
in all MySQL configurations, and is the default storage engine
unless you have configured MySQL to use a different one by
default.
The MEMORY
storage engine provides in-memory
tables. The MERGE
storage engine allows a
collection of identical MyISAM
tables to be
handled as a single table. Like MyISAM
, the
MEMORY
and MERGE
storage
engines handle non-transactional tables, and both are also
included in MySQL by default.
Note: The
MEMORY
storage engine was formerly known as
the HEAP
engine.
The InnoDB
and BDB
storage
engines provide transaction-safe tables. BDB
is included in MySQL-Max binary distributions on those operating
systems that support it. InnoDB
is also
included by default in all MySQL 5.0 binary
distributions. In source distributions, you can enable or
disable either engine by configuring MySQL as you like.
The EXAMPLE
storage engine is a
“stub” engine that does nothing. You can create
tables with this engine, but no data can be stored in them or
retrieved from them. The purpose of this engine is to serve as
an example in the MySQL source code that illustrates how to
begin writing new storage engines. As such, it is primarily of
interest to developers.
NDB Cluster
is the storage engine used by
MySQL Cluster to implement tables that are partitioned over many
computers. It is available in MySQL-Max 5.0 binary
distributions. This storage engine is currently supported on
Linux, Solaris, and Mac OS X only. We intend to add support for
this engine on other platforms, including Windows, in future
MySQL releases.
The ARCHIVE
storage engine is used for
storing large amounts of data without indexes with a very small
footprint.
The CSV
storage engine stores data in text
files using comma-separated-values format.
The BLACKHOLE
storage engine accepts but does
not store data and retrievals always return an empty set.
The FEDERATED
storage engine was added in
MySQL 5.0.3. This engine stores data in a remote database. In
this release, it works with MySQL only, using the MySQL C Client
API. In future releases, we intend to enable it to connect to
other data sources using other drivers or client connection
methods.
This chapter describes each of the MySQL storage engines except for
InnoDB
and NDB Cluster
, which
are covered in Chapter 15, The InnoDB
Storage Engine and
Chapter 16, MySQL Cluster.
When you create a new table, you can tell MySQL what type of table
to create by adding an ENGINE
or
TYPE
table option to the CREATE
TABLE
statement:
CREATE TABLE t (i INT) ENGINE = INNODB; CREATE TABLE t (i INT) TYPE = MEMORY;
While TYPE
is still supported in MySQL
5.0, ENGINE
is now the preferred
term.
If you omit the ENGINE
or TYPE
option, the default storage engine is used. Normally this is
MyISAM
, but you can change it by using the
--default-storage-engine
or
--default-table-type
server startup option, or by
setting the storage_engine
or
table_type
system variable.
When MySQL is installed on Windows using the MySQL Configuration
Wizard, the InnoDB
storage engine is the default
instead of MyISAM
. See
Section 2.3.5.1, “Introduction”.
To convert a table from one type to another, use an ALTER
TABLE
statement that indicates the new type:
ALTER TABLE t ENGINE = MYISAM; ALTER TABLE t TYPE = BDB;
See Section 13.1.5, “CREATE TABLE
Syntax” and
Section 13.1.2, “ALTER TABLE
Syntax”.
If you try to use a storage engine that is not compiled in or that
is compiled in but deactivated, MySQL instead creates a table of
type MyISAM
. This behavior is convenient when you
want to copy tables between MySQL servers that support different
storage engines. (For example, in a replication setup, perhaps your
master server supports transactional storage engines for increased
safety, but the slave servers use only non-transactional storage
engines for greater speed.)
This automatic substitution of the MyISAM
table
type when an unavailable type is specified can be confusing for new
MySQL users. In MySQL 5.0, a warning is generated when
a table type is automatically changed.
MySQL always creates an .frm
file to hold the
table and column definitions. The table's index and data may be
stored in one or more other files, depending on the table type. The
server creates the .frm
file above the storage
engine level. Individual storage engines create any additional files
required for the tables that they manage.
A database may contain tables of different types.
Transaction-safe tables (TSTs) have several advantages over non-transaction-safe tables (NTSTs):
They are safer. Even if MySQL crashes or you get hardware problems, you can get your data back, either by automatic recovery or from a backup plus the transaction log.
You can combine many statements and accept them all at the same
time with the COMMIT
statement (if autocommit
is disabled).
You can execute ROLLBACK
to ignore your
changes (if autocommit is disabled).
If an update fails, all of your changes are reverted. (With non-transaction-safe tables, all changes that have taken place are permanent.)
Transaction-safe storage engines can provide better concurrency for tables that get many updates concurrently with reads.
Although MySQL supports several transaction-safe storage engines,
for best results, you should not mix different table types within a
transaction. For information about the problems that can occur if
you do this, see Section 13.4.1, “START TRANSACTION
, COMMIT
, and ROLLBACK
Syntax”.
In MySQL 5.0, InnoDB
uses default
configuration values if you specify none. See
Section 15.3, “InnoDB
Configuration”.
Non-transaction-safe tables have several advantages of their own, all of which occur because there is no transaction overhead:
Much faster
Lower disk space requirements
Less memory required to perform updates
You can combine transaction-safe and non-transaction-safe tables in the same statements to get the best of both worlds. However, within a transaction with autocommit disabled, changes to non-transaction-safe tables still are committed immediately and cannot be rolled back.
MyISAM
is the default storage engine. It is
based on the older ISAM
code but has many
useful extensions. (Note that MySQL 5.0 does not
support ISAM
.)
Each MyISAM
table is stored on disk in three
files. The files have names that begin with the table name and
have an extension to indicate the file type. An
.frm
file stores the table definition. The
data file has an .MYD
(MYData
) extension. The index file has an
.MYI
(MYIndex
) extension.
To specify explicitly that you want a MyISAM
table, indicate that with an ENGINE
table
option:
CREATE TABLE t (i INT) ENGINE = MYISAM;
(Note: Older versions of MySQL
used TYPE
rather than ENGINE
(for example: TYPE = MYISAM
). MySQL
5.0 supports this syntax for backwards compatibility
but TYPE
is now deprecated and
ENGINE
is the preferred usage.)
Normally, the ENGINE
option is unnecessary;
MyISAM
is the default storage engine unless the
default has been changed.
You can check or repair MyISAM
tables with the
myisamchk utility. See
Section 5.9.5.6, “Using myisamchk for Crash Recovery”. You can also compress
MyISAM
tables with
myisampack to take up much less space. See
Section 8.2, “myisampack — Generate Compressed, Read-Only MyISAM Tables”.
The following are some characteristics of the
MyISAM
storage engine:
All data values are stored with the low byte first. This makes the data machine and operating system independent. The only requirement for binary portability is that the machine uses two's-complement signed integers (as every machine for the last 20 years has) and IEEE floating-point format (also totally dominant among mainstream machines). The only area of machines that may not support binary compatibility are embedded systems, which sometimes have peculiar processors.
There is no big speed penalty for storing data low byte first; the bytes in a table row normally are unaligned and it doesn't take that much more power to read an unaligned byte in order than in reverse order. Also, the code in the server that fetches column values is not time critical compared to other code.
Large files (up to 63-bit file length) are supported on filesystems and operating systems that support large files.
Dynamic-sized rows are much less fragmented when mixing deletes with updates and inserts. This is done by automatically combining adjacent deleted blocks and by extending blocks if the next block is deleted.
The maximum number of indexes per MyISAM
table in MySQL 5.0 is 64. This can be changed by
recompiling. The maximum number of columns per index is 16.
The maximum key length is 1000 bytes. This can also be changed by recompiling. For the case of a key longer than 250 bytes, a larger key block size than the default of 1024 bytes is used.
BLOB
and TEXT
columns
can be indexed.
NULL
values are allowed in indexed columns.
This takes 0-1 bytes per key.
All numeric key values are stored with the high byte first to allow better index compression.
When records are inserted in sorted order (as when you are
using an AUTO_INCREMENT
column), the index
tree is split so that the high node only contains one key.
This improves space utilization in the index tree.
Internal handling of one AUTO_INCREMENT
column per table. MyISAM
automatically
updates this column for INSERT
and
UPDATE
operations. This makes
AUTO_INCREMENT
columns faster (at least
10%). Values at the top of the sequence are not reused after
being deleted. (When an AUTO_INCREMENT
column is defined as the last column of a multiple-column
index, reuse of values deleted from the top of a sequence does
occur.) The AUTO_INCREMENT
value can be
reset with ALTER TABLE
or
myisamchk.
If a table has no free blocks in the middle of the data file,
you can INSERT
new rows into it at the same
time that other threads are reading from the table. (These are
known as concurrent inserts.) A free block can occur as a
result of deleting rows or an update of a dynamic length row
with more data than its current contents. When all free blocks
are used up (filled in), future inserts become concurrent
again.
You can put the data file and index file on different
directories to get more speed with the DATA
DIRECTORY
and INDEX DIRECTORY
table options to CREATE TABLE
. See
Section 13.1.5, “CREATE TABLE
Syntax”.
Each character column can have a different character set. See Chapter 10, Character Set Support.
There is a flag in the MyISAM
index file
that indicates whether the table was closed correctly. If
mysqld is started with the
--myisam-recover
option,
MyISAM
tables are automatically checked
when opened, and are repaired if the table wasn't closed
properly.
myisamchk marks tables as checked if you
run it with the --update-state
option.
myisamchk --fast checks only those tables
that don't have this mark.
myisamchk --analyze stores statistics for portions of keys, as well as for entire keys.
myisampack can pack BLOB
and VARCHAR
columns.
MyISAM
also supports the following features:
Support for a true VARCHAR
type; a
VARCHAR
column starts with a length stored
in two bytes.
Tables with VARCHAR
may have fixed or
dynamic record length.
VARCHAR
and CHAR
columns
may be up to 64KB.
A hashed computed index can be used for
UNIQUE
. This allows you to have
UNIQUE
on any combination of columns in a
table. (However, you cannot search on a
UNIQUE
computed index.)
For the MyISAM
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?21.
The following options to mysqld can be used
to change the behavior of MyISAM
tables:
--myisam-recover=
mode
Set the mode for automatic recovery of crashed
MyISAM
tables.
--delay-key-write=ALL
Don't flush key buffers between writes for any
MyISAM
table.
Note: If you do this, you
should not use MyISAM
tables from another
program (such as from another MySQL server or with
myisamchk) when the table is in use.
Doing so leads to index corruption.
Using --external-locking
does not help for
tables that use --delay-key-write
.
See Section 5.3.1, “mysqld Command-Line Options”.
The following system variables affect the behavior of
MyISAM
tables:
bulk_insert_buffer_size
The size of the tree cache used in bulk insert optimization. Note: This is a limit per thread!
myisam_max_extra_sort_file_size
Used to help MySQL to decide when to use the slow but safe key cache index creation method. Note: This parameter was given in bytes before MySQL 5.0.6, when it was removed.
myisam_max_sort_file_size
Don't use the fast sort index method to create an index if the temporary file would become larger than this. Note: In MySQL 5.0, this parameter is given in bytes.
myisam_sort_buffer_size
Set the size of the buffer used when recovering tables.
See Section 5.3.3, “Server System Variables”.
Automatic recovery is activated if you start
mysqld with the
--myisam-recover
option. In this case, when the
server opens a MyISAM
table, it checks
whether the table is marked as crashed or whether the open count
variable for the table is not 0 and you are running the server
with --skip-external-locking
. If either of
these conditions is true, the following happens:
The table is checked for errors.
If the server finds an error, it tries to do a fast table repair (with sorting and without re-creating the data file).
If the repair fails because of an error in the data file (for example, a duplicate-key error), the server tries again, this time re-creating the data file.
If the repair still fails, the server tries once more with the old repair option method (write row by row without sorting). This method should be able to repair any type of error and has low disk space requirements.
If the recovery wouldn't be able to recover all rows from a
previous completed statement and you didn't specify
FORCE
in the value of the
--myisam-recover
option, automatic repair
aborts with an error message in the error log:
Error: Couldn't repair table: test.g00pages
If you specify FORCE
, a warning like this is
written instead:
Warning: Found 344 of 354 rows when repairing ./test/g00pages
Note that if the automatic recovery value includes
BACKUP
, the recovery process creates files
with names of the form
.
You should have a cron script that
automatically moves these files from the database directories to
backup media.
tbl_name-datetime
.BAK
MyISAM
tables use B-tree indexes. You can
roughly calculate the size for the index file as
(key_length+4)/0.67
, summed over all keys.
This is for the worst case when all keys are inserted in sorted
order and the table doesn't have any compressed keys.
String indexes are space compressed. If the first index part is
a string, it is also prefix compressed. Space compression makes
the index file smaller than the worst-case figure if the string
column has a lot of trailing space or is a
VARCHAR
column that is not always used to the
full length. Prefix compression is used on keys that start with
a string. Prefix compression helps if there are many strings
with an identical prefix.
In MyISAM
tables, you can also prefix
compress numbers by specifying PACK_KEYS=1
when you create the table. This helps when you have many integer
keys that have an identical prefix when the numbers are stored
high-byte first.
MyISAM
supports three different storage
formats. Two of them (fixed and dynamic format) are chosen
automatically depending on the type of columns you are using.
The third, compressed format, can be created only with the
myisampack utility.
When you CREATE
or ALTER
a
table that has no BLOB
or
TEXT
columns, you can force the table format
to FIXED
or DYNAMIC
with
the ROW_FORMAT
table option. This causes
CHAR
and VARCHAR
columns
to become CHAR
for FIXED
format, or VARCHAR
for
DYNAMIC
format.
You can compress or decompress tables by specifying
ROW_FORMAT={COMPRESSED | DEFAULT}
with
ALTER TABLE
. See
Section 13.1.5, “CREATE TABLE
Syntax”.
Static format is the default for MyISAM
tables. It is used when the table contains no variable-length
columns (VARCHAR
, BLOB
,
or TEXT
). Each row is stored using a fixed
number of bytes.
Of the three MyISAM
storage formats, static
format is the simplest and most secure (least subject to
corruption). It is also the fastest of the on-disk formats.
The speed comes from the easy way that rows in the data file
can be found on disk: When looking up a row based on a row
number in the index, multiply the row number by the row
length. Also, when scanning a table, it is very easy to read a
constant number of records with each disk read operation.
The security is evidenced if your computer crashes while the
MySQL server is writing to a fixed-format
MyISAM
file. In this case,
myisamchk can easily determine where each
row starts and ends, so it can usually reclaim all records
except the partially written one. Note that
MyISAM
table indexes can always be
reconstructed based on the data rows.
General characteristics of static format tables:
CHAR
columns are space-padded to the
column width. This is also true for
NUMERIC
, and DECIMAL
columns created before MySQL 5.0.3.
Very quick.
Easy to cache.
Easy to reconstruct after a crash, because records are located in fixed positions.
Reorganization is unnecessary unless you delete a huge
number of records and want to return free disk space to
the operating system. To do this, use OPTIMIZE
TABLE
or myisamchk -r.
Usually require more disk space than for dynamic-format tables.
Dynamic storage format is used if a MyISAM
table contains any variable-length columns
(VARCHAR
, BLOB
, or
TEXT
), or if the table was created with the
ROW_FORMAT=DYNAMIC
option.
This format is a little more complex because each row has a header that indicates how long it is. One record can also end up at more than one location when it is made longer as a result of an update.
You can use OPTIMIZE TABLE
or
myisamchk to defragment a table. If you
have fixed-length columns that you access or change frequently
in a table that also contains some variable-length columns, it
might be a good idea to move the variable-length columns to
other tables just to avoid fragmentation.
General characteristics of dynamic-format tables:
All string columns are dynamic except those with a length less than four.
Each record is preceded by a bitmap that indicates which
columns contain the empty string (for string columns) or
zero (for numeric columns). Note that this does not
include columns that contain NULL
values. If a string column has a length of zero after
trailing space removal, or a numeric column has a value of
zero, it is marked in the bitmap and not saved to disk.
Non-empty strings are saved as a length byte plus the
string contents.
Much less disk space usually is required than for fixed-length tables.
Each record uses only as much space as is required.
However, if a record becomes larger, it is split into as
many pieces as are required, resulting in record
fragmentation. For example, if you update a row with
information that extends the row length, the row becomes
fragmented. In this case, you may have to run
OPTIMIZE TABLE
or myisamchk
-r from time to time to improve performance. Use
myisamchk -ei to obtain table
statistics.
More difficult than static-format tables to reconstruct after a crash, because a record may be fragmented into many pieces and a link (fragment) may be missing.
The expected row length for dynamic-sized records is calculated using the following expression:
3 + (number of columns
+ 7) / 8 + (number of char columns
) + (packed size of numeric columns
) + (length of strings
) + (number of NULL columns
+ 7) / 8
There is a penalty of 6 bytes for each link. A dynamic record is linked whenever an update causes an enlargement of the record. Each new link is at least 20 bytes, so the next enlargement probably goes in the same link. If not, another link is created. You can find the number of links using myisamchk -ed. All links may be removed with myisamchk -r.
Compressed storage format is a read-only format that is generated with the myisampack tool.
All MySQL distributions include myisampack by default. Compressed tables can be uncompressed with myisamchk.
Compressed tables have the following characteristics:
Compressed tables take very little disk space. This minimizes disk usage, which is helpful when using slow disks (such as CD-ROMs).
Each record is compressed separately, so there is very little access overhead. The header for a record takes up 1 to 3 bytes depending on the biggest record in the table. Each column is compressed differently. There is usually a different Huffman tree for each column. Some of the compression types are:
Suffix space compression.
Prefix space compression.
Numbers with a value of zero are stored using one bit.
If values in an integer column have a small range, the
column is stored using the smallest possible type. For
example, a BIGINT
column (eight
bytes) can be stored as a TINYINT
column (one byte) if all its values are in the range
from -128
to
127
.
If a column has only a small set of possible values,
the column type is converted to
ENUM
.
A column may use any combination of the preceding compression types.
Can handle fixed-length or dynamic-length records.
The file format that MySQL uses to store data has been extensively tested, but there are always circumstances that may cause database tables to become corrupted.
Even though the MyISAM
table format is very
reliable (all changes to a table made by an SQL statement are
written before the statement returns), you can still get
corrupted tables if any of the following events occur:
The mysqld process is killed in the middle of a write.
Unexpected computer shutdown occurs (for example, the computer is turned off).
Hardware failures.
You are using an external program (such as myisamchk) on a table that is being modified by the server at the same time.
A software bug in the MySQL or MyISAM
code.
Typical symptoms of a corrupt table are:
You get the following error while selecting data from the table:
Incorrect key file for table: '...'. Try to repair it
Queries don't find rows in the table or return incomplete data.
You can check the health of a MyISAM
table
using the CHECK TABLE
statement, and repair
a corrupted MyISAM
table with
REPAIR TABLE
. When
mysqld is not running, you can also check
or repair a table with the myisamchk
command. See Section 13.5.2.3, “CHECK TABLE
Syntax”,
Section 13.5.2.6, “REPAIR TABLE
Syntax”, and
Section 5.9.5, “myisamchk — MyISAM Table-Maintenance Utility”.
If your tables become corrupted frequently, you should try to
determine why this is happening. The most important thing to
know is whether the table became corrupted as a result of a
server crash. You can verify this easily by looking for a
recent restarted mysqld
message in the
error log. If there is such a message, it is likely that table
corruption is a result of the server dying. Otherwise,
corruption may have occurred during normal operation. This is
a bug. You should try to create a reproducible test case that
demonstrates the problem. See Section A.4.2, “What to Do If MySQL Keeps Crashing” and
Section E.1.6, “Making a Test Case If You Experience Table Corruption”.
Each MyISAM
index
(.MYI
) file has a counter in the header
that can be used to check whether a table has been closed
properly. If you get the following warning from CHECK
TABLE
or myisamchk, it means that
this counter has gone out of sync:
clients are using or haven't closed the table properly
This warning doesn't necessarily mean that the table is corrupted, but you should at least check the table.
The counter works as follows:
The first time a table is updated in MySQL, a counter in the header of the index files is incremented.
The counter is not changed during further updates.
When the last instance of a table is closed (because of a
FLUSH TABLES
operation or because there
isn't room in the table cache), the counter is decremented
if the table has been updated at any point.
When you repair the table or check the table and it is found to be okay, the counter is reset to zero.
To avoid problems with interaction with other processes that might check the table, the counter is not decremented on close if it was zero.
In other words, the counter can go out of sync only under these conditions:
The MyISAM
tables are copied without
first issuing LOCK TABLES
and
FLUSH TABLES
.
MySQL has crashed between an update and the final close. (Note that the table may still be okay, because MySQL always issues writes for everything between each statement.)
A table was modified by myisamchk --recover or myisamchk --update-state at the same time that it was in use by mysqld.
Multiple mysqld servers are using the
table and one server performed a REPAIR
TABLE
or CHECK TABLE
on the
table while it was in use by another server. In this
setup, it is safe to use CHECK TABLE
,
although you might get the warning from other servers.
However, REPAIR TABLE
should be avoided
because when one server replaces the data file with a new
one, this is not signaled to the other servers.
In general, it is a bad idea to share a data directory among multiple servers. See Section 5.12, “Running Multiple MySQL Servers on the Same Machine” for additional discussion.
The MERGE
storage engine, also known as the
MRG_MyISAM
engine, is a collection of identical
MyISAM
tables that can be used as one.
“Identical” means that all tables have identical
column and index information. You cannot merge tables in which the
columns are listed in a different order, do not have exactly the
same columns, or have the indexes in different order. However, any
or all of the tables can be compressed with
myisampack. See Section 8.2, “myisampack — Generate Compressed, Read-Only MyISAM Tables”.
Differences in table options such as
AVG_ROW_LENGTH
, MAX_ROWS
, or
PACK_KEYS
do not matter.
When you create a MERGE
table, MySQL creates
two files on disk. The files have names that begin with the table
name and have an extension to indicate the file type. An
.frm
file stores the table definition, and an
.MRG
file contains the names of the tables
that should be used as one. The tables do not have to be in the
same database as the MERGE
table itself.
You can use SELECT
, DELETE
,
UPDATE
, and INSERT
on the
collection of tables. You must have SELECT
,
UPDATE
, and DELETE
privileges on the tables that you map to a
MERGE
table.
If you DROP
the MERGE
table,
you are dropping only the MERGE
specification.
The underlying tables are not affected.
When you create a MERGE
table, you must specify
a
UNION=(
clause that indicates which tables you want to use as one. You can
optionally specify an list-of-tables
)INSERT_METHOD
option if
you want inserts for the MERGE
table to take
place in the first or last table of the UNION
list. Use a value of FIRST
or
LAST
to cause inserts to be made in the first
or last table, respectively. If you do not specify an
INSERT_METHOD
option or if you specify it with
a value of NO
, attempts to insert records into
the MERGE
table result in an error.
The following example shows how to create a
MERGE
table:
mysql>CREATE TABLE t1 (
->a INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
->message CHAR(20));
mysql>CREATE TABLE t2 (
->a INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
->message CHAR(20));
mysql>INSERT INTO t1 (message) VALUES ('Testing'),('table'),('t1');
mysql>INSERT INTO t2 (message) VALUES ('Testing'),('table'),('t2');
mysql>CREATE TABLE total (
->a INT NOT NULL AUTO_INCREMENT,
->message CHAR(20), INDEX(a))
->TYPE=MERGE UNION=(t1,t2) INSERT_METHOD=LAST;
Note that the a
column is indexed in the
MERGE
table, but is not declared as a
PRIMARY KEY
as it is in the underlying
MyISAM
tables. This is necessary because a
MERGE
table cannot enforce uniqueness over the
set of underlying tables.
After creating the MERGE
table, you can issue
queries that operate on the group of tables as a whole:
mysql> SELECT * FROM total;
+---+---------+
| a | message |
+---+---------+
| 1 | Testing |
| 2 | table |
| 3 | t1 |
| 1 | Testing |
| 2 | table |
| 3 | t2 |
+---+---------+
Note that you can also manipulate the .MRG
file directly from outside of the MySQL server:
shell>cd /
shell>mysql-data-directory
/current-database
ls -1 t1 t2 > total.MRG
shell>mysqladmin flush-tables
To remap a MERGE
table to a different
collection of MyISAM
tables, you can perform
one of the following:
DROP
the MERGE
table and
re-create it.
Use ALTER TABLE
to change the list of underlying tables.
tbl_name
UNION=(...)
Change the .MRG
file and issue a
FLUSH TABLE
statement for the
MERGE
table and all underlying tables to
force the storage engine to read the new definition file.
MERGE
tables can help you solve the following
problems:
Easily manage a set of log tables. For example, you can put
data from different months into separate tables, compress some
of them with myisampack, and then create a
MERGE
table to use them as one.
Obtain more speed. You can split a big read-only table based
on some criteria, and then put individual tables on different
disks. A MERGE
table on this could be much
faster than using the big table.
Perform more efficient searches. If you know exactly what you
are looking for, you can search in just one of the split
tables for some queries and use a MERGE
table for others. You can even have many different
MERGE
tables that use overlapping sets of
tables.
Perform more efficient repairs. It is easier to repair
individual tables that are mapped to a
MERGE
table than to repair a single large
table.
Instantly map many tables as one. A MERGE
table need not maintain an index of its own because it uses
the indexes of the individual tables. As a result,
MERGE
table collections are
very fast to create or remap. (Note that
you must still specify the index definitions when you create a
MERGE
table, even though no indexes are
created.)
If you have a set of tables that you join as a big table on
demand or batch, you should instead create a
MERGE
table on them on demand. This is much
faster and saves a lot of disk space.
Exceed the file size limit for the operating system. Each
MyISAM
table is bound by this limit, but a
collection of MyISAM
tables is not.
You can create an alias or synonym for a
MyISAM
table by defining a
MERGE
table that maps to that single table.
There should be no really notable performance impact from
doing this (only a couple of indirect calls and
memcpy()
calls for each read).
The disadvantages of MERGE
tables are:
You can use only identical MyISAM
tables
for a MERGE
table.
You cannot use a number of MyISAM
features
in MERGE
tables. For example, you cannot
create FULLTEXT
indexes on
MERGE
tables. (You can, of course, create
FULLTEXT
indexes on the underlying
MyISAM
tables, but you cannot search the
MERGE
table with a full-text search.)
If the MERGE
table is non-temporary, all
underlying MyISAM
tables have to be
permanent, too. If the MERGE
table is
temporary, the MyISAM
tables can be any mix
of temporary and non-temporary.
MERGE
tables use more file descriptors. If
10 clients are using a MERGE
table that
maps to 10 tables, the server uses (10*10) + 10 file
descriptors. (10 data file descriptors for each of the 10
clients, and 10 index file descriptors shared among the
clients.)
Key reads are slower. When you read a key, the
MERGE
storage engine needs to issue a read
on all underlying tables to check which one most closely
matches the given key. If you then do a read-next, the
MERGE
storage engine needs to search the
read buffers to find the next key. Only when one key buffer is
used up does the storage engine need to read the next key
block. This makes MERGE
keys much slower on
eq_ref
searches, but not much slower on
ref
searches. See Section 7.2.1, “EXPLAIN
Syntax (Get Information About a SELECT
)”
for more information about eq_ref
and
ref
.
The following are known problems with MERGE
tables:
If you use ALTER TABLE
to change a
MERGE
table to another table type, the
mapping to the underlying tables is lost. Instead, the rows
from the underlying MyISAM
tables are
copied into the altered table, which is then assigned the
new type.
REPLACE
does not work.
You cannot use DROP TABLE
, ALTER
TABLE
, DELETE FROM
without a
WHERE
clause, REPAIR
TABLE
, TRUNCATE TABLE
,
OPTIMIZE TABLE
, or ANALYZE
TABLE
on any of the tables that are mapped into an
open MERGE
table. If you do so, the
MERGE
table may still refer to the
original table, which yields unexpected results. The easiest
way to work around this deficiency is to issue a
FLUSH TABLES
statement prior to
performing any of these operations to ensure that no
MERGE
tables remain open.
A MERGE
table cannot maintain
UNIQUE
constraints over the whole table.
When you perform an INSERT
, the data goes
into the first or last MyISAM
table
(depending on the value of the
INSERT_METHOD
option). MySQL ensures that
unique key values remain unique within that
MyISAM
table, but not across all the
tables in the collection.
When you create a MERGE
table, there is
no check to insure that the underlying tables exist and have
identical structures. When the MERGE
table is used, MySQL checks that the record length for all
mapped tables is equal, but this is not foolproof. If you
create a MERGE
table from dissimilar
MyISAM
tables, you are very likely to run
into strange problems.
The order of indexes in the MERGE
table
and its underlying tables should be the same. If you use
ALTER TABLE
to add a
UNIQUE
index to a table used in a
MERGE
table, and then use ALTER
TABLE
to add a non-unique index on the
MERGE
table, the index ordering is
different for the tables if there was already a non-unique
index in the underlying table. (This is because
ALTER TABLE
puts
UNIQUE
indexes before non-unique indexes
to facilitate rapid detection of duplicate keys.)
Consequently, queries on tables with such indexes may return
unexpected results.
DROP TABLE
on a table that is in use by a
MERGE
table does not work on Windows
because the MERGE
storage engine's table
mapping is hidden from the upper layer of MySQL. Since
Windows does not allow the deletion of open files, you first
must flush all MERGE
tables (with
FLUSH TABLES
) or drop the
MERGE
table before dropping the table.
For the MERGE
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?93.
The MEMORY
storage engine creates tables with
contents that are stored in memory. These were formerly known as
HEAP
tables. In MySQL 5.0,
MEMORY
is the preferred term, although
HEAP
remains supported for backwards
compatibility.
Each MEMORY
table is associated with one disk
file. The filename begins with the table name and has an extension
of .frm
to indicate that it stores the table
definition.
To specify explicitly that you want a MEMORY
table, indicate that with an ENGINE
option:
CREATE TABLE t (i INT) ENGINE = MEMORY;
As indicated by their name, MEMORY
tables are
stored in memory and use hash indexes by default. This makes them
very fast, and very useful for creating temporary tables. However,
when the server shuts down, all data stored in
MEMORY
tables is lost. The tables themselves
continue to exist because their definitions are stored in
.frm
files on disk, but they are empty when
the server restarts.
This example shows how you might create, use, and remove a
MEMORY
table:
mysql>CREATE TABLE test ENGINE=MEMORY
->SELECT ip,SUM(downloads) AS down
->FROM log_table GROUP BY ip;
mysql>SELECT COUNT(ip),AVG(down) FROM test;
mysql>DROP TABLE test;
MEMORY
tables have the following
characteristics:
Space for MEMORY
tables is allocated in
small blocks. Tables use 100% dynamic hashing for inserts. No
overflow area or extra key space is needed. No extra space is
needed for free lists. Deleted rows are put in a linked list
and are reused when you insert new data into the table.
MEMORY
tables also have none of the
problems commonly associated with deletes plus inserts in
hashed tables.
MEMORY
tables can have up to 32 indexes per
table, 16 columns per index and a maximum key length of 500
bytes.
In MySQL 5.0, the MEMORY
storage engine implements both HASH
and
BTREE
indexes. You can specify one or the
other for a given index by adding a USING
clause as shown here:
CREATE TABLE lookup (id INT, INDEX USING HASH (id)) ENGINE = MEMORY; CREATE TABLE lookup (id INT, INDEX USING BTREE (id)) ENGINE = MEMORY;
General characteristics of B-tree and hash indexes are described in Section 7.4.5, “How MySQL Uses Indexes”.
You can have non-unique keys in a MEMORY
table. (This is an uncommon feature for implementations of
hash indexes.)
In MySQL 5.0, you can use INSERT
DELAYED
with MEMORY
tables. See
Section 13.2.4.2, “INSERT DELAYED
Syntax”.
If you have a hash index on a MEMORY
table
that has a high degree of key duplication (many index entries
containing the same value), updates to the table that affect
key values and all deletes are significantly slower. The
degree of this slowdown is proportional to the degree of
duplication (or, inversely proportional to the index
cardinality). You can use a BTREE
index to
avoid this problem.
MEMORY
tables use a fixed record length
format.
MEMORY
doesn't support
BLOB
or TEXT
columns.
MEMORY
in MySQL 5.0 includes
support for both AUTO_INCREMENT
columns and
indexes on columns that can contain NULL
values.
MEMORY
tables are shared between all
clients (just like any other non-TEMPORARY
table).
MEMORY
table contents are stored in memory,
which is a property that MEMORY
tables
share with internal tables that the server creates on the fly
while processing queries. However, the two types of tables
differ in that MEMORY
tables are not
subject to storage conversion, whereas internal tables are:
If an internal table becomes too large, the server
automatically converts it to an on-disk table. The size
limit is determined by the value of the
tmp_table_size
system variable.
MEMORY
tables are never converted to
disk tables. To ensure that you don't accidentally do
anything foolish, you can set the
max_heap_table_size
system variable to
impose a maximum size on MEMORY
tables.
For individual tables, you can also specify a
MAX_ROWS
table option in the
CREATE TABLE
statement.
The server needs sufficient memory to maintain all
MEMORY
tables that are in use at the same
time.
To free memory used by a MEMORY
table when
you no longer require its contents, you should execute
DELETE FROM
or TRUNCATE
TABLE
, or remove the table altogether (using
DROP TABLE
).
If you want to populate a MEMORY
table when
the MySQL server starts, you can use the
--init-file
option. For example, you can put
statements such as INSERT INTO ... SELECT
or LOAD DATA INFILE
into this file in order
to load the table from a persistent data source. See
Section 5.3.1, “mysqld Command-Line Options” and
Section 13.2.5, “LOAD DATA INFILE
Syntax”.
If you are using replication, the master server's
MEMORY
tables become empty when it is shut
down and restarted. However, a slave is not aware that these
tables have become empty, so it returns out-of-date content if
you select data from them. In MySQL 5.0, when a
MEMORY
table is used on the master for the
first time since the master was started, a DELETE
FROM
statement is written to the master's binary log
automatically, thus synchronizing the slave to the master
again. Note that even with this strategy, the slave still has
outdated data in the table during the interval between the
master's restart and its first use of the table. However, if
you use the --init-file
option to populate
the MEMORY
table on the master at startup,
it ensures that this time interval is zero.
The memory needed for one row in a MEMORY
table is calculated using the following expression:
SUM_OVER_ALL_BTREE_KEYS(max_length_of_key
+ sizeof(char*) * 4) + SUM_OVER_ALL_HASH_KEYS(sizeof(char*) * 2) + ALIGN(length_of_row
+1, sizeof(char*))
ALIGN()
represents a round-up factor to
cause the row length to be an exact multiple of the
char
pointer size.
sizeof(char*)
is 4 on 32-bit machines and 8
on 64-bit machines.
For the MEMORY
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?92.
Sleepycat Software has provided MySQL with the Berkeley DB
transactional storage engine. This storage engine typically is
called BDB
for short. Support for the
BDB
storage engine is included in MySQL source
distributions is activated in MySQL-Max binary distributions.
BDB
tables may have a greater chance of
surviving crashes and are also capable of
COMMIT
and ROLLBACK
operations on transactions. The MySQL source distribution comes
with a BDB
distribution that is patched to make
it work with MySQL. You cannot use a non-patched version of
BDB
with MySQL.
We at MySQL AB work in close cooperation with Sleepycat to keep the quality of the MySQL/BDB interface high. (Even though Berkeley DB is in itself very tested and reliable, the MySQL interface is still considered gamma quality. We continue to improve and optimize it.)
When it comes to support for any problems involving
BDB
tables, we are committed to helping our
users locate the problem and create reproducible test cases. Any
such test case is forwarded to Sleepycat, who in turn help us find
and fix the problem. As this is a two-stage operation, any
problems with BDB
tables may take a little
longer for us to fix than for other storage engines. However, we
anticipate no significant difficulties with this procedure because
the Berkeley DB code itself is used in many applications other
than MySQL.
For general information about Berkeley DB, please visit the Sleepycat Web site, http://www.sleepycat.com/.
Currently, we know that the BDB
storage
engine works with the following operating systems:
Linux 2.x Intel
Sun Solaris (SPARC and x86)
FreeBSD 4.x/5.x (x86, sparc64)
IBM AIX 4.3.x
SCO OpenServer
SCO UnixWare 7.1.x
Windows NT/2000/XP
BDB
does not work with
the following operating systems:
Linux 2.x Alpha
Linux 2.x AMD64
Linux 2.x IA-64
Linux 2.x s390
Mac OS X
Note: The preceding lists are not complete. We update them as we receive more information.
If you build MySQL from source with support for
BDB
tables, but the following error occurs
when you start mysqld, it means
BDB
is not supported for your architecture:
bdb: architecture lacks fast mutexes: applications cannot be threaded Can't init databases
In this case, you must rebuild MySQL without
BDB
table support or start the server with
the --skip-bdb
option.
If you have downloaded a binary version of MySQL that includes
support for Berkeley DB, simply follow the usual binary
distribution installation instructions. (MySQL-Max distributions
include BDB
support.)
If you build MySQL from source, you can enable
BDB
support by running
configure with the
--with-berkeley-db
option in addition to any
other options that you normally use. Download a MySQL
5.0 distribution, change location into its
top-level directory, and run this command:
shell> ./configure --with-berkeley-db [other-options
]
For more information, see Section 2.7, “Installing MySQL on Other Unix-Like Systems”, Section 5.1.2, “The mysqld-max Extended MySQL Server”, and Section 2.8, “MySQL Installation Using a Source Distribution”.
The following options to mysqld can be used
to change the behavior of the BDB
storage
engine:
--bdb-home=
path
The base directory for BDB
tables. This
should be the same directory you use for
--datadir
.
--bdb-lock-detect=
method
The BDB
lock detection method. The option
value should be DEFAULT
,
OLDEST
, RANDOM
, or
YOUNGEST
.
--bdb-logdir=
path
The BDB
log file directory.
--bdb-no-recover
Do not start Berkeley DB in recover mode.
--bdb-no-sync
Don't synchronously flush the BDB
logs.
This option is deprecated; use
--skip-sync-bdb-logs
instead (see the
description for --sync-bdb-logs
).
--bdb-shared-data
Start Berkeley DB in multi-process mode. (Do not use
DB_PRIVATE
when initializing Berkeley
DB.)
--bdb-tmpdir=
path
The BDB
temporary file directory.
--skip-bdb
Disable the BDB
storage engine.
--sync-bdb-logs
Synchronously flush the BDB
logs. This
option is enabled by default; use
--skip-sync-bdb-logs
to disable it.
See Section 5.3.1, “mysqld Command-Line Options”.
If you use the --skip-bdb
option, MySQL does
not initialize the Berkeley DB library and this saves a lot of
memory. However, if you use this option, you cannot use
BDB
tables. If you try to create a
BDB
table, MySQL creates a
MyISAM
table instead.
Normally, you should start mysqld without the
--bdb-no-recover
option if you intend to use
BDB
tables. However, this may cause problems
when you try to start mysqld if the
BDB
log files are corrupted. See
Section 2.9.2.3, “Starting and Troubleshooting the MySQL Server”.
With the bdb_max_lock
variable, you can
specify the maximum number of locks that can be active on a
BDB
table. The default is 10,000. You should
increase this if errors such as the following occur when you
perform long transactions or when mysqld has
to examine many rows to execute a query:
bdb: Lock table is out of available locks Got error 12 from ...
You may also want to change the
binlog_cache_size
and
max_binlog_cache_size
variables if you are
using large multiple-statement transactions. See
Section 5.11.3, “The Binary Log”.
See also Section 5.3.3, “Server System Variables”.
Each BDB
table is stored on disk in two
files. The files have names that begin with the table name and
have an extension to indicate the file type. An
.frm
file stores the table definition, and
a .db
file contains the table data and
indexes.
To specify explicitly that you want a BDB
table, indicate that with an ENGINE
or
TYPE
table option:
CREATE TABLE t (i INT) ENGINE = BDB; CREATE TABLE t (i INT) TYPE = BDB;
BerkeleyDB
is a synonym for
BDB
in the ENGINE
or
TYPE
option.
The BDB
storage engine provides transactional
tables. The way you use these tables depends on the autocommit
mode:
If you are running with autocommit enabled (which is the
default), changes to BDB
tables are
committed immediately and cannot be rolled back.
If you are running with autocommit disabled, changes do not
become permanent until you execute a
COMMIT
statement. Instead of committing,
you can execute ROLLBACK
to forget the
changes.
You can start a transaction with the BEGIN
WORK
statement to suspend autocommit, or with
SET AUTOCOMMIT=0
to disable autocommit
explicitly.
See Section 13.4.1, “START TRANSACTION
, COMMIT
, and ROLLBACK
Syntax”.
The BDB
storage engine has the following
characteristics:
In MySQL 5.0, BDB
tables can
have up to 31 indexes per table, 16 columns per index, and a
maximum key size of 1024 bytes.
MySQL requires a PRIMARY KEY
in each
BDB
table so that each row can be
uniquely identified. If you don't create one explicitly,
MySQL creates and maintains a hidden PRIMARY
KEY
for you. The hidden key has a length of five
bytes and is incremented for each insert attempt. This key
does not appear in the output of SHOW CREATE
TABLE
or DESCRIBE
.
The PRIMARY KEY
is faster than any other
index, because the PRIMARY KEY
is stored
together with the row data. The other indexes are stored as
the key data + the PRIMARY KEY
, so it's
important to keep the PRIMARY KEY
as
short as possible to save disk space and get better speed.
This behavior is similar to that of
InnoDB
, where shorter primary keys save
space not only in the primary index but in secondary indexes
as well.
If all columns you access in a BDB
table
are part of the same index or part of the primary key, MySQL
can execute the query without having to access the actual
row. In a MyISAM
table, this can be done
only if the columns are part of the same index.
Sequential scanning is slower than for
MyISAM
tables because the data in
BDB
tables is stored in B-trees and not
in a separate data file.
Key values are not prefix- or suffix-compressed like key
values in MyISAM
tables. In other words,
key information takes a little more space in
BDB
tables compared to
MyISAM
tables.
There are often holes in the BDB
table to
allow you to insert new rows in the middle of the index
tree. This makes BDB
tables somewhat
larger than MyISAM
tables.
SELECT COUNT(*) FROM
is slow for
tbl_name
BDB
tables, because no row count is
maintained in the table.
The optimizer needs to know the approximate number of rows
in the table. MySQL solves this by counting inserts and
maintaining this in a separate segment in each
BDB
table. If you don't issue a lot of
DELETE
or ROLLBACK
statements, this number should be accurate enough for the
MySQL optimizer. However, MySQL stores the number only on
close, so it may be incorrect if the server terminates
unexpectedly. It should not be fatal even if this number is
not 100% correct. You can update the row count by using
ANALYZE TABLE
or OPTIMIZE
TABLE
. See Section 13.5.2.1, “ANALYZE TABLE
Syntax” and
Section 13.5.2.5, “OPTIMIZE TABLE
Syntax”.
Internal locking in BDB
tables is done at
the page level.
LOCK TABLES
works on
BDB
tables as with other tables. If you
do not use LOCK TABLES
, MySQL issues an
internal multiple-write lock on the table (a lock that does
not block other writers) to ensure that the table is
properly locked if another thread issues a table lock.
To be able to roll back transactions, the
BDB
storage engine maintains log files.
For maximum performance, you can use the
--bdb-logdir
option to place the
BDB
logs on a different disk than the one
where your databases are located.
MySQL performs a checkpoint each time a new
BDB
log file is started, and removes any
BDB
log files that are not needed for
current transactions. You can also use FLUSH
LOGS
at any time to checkpoint the Berkeley DB
tables.
For disaster recovery, you should use table backups plus MySQL's binary log. See Section 5.9.1, “Database Backups”.
Warning: If you delete old
log files that are still in use, BDB
is
not able to do recovery at all and you may lose data if
something goes wrong.
Applications must always be prepared to handle cases where
any change of a BDB
table may cause an
automatic rollback and any read may fail with a deadlock
error.
If you get a full disk with a BDB
table,
you get an error (probably error 28) and the transaction
should roll back. This contrasts with
MyISAM
tables, for which
mysqld waits for sufficient free disk
space before continuing.
Opening many BDB
tables at the same time
may be quite slow. If you are going to use
BDB
tables, you should not have a very
large table cache (for example, with a size larger than 256)
and you should use the --no-auto-rehash
option when you use the mysql client.
SHOW TABLE STATUS
does not provide some
information for BDB
tables:
mysql> SHOW TABLE STATUS LIKE 'bdbtest'\G
*************************** 1. row ***************************
Name: bdbtest
Engine: BerkeleyDB
Version: 10
Row_format: Dynamic
Rows: 154
Avg_row_length: 0
Data_length: 0
Max_data_length: 0
Index_length: 0
Data_free: 0
Auto_increment: NULL
Create_time: NULL
Update_time: NULL
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:
Optimize performance.
Change to use no page locks for table scanning operations.
The following list indicates restrictions that you must observe
when using BDB
tables:
Each BDB
table stores in the
.db
file the path to the file as it was
created. This was done enable detection of locks in a
multi-user environment that supports symlinks. As a
consequence of this, it is not possible to move
BDB
table files from one database
directory to another.
When making backups of BDB
tables, you
must either use mysqldump or else make a
backup that includes the files for each
BDB
table (the .frm
and .db
files) as well as the
BDB
log files. The BDB
storage engine stores unfinished transactions in its log
files and requires them to be present when
mysqld starts. The BDB
logs are the files in the data directory with names of the
form log.XXXXXXXXXX
(ten digits).
If a column that allows NULL
values has a
unique index, only a single NULL
value is
allowed. This differs from other storage engines.
If the following error occurs when you start
mysqld after upgrading, it means that the
new BDB
version doesn't support the old
log file format:
bdb: Ignoring log file: .../log.XXXXXXXXXX: unsupported log version #
In this case, you must delete all BDB
logs from your data directory (the files with names that
have the format log.XXXXXXXXXX
) and
restart mysqld. We also recommend that
you then use mysqldump --opt to dump your
BDB
tables, drop the tables, and restore
them from the dump file.
If autocommit mode is disabled and you drop a
BDB
table that is referenced in another
transaction, you may get error messages of the following
form in your MySQL error log:
001119 23:43:56 bdb: Missing log fileid entry 001119 23:43:56 bdb: txn_abort: Log undo failed for LSN: 1 3644744: Invalid
This is not fatal, but until the problem is fixed, we
recommend that you not drop BDB
tables
except while autocommit mode is enabled. (The fix is not
trivial.)
The EXAMPLE
storage engine is a stub engine
that does nothing. Its purpose is to serve as an example in the
MySQL source code that illustrates how to begin writing new
storage engines. As such, it is primarily of interest to
developers.
To examine the source for the EXAMPLE
engine,
look in the sql/examples
directory of a MySQL
5.0 source distribution.
To enable this storage engine, use the
--with-example-storage-engine
option to
configure when you build MySQL.
When you create an EXAMPLE
table, the server
creates a table definition file in the database directory. The
file begins with the table name and has an
.frm
extension. No other files are created.
No data can be stored into the table or retrieved from it.
mysql>CREATE TABLE test (i INT) ENGINE = EXAMPLE;
Query OK, 0 rows affected (0.78 sec) mysql>INSERT INTO test VALUES(1),(2),(3);
ERROR 1031 (HY000): Table storage engine for 'test' doesn't have this option mysql>SELECT * FROM test;
Empty set (0.31 sec)
The EXAMPLE
storage engine does not support
indexing.
The FEDERATED
storage engine is available
beginning with MySQL 5.0.3. It is a storage engine that accesses
data in tables of remote databases rather than in local tables.
The FEDERATED
storage engine is available only
in the -Max
version of MySQL.
To examine the source for the FEDERATED
engine,
look in the sql
directory of a source
distribution for MySQL 5.0.3 or newer.
For the FEDERATED
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?105.
To enable this storage engine, use the
--with-federated-storage-engine
option to
configure when you build MySQL.
When you create a FEDERATED
table, the server
creates a table definition file in the database directory. The
file begins with the table name and has an
.frm
extension. No other files are created,
because the actual data is in a remote database. This differs
from the way that storage engines for local tables work.
For local database tables, data files are local. For example, if
you create a MyISAM
table named
users
, the MyISAM
handler
creates a data file named users.MYD
. A
handler for local tables reads, inserts, deletes, and updates
data in local data files, and records are stored in a format
particular to the handler. To read records, the handler must
parse data into columns. To write records, column values must be
converted to the row format used by the handler and written to
the local data file.
With the MySQL FEDERATED
storage engine,
there are no local data files for a table (for example, there is
no .MYD
file). Instead, a remote database
stores the data that normally would be in the table. This
necessitates the use of the MySQL client API to read, delete,
update, and insert data. Data retrieval is initiated via a
SELECT * FROM
SQL statement. To
read the result, rows are fetched one at a time by using the
tbl_name
mysql_fetch_row()
C API function, and then
converted from the columns in the SELECT
result set to the format that the FEDERATED
handler expects.
The basic flow is as follows:
SQL calls issued locally
MySQL handler API (data in handler format)
MySQL client API (data converted to SQL calls)
Remote database -> MySQL client API
Convert result sets (if any) to handler format
Handler API -> Result rows or rows-affected count to local
The procedure for using FEDERATED
tables is
very simple. Normally, you have two servers running, either both
on the same host or on different hosts. (It is also possible for
a FEDERATED
table to use another table that
is managed by the same server, though there is little point in
doing so.)
First, you must have a table on the remote server that you want
to access with the FEDERATED
table. Suppose
that the remote table is in the federated
database and is defined like this:
CREATE TABLE test_table ( id int(20) NOT NULL auto_increment, name varchar(32) NOT NULL default '', other int(20) NOT NULL default '0', PRIMARY KEY (id), KEY name (name), KEY other_key (other) ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
The ENGINE
table option could name any
storage engine; the table need not be a
MyISAM
table.
Next, create a FEDERATED
table on the local
server for accessing the remote table:
CREATE TABLE federated_table ( id int(20) NOT NULL auto_increment, name varchar(32) NOT NULL default '', other int(20) NOT NULL default '0', PRIMARY KEY (id), KEY name (name), KEY other_key (other) ) ENGINE=FEDERATED DEFAULT CHARSET=latin1 CONNECTION='mysql://root@remote_host:9306/federated/test_table';
(Before MySQL 5.0.13, use COMMENT
rather than
CONNECTION
.)
The structure of this table must be exactly the same as that of
the remote table, except that the ENGINE
table option should be FEDERATED
and the
CONNECTION
table option is a connection
string that indicates to the FEDERATED
engine
how to connect to the remote server.
The FEDERATED
engine creates only the
test_table.frm
file in the
federated
database.
The remote host information indicates the remote server to which
your local server connects, and the database and table
information indicates which remote table to use as the data
file. In this example, the remote server is indicated to be
running as remote_host
on port 9306, so you
want to start that server so that it listens to port 9306.
The general form of the connection string in the
CONNECTION
option is as follows:
scheme
://user_name
[:password
]@host_name
[:port_num
]/db_name
/tbl_name
Only mysql
is supported as the
scheme
at this point; the password
and port number are optional.
Here are some example connection strings:
CONNECTION='mysql://username:password@hostname:port/database/tablename' CONNECTION='mysql://username@hostname/database/tablename' CONNECTION='mysql://username:password@hostname/database/tablename'
The use of CONNECTION
for specifying the
connection string is non-optimal and is likely to change in
future. Keep this in mind when you use
FEDERATED
tables, because it means that
modifications are likely to be required when that happens.
Because any password used is stored in the connection string as
plain text, it can be seen by any user who can use SHOW
CREATE TABLE
or SHOW TABLE STATUS
for the FEDERATED
table, or query the
TABLES
table in the
INFORMATION_SCHEMA
database.
For the FEDERATED
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?105.
What the FEDERATED
storage engine does and
does not support:
In the first version, the remote server must be a MySQL
server. Support by FEDERATED
for other
database engines may be be added in the future.
The remote table that a FEDERATED
table
points to must exist before you try to
access the table through the FEDERATED
table.
It is possible for one FEDERATED
table to
point to another, but you must be careful not to create a
loop.
There is no support for transactions.
There is no way for the FEDERATED
engine
to know if the remote table has changed. The reason for this
is that this table must work like a data file that would
never be written to by anything other than the database. The
integrity of the data in the local table could be breached
if there was any change to the remote database.
The FEDERATED
storage engine supports
SELECT
, INSERT
,
UPDATE
, DELETE
, and
indexes. It does not support ALTER TABLE
,
DROP TABLE
, or any other Data Definition
Language statements. The current implementation does not use
Prepared statements.
The implementation uses SELECT
,
INSERT
, UPDATE
, and
DELETE
, but not
HANDLER
.
FEDERATED
tables do not work with the
query cache.
Some of these limitations may be lifted in future versions of
the FEDERATED
handler.
The ARCHIVE
storage engine is used for storing
large amounts of data without indexes in a very small footprint.
To enable this storage engine, use the
--with-archive-storage-engine
option to
configure when you build MySQL. You can see if
this storage engine is available with this statement:
mysql> SHOW VARIABLES LIKE 'have_archive';
When you create an ARCHIVE
table, the server
creates a table definition file in the database directory. The
file begins with the table name and has an
.frm
extension. The storage engine creates
other files, all having names beginning with the table name. The
data and metadata files have extensions of
.ARZ
and .ARM
. A
.ARN
file may appear during optimization
operations.
The ARCHIVE
engine supports only
INSERT
and SELECT
(no
deletes, replaces, or updates). It does support ORDER
BY
operations, BLOB
fields, and
basically all data types except geometry data types (see
Section 17.4.1, “MySQL Spatial Data Types”). The
ARCHIVE
engine uses row-level locking.
Storage: Records are compressed
as they are inserted. The ARCHIVE
engine uses
zlib lossless data
compression. Use of OPTIMIZE TABLE
can analyze
the table and pack it into a smaller format (for a reason to use
OPTIMIZE TABLE
, see below). Beginning with
MySQL 5.0.15, the ARCHIVE
engine supports
CHECK TABLE
. There are several types of
insertions that are used:
A straight INSERT
just pushes rows into a
compression buffer, and that buffer flushes as it needs. The
insertion into the buffer is protected by a lock. A
SELECT
forces a flush to occur, unless the
only insertions that have come in were INSERT
DELAYED
(those flush as need be). See
Section 13.2.4.2, “INSERT DELAYED
Syntax”.
A bulk insert is only visible after it completes, unless other
inserts occur at the same time, in which case it can be seen
partially. A SELECT
never causes a flush of
a bulk insert unless a normal insert occurs while it is
loading.
Retrieval: On retrieval, records
are uncompressed on demand; there is no row cache. A
SELECT
operation performs a complete table
scan: When a SELECT
occurs, it finds out how
many rows are currently available and reads that number of rows.
SELECT
is performed as a consistent read. Note
that lots of SELECT
statements during insertion
can deteriorate the compression, unless only bulk or delayed
inserts are used. To fix any compression issues that have occurred
you can always do an OPTIMIZE TABLE
(REPAIR TABLE
also is supported). The number of
rows in ARCHIVE
tables reported by
SHOW TABLE STATUS
is always accurate. See
Section 13.5.2.6, “REPAIR TABLE
Syntax”, Section 13.5.2.5, “OPTIMIZE TABLE
Syntax”,
Section 13.5.4.18, “SHOW TABLE STATUS
Syntax”.
For the ARCHIVE
storage engine, there's a
dedicated forum available on
http://forums.mysql.com/list.php?112.
The CSV
storage engine stores data in text
files using comma-separated values format.
To enable this storage engine, use the
--with-csv-storage-engine
option to
configure when you build MySQL.
When you create a CSV
table, the server creates
a table definition file in the database directory. The file begins
with the table name and has an .frm
extension. The storage engine also creates a data file. Its name
begins with the table name and has a .CSV
extension. The data file is a plain text file. When you store data
into the table, the storage engine saves it into the data file in
CSV format.
mysql>CREATE TABLE test(i INT, c CHAR(10)) ENGINE = CSV;
Query OK, 0 rows affected (0.12 sec) mysql>INSERT INTO test VALUES(1,'record one'),(2,'record two');
Query OK, 2 rows affected (0.00 sec) Records: 2 Duplicates: 0 Warnings: 0 mysql>SELECT * FROM test;
+------+------------+ | i | c | +------+------------+ | 1 | record one | | 2 | record two | +------+------------+ 2 rows in set (0.00 sec)
If you examine the test.CSV
file in the
database directory created by executing the preceding statements,
its contents should look like this:
"1","record one" "2","record two"
The CSV
storage engine does not support
indexing.
The BLACKHOLE
storage engine acts as a
“black hole” that accepts data but throws it away and
does not store it. Retrievals always return the empty set:
mysql>CREATE TABLE test(i INT, c CHAR(10)) ENGINE = BLACKHOLE;
Query OK, 0 rows affected (0.03 sec) mysql>INSERT INTO test VALUES(1,'record one'),(2,'record two');
Query OK, 2 rows affected (0.00 sec) Records: 2 Duplicates: 0 Warnings: 0 mysql>SELECT * FROM test;
Empty set (0.00 sec)
When you create a BLACKHOLE
table, the server
creates a table definition file in the database directory. The
file begins with the table name and has an
.frm
extension. There are no other files
associated with the table.
The BLACKHOLE
storage engine supports all kinds
of indexing.
To enable this storage engine, use the
--with-blackhole-storage-engine
option to
configure when you build MySQL. The
BLACKHOLE
storage engine is available in
MySQL-supplied server binaries; you can determine whether or not
your version supports this engine by viewing the output of
SHOW ENGINES
or SHOW VARIABLES LIKE
'have%'
.
Inserts into a BLACKHOLE
table do not store any
data, but if the binary log is enabled, the SQL statements are
logged (and replicated to slave servers). This can be useful as a
repeater or filter mechanism. For example, suppose that your
application requires slave-side filtering rules, but transfering
all binlog data to the slave first results in too much traffic. In
such a case, it is possible to set up on the master host a
“dummy” slave process whose default storage engine is
BLACKHOLE
, depicted as follows:
The master writes to its binary log. The “dummy”
mysqld
process acts as a slave, applying the
desired combination of replicate-do
and
replicate-ignore
rules, and writes a new,
filtered binlog of its own. (See
Section 6.8, “Replication Startup Options”.) This filtered log is
provided to the slave.
Since the dummy process does not actually store any data, there is
little processing over head incurred by running the additional
mysqld
process on the replication master host.
This type of setup can be repeated with additional replication
slaves.
Other possible uses for the BLACKHOLE
storage
engine include:
Verification of dumpfile syntax.
Measurement of the overhead from binary logging, by
comparing performance using BLACKHOLE
with and without binary logging enabled.
since BLACKHOLE
is essentially a
“no-op” storage engine, it could be used for
finding performance bottlenecks not related to the storage
engine itself.