Text Table support for HSQLDB was originally developed by Bob Preston independently from the Project. Subsequently Bob joined the Project and incorporated this feature into version 1.7.0, with a number of enhancements, especially the use of conventional SQL commands for specifying the files used for Text Tables.
In a nutshell, Text Tables are CSV or other delimited files treated as SQL tables. Any ordinary CSV or other delimited file can be used. The full range of SQL queries can be performed on these files, including SELECT, INSERT, UPDATE and DELETE. Indexes and unique constraints can be set up, and foreign key constraints can be used to enforce referential integrity between Text Tables themselves or with conventional tables.
HSQLDB with Text Table support is the only comprehensive solution that employs the power of SQL and the universal reach of JDBC to handle data stored in text files and will have wide-ranging use way beyond the currently established Java realm of HSQLDB.
Text Tables are defined similarly to conventional tables with the added TEXT keyword:
CREATE [TEMP] TEXT TABLE <tablename> (<column definition>… [<constraint definition>…])
In addition, a SET command specifies the file and the separator character that the Text table uses:
SET TABLE <tablename> SOURCE <quoted_filename_and_options> [DESC]
Text Tables cannot be created in memory-only databases (databases that have no script file).
From version 1.7.2 the new source file is scanned and indexes are built when it is assigned to the table. At this point any violation of NOT NULL, UNIQUE or PRIMARY KEY constrainst are caught and the assignment is aborted.
This has changed in 1.7.2 to support both null values and empty strings.
The default field separator is a comma (,). A different field separator can be specified within the SET TABLE SOURCE statement. For example, to change the field separator for the table mytable to a vertical bar, place the following in the SET TABLE SOURCE statement, for example:
SET TABLE mytable SOURCE "myfile;fs=|"
Since HSQLDB treats CHAR’s, VARCHAR’s, and LONGVARCHAR’s the same, the ability to assign different separators to the latter two is provided. When a different separator is assigned to a VARCHAR or LONGVARCHAR field, it will terminate any CSV field of that type. For example, if the first field is CHAR, and the second field LONGVARCHAR, and the separator fs has been defined as the pipe (|) and vs as the period (.) then the data in the CSV file for a row will look like:
First field data|Second field data.Third field data
The following example shows how to change the default separator to the pipe (|), VARCHAR separator to the period (.) and the LONGVARCHAR separator to the tilde (~). Place the following within the SET TABLE SOURCE statement, for example:
SET TABLE mytable SOURCE "myfile;fs=|;vs=.;lvs=~"
HSQLDB also recognises the following special indicators for separators:
\semi - semicolon
\quote - quote
\space - space character
\apos - apostrophe
\n - newline - Used as an end anchor (like $ in regular expressions)
\r - carriage return
\t - tab
\\ - backslash
\u#### - a Unicode character specified in hexadecimal
Furthermore, HSQLDB provides csv file support with three additional boolean options: ignore_first, quoted and all_quoted. (1.7.2) The ignore_first option (default false) tells HSQLDB to ignore the first line in a file. This option is used when the first line of the file contains column headings. The all_quoted option (default false) tells the program that it should use quotes around all character fields when writing to the source file. The quoted option (default true) uses quotes only when necessary to distinguish a field that contains the separator character. It can be set to false to prevent the use of quoting altogether. These options may be specified within the SET TABLE SOURCE statement:
SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true"
When the default options all_quoted=false and quoted=true are in force, fields that are written to a line of the csv file will be quoted only if they contain the separator or the quote character. The quote character is doubled when used inside a string. When all_quoted=false and quoted=false the quote character is not doubled. With this option, it is not possible to insert any string containing the separator into the table, as it would become impossible to distinguish from a separator. While reading an existing data source file, the program treats each individual field separately. It determines that a field is quoted only if the first character is the quote character. It interprets the rest of the field on this basis.
The character encoding for the source file is ASCII by default. To support UNICODE or source files preprared with different encodings this can be changed to UTF-8 or any other encoding. The default is encoding=ASCII and the option encoding=UTF-8 or other supported encodings can be used.
Finally, HSQLDB provides the ability to read a text file from the bottom up, by placing the keyword "DESC" at the end of the SET TABLE SOURCE statement:
SET TABLE mytable SOURCE "myfile" DESC
This feature provides functionality similar to the Unix tail command, by re-reading the file each time a select is executed. Using this feature sets the table to read-only mode. Afterwards, it will no longer be possible to change the read-only status with SET TABLE <tablename> READONLY TRUE.
Text table source files are cached in memory. The maximum number of rows of data that are in memory at any time is controlled by the textdb.cache_scale property. The default value for textdb.cache_scale is 10 and can be changed by setting the property in the .properties file for the database. The number of rows in memory is calculated as 3*(2**scale), which translates to 3072 rows for the default textdb.cache_scale setting (10). The property can also be set for individual text tables:
SET TABLE mytable SOURCE "myfile;ignore_first=true;all_quoted=true;cache_scale=12"
textdb.allow_full_path=true. (* deprecated: see below)
SELECT <select list> INTO TEXT <tablename> FROM …
is the directory that contains the database and the file name is based on the table name. The table name is converted into the file name by replacing all the non-alphanumeric characters with the underscore character, conversion into lowercase, and adding the ".csv" suffix.
ignore_first=true option, the first, ignored line is replaced with a blank line after a SHUTDOWN COMPACT.textdb.<tablename>.property=value will be disallowed. Only properties that affect all the text tables in the database will be allowed in the *.properties files. These global properties can then be overridden for individial tables with definitions in the SET TABLE <tablename> SOURCE strings.
The full list of supported global properties in *.properties files is as follows:
textdb.fs
textdb.vs
textdb.lvs
textdb.quoted
textdb.all_quoted
textdb.ignore_first
textdb.encoding
textdb.cache_scale
Authors: Bob Preston and Fred Toussi - 29 May 2002 - last updated by FT 15 Feb 2004
Copyright 2002-2003 Bob Preston and Fred Toussi. Permission is granted to distribute this document without any alteration under the terms of the HSQLDB license. Additional permission is granted to the HSQLDB Development Group to distribute this document with or without alterations under the terms of the HSQLDB license.