Squat C Template Library Overview


The C Template Library is a system that serves approximately the same role as the C++ Standard Template Library for C, and to extend that role to include some more useful forms of code generation, which will shortly be ported over to C++, and probably added to the 'Boost C++' library, sooner or later. (This document was generated from ctldef.c)

Code generators support records in such a way that you don't have to write much of the initialization, destruction, serialization, XML, etc. kinds of repetitive code for your data to have useful features. This frees a lot of effort from ongoing development as you can rest assured that once you've declared your record (in its own curious format), you won't have to write all of the glue that makes creating, destroying, copying, saving, loading, passing around, etc. the record is written for you.

Two kinds of serialization and XML are provided. One serialization is called 'record', and the other is simply called 'serial'. The 'record' version labels and tags things such that certain kinds of changes can be made to the underlying structures and code without invalidating the data that was saved from a previous version. For instance, a stored integer could be turned into a floating point value, or a member can be added or removed from a structure, or an array could be replaced with a different container set, and the data can still be read. The 'serial' version is basically just the raw data concatenated together on byte boundaries, with a few lengths and counts thrown in to keep things safe, and is intended for situations where two versions of the software will definitely match. XML is well, just an XML-ish text data format written to be as simple as possible, and is well suited for saving/loading user editable configuration data, exporting to databases (with lots of caveats, according to the import process), or just dumping readable logs of information which CAN in fact be used to drive the same process to repeat certain actions, such as when testing a server's reaction to certain inputs, the XML data could be used to record packets, allow a programmer to tinker with their contents, and replay them to the server.

Records may contain containers, unions, and other records made with or made compatible to the system. The data is patterned and bound together such that members of a record are properly initialized or destroyed when the record is initialized or destroyed. Also correct record copying (allocating buffers and initializing other record types) and record comparison and validation functions are provided.

Both 'record' and 'XML' formats provide some flexibility for data, allowing certain classes of changes to the data to read data from the older to the newer code. Both are intended for file/database records, rather larger than the 'serial' mode, which is intended for serial communications, but both will also compress much smaller. From the perspective of this library, the binary records are smaller and faster than XML to decode, while XML is very useful for user editable configuration data, or viewing dumps of data, or half-baked compatibility with other XML based doohickies.

The primary place to start when looking for examples is in the unit project, which contains the unit tests for the various elements of the library. These tests are mostly functional, but have some value as examples for use.

Overview Of Parts

The distribution comes in several directories.


Unzip the library somewhere and build it. For Linux or Cygwin, use the Jamfile. For Visual Studio, use the project files. Run Doxygen on the Doxyfile to generate pretty documents, if groping around for comments isn't for you. Doxygen also happens to be the reason for all the curious backslashes and notations and stuff in the comments, basically everywhere. If you're not using doxygen, or something very much like it for your code documentation, you should be.

Step 1: Getting Started

Once you've built it, decide how you'll integrate it into your project.

Hopefully, you haven't started on your project yet, so you should have relative freedom to carry on developent by adding a new folder(s), Jamfile(s) and/or Visual Studio project(s) to it. I use 'jam' instead of 'make' because it's just less kinky. So far, there haven't been 1001 versions of 'Jamfile' syntax evolved, whereas there are seemingly that many and more 'Makefile' solutions, all incompatible with each other. Jam is based on building projects with lots of parts in a hierarchy, and it encapsulates a lot of mundane platform-specific garbage a Makefile would otherwise need to be cluttered with.

Data Generation Patterns

Basically, the pattern used by this library for data generation is 'begin-body-end'. Generally, the label for what a piece of data is, is specified first within any given line of definition. This is because the label is the most significant thing. A scalar may change from one type to another, but if it's 'y', it's probably still 'y'.

// If you write this in some header named "datatest.h"...
#include "ctl/dg_patterns.h"
#include "unit/datatest.h"

	record_var(	z,	float32,	0.0f ) // Z is first because it's most significant for comparisons 
	record_var(	y,	float32,	0.0f )
	record_var(	x,	float32,	0.0f )

// You get this...
typedef struct XYZ
	float32 z;
	float32 y;
	float32 x;
} XYZ;
void XYZ_init (XYZ * self);
void XYZ_copy (XYZ * dst, const XYZ * src);
void XYZ_move (XYZ * dst, XYZ * src);
void XYZ_array_qsort (XYZ * low, XYZ * high, int (*compare) (const XYZ * p1, const XYZ * p2));
int XYZ_compare (const XYZ * p1, const XYZ * p2);
int XYZ_rcompare (const XYZ * p1, const XYZ * p2);
void XYZ_destroy (XYZ * self);
bool XYZ_valid (const XYZ * self);
size_t XYZ_size (const XYZ * self);
size_t XYZ_serial_size (const XYZ * self);
void XYZ_serial_write (const XYZ * self, ctl_serial * serial);
void XYZ_serial_read (XYZ * self, ctl_serial * serial);
size_t XYZ_record_size (const XYZ * self, const char *szlabel);
void XYZ_record_write (const XYZ * self, ctl_serial * serial, const char *szlabel);
bool XYZ_record_read (XYZ * self, ctl_serial * serial, const char *szlabel);
bool XYZ_xml_read (XYZ * self, ctl_xmlread * xml, const char *szlabel);
void XYZ_xml_write (const XYZ * self, ctl_xmlwrite * xml, const char *szlabel);

// ... and the correct implementations for each where you type this into a C file.
#define ctl_declarations "datatest.h"
#include "ctl/dg_patterns.temp.h"

// If you add this...
	record_string( name, char, 32, "unnamed" )
	record_container( stars, starmap )

// You also get this... 
typedef struct starmap_node
	ctl_tree_after_unique_node link;
	XYZ obj;
} starmap_node;
typedef ctl_tree_after_unique starmap;
extern ctl_pool_base starmap_pool;
typedef XYZ starmap_type;
typedef ctl_tree_after_unique_iterator starmap_iterator;
void starmap_init (starmap * self);
void starmap_copy (starmap * dst, const starmap * src);
void starmap_move (starmap * dst, starmap * src);
void starmap_array_qsort (starmap * low, starmap * high, int (*compare) (const starmap * p1, const starmap * p2));
int starmap_compare (const starmap * p1, const starmap * p2);
int starmap_rcompare (const starmap * p1, const starmap * p2);
void starmap_destroy (starmap * self);
bool starmap_valid (const starmap * self);
size_t starmap_size (const starmap * self);
size_t starmap_serial_size (const starmap * self);
void starmap_serial_write (const starmap * self, ctl_serial * serial);
void starmap_serial_read (starmap * self, ctl_serial * serial);
size_t starmap_record_size (const starmap * self, const char *szlabel);
void starmap_record_write (const starmap * self, ctl_serial * serial, const char *szlabel);
bool starmap_record_read (starmap * self, ctl_serial * serial, const char *szlabel);
bool starmap_xml_read (starmap * self, ctl_xmlread * xml, const char *szlabel);
void starmap_xml_write (const starmap * self, ctl_xmlwrite * xml, const char *szlabel);
void starmap_sort (starmap * self, int (*compare) (const XYZ * p1, const XYZ * p2));
size_t starmap_count (const starmap * self);
bool starmap_ismember (const starmap * self, const XYZ * mbr);
void starmap_clear (starmap * self);
XYZ *starmap_back (const starmap * self);
XYZ *starmap_front (const starmap * self);
void starmap_pop_back (starmap * self);
void starmap_pop_front (starmap * self);
XYZ *starmap_insert (starmap * self, const XYZ * key);
XYZ *starmap_at (const starmap * self, const XYZ * key);
void starmap_erase (starmap * self, XYZ * mbr);
void starmap_erase_key (starmap * self, XYZ * key);
void starmap_splice (starmap * self, starmap * from);

typedef struct Space
	char name[32];
	starmap stars;
} Space;
void Space_init (Space * self);
void Space_copy (Space * dst, const Space * src);
void Space_move (Space * dst, Space * src);
void Space_array_qsort (Space * low, Space * high, int (*compare) (const Space * p1, const Space * p2));
int Space_compare (const Space * p1, const Space * p2);
int Space_rcompare (const Space * p1, const Space * p2);
void Space_destroy (Space * self);
bool Space_valid (const Space * self);
size_t Space_size (const Space * self);
size_t Space_serial_size (const Space * self);
void Space_serial_write (const Space * self, ctl_serial * serial);
void Space_serial_read (Space * self, ctl_serial * serial);
size_t Space_record_size (const Space * self, const char *szlabel);
void Space_record_write (const Space * self, ctl_serial * serial, const char *szlabel);
bool Space_record_read (Space * self, ctl_serial * serial, const char *szlabel);
bool Space_xml_read (Space * self, ctl_xmlread * xml, const char *szlabel);
void Space_xml_write (const Space * self, ctl_xmlwrite * xml, const char *szlabel);

// ... and the correct implementations for each where you type this into a C file. 
#define ctl_declarations "unit/datatest.h"
#include "ctl/dg_patterns.temp.h"

See Also: datagen.h datagen.c dg_patterns.h dg_patterns.temp.h datatest.h

Each kind of definition builds on the assumptions from the previous patterns already established. If you fill up the star map with XYZ 'stars', all of those will be correctly serialized or read/written as XML when you invoke the appropriare functions on Space, because they're part of space. Functions to correctly initialize and destroy, copy and sort are also provided. Since the XYZ items are defined as z-first, the stars will be automatically sorted by z,y,x by default when added to the set. The sort order of the set can be changed by invoking starmap_sort, even after there is data in the set. If you wanted a record defined named 'Space' with sorted stars, you are finished with everything but the application for what to do with the stars. Even better, if you change your mind, you can make a 'star' record with a color, make the set contain that, and all of this code will be re-manufactured to make sure stars with colors are handled correctly, saved to files correctly, loaded from files correctly, etc.

If it seems like a lot of code, it is. Thousands of lines of repetitive code you didn't have to write, and won't have to debug (if the library is working right) or maintain. Of course, if you don't use something, any modern compiler's linker is 'smart' enough not to link it into the executable until some code references it by name.

Scalars are defined by int/uint for signed/unsigned integer, and float, followed by a count of bits. If it wasn't repetitively defined in datagen.h, it's not going to work. For example, though it's tempting to add 'int' to the supported types, just what is an int, anyway? For different compilers and platforms, an 'int' could be any size that represents the native register size, or an OS dependant size, or just whatever some pointy haired boss dictated it should be.

Compounds are defined with the 'begin-body-end' pattern established above. Compounds may contain compounds and a mixture of other kinds of data, so long as that data is DEFINED within this system, or DEFINED just like this system expects. The ctl/dg_patterns.h file contains a macro called ctl_datagen_declare_functions that basically gives you a to-do list of what needs to exist for any data type to be fully compatible with this system. The ctl/datagen.h file contains another macro called DEF_ENUMTYPES that shows what scalars are supported.


One oddity in this library is that it uses little-endian (least significant first) byte ordering by default. This is because MOST of the machines I personally plan to run this using will be based on the Intel memory model, where it's little endian, and doesn't care if a data access is odd. This means that in many cases, the code can be...
*((*int32)serial.curr) = value;
serial.curr += 4;
... instead of the equivalent shifting and offsetting you'd otherwise need. That's potentially a lot of CPU cycles that could be used for other things. There are preprocessor directives to force the equivalent serializing for scalar data, and if you like, you can set a preprocessor flag to do 'big-endian'. It's all just code, but most big-endian machines don't generally do odd aligned reads/writes, and so they would have to do exactly the same steps whether the target data is big-endian or little-endian, so for my purposes making it little-endian seems more practical.

Another oddity in this library is that simple arrays of scalars are written by the basic serial handling as ALL of the least significant bytes for all of the members first, up towards the most significant bytes. This is because normally an array will contain a lot of values that don't express the whole range of the possible values that they could. An array of { 0001 0003 0005 0002 } becomes serial data of {01 03 05 02 00 00 00 00}. For larger types and longer arrays, the pattern becomes much longer. This compresses better, on average.

Generated on Fri Jan 2 15:28:34 2009 for Squat by  doxygen 1.5.6