# libpqxx tutorial

## About libpqxx

libpqxx is a C++ API for PostgreSQL.

Since libpqxx builds on top of the C frontend libpq, you will need a working libpq library on your system prior to getting started with libpqxx. This is similar to the situation with the older libpq++ interface.

See the Github page for more information.

## Class overview

To work with libpqxx, you need to be aware of at least the following classes. These are all introduced in a dedicated namespace, `pqxx`:

- `connection` represents a connection from your program to a PostgreSQL back-end server. Naturally your program may open multiple connections, simultaneous or not, to any number of databases.

- `work`, actually an alias for the `transaction` class template, represents a transaction being executed in the context of a `connection`. This is a unit of work that must be completed as a whole by the database backend. If execution fails, everything done within that `transaction` is undone by the backend as if it had never happened.

  Actually libpqxx provides three classes doing this at various levels of reliability, called `nontransaction`, `transaction`, and `robusttransaction` for no, standard, and best reliability respectively. You probably shouldn't mess with `robusttransaction` though, because it comes with complications.

  You may execute any number of transactions on a single `connection`, but only one at a time.

  I recommend that you do not set up your own `transaction`s, but have them managed for you by a `transactor` instead (see below).

- A `result` is a container holding the resulting data coming out of a query or command executed in a `transaction`. It behaves just like a standard C++ container as found in the STL, although its contents can't be modified in any way. (You can assign or copy it cheaply though; it's basically a smart pointer). The rows held in the result in turn behave much like non-modifiable containers themselves.

I would also like to use this opportunity to plug the `esc` function, which you should use whenever you want to include a variable as a string in your SQL (eg. insert it into a text field in a table). See below.

There are other classes that may be of interest to you, but which you don't necessarily need to be aware of for writing programs. You will probably want to look them up at some point when it becomes necessary to figure out complex compiler warnings.

Some of the classes you may become interested in fairly quickly are:

- `broken_connection` is an exception class that is thrown if libpqxx loses its connection to the back-end. It is derived from the standard C++ exception `std::runtime_error`, and can generally be treated as such.

- `sql_error` is an exception class that is thrown if a query to the back-end fails. Like `broken_connection` it is derived from `std::runtime_error`, but it also carries information about the failed query. This may be useful for debugging.

- `icursorstream` is a way of iterating over a query result using an SQL cursor. This can be useful for selectively retrieving parts of a query, or for retrieving and processing results in chunks so your program can do useful work on one chunk while the next is still coming in over a network connection. Or you may want to use this to give progress reports to the user between chunk transfers.

- `in_doubt_error` is an exception class to indicate a rare and difficult error condition. Explaining this is a bit painful, since it invalidates some of what was said above (and is commonly said in the database world).

  In theory, a "transaction" is an integral, atomic unit of work that either fails or succeeds. This is a beautiful idea, but in practice there is nothing that cannot be broken. There is a tiny risk that, like Schroedinger's Cat, the transaction can end up in an indeterminate state. This happens when the connection to the backend is lost just when you were waiting for confirmation after completing your transaction. Your transaction may have succeeded (in fact it probably has), but if the connection is lost during just this small time window, there is no way to confirm this to your application.

  When this happens, libpqxx throws an `in_doubt_error` on the client side, which you may want to `catch` as a special case. If you do, make sure all alarm bells go off!

  See also the description for `robusttransaction`, which sacrifices some performance to minimize this risk.

- `nontransaction` is just like a regular `transaction` as far your code is concerned (except that it's not a template). The big difference is that where the latter opens a back-end transaction to keep your view and modifications of the database atomic, the former does nothing to maintain integrity. It just passes your queries directly to the database, with no enclosing transaction. This difference is expressed by the fact that the other transaction classes are derived from `dbtransaction`, an abstract class that expresses the presence of a real backend transaction, whereas `nontransaction` is not.

  This may give you better performance if your transaction does not modify the database, but if it does, may have unforeseen implications if you expect certain levels of consistency in the database--especially if other processes or threads may be modifying the database at the same time. You'll probably only want to use this in the following cases:

  - If you only want to read from the database, not modify it, and you know that the data is not going to be updated while your program is running.

  - If you are interested in always getting the very latest information out of your database, even if that means that you may not get a temporally consistent view of the database.

  - When making changes to the database schema; some operations may not be supported by the backend inside backend transactions.

- `robusttransaction` (a template just like `transaction`) is essentially the same as a `transaction`, ie. an atomic unit of work, except that it works harder to prevent `in_doubt_error`s from happening. This comes at the cost of some performance, which is one reason why this behaviour isn't the default. It also creates some extra tables in your database to keep track of what's happening.

  Use this class instead of plain old `transaction` if you get `in_doubt_error`s in your application--but only after thoroughly inspecting your code for bugs that might be causing the problem, or you'll only be hiding glitches that should be fixed instead.

- `transaction_base` defines the common public interface for the `transaction`-like classes: `nontransaction`, `robusttransaction`, and `transaction` itself. To look up methods available in these classes, see the definition (or documentation) for `transaction_base` instead.

  You'll usually want to refer to the default `transaction` type as `work`, an alias that means "`transaction` with default (read-committed) isolation level."

- `transactor<>` provides a framework that makes it easier for you to write correct, robust transaction code to be executed in a transaction. You should generally try to write your code as a class derived from `transactor<>`, but this is not required.

  A `transactor`-derived class may select its own "quality of service" in transactional integrity by choosing a transaction type as its template argument, e.g. `transactor<robusttransaction<> >`. The default `transactor<>` is equivalent to `transactor<work>`. Within your class you can refer to the transaction type used as `argument_type`.

- `notification_receiver` is an instance of the Observer design pattern. Any code connected to a backend may flag an event using the PostgreSQL `NOTIFY` command. Frontends connected to that same backend database may be listening for an event of that name to occur. A frontend using libpqxx does this by registering an observer object derived from `notification_receiver`, which will be invoked by libpqxx to handle the event when it occurs.

  Something every programmer using notification receivers in PostgreSQL should know, by the way, is that notifications are not delivered to your program while it is inside a back-end transaction. Nor are they sent until the notifying process has committed any transaction it may be in. This makes sense from a correctness point of view (otherwise your processing of the event might get rolled back just because the transaction you're in happens to fail), but may be a little confusing to the unaware programmer.

- `row` refers to a row of data within a `result` object.

- `field` refers to one field within a `row`. A `field` contains one retrieved value such as a number or a string, and can be read into variables of various types such as `int`s or `string`s.

- `result::const_iterator` allows you to enumerate the `row`s in a `result` in a similar way to the iterators defined by any standard C++ container.

- `result::row::const_iterator` similarly iterates over the `field`s in a `row`.

Finally, there are also some functions that you may want to know about, that live alone in the `pqxx` namespace without being part of any class:

- `esc`, found in the transaction classes, is the canonical way to use strings in your SQL. This function escapes any quotes, backslashes, and other characters in your input string that would otherwise cause syntax errors (or worse, security holes) in your queries when included directly.

  Use these functions whenever you want to have a variable string in your SQL queries. If you don't, you'll be left with annoying bugs or possibly even security holes in your program. Use this function, use this function, use this function. Please. Make me proud.

- `to_string` knows how to render many types of values as strings. This is used internally by `libpqxx` to convert values coming out of or going into the backend, and doesn't respect locale settings.

- `from_string` is the inverse of `to_string`. It is used by eg. `field::to()` to convert the incoming field data (which is in text format) to the desired C++ type.

## Tutorial

### Before we begin programming

#### This document

This frontend library is built on top of the `libpq` frontend library, which defines PostgreSQL's C API. Therefore this manual will sometimes refer to the `libpq` documentation rather than repeat what is said there. This was not done to annoy you, but to ensure that the libpqxx documentation remains up-to-date with any changes in the C frontend. Please bear with us.

Furthermore, if you're ever in doubt about how to use the basic features of libpqxx, take a look at the example programs provided; they're called `test000`, `test001` etc. Those should help give you an idea of how simple the library can be in practice. Don't let that keep you from submitting a request for improved documentation though; your questions define what this tutorial needs to say!

#### C++ conventions

All definitions made by libpqxx reside in a namespace `pqxx`. If you're not familiar with C++ namespaces, you have two options:

- Always import the whole of the `pqxx` namespace in your programs.

  This tells the compiler to always look in the `pqxx` namespace when trying to resolve a name that doesn't occur in the global (ie. the "default") namespace). To do this, include the directive

      		using namespace pqxx;

  near the top of each source file that uses libpqxx. This is typically best for your source files (you're probably doing it already for the `std` namespace), though it is not considered good practice in header files.

- Explicitly qualify each libpqxx name with the `pqxx` namespace.

  This will tell the compiler exactly which names are supposed to be in the `pqxx` namespace, avoiding any confusion over multiple classes or functions used in your program possibly having the same name. As long as they're in different namespaces, the compiler will be able to tell them apart without trouble. To do this, write `pqxx::connection` instead of `connection`, `pqxx::result::const_iterator` instead of `result::const_iterator`, and so on.

  This is the mode of address you will typically want to use in header files, to avoid polluting the namespaces of the actual source files with the definitions of the extra namespace.

### Setting up a connection

The very first thing to do before we can begin working with a database, is actually connect to one. We do this by creating a `connection` object that will serve as our "handle" on the connection:

    	connection Conn("dbname=test");

This gives us a connection object called `Conn`.

The `connection` constructor here takes one argument, the "connect string." This string may be used to specify which host on the network runs the database backend we wish to connect to, which database we're interested in, which user name we'll be using to log in, etc. Refer to the libpq `connect` call for a complete definition of what may go into the connect string. In this case we're connecting to a database `test` residing on the local machine. By default the client will try to connect to a server running on the local machine.

If no connection could be established, the `connection` constructor may throw an exception immediately; or it may decide to retry the connection later in case the problem is only temporary. In the latter case, an exception may occur at some later point if the problem turns out not to be so temporary after all.

The `connection` can now act as a "service counter" for our database; your client will use it to perform one or more transactions related to the database.

Connections cannot be copied or assigned. Any attempt to do so will be met with a compiler error.

As a matter of design, libpqxx will not allow you to perform queries on the `connection` directly. You will need to open a transaction instead.

See the section on transactions below.

### Performing a transaction

#### What's a Transaction?

A transaction demarcates a "unit of work" operating on the database, ie. a series of operations that must either complete successfully, or if it is aborted at any stage, must not affect the data in the database at all. Hence the process of completing a transaction is called the "commit" operation, and the process of aborting it and undoing any changes it may have made is called "rollback."

All access to the database in libpqxx must go through a transaction object, so familiarity with this class family's interface is essential.

#### Creating a transaction object

Creating a transaction object is not normally something you will have to do. The `transactor` class, discussed below, will take care of that and some other red tape besides. Still, there are cases where creating the object is unavoidable (when acting on multiple database connections simultaneously) or just plain easier to do. Please consider using a `transactor` whereever possible.

When creating a transaction, pass it the `connection` object it is to act on, and optionally an identifying name for your transaction. The name, which need not be unique but should begin with a letter and may contain only letters, digits and underscores, can be used by libpqxx to make some error messages more specific.

    	    transaction<> Xaction(Conn, "DemoTransaction");

Or, alternatively (once you get fed up with typing `transaction<>`):

    	    work Xaction(Conn, "DemoTransaction");

The lifetime of the transaction object demarcates the unit of work. Its construction marks the beginning of the transaction, and its destruction means that the transaction is ended--whether through commit or rollback.

The transaction class hierarchy is built on the principle of "explicit commit," ie. the commit operation must always be explicit in the code. If the transaction is destroyed before a commit was given, the transaction is implicitly aborted (rolled back). If your transaction makes no changes to the database, however, there is nothing to commit or roll back and the commit may safely be omitted.

Destroying the connection object while the transaction still exists is an error that may result in a program crash, although the library will try to log an error message. Transactions cannot be copied, nor assigned, nor constructed without a connection (default-constructed); attempts to do any of these will result in compile errors.

### Ending a transaction

A transaction ends either successfully through an explicit commit command, or unsuccessfully in any of a number of ways. The following are the ways to end a transaction:

- The transaction may be committed through its `commit` member function:

      	    Xaction.commit();

  The commit operation is sent to the backend at the point where the `commit` call occurs. Any exceptions generated by the database transaction will be thrown from here at the latest. The only exceptions that may be generated by `Xaction` beyond this point are related to incorrect handling of the transaction object, eg. if an attempt is made to abort `Xaction` after it has been committed, or runtime errors such as memory running out.

  As a consequence, any streams or cursors nested within the transaction (to be discussed later) must have been closed before the `commit()`. To do otherwise could possibly allow a transaction to be committed before all related actions had completed. The library will throw an exception if any streams are still open when the transaction is ended.

- A transaction is aborted if it is destroyed without having been explicitly committed:

  -     		{
        		work Xaction(Conn, "DemoTransaction");

        		// (Queries)

        		} // Xaction destroyed here

  -     		work *XactionP = new work(Conn, "DemoTransaction");

        		// (Queries)

        		delete XactionP; // Xaction destroyed here

  -     		try
        		{
        		work Xaction(Conn, "DemoTransaction");

        		// (Queries)

        		Xaction.commit();
        		// If we get here, Xaction is committed
        		}
        		catch (...)
        		{
        		// If we get here, Xaction has been rolled back
        		}

  No matter where exactly the decision to abort is made, the actual abort operation is sent to the backend when the transaction's destructor is called. If the abort fails, eg. because the network connection has been lost, no error is reported

  Throwing an exception from a destructor to report the error would have serious effects on program correctness. Never throw exceptions from a destructor.

  and the transaction will die of natural causes (either it has been closed by the backend already, or it soon will be if the connection is lost).

- If a database error occurs during the transaction, such as an SQL syntax error or lost connection to the backend, the transaction is aborted.

      	    work Xaction(Conn, "DemoTransaction");
      	    try
      	    {
      	    // (Queries)
      	    Xaction.exec("SELECT !?^H^H^H^H");	// Fails: SQL syntax error
      	    }
      	    catch (...)
      	    {
      	    }
      	    Xaction.commit(); // ERROR: Xaction has already aborted!

  For this reason, it is recommended always to include the "commit" operation inside the `try` block (if any) surrounding the transaction code, not after the `catch` block.

  Think of it as a natural extension of structural programming: the transaction is "nested" within the connection, and the transaction code can be "nested" in a `try`/`catch` block.

No more queries may be issued to the transaction regardless of how it ended; an exception will be thrown if the application attempts to continue the transaction after that time. Ending a transaction more than once is an error, except that aborting it multiple times is tolerated to facilitate error handling.

### Executing queries

So let's execute an actual query. There is no "query class" in libpqxx; we really do try to keep your life simple.

A query is executed within a transaction by passing the query string to the transaction object's `exec` method. If the query fails to complete successfully, this method will throw the appropriate exception.

The query itself is a standard C string in this case, ie. a `const char *` but you'll frequently want to use a C++ `string` to make it easy to include variables:

    	void DeleteEntry(work &T, string Table, long ID)
    	{
    	T.exec("DELETE FROM " + Table + " WHERE ID=" + to_string(ID));
    	}

In some places, even `stringstream`s will work, so you can use the full stream formatting capabilities, locales etc. in the standard C++ library to compose your SQL queries.

### Getting query results

Obviously not all queries are commands--the more common kind actually returns useful data. Result data in libpqxx are encapsulated in a `result` object, which acts as a container similar to the STL's `vector` template.

    	result R = T.exec("SELECT firstname FROM employee WHERE lastname='Ng'");

This executes a query on the database, collects all matching data, and stores it in the form of a `result`.

Two ways are provided to get at individual rows in a `result`: first, through indexing with the array index operator `[]` or the `at` member function, and second, through random-access iterators. Either will give you a `row` object that in turn can be addressed with the array index operator (or the `at` member function) or using iterators to get at its individual fields

The difference between `[]` and `at` is that the latter is guaranteed to perform bounds-checking, throwing an exception if you pass it an illegal index value. With the array index operator you may get slightly better performance, but attempting to address a nonexistent row or field will result in undefined behaviour such as program crashes or inexplicably strange results.

.

Thus, `R[0]` will return the first ("zeroth") row in R. You won't normally want to bother with `row`s though; they don't contain the actual data, but rather serve as placeholders to later tell `result` which fields to fetch when field values from the row are requested. The class exists mostly for technical reasons related to the array index operators

This is an instance of the Proxy implementation pattern, needed to allow a `result` to be indexed as if it were a two-dimensional array. C++'s array index operator doesn't allow this usage directly, but the operator can be applied to the result of a previous use of the same operator. The "result in the middle" needs to have a type of its own, though, and that's what `row` is for.

. What you'll usually do is index the row directly to get at the field you want, e.g. `R[0][0]` to get the first field of the first row.

Array indexing of rows also works with the fields' names instead of their numbers, eg.:

    	// Process employees' names one by one.  ProcessNames() doesn't know exactly
    	// what columns are going to be in R, but there must be one called "lastname".
    	void ProcessNames(result R)
    	{
    	for (result::size_type i = 0; i != R.size(); ++i)
    	Process(R[i]["lastname"]);
    	}

As for the alternative, accessing the result rows through an iterator, only `const` iterators are provided so the contents of the result cannot be modified. Use these iterators as random-access iterators like with any STL-like container:

    	for (result::const_iterator i = R.begin(); i != R.end(); ++i)
    	Process(*i);

Iterators may be incremented or decremented (whether pre- or post-), they may be added to or subtracted from to jump through the result rows; their positions may be compared (provided they point into the same `result`), and they may be dereferenced through the `*` or `->` operators.

Finally, the iterated rows' fields may be addressed using the array index operator on the iterator directly, eg. `R.begin()[0]` gets you the first field of `R`'s first row, just like `R[0][0]` would

Or `(*R.begin())[0]`. It may seem quirky to have `R.begin()` and `*R.begin()` mean the same thing, but it makes practical use of iterators a lot easier. In fact it matches how arrays and pointers work in C and C++.

.

Either way, once you've indexed the `row` you get a `field`--which is another placeholder, but this time encapsulates an actual field value in our query result. A `field` `F` also knows its column name, which can be obtained as `F.Name()`.

Again, there is more than one way to read the field's value. Let's start out with the easy one, `c_str`, which reads the value as a C string:

    	cout << "Name: " << F.c_str() << endl;

This will return the empty string (`""`) if field F has the null value. Use `is_null` to see if this is the case:

    	if (!F.is_null())
    	cout << "Name: " << F.c_str() << endl;

In practice of course, not all data is going to consist of strings. Many fields will be integer values, or decimals, or Booleans. To convert the field's value to one of these, use its `to` method. This adapts itself to the type of variable you pass it, expecting the field value to be of an appropriate form for that type. For convenience, `to` returns `false` if the field had the null value, and `true` otherwise. In the former case, the variable will retain the value it had before the call.

    	// Pay an employee his salary.  Add bonus for managers.
    	// The employee row must contain the fields
    	void PaySalary(result::const_iterator empl)
    	{
    	long id;
    	float salary;
    	bool is_manager=false;

    	// Get id.  Must never be null.
    	if (!empl[0].to(id)) throw runtime_error("No id!");

    	// Get salary.  If this employee has no salary, skip payment.
    	if (!empl[1].to(salary)) return;

    	// Get manager status.  If field is null, variable's original value (false)
    	// will be preserved.
    	empl[2].to(is_manager);

    	if (is_manager) salary += Bonus;

    	TransferMoney(id, salary);
    	}

If conversion fails, e.g. when trying to convert a floating-point value to an integer variable, `to` will throw a `runtime_error` reporting the problem in its `what` message.

### Doing Transactions Right: Transactors

Writing database code can be tricky. One of the most complicated areas is dealing with unexpected error conditions, such as losing one's connection to the database server. For long-running processes you'll frequently find yourself rewriting code for a simple transaction to make it:

- Attempt to perform the transaction.

- Check for "connection lost" errors.

- Attempt to restore the connection.

- Repeat until the transaction succeeds.

This is bad for the heart, and clutters up your code besides. The `transactor` framework will take this work out of your hands if you let it.

#### Functors

The mechanism is based on the concept of Functors, a powerful object-oriented design pattern that replaces the older practice of passing callback functions (or hooks, as they're sometimes called, or exits) to foreign code. Unlike classic callback functions, Functors provide an elegant way of maintaining custom state in your callback code, when the exact form or size of that state was not known in advance to the writer of the foreign code that will eventually invoke your callback.

Functors in C++ are simple objects that can be invoked just like functions or function pointers can, by virtue of providing the function invocation operator, `operator()`.

A simple functor could look like this:

    	  struct HelloFunctor
    	  {
    	  void operator()() { cout << "Hello World" << endl; }
    	  };

...And once an object of this functor type has been created, it can be "invoked" just as if it were a function:

    	  HelloFunctor Hi;
    	  Hi();

But the invoking code may also be foreign code knowing nothing about the type of your functor. The foreign code is usually a template, so it automatically becomes "specialized" to your type of functor when the one meets the other:

    	  template<typename FUNCTOR> void DoFunctor(const FUNCTOR &Hi)
    	  {
    	  Hi();
    	  }

The great thing about functors is that they can carry state. This is most useful when you need to pass a functor object to foreign code like `DoFunctor` above, but you need certain extra parameters to be passed to your functor that the foreign code isn't going to pass. The classic C solution to this problem is to let you pass both a function pointer and a pointer-to-`void` as a method of letting you provide any type of data of your liking to it. This generally makes programs a little harder to read, and is not very safe or convenient.

With functors, there is a better way to do the same thing. Let's say you want to adapt the `HelloFunctor` class to print its output to a different output stream. What you'd really like to do is add a parameter to your `()` operator to indicate which stream to print to:

    	  void operator()(ostream &Stream)
    	  {
    	  Stream << "Hello World" << endl;
    	  }

Unfortunately, `DoFunctor` doesn't know about this new parameter, let alone what argument to pass! So instead, let's make this `Stream` a class member:

    	  struct HelloFunctor
    	  {
    	  ostream &Stream;

    	  // Set Stream when creating a HelloFunctor
    	  explicit HelloFunctor(ostream &S) : Stream(S) { }

    	  // Print to output stream selected at construction time
    	  void operator()()
    	  {
    	  Stream << "Hello World" << endl;
    	  }
    	  };

We can now provide the necessary information (ie., which stream to print to) to our `HelloFunctor` before we pass it to `DoFunctor`:

    	  HelloFunctor Hi1(cout), Hi2(cerr);

    	  DoFunctor(Hi1);  // Print to cout
    	  DoFunctor(Hi2);  // Print to cerr

Naturally a functor's `()` operator may also return some other type than `void`, and it may take arguments just like any other function. This mechanism is used extensively by the STL to sort containers, to find items in sorted containers like `set` or `map`, and so on.

#### How Transactors Work

A transactor is a functor derived from an instantiation of `libpqxx`'s `transactor` class template. Instead of writing your database transaction inline with the rest of your code, you encapsulate it in your functor's `()` operator. When the time comes to execute your transaction, you create an object of your functor type and pass it to the `perform` method of your `connection` to the database.

`perform` will make a copy of your transactor (which means it needs to be copy-constructible, by the way). It will also create a transaction object and invoke your transactor's `()` operator, passing the transaction object to it. All your transactor needs to do is perform its queries on this object and return, after which `perform` will commit the transaction. If your `()` operator throws an exception instead, `perform` will discard the copy of your original transactor, and try again with a new copy until it either succeeds or eventually gives up

`perform` gives up if the transaction fails too many times in succession. The maximum number of attempts that are made can be passed to `perform` as an optional second argument.

. If the connection is lost, the transaction will fail but `perform` will restore it transparently and simply try again.

To make all this work, your transactor's `()` operator must make no changes to the rest of your program's state. Any intermediate results, data to be processed, and so on, must stay within your transactor, and be destroyed when the transactor is. That is the magic trick that allows `perform` to create copies of your transactor and use them to rerun the transaction as many times as needed, without your program noticing.

So how does your transactor pass query results back to the outside world once it's done? For this purpose, you may redefine `transactor`'s `on_commit` member function to pass any data back to the rest of your program. This member function will be called only if your transaction succeeded.

(You may also wish to go the other route, storing data to variables outside the transactor right away, and override the `on_abort` and `on_doubt` functions to remove the data again if the transaction failed, but this is much more likely to cause subtle bugs.)

Please refer to the reference manual, the source code, and the test programs that come with `libpqxx` to learn more about how transactors work.