Hope you enjoy, it should not be necessary to comment much if I got to the point!

Of course I commited the text source file to the Git repository. The only problem I ran into is that ditaa defaults to ouputing a quite big right margin containing only white pixels, and that didn't fit well, visually, in this blog. So I had to resort to ImageMagik crop command in order to avoid any mouse usage in the production of this diagram.

convert .../pg_staging/bird-view.png -crop '!550' bird-view.png
mv bird-view-0.png pg_staging.png

Quicker than learning to properly use a mouse, at least for me :)

Here is my try at some nice graphics. Well I'll let you decide how nice the result is:

If you want to see the little python script I used, here it is: feedback.py, with the data embedded and all...

Now, how to read it? Well, the darker the color the better the score. For example I had 5 people score me 5 for Topic Importance on the Hi-Media talk (in french) and only 3 people at this same score and topic for pg_staging talk. The scores are from 1 to 5, 5 being the best.

The comitee accepted interesting enough topics and it seems I managed to deliver acceptable content from there. Not very good content, because reading the comments I missed some nice birds-eye pictures to help the audience get into the subject. As I'm unable to draw (with or without a mouse) I plan to fix this in latter talks by using ditaa, the DIagrams Through Ascii Art tool. I already used it and together with Emacs picture-mode it's very nice.

Oh yes the baseline of this post is that there will be later talks. I seem to be liking those and the audience feedback this time is saying that it's not too bad for them. See you soon :)

Note: in 8.5 we should have a way to express contraint exclusion and have PostgreSQL forbids overlapping entries for us. Not being there yet, you could write a constraint trigger and use the GiST index to have nice speed there, which is exactly what this constraint exclusion support is about.

It turns out the code change required is pretty simple:

-    return (a->first == b->first) ? (a->last - b->last) : (a->first - b->first);
+    /*
+     * we are comparing e.g. '1' and '12' (the shorter contains the
+     * smaller), so let's pretend '12' < '1' as it contains less elements.
+     */
+    return (alen == mlen) ? 1 : -1;

This happens in the compare support function (see Interfacing Extensions To Indexes) so that means you now have to rebuild your prefix_range btree indexes, hence the version number bump.

The tool provides a lot of commands to either dump or restore a database. It comes with documentation covering about it all, except for the londiste support part, which will be there in time for 1.0.0 release. The Todo list is getting smaller and smaller, the version you'll soon find in debian sid is already called 0.9.

So, how do you go about using this software, and what service it implements?

it's all about deriving a staging environment from your backups

To validate backups, you want to restore them and check the database you get from them. And your developers will want to sometime refresh the database they're working with. And you could have both an integration environment and a pre-live one: On the former, you develop new code atop a stable set of data; while on the latter you test stable enough code (ready to go live) on a set of data as near as live data as possible.

And you want to be flexible about it, so that there's not a fulltime job to handle retoring databases each and every days, for project A integration or project B pre-live testing, or project C accounting snapshot. Or you name it.

And of course you want to have a single point of control of all your databases. Let's call it the controler.

setting up pg_staging

The pg_staging setup consists of one pg_staging.ini file wherein you describe your different target databases (those dev and prelive ones), and of course where to get the production backups from. Currently you have to serve the backups file in a format suitable for pg_restore (that means you use either pg_dump -Ft or pg_dump -Fc) on an apache folder. The produced HTML will get parsed.

So you setup the DEFAULT section with common settings, then one section per target: the databases you want to restore. Tell pg_staging where they are (host), etc, and it'll be able to drive them.

In order to being able to host more than a single restored dump on a staging server, for the same database, we use pgbouncer:

pg_staging> pgbouncer some_db.dev
              some_db      some_db_20091029 :5432
     some_db_20090717      some_db_20090717 :5432
     some_db_20091029      some_db_20091029 :5432

So as explained into the pg_staging(1) man page, you have to open non-interactive SSH connection from the controler to the hosts where the databases will get restored. Then you have to do a minimal setup pgbouncer on the hosts with a trust connection. It'll get used from pg_staging for adding newly restored database and have them accessible. Then you can also switch the new database to being the virtual some_db so that you avoid editing any connection string on your softwares.

Also, install the pgstaging-client package on every host you target. The client is a simple shell script that must run as root (sudo is used) in order to replace your pgbouncer setup or manage your londiste services.

See man 5 pg_staging for available options, including schemas to filter out either completely or just skipping data restoring in those.

pg_staging usage

Now you're all setup, you can begin to enjoy using pgstaging. Enter the console and see what you have in there.

$ pg_staging
Welcome to pg_staging 0.9.
pg_staging> databases
...
pg_staging> restore some_db.dev
...
pg_staging> pgbouncer some_db.dev
...
pg_staging> dbsizes --all some_db.dev
...
pg_staging> psql some_db.dev
some_db_20091125=#

And as you can see in man pg_staging there are a lot of commands already. You can for example obtain a new pg_restore catalog from a dump file, with some schemas commented out. It will even comment out triggers that are using a function which is defined in a filtered out schema, for example a PGQ trigger. And much much more.

pg_staging will even allow you to dump your production databases, but consider installing a separate instance of it on the machine serving the backups to your local network thanks to an apache directory listing!

Roadmap to 1.0.0

What's remain to be done is testing and having PITR based restoring to work, and adding some documentation (tutorial, which this blog post about is; and londiste support). At this point, unless some reader here asks for a new feature (set), I'll consider pg_staging ready for 1.0.0. After all, we're using it about daily here :)

Consider commenting, you should be able to easily spot my private mail address...

It also was for me the occasion to first talk about this tool I've been working on for months, called pg_staging, which aims to empower those boring production backups to help maintaining staging environments (for your developers and testers).

All in all such events keep reminding me what it means exactly when we way that one of the greatest things about PostgreSQL is its community. If you don't know what I'm talking about, consider joining!

In order to make it clear that I intend to maintain this version, the number has 3 digits rather than 2... which is also what PostgreSQL users will expect.

The only last minute change is that you can now use the first version of the two following rather than the second one:

-  create index idx_prefix on prefixes using gist(prefix gist_prefix_range_ops);
+  create index idx_prefix on prefixes using gist(prefix);

For you information, I'm thinking about leaving pgfoundry as far as the source code management goes, because I'd like to be done with CVS. I'd still use the release file hosting though at least for now. It's a burden but it's easier for the users to find them, when they are not using plain apt-get install. That move would lead to host prefix and pgloader and the backports over there at github, where my next pet project, pg_staging, will be hosted too.

The way to see this pgfoundry leaving is that if everybody does the same, then migrating the facility to some better or more recent hosting software will be easier. Maybe some other parts of the system are harder than the sources to migrate, though. If that's the case I'll consider moving them out too, maybe getting listed on the PostgreSQL Software Catalogue will prove enough as far as web presence goes?

Time to do some bug fixing already, as hstore-new packaging is using a bash'ism I shouldn't rely on (or so the debian buildfarm is telling me) and for preprepare I was waiting for inclusion before to go improving the GUC management, stealing some code from Selena's pgGearman :)

As some of you wonder about prefix 1.0 scheduling, it should soon get there now it's been in testing long enough and no bug has been reported. Of course releasing 1.0 in august isn't good timing, so maybe I should just wait some more weeks.

Anyway, the plan for prefix is to now wait something like 2 weeks, then, baring showstopper bugs, release the 1.0 final version. If you have a use for it, now is the good time for testing it!

About upgrading a current prefix installation, the advice is to save data as text instead of prefix_range, remove prefix support, install new version, change again the columns data type:

BEGIN;
  ALTER TABLE foo
     ALTER COLUMN prefix
             TYPE text USING text(prefix);

  DROP TYPE prefix_range CASCADE;
  \i prefix.sql

  ALTER TABLE foo
     ALTER COLUMN prefix
             TYPE prefix_range USING prefix_range(prefix);

  CREATE INDEX idx_foo_prefix ON foo
         USING gist(prefix gist_prefix_range_ops);
COMMIT;

Note: I just added the gist_prefix_range_ops as default for type prefix_range so it'll be optional to specify this in final 1.0. I got so used to typing it I didn't realize we don't have to :)

  SELECT foo, bar
    FROM prefixes
   WHERE prefix @> '012345678'
ORDER BY length(prefix) DESC
   LIMIT 1;

And here's a simple stupid benchmark of the new function, here in prefix-1.0~rc2.tar.gz. And it'll soon reach debian, if my QA dept agrees (my sponsor is a QA dept all by himself!).

First some preparation:

dim=#   create table prefixes (
dim(#          prefix    prefix_range primary key,
dim(#          name      text not null,
dim(#          shortname text,
dim(#          status    char default 'S',
dim(#
dim(#          check( status in ('S', 'R') )
dim(#   );
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "prefixes_pkey" for
 table "prefixes"
CREATE TABLE
Time: 74,357 ms
dim=#   \copy prefixes from 'prefixes.fr.csv' with delimiter ; csv quote '"'
Time: 200,982 ms
dim=# select count(*) from prefixes ;
 count
-------
 11966
(1 row)
Time: 3,047 ms

And now for the micro-benchmark:

dim=# \o /dev/null
dim=# select length(prefix) from prefixes;
Time: 16,040 ms
dim=# select length(prefix::text) from prefixes;
Time: 23,364 ms
dim=# \o

Hope you enjoy!

If you have any use for it (as some VoIP companies have already), please consider testing it, in order for me to release a shiny 1.0 next week! :)

Recent changes include getting rid of those square brackets output when it's not neccesary, fixing btree operators, adding support for more operators in the GiST support code (now supported: @>, <@, =, &&). Enjoy!

I've met a lot of known people in the community, some of them I already had the chance to run into at Toronto or Prato, but this was the first time I got to talk to many of them about interresting projects and ideas. That only was awesome already, and we also had a lot of talks to listen to: as others have said, it was really hard to get to choose to go to only one place out of three.

I'm now back home and seems to be recovering quite fine from jet lag, and I even begun to move on the todo list from the conference. It includes mainly Skytools 3 testing and contributions (code and documentation), Extension Packaging work (Stephen Frost seems to be willing to help, which I highly appreciate) begining with search_path issues, and posting some backtrace to help fix some SPI_connect() bug at _PG_init() time in an extension.

The excellent lightning talk about How not to Review a Patch by Joshua Tolley took me out of the dim, I'll try to be bright enough and participate as a reviewer in later commit fests (well maybe not the first next ones as some personal events on the agenda will take all my "free" time)...

Oh and the Golconde presentation gave some insights too: this queueing based solution is to compare to the listen/notify mechanisms we already have in PostgreSQL, in the sense that's it's not transactional, and the events are kept in memory only to achieve very high distribution rates. So it's a very fine solution to manage a distributed caching system, for example, but not so much for asynchronous replication (you need not to replicate events tied to rollbacked transactions).

So all in all, spending last week in Ottawa was a splendid way to get more involved in the PostgreSQL community, which is a very fine place to be spending ones free time, should you ask me. See you soon!

The problem is that while conbining pgbouncer and prepared statements is possible, it requires the application to check at connection time if the statements it's interrested in are already prepared. This can be done by a simple catalog query of this kind:

  SELECT name
    FROM pg_prepared_statements
   WHERE name IN ('my', 'prepared', 'statements');

Well, this is simple but requires to add some application logic. What would be great would be to only have to EXECUTE my_statement(x, y, z) and never bother if the backend connection is a fresh new one or an existing one, as to avoid having to check if the application should prepare.

The preprepare pgfoundry project is all about this: it comes with a prepare_all() function which will take all statements present in a given table (SET preprepare.relation TO 'schema.the_table';) and prepare them for you. If you now tell pgbouncer to please call the function at backend creation time, you're done (see connect_query).

There's even a detailed README file, but no release yet (check out the code in the CVS, pgfoundry project page has clear instruction about how to do so.

The other things people miss out a lot in current stable Skytools (version 2.1.9 currently) are cascading replication (which allows for switchover and failover) and DDL support. The new incarnation of skytools, version 3.0 reaches alpha1 today. It comes with full support for cascading and DDL, so you might want to give it a try.

It's a rough release, documentation is still to get written for a large part of it, and bugs are still to get fixed. But it's all in the Skytools spirit: simple and efficient concepts, easy to use and maintain. Think about this release as a developer preview and join us :)

Maybe you didn't come across this project before, so here's the typical query you want to run to benefit from the special indexing, where the @> operator is read contains or is a prefix of:

  SELECT * FROM prefixes WHERE prefix @> '0123456789';

Now, a user asked about an 8.1 version of the module, as it's what some distributions ship (here, Red Hat Enterprise Linux 5.2). It turned out it was easy to support 8.1 when you already support 8.2, so the CVS now hosts 8.1 support code. And here's what the user asking about the feature has to say:

It's works like a charm now with 3ms queries over 200,000+ rows. The speed also stays less than 4ms when doing complex queries designed for fallback, priority shuffling, and having multiple carriers.

The usual tool used to import files is COPY, but it'll want each line of the file to host a text representation of a database tuple, so it doesn't apply to the case at hand. RhodiumToad was online and offered the following code to solve the problem:

create or replace function xml_import(filename text)
  returns xml
  volatile
  language plpgsql as
$f$
    declare
        content bytea;
        loid oid;
        lfd integer;
        lsize integer;
    begin
        loid := lo_import(filename);
        lfd := lo_open(loid,262144);
        lsize := lo_lseek(lfd,0,2);
        perform lo_lseek(lfd,0,0);
        content := loread(lfd,lsize);
        perform lo_close(lfd);
        perform lo_unlink(loid);

        return xmlparse(document convert_from(content,'UTF8'));
    end;
$f$;

As you can see, the trick here is to use the large objects API to load the file content into memory (content variable), then to parse it knowing it's an UTF8 encoded XML file and return an XML datatype object.

The presentation page is in russian but the slides are in English, so have a nice read!

So HOT will get used each time you update a row without changing an indexed value of it, and the benefit is skipping index maintenance, and as far as I understand it, easying vacuum hard work too. To get the benefit, HOT will need some place where to put new version of the UPDATEd tuple in the same disk page, which means you'll probably want to set your table fillfactor to something much less than 100.

Now, here's how to check you're benefitting from HOT:

SELECT schemaname, relname,
       n_tup_upd,n_tup_hot_upd,
       case when n_tup_upd > 0
            then ((n_tup_hot_upd::numeric/n_tup_upd::numeric)*100.0)::numeric(5,2)
            else NULL
       end AS hot_ratio

 FROM pg_stat_all_tables;

 schemaname | relname | n_tup_upd | n_tup_hot_upd | hot_ratio
------------+---------+-----------+---------------+-----------
 public     | table1  |         6 |             6 |    100.00
 public     | table2  |   2551200 |       2549474 |     99.93

Here's even an extended version of the same request, displaying the fillfactor option value for the tables you're inquiring about. This comes separated from the first example because you get the fillfactor of a relation into the pg_class catalog reloptions field, and to filter against a schema qualified table name, you want to join against pg_namespace too.

SELECT t.schemaname, t.relname, c.reloptions,
       t.n_tup_upd, t.n_tup_hot_upd,
       case when n_tup_upd > 0
            then ((n_tup_hot_upd::numeric/n_tup_upd::numeric)*100.0)::numeric(5,2)
            else NULL
        end AS hot_ratio
FROM pg_stat_all_tables t
      JOIN (pg_class c JOIN pg_namespace n ON c.relnamespace = n.oid)
        ON n.nspname = t.schemaname AND c.relname = t.relname

 schemaname | relname |   reloptions    | n_tup_upd | n_tup_hot_upd | hot_ratio
------------+---------+-----------------+-----------+---------------+-----------
 public     | table1  | {fillfactor=50} |   1585920 |       1585246 |     99.96
 public     | table2  | {fillfactor=50} |   2504880 |       2503154 |     99.93

Don't let the HOT question affect your sleeping no more!

The following query allows you to count how many events that represents, with the magic tick numbers coming from pgq.subscription in columns sub_last_tick and sub_next_tick.

SELECT count(*)
  FROM pgq.event_1,
      (SELECT tick_snapshot
         FROM pgq.tick
        WHERE tick_id BETWEEN 5715138 AND 5715139
      ) as t(snapshots)
WHERE txid_visible_in_snapshot(ev_txid, snapshots);

In our case, this was more than 5 millions and 400 thousands of events. With this many events to care about, if you start londiste, it'll eat as many memory as needed to have them all around, which might be more that what your system is able to give it. So you want a way to tell londiste not to load all events at once. Here's how: add the following knob to your .ini configuration file before to restart the londiste daemon:

    pgq_lazy_fetch = 500

Now, londiste will lazyly fetch 500 events at once or less, even if a single batch (which contains all events between two ticks) contains a huge number of events. This number seems a good choice as it's the default PGQ setting of number of events in a single batch. This number is only outgrown when the ticker is not running or when you're producing more events than that in a single transaction.

Hope you'll find the tip useful!

With some SQL inside:

SELECT * FROM planet.postgresql.org WHERE author = "dim";