How to build a mail search engine using Nitro

Introduction

Thanks:

I thank George for creating Og and Nitro in the first place and for his curiosity about the mail search engine which made me do this tutorial, Brian for driving so much forward, Zimba for devlab and everything else, Trans for his wonderful facets (which is used in here), manveru for getting me to write less evil SQL and the many other people on the Nitro-General Mailing List and the cozy #nitro channel on freenode.net.

Special thanks go to olleolleolle from #nitro who was so friendly as to make error corrections on this document, I suck at this ;/ (do these two paragraphs make sense? If they don't, it's because I wrote this after the correction ;))

What does it do?

It uses the public archives of a Mailing List to create an indexed and searchable version of it for fast and easy access.

Motivation?

Uh... because I can. No seriously, being able to create a quick search for anything you like is quite handy. Using Nitro and some other tools you can hack together such an engine within an hour. My initial motivation was, that the Nitro-General Mailing List wasn't searchable through a web interface. (It is, through http://news.gmane.org/, but I didn't know that. What did DHH say? Reuse is overrated? Yes, I know - a lame excuse for not enough knowledge ;D )

Here is how I did it, it's quick and dirty, but it does the job pretty well. The functionality is quite limited, but with little work it can be extended to anything you like.

What you should know about

Basic knowledge in:

• Ruby
• Regular Expressions
• SQL
• Shell
• MVC (Model View Controller) paradigm

If you have a little knowledge in Nitro and Og already, that helps as well.

• Og Tutorial

This is a good link, although old it is a good introduction to Og.

Installed programs:

• Ruby (+ rubygems, facets, nitro, og, ruby-postgres)
• PostgreSQL (+ TSearch2)

OK, you got me interested

Glad you are!

But, if you not only want to learn Nitro and Og, but also want to feel the power of PostgreSQL but have no idea where to start, go to PostgreSQL and its excellent documentation.

Remember! Knowledge is the first step to world domination! Gimme cheese now.

Who should read this document

Anyone interested in the above. If you don't like PostgreSQL, but are interested into the other techniques, you can just skip the psql parts and just use MySQL fulltext searching (refer to MySQL Fulltext Search if nessessary). There will be no support from me for this whatsoever.

Who should not read this document

People without humor... well, I guess those people stopped reading already :D, given the way I write. Good, no coffee for you people ;/.

People afraid of working. Nope, this tutorial isn't going to be a piece of cake, there will be no use just copying code and looking if it works...
Because it simply won't. I won't even include every code block which is needed. People need to think to learn, copy-pasting does not encourage thinking.

How to read this document

The initial ordering of this document is based on a Bottom-Up way of programming, if you don't like this, first read Part 1 and 2, then read the whole document partwise backwards.