Skip to article
Convex explained Build Your Own Convex

Build your own Convex

Convex is not just a wrapper around Postgres. It's doing something much deeper and, in my opinion, kind of brilliant.

Rather than me explaining how it works, I thought it might be more fun if we attempt to build a database with the same properties as Convex and see what we learn.

So grab yourself a lovely cup of tea as this is sure to be a fun one!

Mike raising a mug of tea in his Convex cap

01The Basics

A database has two jobs:

  1. Remember things.
  2. Give them back.

Those are the main things, but a really good database has a bunch of other properties that we care about too, such as:

  1. Correctness
  2. Performance
  3. Auditability
  4. Usability
  5. Realtime
  6. Etc.

For now, let's not worry about those and just focus on the two basics: storing and retrieving data.

Let's tackle remembering, or "storing", first. To do that, we'll do the simplest thing possible: append the JSON to a file.

Go on, press insert a few times to see it in action.

Interactive demo. Press "insert a document" and one line of JSON such as { id: "hat", stock: 3 } is appended to the bottom of a file called db.json. Each press adds one more document to the end of the file.

Okay, nice, we've stored some data in our "database". Now let's see if we can get it back.

The simplest way to do that is a scan: start at the top and read line by line until we find what we are looking for.

Give it a crack. Press find "plushie", then try asking for something we don't sell, like "socks".

Interactive demo. Press find "plushie" and the database scans db.json from the top, one line at a time, until it reaches the plushie document. Press find "socks" and it scans every line and finds nothing. "add items" makes the file longer so the scan takes more steps.

Notice how we had to scan the file just to find the one document we were after.

That's no problem at 6 documents, computers are fast, but what if we had 6 million documents?

Let's put a pin in that performance problem for now. We'll fix it properly later. We have a bigger issue to deal with first.

02Updates

Okay, so we can write stuff to our database and read it back. Database done now, right? Not quite. We still need to handle what happens when we want to change something.

Let's take a look at the simplest way we could do that: find the place in the file to change and update it.

Interactive demo. Press "update hat stock" and the database finds the hat's line in the middle of db.json and edits the stock value in place.

Now this would work, but the problem is that files are not really great at editing stuff in the middle of them.

If we change that stock level from 9 to 10, we have just updated the "9" to a "1" and added an extra character "0", which means we have to shift everything that comes after it.

This can get messy fast.

Interactive demo showing db.json as individual characters, one byte per cell. Changing stock 9 to 10 needs one extra character, so every byte after it has to shift right.

So editing stuff in the middle of a file isn't great, but what is much faster is to simply append data to the end of the file.

Let's see what that looks like if we try that instead.

Interactive demo. Press "update item" and instead of editing in place, a new version of the document is appended to the end of db.json. The old line stays where it was.

Okay, that's great. We have turned our database into an append-only log, but how do we now get the current stock for our item?

Well, if we think about it, if we start at the bottom of the log, then read back to the start, the first document we meet with our item's id is the current value.

Interactive demo. Press "find hat" and the database reads db.json from the bottom up, stopping at the first hat document it meets, which is the current value. "update hat" appends another version so the read has to look at the newest line.

03Transactions

Okay, this works for updating a single item, but it's not realistic for what we likely want to do in the real world, is it?

Instead of randomly setting a stock level, what we usually want to do is a couple of things at once. For this example:

  1. Reduce the stock level when a user buys a hat.
  2. Put it in their cart.

To make that happen, we need to combine reads and writes in a sequence.

Let's take a look.

Step-through demo of buyHat(): read the hat and the cart, then write cart hats + 1 and hat stock - 1 to db.json. Each step highlights the running line of code and the line it touches in the file.

So now we read both the hat stock and the cart size at the start, then we do the increment and decrement and save the new values to the database.

So far so good, but there is an issue here. Can you see what it is?

Try stepping through the 4 lines again:

Step-through demo of the same buyHat() function where a power cut happens after the cart write and before the stock write. The cart shows a hat, but stock was never decremented, so the file is now corrupted.

Uh oh. There was a power cut right as we were about to write the stock decrement, so it never reached the database.

We now have corrupted data: the hat is both in the cart and in the stock. Not good.

You know what we really need? Either both changes become visible, or neither does. Both pass or both fail, so there is no halfway point.

Fortunately, we aren't inventing a new thing here. This is what database folks call a transaction and the all-or-nothing property is called atomicity.

Let's wrap our two database writes in a transaction block so that either both changes become visible, or neither does.

Step-through demo where the two writes are wrapped in a transaction block, so both land in db.json at the same moment or not at all.

Great, this works and solves the power-cut issue because the data can never be in a state where one part is written but not another, but there is still a major issue lurking in our buyHat() function that is more subtle and harder to find.

Can you see it?

04Mutations

Let's just see what happens if we call that buyHat function twice at the same time.

Step-through demo where User A and User B run buyHat() at the same time against the same db.json. Both read stock 1, both write stock 0, and both believe they bought the last hat.

Because the two users read the hat at exactly the same time, they both saw stock 1 and thus both thought it was safe to buy it. So both end up setting the stock value to 0, and now we have a nightmare on our hands as both users think they have purchased the hat.

The source of the problem is that we allowed two users to execute the buyHat() function at the same time.

How do we fix this?

Well, a super simple solution would be to make our buyHat() function single-threaded. If two users try to buy the hat at the same time, then one must queue and wait.

Step-through demo where buyHat() is single-threaded. User A runs while User B waits in a queue, then B runs and reads the updated stock.

This works, but we have now massively hurt the throughput of our system, haven't we?

Imagine if our buyHat() function had some operation in it that took a while to perform. 

async function buyHat() {
  const hat = await db.get("hat");
  const cart = await db.get("cart");

  await doSomethingThatTakesSomeTime();

  await transaction(async () => {
    await db.set("cart", { hats: cart.hats + 1 });
    await db.set("hat", { stock: hat.stock - 1 });
  });
}

Everyone else
has to wait!

Well, then the queueing user would have to wait for the buyHat() call to finish because we only allow one instance of it to run at once. Not ideal.

Let's think about this a bit harder.

What if we make our transaction block bigger so that it includes not just the writes but the reads too? 

async function buyHat() {
  await transaction(async () => {
    const hat = await db.get("hat");
    const cart = await db.get("cart");

    if (hat.stock <= 0) throw "no hats left";

    await db.set("cart", { hats: cart.hats + 1 });
    await db.set("hat", { stock: hat.stock - 1 });
  });
}

reads

business logic

writes

Now we make a new requirement that before our transaction can write to the JSON file, we first check whether anything it read has changed. If it has, then the whole transaction block gets retried.

It's quite a lot to think through, I know, but I think if we do this, then we can once again allow two users to call the buyHat() function at the same time.

Let's take a look and see.

Step-through demo where the transaction block includes the reads. A and B both read stock 1. A commits first. B's reads no longer match the database, so B's whole block is retried and the retry sees no hats left.

My friends, we have just reinvented the Convex mutation.

Static code listing of a real Convex mutation, buyHat, in convex/shop.ts: it reads the hat and the cart with ctx.db.get, throws if there are no hats, then patches both documents with ctx.db.patch.

A Convex Mutation is just a transactional function. If anything in a transaction fails, we abort the transaction, as if it never even ran. You are guaranteed to never be in that no-man's-land corrupted state we saw earlier.

Now you may have noticed I did do a bit of a sleight of hand above. How can "db.json", a file on disk, check to see if the reads from one transaction don't conflict with the writes from another? The answer is it can't, it's just a file.

What we need is something to sit between our mutation and db.json. Something that is going to perform these checks for us and commit the transaction only if nothing changed. That thing is called...

05The Committer

The committer is a little service that sits between our mutations and the database. It manages a queue of transactions that it works on one at a time.

For a transaction to succeed and its writes to be committed to the database, the committer must first ensure that none of the reads of the transaction have changed since the transaction began.

To achieve this, each transaction is going to need to keep a list of the things that it reads and writes. These lists are called the read set and the write set:

Step-through demo of one transaction recording its read set (hat, cart) and its write set (cart hats 1, hat stock 0) as each line of the buyHat mutation runs. The read and write sets sit inside the mutation card. The database is not touched yet.

Okay, so now we have those read and write sets for each transaction. We can pass them to the committer to do its thing.

Let's take a look at that situation again where two users call the mutation at the same time:

Step-through demo with User A's and User B's mutations on the left, a Committer with a commit queue in the middle, and db.json on the right. Both read the same hat. A's transaction is checked first and commits. The Committer then finds B's read set has changed, rejects B, and B retries and reads stock 0.

Mutation A commits first. Mutation B's read set is now stale, so the committer rejects B's proposed writes.

We will take a look at what we do with User B's mutation in a second, but first, you might be thinking, haven't we just slowed everything down by making every transaction go through a single point, the Committer?

To an extent, yes. We have re-introduced a single place where all writes must go through in sequential order. But unlike mutations, the committer has a very narrow job: check the reads and commit the writes.

Keeping that task small means the code can be heavily optimised for just this one task and thus can smash through those transactions very quickly.

All the more complicated business logic can still run in parallel in your mutations. A and B can do that work at the same time. It's only this final check and commit that has to take turns.

Dealing with failed mutations

Okay, cool. So now User B's mutation failed. How do we handle that?

Well, we could just reject the mutation and return an error to the client, but that's not a great experience for the user as there may still be hats in stock. It's just that this particular transaction conflicted with another.

A better solution would be to simply have mutation B try again.

There's a catch, though. To be able to safely retry a function, its logic must be deterministic and effect free. This means it must not have an effect on the outside world such as calling an external service like Stripe.

To illustrate why this is needed, let's take a look at what would happen if we retried a function that has something effectual within it.

It's the same two users and the same committer as before. The only thing that has changed is one new line in buyHat(), which charges the card before the stock gets updated:

Step-through demo that repeats the committer example with one new line in buyHat: chargeCard(20) runs before the database write. User A's and User B's mutations sit on the left, each with its code, read set, write set and a side effects list. Stripe sits below them, outside Convex. Both read stock 1, both charge $20 at Stripe, and both send their transactions to the Committer in the middle. A commits to db.json on the right. B's read set is stale, so the Committer discards B's write set, but the $20 charge in B's side effects list cannot be undone. B retries, reads stock 0 and stops, so B has paid $20 for no hat.

So you can see this time, in addition to the database reads and writes, the mutation is also attempting to charge the user's bank account before patching the database.

This is fine so long as the write goes through, but if the committer rejects it and we try to retry the function, then we can end up in a state where we have charged the user but not given them the hat.

And if there had been more hats in stock, the retry would have run chargeCard() a second time and billed them twice.

I think you would agree that performing side effects is a pretty important part of an application, without them, our application would be kinda limited.

So how would we allow our mutation to call an external service like Stripe but also keep that nice retryability property?

Actually, let's just park that thought for a second, as I want to take a look at something we skipped past first.

06The Transaction Log

So our database is starting to look pretty good. Rather than a simple, dumb file that we write edits to in the middle, it now is a log of transactions that the committer approves.

There is an issue, however.

We saw earlier that for the committer to approve a transaction, it must first make sure that nothing in the read set has changed.

Makes sense, right?

Yeah, well, we neglected to mention what exactly "nothing changed in the read set" means.

As always, the devil is in the details, and the details are quite subtle here, so let's take a look at an example:

Interactive demo. A transaction log shows hat stock 1, then 2, then 1, then a mug write. A mutation says it read stock 1. Choose whether it read near the top or lower down: without timestamps the log cannot tell whether the hat changed while the mutation ran.

Suppose when the mutation started, the hat's stock was 1. Then another mutation in another part of our system changed the value to 2, then yet another mutation changed it back to 1 again, all before our original mutation finished.

Our original mutation tells the Committer that its read-set showed the hat having a value of 1, but the Committer has no idea that the hat actually changed in the intervening time.

"Nothing changed" since when?

To fix this, we need to introduce the concept of time to the Transaction Log.

Let's give each committed transaction a timestamp.

Static table of a transaction log with a timestamp column: ts 10 hat stock 1, ts 11 hat stock 2, ts 12 hat stock 1, ts 13 mug stock 5.

Now let's make it so that when a mutation begins, it's given a start time. Let's call this the snapshot timestamp as it's the "snapshot" or "view" of the database at that instant in time.

Interactive demo. Pick the mutation's snapshot timestamp (10 or 12), then press "check next write" to walk the committer through every write after that snapshot. From 10, the hat write at 11 conflicts so the mutation must retry. From 12, only the mug changed, so it commits.

Now, along with the write set and read set, the snapshot timestamp is passed along to the Committer. The Committer can now definitively say whether something has or has not changed in the mutation's read set because it knows when the mutation started.

07Indexes

Okay, our database is starting to look pretty good from a correctness standpoint, but remember the slow reads issue we put aside way back in section 01? Yeah, I think it's time we addressed that now.

Up till now, the way we have been finding the "latest" version of a document is by starting at the bottom, then reading back to the top until we find the one we want:

Interactive demo. Press "find hat" and the database scans the transaction log from the bottom up, past unrelated writes, until it finds the hat. "add unrelated writes" adds more mug and cart entries so the scan gets longer.

The problem is that every time somebody changes something else, there's more stuff between us and that answer that we have to scan past to get to the value we are looking for.

What if we keep a little lookup table that tells us where the latest version of each document lives? That way we can instantly jump to the right place in the transaction log to get the data we need. It would be a bit like a table of contents or index in a book. It lets you jump to the right page for a given chapter.

The important bit: in our little database, this index lives in memory, as a dictionary (a hash map) keyed by document ID. A lookup like index["hat"] gives us the hat's log position directly. We don't scan the file, or walk through the dictionary's entries. The index holds IDs and pointers, not another copy of every document.

Let's check it out: the same example, but with an index.

Interactive demo. Press "build the index" to create an in-memory dictionary mapping each document id to the log position of its latest version. "find hat" then jumps straight to that line instead of scanning. "find socks" is a miss in the index.

Sweet! We scan the log once to build the dictionary. After that, a read is a quick in-memory lookup followed by reading the log entry it points to. If the process restarts, we can rebuild the dictionary from the durable transaction log.

Now this example is pretty simple, and in real databases things are a bit more complicated, but the basic idea is the same: make reads more efficient by storing an index that points back to the original data.

Before we move on, let's just quickly look at how the index gets updated:

Step-through demo. A mutation submits its transaction to the Committer's queue. The Committer processes that transaction, appending the new hat version to the transaction log as line 2 (ts 11) and moving the hat's index pointer from line 1 to line 2 together in the same commit.

So at the same time the committer writes to the transaction log, it is also responsible for updating the index.

Versioned indices

Now there is a bit of a complication we should talk about.

Mutation A starts at timestamp 10. While it is still running, mutation B starts, finishes its work, and gets committed at timestamp 11. Starting first doesn't mean finishing first.

Now A reads the hat. It needs the database as it was at timestamp 10, but our index only points to the latest version. Let's see what goes wrong:

Step-through demo. Start mutation A at ts 10, then run B and queue its transaction. The committer commits B at ts 11, so the committer appends line 2 to the log and moves the hat's only index pointer to it. When A resumes, db.get looks up the hat in the index and follows its pointer to line 2, stock 0 from ts 11, instead of the stock 1 it should see at ts 10.

So the old version is still in the log, but our latest-only index no longer points to it. A ends up reading data newer than its snapshot. The read uses the index to find a log record; only the committer writes the transaction to the log and updates the index.

We could solve this if we could "timestamp" our index just like we did with the Transaction Log.

Step-through demo. Start A, run B, commit B, then resume A: the same order, but every index row now carries a timestamp. B's commit adds a second row, ts 11 to line 2, and keeps ts 10 to line 1. When A reads at ts 10 it uses the ts 10 row and gets stock 1.

Now A can read the database as it was when it started, even after B has committed. We look up the newest index version at or before A's snapshot timestamp, then follow that pointer to the log.

That gives A a consistent snapshot, not permission to commit stale writes. When A submits its transaction, the committer still has to validate its read set, just as we saw earlier.

You might be able to see a problem with this, though.

If our index has to keep track of all the changes to a given row over time, then things could get out of control quite quickly as the index would grow and grow, using more and more memory.

Interactive demo. Each commit appends one line to the transaction log and one row to the versioned index. The index keeps a row for every version of the hat, so it grows exactly as fast as the log.

This would work, but I hope you can see that it would be quite inefficient.

Let's just think about this a bit more. We don't actually need to keep every index version forever, do we? We only really need the index versioned as far back as a potential mutation start point. So if we cap mutations to only run for a brief time (it's about 1 second on Convex), we can cap how far back we need an index for.

Interactive demo with a time slider. The versioned index for the hat keeps only the versions inside the last five minutes, plus the version that was current at the start of that window. Older versions are marked removed.

In the case of Convex, it's about 5 minutes. Versions older than that get dropped until there is only one version remaining in the index.

In the absolute worst case, if there is a missing version from the index, we can always fall back to a scan of the transaction log.

08Queries

So now we have a simple transactional database that we can write things to and get the data back in an efficient way thanks to indices. There's one other property that falls nicely out of this, and that is subscribable queries.

A query is just like a mutation except it has no write set. Pretty simple. Let's take a look:

The query reads only the hat at snapshot 12, records the hat in its read set, and returns stock 2 to the browser. It has no write set.

So you can see we have introduced a "Browser" that is going to call our query, which in turn is going to read from the database and return the result. The key things to note are that there is no write set, but there is a read set: the hat's stock level.

Now, if instead of the browser just calling the Query once and getting the value back once, we were somehow able to "hold open" that connection to the query so that whenever anything that query depends on changes, we get a new value in the browser, then that would be pretty cool, right? Well, how exactly would we do that?

If we had some sort of service that kept track of those Queries and their read sets, then we could check them against committed changes. If a change overlaps a query’s read set, we can re-run that query and send the new result to the client!

One browser subscribes to the hat query. A committed stock change overlaps its read set. The query reruns, and the browser receives stock 1.

Wow, surprisingly simple! This powerful feature just falls neatly out of our transactional database.

So now we have real-time queries along with our transaction-based mutations, what's left for us to cover?

Well we parked a problem earlier around how we might call external services from our mutations. Let's take a look at that next.

09Scheduling, Actions and Effects

If you remember from earlier, we limited our mutations to being "side-effect-free"; that is, they can't talk to the outside world. We did that because we needed to be able to "retry" them if they fail.

We parked the question of how we were going to allow our application to talk to the outside world but still keep those nice transactional properties.

Let's just have a think now and see if there is a way we can achieve this.

If we introduced another kind of function, let's call it an action, then we could allow effects inside of it.

They wouldn't be transactional like mutations, but if we allowed actions to call into mutations and wait for their return, then we could at least still interact with the database that way.

Let's take a look at what that might look like:

The browser calls checkout. The action waits for buyHat to commit, then calls Stripe. The mutation is shown as a single familiar operation.

So now instead of calling the mutation from the client, we can just call the action, which will in turn call into the mutation itself.

Is this how I would do it in real Convex?

Probably not. I would more likely record a "hold" on the hat pending payment, then call Stripe for the payment, then, if that succeeds, release the hold and award the user the hat. But lets keep this simple for this example!

Now this does work, but it's a bit inconvenient as we have to call the action first.

What if the mutation could schedule the action itself, only if the mutation succeeds?

Enter the scheduler:

The proposed write set includes stock 0 and a scheduled chargeCard call. The committer accepts both together. Five seconds later, the queued action runs.

The scheduled call becomes part of the write set. It enters the queue only when the transaction commits.

If the committer rejects the transaction, nothing gets scheduled. We can retry the mutation without leaving a scheduled call behind from the rejected attempt.

10Putting it all together

Alright, I hope you are all still with me. If you are, congratulations! You have just built the key pieces of Convex.

From nothing, we have constructed the foundational elements that let us build scalable, safe applications that have a fantastic user experience.

We have Mutations that commit safely to a transactional database. We have realtime subscribable Queries, and we have the ability to call the outside world via Actions.

I think it's time for one last demo showing everything working together. Try buying two hats at once and see how the system handles it:

Live simulation. Two browsers can buy and sell hats. A click runs the mutation at a snapshot, the committer validates its read set and retries it on conflict, the subscription manager re-runs the query for both browsers, and the scheduled action charges the card through Stripe two seconds later.

Now, obviously, there is a lot more we could talk about here, and there are many other features that Convex supports, such as Full Text Search, Vector Search, Components, Files and so on, but we have covered what I think are the most important parts.

I hope you found this walkthrough useful. If you did, then share it with whoever you think will enjoy reading it. You can also come say hi to me on X or Discord at any time.

But until next time, thanks for reading.

Cheerio!

Mike waving goodbye in his Convex cap