Did you know that a Cloud-based relational database could have branches? Neon does, by default, and is built very differently than other cloud databases. If that sounds intriguing to you, we hope you’ll join us in exploring the concept practically with a brief overview followed by a hands-on simple demo of how you could use it.
Branching in Neon is extremely similar to branching in git: a Neon branch is an inexpensive copy-on-write operation that allows you to diverge both your structure and your data from your main branch at the head, or from a specific point in time. But as we dive into how practical and simple Neon makes this for developers to use, you’ll start to gain a great appreciation for what’s actually happening under the hood.
In this overview, we will briefly explore Neon’s branching concepts, then create a quick-and-dirty Discord bot that gathers shipping info for prize fulfillment as a use case for branches that are designed to be deleted. And since every coding event is unique, we’ll have at least one migration per event, so you’ll hopefully walk away with a better sense of what it’s like to work with Neon day-to-day.
First, a compact primer on branching!
Branch Immediately; Branch Often!
Neon is designed to be branched. In fact, every new Project you create leads you straight to its first (and main) branch. Branching in Neon is an inexpensive copy-on-write operation under the hood. Because Neon decouples data and storage, your limit for branching is determined by your plan, not by you having to worry about I/O cost and overhead. As far as we’re concerned, you can have as many databases as you want; it’s your cluster, after all!
With Neon branches, you also have the freedom to stop thinking about point-in-time snapshots as logical dumps, and more like non-blocking file system operations. Working day-to-day with Neon, you’ll find yourself creating branches like ticket/112_fix_bug or feature/add_user_properties.
But you’ll also find yourself creating branches like `wonder_if_this_works` and `wild_idea` too, because branches are great ways to give yourself a waypoint for point-in-time recovery, or (if the idea works) the starting point for something new.
Not All Branches Are Equal
Because all projects in Neon are (under the hood) a Postgresql cluster with storage and data separated, you get to pick which branches need to be available to query by controlling which branches are active with Compute nodes.
Compute nodes that run the Postgresql server are optional for branches, and can be added to branches at any time to make the branch available. Compute nodes can optionally scale to meet additional demand if needed, just like modern applications do.
Branches can be backups, production, or part of a strategy to merge lots of data together. They provide way more utility beyond being fast and inexpensive to create.
Branches Can Be Created From The Past
Not in the past because physical time-travel still hasn’t happened yet that we know of, but because Neon uses extremely efficient write-ahead-logging (WAL), log events can very easily become a waypoint from which a branch can be created.
The confidence you gain from knowing that you do have the ability to take a point-in-time backup easily even if alarms are already going off is a big part of how Neon actively makes the day-to-day inner development loop better, especially when you’re at a startup and wear many hats.
Now: Our Project, And How Neon Branching Impacts The Design
We will create a Discord bot to automate coding events and contest fulfillment so that doing fun stuff with the community is easy to stage. The thing is, we don’t have tons of time to send out prizes manually, and we don’t want lots of random Google sheets with people’s details hanging around, so automating these things removes almost all the friction around hosting them. If you don’t have time to run the image and play, here’s a quick video showing the bot operating.
We’ll head to our Neon console and create our project, which will spin up a database cluster ready for branching:
Once successful, you’ll be shown the connection string to copy, but don’t worry if you don’t have a place to put it right away, it’s easy to get later. You’ll see that your project has a cluster with a default branch named ‘main’ and a default database named neondb; feel free to use the default, or create another.
We’ll use the built-in SQL query console to create a database named events:
Now we’re ready to head to our app, but let’s check out our main Neon console display, where you can find a lot of handy information and manage everything. There’s a convenient way to grab the connection URL so you can feed it to your secret manager:
Notice the branch selector? It’s pretty simple. But as we work through this and you start to think about what Neon might look like in your workflow, it’s a good time to point out that Neon does have a REST API and portable CLI tool that are accessible on your CI server.
Sample Code Repo
Our fully-working example application and bot can be found in this Github repository. We recommend you grab a copy to follow along. There’s a docker build script included so you don’t have to install any dependencies.
Our App Flow, Development And Deployment
Since we’re using Neon, we can count on cheap, disposable branches. We’ll make this part of our deployment strategy as shown in this flow chart:
Our ideal deployment scenario and lifecycle looks like this:
- We create a bot from a template Github repository
- We create a branch from a template database to deploy with
- We modify the bot for any particulars that the contest will require
- We push any schema changes & deploy a Docker image
- We re-assign our branch to be our main Compute node so it remains responsive
- We run the contest and fulfill the prizes
- We delete the branch
We’re going to use PostgresJS to manage our Database connection along with Prisma to manage our migrations. Our Discord bot will be implemented with Harmony in TypeScript, and the back-end shipping form will be handled with ExpressJS. Deno is going to provide a fantastic TypeScript installation out-of-the-box, but we could almost as easily use Node with any of these components.
Let’s start by installing our database schema (found in `prisma/schema.prisma`) by running our first migration, a-la:
npx prisma migrate devOnce successful, you’ll see a screen like this:
You can (optionally) use the generated client in lieu of our simple lightweight wrapper if you want to use node. Right now, we’ll just use it for migrations.
Discord Is Our “Front End” When Writing A Bot
First, let’s look at Harmony, which is going to be the brains of our operation. Harmony is a very easy to leverage TypeScript client that makes interacting with Discord safe and easy. In our flow, interested developers will join coding events by promising to build something that meets a certain criteria, then come back and submit a link to it once done.
To accomplish this, we define four commands: pledge, submit, list and help. We define these in a pretty straight-forward object as shown:
const botCommands = [
{
name: "pledge",
description: "Use this command to promise a new submission.",
options: [
{
name: "summary",
description: "Short Summary Description",
required: true,
type: ApplicationCommandOptionType.STRING,
},
],
},
{
name: "submit",
description: "Submit Your Entry",
options: [
{
name: "url",
description: "A link (URL) to your submission.",
required: true,
type: ApplicationCommandOptionType.STRING,
},
],
},
{
name: "list",
description: "Show your current pledge, if any.",
},
{
name: "help",
description: "Get the tl;dr on how I work",
},
];
Now we need a class to implement all the magic, and the easiest way to do that is to just extend the base Harmony client class. Our commands and their structure is treated like a schema by Discord, so once we publish them they’ll be remembered.
Harmony provides us with a hook to run code once we’ve authenticated using the `@` decorator, so we first check to see our list of commands needs to be re-published:
@event()
async ready(): Promise<void> {
console.log(`Bot: Logged in as ${this.user?.tag}!`);
const commands = await this.interactions.commands.all();
if (commands.size !== botCommands.length || Deno.env.get("DISCORD_BOT_REFRESH")) {
console.info("App: Refreshing list of provided commands ...");
this.interactions.commands.bulkEdit(botCommands);
}
}
Next, in the same class, we create some code to handle these commands using friendly, easy-to-read decorators. Like other things, we use async methods when we’re doing any fetching – let’s just look at the entry for one of the commands to point out what’s worth noticing:
@slash()
async pledge(d: ApplicationCommandInteraction): Promise<void> {
const summary: string = d.option<string>("summary");
const discord_name: string = d.user.username;
const discord_id: string | number = d.user.id;
Harmony provides everything we need from our required command arguments, as well as all of the metadata that we need to know about and interact with the user, like in the fulfillment method when we use the response to create a DM:
if (pledge.length) {
if (this.formUrl) {
const channel = await d.user.createDM();
channel.send(
`Hi! I'm a bot, so you can't reply to this. Please ….: (snipped) `,
);
}
It really is worth checking out Harmony’s documentation as it has a ton of easy capabilities.
Next, Our Express Micro Service
We now have a bot that tracks pledges and DMs links to a form when people hand in their challenge, so we’re going to need something to handle the form submission. Express is a mature, compact, tried and tested framework for this, but any micro server that can handle POST requests would work.
We only have two routes, / which serves an index, and /secret which serves and processes the shipping form. The only middleware we register is a logger, which could also be stored in Neon if this was going to be used for real work:
const reqLogger = function (req: Request, _res: Response, next: NextFunction) {
console.info(`${req.method} request to "${req.url}" by ${req.hostname}`);
next();
};
Bringing Both Parts Together & Deployment
Let’s roll these up into a single application we can deploy with Docker. Our entry point for this is main.ts which loads the bot and application code, and starts both services. We can confirm everything is working by typing `deno task dev`, which will also watch files for changes and reload:
We’re now ready to package it all up by creating and testing our app in a Docker image. Once you’re certain Docker and Docker Composer are installed, it’s as easy as issuing the following command:
docker compose up –build We now have our template bot! All we need to do to run a contest is kick off a branch in our repo to support whatever customizations we want to make, and create a temporary Neon branch to retain the ephemeral data.
Let’s Kick Off Our First Event!
The first event is going to have a lot of international participation, so we need to run a migration in order to accommodate some additional information that’s not in the first iteration of our form. We need to collect people’s country information, as well as a phone number for delivery to be arranged.
We start by branching off our code, a-la git checkout -b event_one. Neon offers two ways to manage branches: from the web, or from the command line (much like we manage code branches). We’ll use the command line interface to create a branch named `event_one`, with the following properties:
- We’ll create from the schema / data head (as opposed to point-in-time, which is the subject of another overview)
- We’re creating our branch from the main branch (yes, branches can also have branches!)
- We need a Compute node to make this branch available for querying.
Let’s make sure we’re authenticated with our CLI:
neonctl authThis will take you to a tab to sign in. The CLI will alternatively use the NEON_API_KEY environmental variable, if it is set. Now, the branch:
neonctl branches create –name event_oneThis returns our endpoint info, which we can use to swap out our connection string now that we’re also on a code branch:
Note that you can just issue neonctl branches create to have a random branch name generated. See neonctl branches create –help, neonctl branches –help and the linked documentation for more information.
One More Quick Migration
Now, we just need to update our schema and our code to handle the additional fields. We’ll assume you know how to modify a form and associated query, and just focus on semantics in this step. We do this just like any other migration with Prisma, we just have the added assurance of knowing that we’re on a branch. Let’s edit the schema to add our fields:
Let’s double-check that our local shadow DB server is running, and we’ll run our migration:
npx prisma migrate dev --name event_oneOutstanding! We’ve verified that it works, and we’re on our disposable branch, ready to receive data.
Updating The Primary Branch
There’s just ONE more thing we want to do, which is to set this branch temporarily as our primary branch, so that it remains consistently responsive even if we exceed our project limits during our coding event:
This can also be done using Neon’s CLI utility when it needs to be automated, which we’ll also use at the VERY end to delete the branch once the prizes have been fulfilled, like so:
neonctl branches delete event_oneBy default, this returns a table with the branch details, with the “Updated At” field set for when the operation completed. You can always see your branches with neonctl branches list. It’s also now really clear how 99% of our application’s lifecycle could easily be automated, all thanks to database branches working in conjunction with code branches.
And with that, we’re just about done. Our app is responsive, and all systems are good to go! We’d need a little more polish if we wanted to push this to production right now, but you’ve hopefully gained some insight into the real value that Neon brings to your workflow from a few different angles while following along.
Wrapping Up
That was definitely a bit to take in! Let’s go over what we covered:
- Branches are inexpensive ways to fork off from your main database for any reason. They can have a Compute node attached to them if you need a server running to work with that branch, or can just be a point-in-time backup.
- Because Neon uses WAL, it’s easy to create branches from recent log events, for cases where you really wish you had a branch before something happened.
- Branches are just as useful for development as they are for backup, and really up the game a bit when it comes to how you approach problems at the database layer.
- Ephemeral branches that are designed to be deleted are a great way to deal with any kind of information that you don’t want to have around any longer than necessary, just like our example.
And while this intro covered all of that, there’s so much more to talk about with branches just focusing on backup strategy alone, which will be the topic of a future overview on them.
Until then, thanks for reading, we hope you found it useful! Remember, to try this demo, you’ll need to sign up for a free account – did we mention that you can create as many databases as you want? Keep up with more things happening at Neon on Twitter.
