Citus Data Blog

Thoughts on scaling out PostgreSQL, big data architectures, distributed systems, and the PostgreSQL community.

Craig Kerstiens
By Craig Kerstiens
November 16, 2017

Citus Cloud 2, Postgres, and scaling out without sacrifice

Relational databases have been a mainstay in applications for decades now. And with their dominance has come a rich set of tools: you have tools to help with monitoring, to gain insight into performance, and to operate the database in safe ways.

The knock against relational databases has long been: what happens when you need to scale? At that point, you usually have to make difficult trade-offs, like having to trade off relational semantics in order to get a database that scales out (think: NoSQL.) Or having to find a way to reduce the amount of data you need to retain, in order to continue to skate by with a single-node relational database. Or having to trade off as much as a year’s worth of application features in order to divert an engineering team away from your core business, to instead spend their time sharding the application. Bottom line: the database trade-offs to get scale can be painful.

Today I’m excited to announce Citus Cloud 2, the newest version of our cloud database. We launched the first release of Citus Cloud 18 months ago as a fully-managed database as a service that enables you to keep right on scaling your relational database. If you’re unfamiliar, Citus is an extension to Postgres that transforms your Postgres database into a distributed system under the covers, while appearing to your application as a single-node database. With Citus, you don’t need to teach your application all about distributed systems to continue scaling. We make Citus available as open source, as on-prem enterprise software, and as a fully-managed database as a service, Citus Cloud. And with Citus Cloud, you have all of your management/backups/etc. taken care of for you.

Continue reading
Sai Krishna Srirampur
By Sai Krishna Srirampur
November 14, 2017

Scaling out your Django Multi-tenant App

There are a number of data architectures you could use when building a multi-tenant app. Some, such as using one database per customer or one schema per customer, have trade-offs when it comes to larger scale. The other option is to build the notion of tenancy directly into the logic of your SaaS application. With django-multitenant and Citus, built-in tenancy becomes much easier to put in place for your application without having to re-invent the wheel yourself.

Our django-multitenant Python library, enables easy scale out of applications that are built on top of Django and follow a multi tenant data model. This Python library has evolved from our experience working with SaaS customers, scaling out their multi-tenant apps.

Continue reading
Craig Kerstiens
By Craig Kerstiens
November 8, 2017

Faster bulk loading in Postgres with copy

If you’ve used a relational database, you understand basic INSERT statements. Even if you come from a NoSQL background, you likely grok inserts. Within the Postgres world, there is a utility that is useful for fast bulk ingestion: \copy. Postgres \copy is a mechanism for you to bulk load data in or out of Postgres.

First, lets pause. Do you even need to bulk load data and what’s it have to do with Citus? We see customers leverage Citus for a number of different uses. When looking at Citus for a transactional workload, say as the system of record for some kind of multi-tenant SaaS app, your app is mostly performing standard insert/updates/deletes.

But when you’re leveraging Citus for real-time analytics, you may already have a separate ingest pipeline. In this case you might be looking at event data, which can be higher in volume than a standard web app workload. If you already have an ingest pipeline that reads off Apache Kafka or Kinesis, you could be a great candidate for bulk ingest.

Back to our feature presentation: Postgres \copy. Copy is interesting because you can achieve much higher throughput than with single row inserts.

Continue reading
Craig Kerstiens
By Craig Kerstiens
October 20, 2017

Monitoring your bloat in Postgres

Postgres under the covers in simplified terms is one giant append only log. When you insert a new record that gets appended, but the same happens for deletes and updates. For a delete a record is just flagged as invisible, it’s not actually removed from disk immediately. For updates the old record is flagged as invisible and a new record is written. What then later happens is Postgres comes through and looks at all records that are invisible and actually frees up the disk storage. This process is known as vacuum.

There are a couple of key levels to VACUUM within Postgres:

  • VACUUM ANALYZE - This one is commonly run when you’ve recently loaded data into your database and want Postgres to update it’s statistics about the data
  • VACUUM FULL - This will take a lock during the operation, but will scan the full table and reclaim all the space it can from dead tuples.
Continue reading
Craig Kerstiens
By Craig Kerstiens
October 17, 2017

A tour of Postgres Index Types

At Citus we spend a lot of time working with customers on data modeling, optimizing queries, and adding indexes to make things snappy. My goal is to be as available for our customers as we need to be, in order to make you successful. Part of that is keeping your Citus cluster well tuned and performant which we take care of for you. Another part is helping you with everything you need to know about Postgres and Citus. After all a healthy and performant database means a fast performing app and who wouldn’t want that. Today we’re going to condense some of the information we’ve shared directly with customers about Postgres indexes.

Postgres has a number of index types, and with each new release seems to come with another new index type. Each of these indexes can be useful, but which one to use depends on 1. the data type and then sometimes 2. the underlying data within the table, and 3. the types of lookups performed. In what follows we’ll look at a quick survey of the index types available to you in Postgres and when you should leverage each. Before we dig in, here’s a quick glimpse of the indexes we’ll walk you through:

  • B-Tree
  • Generalized Inverted Index (GIN)
  • Generalized Inverted Seach Tree (GiST)
  • Space partitioned GiST (SP-GiST)
  • Block Range Indexes (BRIN)
  • Hash

Now onto the indexing

Continue reading
Craig Kerstiens
By Craig Kerstiens
October 11, 2017

Indexing all the things in Postgres

Postgres indexes make your application fast. And while one option is to analyze each of your relational database queries with pg_stat_statements to see where you should add indexes… an alternative fix (and a quick one at that) could be to add indexes to each and every database table—and every column—within your database. To make this easy for you, here’s a query you can run that will create the CREATE INDEX commands for every table and column in your Postgres database.

Disclaimer: Adding an index to every column of every table is not a best practice for production databases. And it’s certainly not something we recommend at Citus Data, where we take scaling out Postgres and database performance very seriously. But indexing all the things is a fun “what if” thought exercise that is well worth thinking about, to understand the value of indexes in Postgres. What if you indexed all the things?

With that disclaimer out of the way, onto my advice of how to index all the things as a way to understand your database’s performance.

Continue reading
Samay Sharma
By Samay Sharma
September 29, 2017

Fermi Estimates On Postgres Performance

A lot of people look to Citus for a solution that scales out their Postgres database, whether on-prem or as open source or in the cloud, as a fully-managed database as a service. And yet, a common question even before looking at Citus is: “what kind of performance can I get with Postgres?” The answer is: it depends. The performance you can expect from single node Postgres comes down to your workload, both on inserts and on the query side and how large that single node is. Unfortunately, “it depends” often leaves people a bit dissatisfied.

Fortunately, there are some fermi estimates, or in laymans terms ballpark, of what performance single node Postgres can deliver. These ballparks apply both to single-node Postgres, but from there you can start to get estimates of how much further you can go when scaling out with Citus. Let’s walk through a simplified guide for what you should expect in terms of the read performance and ingest performance for queries in Postgres.

Continue reading
Marco Slot
By Marco Slot
September 22, 2017

Podyn: DynamoDB to PostgreSQL replication and migration tool

Wouldn’t it be great if you could run SQL queries on your data in DynamoDB? While this isn’t possible directly, there is an alternative: With Podyn, you can automatically replicate the schema, data, and changes in your DynamoDB tables to Postgres. Once your data is flowing into Postgres, you can start using a wide array of features including views, indexes, rollup tables, and advanced SQL queries. And if you chose Dynamo because you needed scale beyond what you thought a single Postgres instance could provide, you can replicate it into Citus.

Continue reading
Craig Kerstiens
By Craig Kerstiens
September 20, 2017

Migrating from single-node Postgres to Citus

There are a lot of things that are everyday occurrences for engineering teams. Deploying new code, deploying a new service, it’s even fairly common to deploy a net new data store or language. But migrating from one database to another is far more rare. While migrating your database can seem like a daunting task, there are lessons you can learn from others—and steps you can take to minimize risk in migrating from one database to another.

At Citus Data, we’ve helped many a customer migrate from single node Postgres, like RDS or Heroku Postgres, to a distributed Citus database cluster, so they can scale out and take advantage of the compute, memory, and disk resources of a distributed, scale-out solution. So we’ve been privy to some valuable lessons learned, and we’ve developed some best practices. Here you can find your guide for steps to follow as you start to create your migration plan to Citus.

Continue reading

Page 1 of 14