Citus Data Blog - Articles by Jason Petersen

What it means to be a Postgres extension

Jason Petersen — 2017-10-25T19:47:00+00:00

Nearly 18 months ago, we open sourced our Citus distributed database and "unforked it" from PostgreSQL by refactoring Citus into a PostgreSQL extension. Seasoned PostgreSQL users likely already know of and use popular PostgreSQL extensions, such as hstore, PostGIS, and pg_stat_statements; however, we realized some of you might appreciate a recap of our journey from fork to extension and beyond.

Before Becoming a PostgreSQL Extension

Prior to the release of Citus 5.0 in the spring of 2016, our codebase was best described as a fork of PostgreSQL. The open source license used by PostgreSQL is quite liberal and generously allows for forks, which has led to a long history of systems being built on top of PostgreSQL, including ParAccel, Truviso, Aster Data, and Greenplum. PostgreSQL releases new major versions each year, which are generally backwards compatible from an end-user perspective, but can come with substantial changes in the lower-level PostgreSQL code.

Being a fork of PostgreSQL means needing to adapt to those low-level changes by rebasing atop each new version after it is released, a complex integration process that can easily consume weeks.

Transforming Citus from a Fork to an Extension of Postgres

Fortunately, PostgreSQL exposes many internal hooks to permit the creation of extensions to expand its capabilities and power. Extensions can be developed against the low-level hooks using C or in higher-level procedural languages. The hooks give direct access to the core of PostgreSQL: scans, utility commands, planning, and execution are just a few of the processes that can be modified or entirely overridden by an extension. Extensions can provide new datatypes, better monitoring, foreign data wrappers, advanced security capabilities, and even entirely new languages for writing stored procedures!

Our Citus database leverages a majority of the low-level PostgreSQL hooks available to us: we create custom nodes, use them to build custom scans, and have custom planner and executor lifecycles to actually carry out your distributed queries. We even override processing of DDL commands to help perform schema modifications on remote nodes and have recently added a background worker within our extension, which performs distributed deadlock detection on in-flight queries. That we can do all of this in a modular “add-on” fashion is a testament to the careful design of PostgreSQL's internals.

Benefits of Being an Extension to PostgreSQL

Because we no longer maintain an entire fork of the complete PostgreSQL codebase, the effort required to remain compatible with new versions of PostgreSQL has been dramatically reduced: after a new PostgreSQL version is released, we need only integrate with changes to the interfaces we use to call out to PostgreSQL. These kinds of changes are often more along the lines of a new parameter has been added to a method, fix the call sites rather than an entire codebase of millions of lines has changed beneath you.

With this reduction in integration effort, we have recently been able to begin supporting major PostgreSQL releases before they land. This even includes support for entirely new features in upcoming releases: for instance, Citus 7 includes some awareness of PostgreSQL 10’s declarative partitioning syntax, even though Citus 7 shipped a full month before PostgreSQL 10 was released. In the past, when Citus was a fork of PostgreSQL, supporting a new PostgreSQL feature like that would have taken months of integration work.

Today: Citus + PostgreSQL = ❤️

All of this history leads us to where we are today with Citus. We continue to push forward with new Citus releases chock-full of features to serve all our customers—whether they be open-source, enterprise, or cloud—making it so you don’t have to worry about your database and can get back to building features.

Citus 7—released in September—was our first release which supported a major PostgreSQL version on the day it was released. As soon as the PostgreSQL 10 packages showed up in PGDG, we released OS packages for Citus that were compatible with the PostgreSQL 10 release. The next step: we immediately began builds against PostgreSQL 11 in our continuous integration environment, the earliest we have ever begun building against a new PostgreSQL version.

Admittedly we’re biased when it comes to PostgreSQL: pretty much everyone on the engineering team at Citus Data is a PostgreSQL fan. Still, we've been impressed by the capabilities afforded us as an extension to the “world’s most advanced open source database.” And we look forward to the many more impressive features on the PostgreSQL roadmap ahead.

This article was originally published on citusdata.com.

Announcing pg_shard 1.2

Jason Petersen — 2015-07-30T00:00:00+00:00

pg_shard continues to gain momentum as a straightforward sharding extension for PostgreSQL. We’re ecstatic each time we hear about a new deployment and are always considering what’s next.

We’ve been hard at work addressing some of those customer needs and have released a new version of pg_shard this week with our latest efforts. The changes in this release include:

Full CitusDB integration — Distribution metadata is always in sync with CitusDB
Better type support — Partition by enumeration or composite types
Planning improvements — Improved internal metadata locking and function checking
Usability enhancements — Better validations and error messages during use

Upgrading or installing is a breeze: see pg_shard’s GitHub page for detailed instructions.

Whether you want a distributed document store alongside your normal PostgreSQL tables or need the extra computational power afforded by a sharded cluster, pg_shard can help. We continue to grow pg_shard’s capabilities and are open to feature requests.

Got questions?

If you have any questions about pg_shard, please contact us using the pg_shard-users mailing list.

If you discover an issue when using pg_shard, please submit it to our issue tracker on GitHub.

Further information is available on our website, where you are free to contact us with any general questions you may have.

This article was originally published on citusdata.com.

PGConf Silicon Valley PostgreSQL Conference Call for Speaker Proposals Closes June 15th

Jason Petersen — 2015-05-29T00:00:00+00:00

PGConf Silicon Valley is a technical conference for the PostgreSQL community scheduled for November 17th and 18th 2015 at the South San Francisco Conference Center. The call for speaking proposals for PGConf Silicon Valley is open until June 15, 2015.

We are pleased to welcome our distinguished Conference Committee which includes PostgreSQL community leaders from throughout the United States:

Josh Berkus, PostgreSQL Experts
Peter van Hardenberg, Heroku
Albert Strasheim, Cloudflare
Simon Riggs, 2nd Quadrant
Elein Mustain, Adobe
Bruce Momjian, EnterpriseDB
Patrick King, Pandora Internet Radio
Ozgun Erdogan, Citus Data
Rafael Solari, Neustar
Aaron Brashears, Twitch

The conference will bring together the PostgreSQL community for an event filled with learning, connection building, and speakers from leading Postgres users and vendors on topics such as performance tuning, backup and replication, and new technologies.

PGConf Silicon Valley is the first solely PostgreSQL-focused conference in Silicon Valley. Sponsorship opportunities are available. Visit the PGConf Silicon Valley website to register or get more information about the conference.

What

PGConf Silicon Valley

Where

November 17–18, 2015

When

South San Francisco Conference Center
255 S Airport Blvd
South San Francisco, CA 94080
(map)

Call for Speakers Deadline

June 15, 2015

This article was originally published on citusdata.com.

Webinar on PostgreSQL Real-Time Analytics Terabyte-Scale Data Ingestion on May 20th

Jason Petersen — 2015-05-14T00:00:00+00:00

Learn how to enable real-time analytics on terabytes of data using PostgreSQL by combining analytics and operational workloads in a single database. Join Utku Azman from Citus Data at 10:00 AM PDT on Wednesday, May 20, 2015 for a webinar which will discuss the challenges of real-time big PostgreSQL and present a case study of a company which has solved this challenge.

In this live webinar, you will hear how users can differentiate themselves in a competitive environment by leveraging the rich and flexible PostgreSQL ecosystem. You will become familiar with the concepts and methods employed to enable processing of billions of events in real-time using PostgreSQL-based solutions. You will also hear about real-life implementations including how Cloudflare uses Citus to power real-time analytics for millions of websites on tens of terabytes of data.

By the end of the webinar, attendees will have a greater understanding of Citus features that enable massive parallelization of analytical queries, real-time ingestion on distributed data sets, columnar storage and advanced data compression, and high availability and dynamic scaling. They will also have a better understanding of a real-time analytics use case on big PostgreSQL.

This article was originally published on citusdata.com.

Webinar on Sharding and Scaling PostgreSQL on April 29th

Jason Petersen — 2015-04-22T00:00:00+00:00

I am hosting a webinar on April 29th at 10:00 am Pacific time to discuss the core principles of PostgreSQL database scaling. The webinar will enable attendees to avoid common mistakes that limit future system capabilities.

During the webinar, I will discuss how to simplify PostgreSQL scaling by using pg_shard, an open-source extension for managing the placement, distribution, and replication of a sharded database. pg_shard is a standalone PostgreSQL extension developed by Citus Data which addresses many NoSQL use cases while also enabling targeted real-time analytical queries. For users who need complex full-cluster queries and JOINs, pg_shard provides an easy upgrade path to CitusDB. The pg_shard extension maintains full compatibility with the standard PostgreSQL tool set, types, indexes, and semi-structured data types.

By the end of the webinar, attendees will understand the core tenets of database scaling as well as which scaling problems are appropriate to address with sharding. In addition, I will cover the features and functionality of pg_shard by presenting use cases where it is being successfully used in production environments today.

I hope you will join me on April 29th.

This article was originally published on citusdata.com.

Announcing pg_shard 1.1

Jason Petersen — 2015-03-18T00:00:00+00:00

Last winter, we open-sourced pg_shard, a transparent sharding extension for PostgreSQL. It brought straightforward sharding capabilities to PostgreSQL, allowing tables and queries to be distributed across any number of servers.

Today we’re excited to announce the next release of pg_shard. The changes in this release include:

Improved performance — INSERT commands run up to four times faster
Shard repair — Easily bring inactive placements back up to speed
Copy script — Quickly import data from CSV and other files from the command line
CitusDB integration — Expose pg_shard’s metadata for CitusDB’s use
Resource improvements — Execute larger queries than ever before

For more information about recent changes, you can view all the issues closed during this release cycle on GitHub.

Upgrading or installing is a breeze: see pg_shard’s GitHub page for detailed instructions.

Got questions?

If you have any questions about pg_shard, please contact us using the pg_shard-users mailing list.

If you discover an issue when using pg_shard, please submit it to our issue tracker on GitHub.

Further information is available on our website, where you are free to contact us with any general questions you may have.

This article was originally published on citusdata.com.