Citus Data Blog - Articles by Gurkan Indibay

Evolving django-multitenant to build scalable SaaS apps on Postgres & Citus

Gurkan Indibay — 2023-05-09T16:28:00+00:00

If you're building a software application that serves multiple tenants, you may have already encountered the challenges of managing and isolating tenant-specific data. That's where the django-multitenant library comes in. This library, actively used since 2017 and now downloaded more than 10K times per month, offers a simple and flexible solution for building multi-tenant Django applications.

In this blog post, we'll dive deeper into the concept of multi-tenancy and explore how Django-multitenant can help you build scalable, secure, and maintainable multi-tenant applications on top of PostgreSQL and the Citus database extension. We'll also provide a practical example of how to use Django-multitenant in a real-world scenario. So, if you're looking to simplify your multi-tenant development process, keep reading.

If you already have a good understanding of multi-tenancy concepts, you can skip this multi-tenancy part and proceed to the sections below:

Automating Django queries with django-multitenant
What's new in django-multitenant version 3.2

What is multi-tenancy?

Multitenancy is an architecture where a single instance of a software application serves multiple customers (tenants). Each tenant has its own data, configuration, and user interface, but shares the same codebase and infrastructure. It's beneficial for SaaS applications as it allows for efficient resource usage, cost savings, and easier maintenance through simultaneous updates for all tenants.

Multitenancy challenges and approaches

When dealing with multitenancy in your application, you must design with a few database issues:

Tenant Isolation : Ensure data and resource separation, route requests correctly, and manage tenant-specific configuration.
Scalability : Design a scalable database architecture, partition data, and manage concurrent access to handle increasing tenant load.
Security : Implementing additional security measures to protect tenant data, secure data in transit and at rest, and manage access control.
Customization : Allow easy customization of the application for each tenant, including configuration, branding, and workflows.
Performance : Optimize application performance through efficient database schema design, caching, and data access mechanisms.

There are three approaches to managing multi-tenancy in applications.

Separate Databases - Each tenant has their database.
Shared Database, Separate Schemas. Tenants have their own schemas but can reside in the same database.
Shared Database, Shared Schema - All tenants share the same schema and data objects should include tenant_id to denote ownership to a tenant.

The first 3 approaches above (separate databases, and separate schemas) are out of scope for this blogpost.

We will focus on the 3rd approach, the "Shared Database, Shared Schema" approach.

Multitenancy with a shared database, shared schema

Django-multitenant supports sharing a schema between tenants, which offers advantages such as easier maintenance, reduced complexity, better resource utilization, improved scalability, and enhanced data security. However, this approach has two challenges: scaling Postgres databases for multiple tenants and dealing with complexity in application development.

If you are currently in the process of developing a multi-tenant application using Django ORM on PostgreSQL using the shared schema approach, there are 2 problems needs to be addressed:

Application complexity due to tenant ID addition into application
Scaling the application while adding more tenants

The django-multitenant library helps with the first problem above—application complexity— while the Citus extension to Postgres can be used to enable distributed scale.

Citus is available as open source, and as managed service as part of Azure Cosmos DB for PostgreSQL

You can use the django-multitenant library with Citus seamlessly whether using Citus open source or on the Azure Cosmos DB for PostgreSQL managed service. django-multitenant helps with data isolation in application development. Data isolation requires careful development process, including filtering queries by tenant_id, including tenant column in join clauses, and recording correct tenant data when saving user input which django-multitenant easy for you.

Automating Django queries with django-multitenant

The django-multitenant library extends the Django ORM to automatically and uniformly scope queries by tenant. It adds the tenant_id in both fetching and manipulating data. It's a mature library which is downloaded over 10,000 times each month.

Features of django-multitenant:

You can set the tenant with one method call, and existing application code will access the proper tenant without extensive changes.
With a tenant set, the library will seamlessly include the tenant ID in join operations, eliminating the need to manually add tenant IDs.
Supports standard Django and the Django Rest Framework to streamline integration.
Implements helper classes for Citus, a distributed Postgres extension—and facilitates table distribution during the database migration process.

What's new in django-multitenant version 3.2

With the start of 2023, the django-multitenant project has been reinvigorated with active development and new releases. The latest release, version 3.2, includes the following critical changes and new features:

Added DjangoRestFramework support, enabling you to easily create RESTful APIs that can handle multi-tenant data.
Improved model migration guidelines, which can help you migrate tenant-specific data seamlessly.
Support for the latest version of Django (version 4.1), allowing you to take advantage of its latest features and improvements.
Ability to automatically set the tenant for ManyToMany related models.
Fixed invalid error messages in case of invalid field names.
Support for getting models using apps.get_model.
Removed reserved tenant_id limitation by introducing TenantMeta usage.
Introduced ReadTheDocs documentation.
Fixed issue with ManyToMany Non tenant model saves.

Quickstart on how to get started with django-multitenant

Let's learn how to use django-multitenant in a sample application which includes newly added features introduced with version 3.2. This example provides a web interface for a fictitious project management application. (The example uses the Django Rest Framework for developer efficiency, but the underlying Django-multitenant library works with vanilla Django too.)

To get started, first install the Python packages:

pip install django-multitenant
pip install Django
pip install djangorestframework

Add a TenantMeta class in your models and specify the field which identifies tenants.

class Country(models.Model):
    name = models.CharField(max_length=255)

class Account(TenantModel):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    name = models.CharField(max_length=255)
    domain = models.CharField(max_length=255)
    subdomain = models.CharField(max_length=255)
    country = models.ForeignKey(Country, on_delete=models.CASCADE)

    class TenantMeta:
        tenant_field_name = 'id'

class Manager(TenantModel):
    name = models.CharField(max_length=255)
    account = models.ForeignKey(
        Account, on_delete=models.CASCADE, related_name="managers"
    )
    class TenantMeta:
        tenant_field_name = 'account_id'
    class Meta:
        constraints = [
            models.UniqueConstraint(fields=['id', 'account_id'], name='unique_manager_account')
        ]

class Project(TenantModel):
    name = models.CharField(max_length=255)
    account = models.ForeignKey(
        Account, related_name="projects", on_delete=models.CASCADE
    )
    managers = models.ManyToManyField(Manager, through="ProjectManager")
    class TenantMeta:
        tenant_field_name = 'account_id'

    class Meta:
        constraints = [
            models.UniqueConstraint(fields=['id', 'account_id'], name='unique_project_account')
        ]

class ProjectManager(TenantModel):
    project = TenantForeignKey(
        Project, on_delete=models.CASCADE, related_name="projectmanagers"
    )
    manager = TenantForeignKey(Manager, on_delete=models.CASCADE)
    account = models.ForeignKey(Account, on_delete=models.CASCADE)

    class TenantMeta:
        tenant_field_name = 'account_id'

Distribute tables by their tenant columns. You need to remove constraints before distribution, distribute tables via tenant_migrations.Distribute(), and reinstate the constraints.

from django_multitenant.db import migrations as tenant_migrations

def get_operations():
    operations = []
    operations += [
        migrations.RunSQL(
            "ALTER TABLE tests_country DROP CONSTRAINT tests_country_pkey CASCADE;"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_manager DROP CONSTRAINT tests_manager_pkey CASCADE;"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_project DROP CONSTRAINT tests_project_pkey CASCADE;"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_projectmanager DROP CONSTRAINT tests_projectmanager_pkey CASCADE;"
        )
    ]

    operations += [
        tenant_migrations.Distribute("Country", reference=True),
        tenant_migrations.Distribute("Account"),
        tenant_migrations.Distribute("Manager"),
        tenant_migrations.Distribute("Project"),
        tenant_migrations.Distribute("ProjectManager"),
    ]

    operations += [
        migrations.RunSQL(
            "ALTER TABLE tests_country ADD CONSTRAINT tests_country_pkey PRIMARY KEY (id);"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_project ADD CONSTRAINT tests_project_pkey PRIMARY KEY (account_id, id);"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_manager ADD CONSTRAINT tests_manager_pkey PRIMARY KEY (account_id, id);"
        ),
        migrations.RunSQL(
            "ALTER TABLE tests_projectmanager ADD CONSTRAINT tests_projectmanager_pkey PRIMARY KEY (account_id, id);"
        ),
    ]

That's all the setup needed for models and migrations. Once the initial setup is done, you must call "set_current_tenant" somewhere in the code.

In the case of Django Rest Framework, our library interoperates to ensure effective calling of "set_current_tenant" and prevent any possibility of inter-tenant data leakage.

To complete our REST application, let's complete the integration steps:

Define a "get_tenant" method in a suitable location in the code. In this example, we use the views file

from django_multitenant import views
def tenant_func(request):
    return Account.objects.filter(user=request.user).first()

views.get_tenant = tenant_func

Construct the viewsets for the multi-tenant models using "TenantModelViewSet.

# views.py
from django_multitenant.views import TenantModelViewSet

class AccountViewSet(TenantModelViewSet):
    """
    API endpoint that allows groups to be viewed or edited.
    """

    model_class = Account
    serializer_class = AccountSerializer
    permission_classes = [permissions.IsAuthenticated]

Add MultitenantMiddleware into the Middleware list in your settings file

# settings.py

MIDDLEWARE = [
    ..
    "django_multitenant.middlewares.MultitenantMiddleware",
]

Thats it. Now you have a running multitenant application. Here's the link to the fully functional sample application using django-multitenant. You can download the application and test all its features:

https://github.com/gurkanindibay/django-mt-examples

For comprehensive guidance on how to use django-multitenant, you can refer to its documentation available at the following link: https://django-multitenant.readthedocs.io/en/stable/

It's important to note that this library is responsible for handling tenancy functions in the ORM. However, you will need to set the tenant by calling "set_current_tenant" throughout your application controllers. If you fail to call it correctly, you may end up getting all tenant results in queries, and tenants won't be properly set in your insert/update operations.

For more information on Multi-tenancy and Django-Multitenant, this PyCon Canada presentation about "Scaling multi-tenant apps using the Django ORM and Postgres" gives a good overview.

django-multitenant can simplify your multi-tenant SaaS app

In conclusion, building multi-tenant applications is a challenging task that requires careful planning and implementation. With the Django-multitenant library, you can simplify your multi-tenant development process by providing a flexible and scalable solution to manage and isolate tenant-specific data.

Finally, we provided a practical example of how to use Django-multitenant in a real-world scenario, demonstrating its ease of use and flexibility.

Overall, with its active development and support, django-multitenant is a valuable tool if you are looking to build scalable, secure, and maintainable multi-tenant applications on Postgres (and with Citus).

Tips for installing Citus and Postgres packages

Gurkan Indibay — 2022-01-22T17:19:00+00:00

Citus is a great extension for scaling out Postgres databases horizontally. You can use Citus either on the cloud on Azure or you can download Citus open source and install it wherever. In this blog post, we will focus on Citus open source packaging and installation.

When you go to the Citus download page to download the Citus packages—or you visit the Citus open source docs—many of you jump straight to the install instructions and the particular OS you’re looking for. That way, you can get straight to sharding Postgres with Citus.

But what if you want to see which operating systems the Citus packages support? Or what if you want to install Citus with an older version of Postgres?

This post will answer these types of nitty-gritty questions about Citus packages and their usages. Specifically, this post will cover these questions:

Do you have to install Postgres separately before installing Citus?
Where can you download Citus packages, from which package repos?
How can you download older versions of Citus (or Citus for older versions of Postgres)?
Which OS versions does Citus support?
What are the docker images used for?
Is there a matrix of all the operating systems supported by Citus?
Where can packaging-related requests be reported?

Do you have to install Postgres separately before installing Citus?

Citus is a Postgres extension which executes side by side with Postgres. You can get detailed information about Postgres extensions on the PostgreSQL wiki and this Citus blog post about what it means to be a Postgres extension. Additionally, you can get lots more information about Citus in the Citus docs

When you install Citus, you execute two commands as shown on the Citus Downloads page. For example, the commands below are the current installation commands for Debian-based systems. At the time of publication of this blog post, these commands will install Postgres 14 and Citus 10.2. We’re using the Debian-based install command as an example... Since both deb and rpm package installations use two commands to do the install, it is enough to illustrate this point with just the deb packages example:

curl https://install.citusdata.com/community/deb.sh | sudo bash
sudo apt-get -y install postgresql-14-citus-10.2

The first command above (that starts with curl) is the package repository installation script. This script installs both the

Postgres repository, and
Citus PackageCloud package installation repository

Before installing these two repos, the script performs some validations such as operating system and release version checks to be able to validate installation environment.

The second command:

sudo apt-get -y install postgresql-14-citus-10.2

is a package installation script which installs the versions specified in the command (i.e. Postgres 14 and Citus 10.2 in the example above.)

So, if you install using commands given in the Citus Download page, the Postgres server and required packages for Citus will both be installed automatically (even though the Citus and Postgres are separate packages that is installed in your system.)

Where can you download Citus packages, from which package repos?

Citus open source packages can be downloaded from 4 different package repositories.

Package Repository	Package Type	Description
PackageCloud	Deb Rpm	Main package repository for Citus. All packages released up to now are stored here. You can search for and download any packages using PackageCloud’s easy-to-use interface.
PGDG	Rpm	Package repository for Postgres and its extensions, graciously supported by the Postgres community’s packaging team. Keep in mind that sometimes the versions available on PGDG will lag the latest release slightly. Also, PGDG does not support multiple versions. Therefore, you can install only the latest version of Citus on PGDG.
PGXN	Zip-based packages	Postgres Extension Network is a central repository for all Postgres extensions. It has its own client and if you have a server which has PGXN client installed on it and want to manage all the Postgres extensions in one place, you can use this option.
Docker Hub	Docker images	Citus Docker images include Postgres and Citus together and are pre-configured which makes them good alternatives for you to try out Citus locally.Note that the docker images for Citus are only for development purposes—and are not optimized for production usage. Therefore, if you need to use Citus for test and production servers, Linux packages from one of the other repositories are the best option.

How can you download older versions of Citus (or Citus for older versions of Postgres)?

The main package repository for Citus open source is PackageCloud. All the Citus versions are stored in there and are ready for deployment.

If you want to install an older version of Citus (or install Citus with an older version of Postgres), first you need to install the package repository with the command below:

Deb	`curl https://install.citusdata.com/community/deb.sh \| sudo bash`
Rpm	`curl https://install.citusdata.com/community/rpm.sh \| sudo bash`

When downloading previous versions, you need to change the version in the second installation command.

Let’s evaluate the second install command for both .deb and rpm based package installation scripts:

Deb	`sudo apt-get -y install postgresql-14-citus-10.2`
Rpm	`sudo yum install -y citus102_14`

The file name pattern for rpm and deb packages is different. Here’s a comparison for the package_name structure for each:

Deb	`sudo apt-get -y install postgresql-<postgres_version>-citus-<citus_major_version>.<citus_minor_version>`
Rpm	`sudo yum install -y citus <citus_major_version><citus_minor_version>_<postgres_version>`

If you want to use an older version of Postgres (or of Citus), you can do a parameter replace and switch out the Citus major minor versions and/or the Postgres versions in the package installation commands that you see on the Citus download page. Assuming you get the syntax correct, and if Citus has a package for that specific version, then your customized package download will start and will use the different/older versions that you want.

It is easy to download with parameter replacement if you are sure about what version you want.

Browsing and searching for release versions on PackageCloud

If you are not sure about which version of Postgres (and/or Citus) you’d like to install, you may want to search for the desired package versions instead. In that case you can use PackageCloud’s interface to browse and search for the version you want—and then download the packages.

Go to the Citus Community on PackageCloud
Enter the version you want to download in the search box

Figure 1: Screenshot of the main search page on PackageCloud for the Citus open source packages . On this page, you can search for the Citus version you want to find all the packages related to this version.

Click on the package name you want to install.

Figure 2: Screenshot of the PackageCloud search page after searching for Citus version 10.0.4. Package count and all related packages related to the version is listed on the page. You can click and access the package detail page by clicking the package link you want.

Copy and execute the command in the install section

Figure 3: Screenshot of the detail page on PackageCloud for the package selected from the package search page. You can see the full description, checksums, upload date, and installation instructions on this page.

Which OS versions does Citus support?

Our Citus open source packages mainly support these stable releases of operating systems:

Debian
Ubuntu
Red Hat
CentOS
Fedora

If a new stable release is added for one of the operating systems above and includes Postgres support, we add support for this release. On occasion, some of you have asked us to add support for non-stable releases. However, we don’t support non-stable releases until they become stable.

When do we decide to stop support of new Citus releases for an operating system? We take 2 criteria into account:

Add/drop support decision of Operating System maintainers for OS releases.
Add/drop support decision of Postgres Maintainers for OS releases.

Let me clarify the cases with some real time examples:

Debian 8 Jessie: Debian dropped support for Debian 8 (code name jessie) as June 2020. Citus dropped support for this release after 9.5 release but kept the packages released before for Debian 8.
Ubuntu Xenial: Postgres team dropped support for Ubuntu Xenial as of September 2021 so that we dropped support for Ubuntu/Xenial after 10.1.1 release as of September 2021 as well and kept the packages released before for Ubuntu/Xenial

What are the docker images used for?

The docker images for Citus include both Postgres and Citus together and are pre-configured, which makes them good alternatives for you to experiment with Citus locally.

As you might expect, the Citus docker images are only meant for development purposes (and to give you a quick way to try out Citus and get started)—and they are not optimized for production usage.

Therefore, if you need to use Citus just for development, docker images are the best option since installation effort is near zero. If you need to use Citus for test and production servers, Linux packages are the best option.

Is there a matrix of all the operating systems supported by Citus?

The support matrix for the currently supported OS releases (at the time of publication of this blog post) that maps the OS release versions to the Citus versions is as follows:

OS/Release	Still Supported	Starting with OS Release #	Ending with OS Release # **
Enterprise Linux/CentOS 8	Yes	9.0.0
Enterprise Linux/ CentOS 7	Yes	5.0.0
Enterprise Linux/ CentOS 6	No	5.0.0	9.4.4
Oracle Linux 8	Yes	10.0.6
Oracle Linux 7	Yes	5.0.0
Oracle Linux 6	No	5.0.0	9.4.4
Debian Bullseye	Yes	10.0.6
Debian Buster	Yes	9.0.0
Debian Stretch	Yes	7.0.2
Debian Jessie	No	5.0.0	8.3.3
Debian Wheezy	No	5.0.0	8.0.2
Ubuntu Bionic	Yes	7.2.2
Ubuntu Focal	Yes	9.3.3
Ubuntu Xenial	No	7.0.2	10.0.5
Ubuntu Precise	No	5.0.0	6.2.3
Ubuntu Trusty	No	5.0.0	8.3.3
Ubuntu Wily	No	5.0.0	5.2.1

** As of date of blog post publication. Blank field = still supported for new releases when this blog post was published.

Where can packaging-related requests be reported?

If you have any requests or questions about Citus open source packaging, you can report it to us by opening an issue in our Citus packaging repo on GitHub.

Oh, and if you’re looking for more information about the Citus extension, here are 4 good places to get started:

This article was originally published on citusdata.com.