Citus Data Blog - Articles by Halil Ozan AkgulHalil Ozan AkgulScaling data and analytics with Postgreshttps://www.citusdata.com/blog/2023-05-12T16:01:00+00:00Tenant monitoring in Citus & Postgres with citus_stat_tenantshttps://www.citusdata.com/blog/2023/05/12/tenant-monitoring-in-citus-and-postgres-with-citus-stat-tenants/2023-05-12T16:01:00+00:002023-05-12T16:01:00+00:00Halil Ozan Akgul<p>If you have ever used a database like Postgres, you know how important optimization is. Some <a href="https://docs.citusdata.com/en/stable/performance/performance_tuning.html">minor changes</a> in how the database is setup make all the difference between long waiting times and satisfied customers. And one crucial thing you need before doing the optimization is to monitor and understand how your database is being used.</p>
<p><a href="/">Citus is an extension to Postgres</a> that improves scalability and parallelization by distributing your Postgres database across nodes in a cluster. The Citus database extension is available as <a href="https://github.com/citusdata/citus">open source</a> and as a managed service on the cloud, as <a href="https://learn.microsoft.com/azure/cosmos-db/postgresql/introduction">Azure Cosmos DB for PostgreSQL</a>. You can track your Citus nodes and the Postgres tables, but Citus 11.3 takes it one step further and introduces a new way to gather insight on your Citus database with tenant monitoring.</p>
<p>The new Citus 11.3 release, <a href="/updates/v11-3/">among many other features</a>, introduces a new <code>citus_stat_tenants</code> view to track your most active tenants, for those with multi-tenant SaaS applications. </p>
<p>In a <a href="https://docs.citusdata.com/en/stable/use_cases/multi_tenant.html">multi-tenant</a> SaaS application, the same database stores data from the multiple customers of the application. Each of the customers are often referred as tenants. Usually the data for each tenant is handled separately. When you distribute your Postgres table using the Citus <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#create-distributed-table">create_distributed_table</a> function, every partition key value represents a tenant.</p>
<p>With the new <code>citus_stat_tenants</code> view you can track:</p>
<ul>
<li>read query count, <code>SELECT</code> queries,</li>
<li>total query count, <code>SELECT</code>, <code>INSERT</code>, <code>DELETE</code>, and <code>UPDATE</code> queries,</li>
<li>total CPU usage in seconds</li>
</ul>
<p>In this post, you’ll learn how to monitor your top tenants to make more-informed decisions on your database—and you’ll learn how to configure <code>citus_stat_tenants</code> to best fit your application. This post includes a quickstart and code examples. Let’s explore:</p>
<ul>
<li><a href="#monitor-top-tenants">Monitor your top tenants with citus_stat_tenants</a></li>
<li><a href="#monitoring-in-real-time">Monitoring tenants in real time</a></li>
<li><a href="#insights-on-active-tenants">Insights on the most active tenants (versus on all tenants)</a></li>
<li><a href="#tenant-level-statistics">Citus stores extra data to ensure consistency of tenant-level statistics</a></li>
<li><a href="#local-stats">Local stats from Citus worker nodes</a></li>
<li><a href="#clear-statistics-history">Optional choice to clear tenant statistics history</a></li>
<li><a href="#getting-started">Getting started with Citus & Citus tenant monitoring for multi-tenant SaaS</a></li>
</ul>
<h2 id="monitor-top-tenants">Monitor your top tenants with citus_stat_tenants</h2>
<p>Let us say you have a multi-tenant app with Citus database, similar to the one on our <a href="/blog/2023/05/05/whats-new-in-citus-11-3-multi-tenant-saas/">Citus 11.3 blog post</a>, and your customers are companies who have some ad campaigns.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">companies</span> <span class="p">(</span><span class="n">id</span> <span class="n">BIGSERIAL</span><span class="p">,</span> <span class="n">name</span> <span class="nb">TEXT</span><span class="p">);</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'companies'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">id</span> <span class="n">BIGSERIAL</span><span class="p">,</span> <span class="n">company_id</span> <span class="nb">BIGINT</span><span class="p">,</span> <span class="n">name</span> <span class="nb">TEXT</span><span class="p">);</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'campaigns'</span><span class="p">,</span> <span class="s1">'company_id'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="CREATE TABLE companies (id BIGSERIAL, name TEXT);
SELECT create_distributed_table ('companies', 'id');
CREATE TABLE campaigns (id BIGSERIAL, company_id BIGINT, name TEXT);
SELECT create_distributed_table ('campaigns', 'company_id');
">Copy</button>
</div>
<p>Each of the companies will be a tenant and the company ids are the tenant ids, or the tenant attributes.</p>
<p>Now we can run some queries to track with the <code>citus_stat_tenants</code> view.</p>
<p><strong>Tip</strong>: You need to set the <code>citus.stat_tenants_track</code> to <code>all</code> in all your nodes to be able to track the statistics. You can put the setting in the <code>postgresql.conf</code> file.</p>
<p>Let us run some queries.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">companies</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="s1">'GigaMarket'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Crazy Wednesday'</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Frozen Food Frenzy'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Spring Cleaning'</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Bread&Butter'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Personal Care Refresh'</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'Lazy Lunch'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">companies</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="s1">'White Bouquet Flowers'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'Bonjour Begonia'</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'April Selection'</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="s1">'May Selection'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">companies</span> <span class="p">(</span><span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="s1">'Smart Pants Co.'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Short Shorts'</span><span class="p">),</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Tailors Cut'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">campaigns</span> <span class="p">(</span><span class="n">company_id</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">VALUES</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="s1">'Smarter Casual'</span><span class="p">);</span>
<span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">campaigns</span> <span class="k">WHERE</span> <span class="n">company_id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">count</span>
<span class="c1">-------</span>
<span class="mi">6</span>
<span class="p">(</span><span class="mi">1</span> <span class="k">row</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">name</span> <span class="k">FROM</span> <span class="n">campaigns</span> <span class="k">WHERE</span> <span class="n">company_id</span> <span class="o">=</span> <span class="mi">2</span> <span class="k">AND</span> <span class="n">name</span> <span class="k">LIKE</span> <span class="s1">'%Selection'</span><span class="p">;</span>
<span class="n">name</span>
<span class="c1">-----------------</span>
<span class="n">April</span> <span class="n">Selection</span>
<span class="n">May</span> <span class="n">Selection</span>
<span class="p">(</span><span class="mi">2</span> <span class="k">rows</span><span class="p">)</span>
<span class="k">UPDATE</span> <span class="n">campaigns</span> <span class="k">SET</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Tailor</span><span class="se">''</span><span class="s1">s Cut'</span> <span class="k">WHERE</span> <span class="n">company_id</span> <span class="o">=</span> <span class="mi">3</span> <span class="k">AND</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'Tailors Cut'</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="INSERT INTO companies (name) VALUES ('GigaMarket');
INSERT INTO campaigns (company_id, name) VALUES (1, 'Crazy Wednesday'), (1, 'Frozen Food Frenzy');
INSERT INTO campaigns (company_id, name) VALUES (1, 'Spring Cleaning'), (1, 'Bread&Butter');
INSERT INTO campaigns (company_id, name) VALUES (1, 'Personal Care Refresh'), (1, 'Lazy Lunch');
INSERT INTO companies (name) VALUES ('White Bouquet Flowers');
INSERT INTO campaigns (company_id, name) VALUES (2, 'Bonjour Begonia'), (2, 'April Selection'), (2, 'May Selection');
INSERT INTO companies (name) VALUES ('Smart Pants Co.');
INSERT INTO campaigns (company_id, name) VALUES (3, 'Short Shorts'), (3, 'Tailors Cut');
INSERT INTO campaigns (company_id, name) VALUES (3, 'Smarter Casual');
SELECT COUNT(*) FROM campaigns WHERE company_id = 1;
count
-------
6
(1 row)
SELECT name FROM campaigns WHERE company_id = 2 AND name LIKE '%Selection';
name
-----------------
April Selection
May Selection
(2 rows)
UPDATE campaigns SET name = 'Tailor''s Cut' WHERE company_id = 3 AND name = 'Tailors Cut';
">Copy</button>
</div>
<p>Now you can check <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#tenant-level-query-statistics-view">citus_stat_tenants</a> to see the statistics.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">tenant_attribute</span><span class="p">,</span>
<span class="n">read_count_in_this_period</span><span class="p">,</span>
<span class="n">query_count_in_this_period</span><span class="p">,</span>
<span class="n">cpu_usage_in_this_period</span>
<span class="k">FROM</span> <span class="n">citus_stat_tenants</span><span class="p">;</span>
<span class="n">tenant_attribute</span> <span class="o">|</span> <span class="n">read_count_in_this_period</span> <span class="o">|</span> <span class="n">query_count_in_this_period</span> <span class="o">|</span> <span class="n">cpu_usage_in_this_period</span>
<span class="c1">------------------+---------------------------+----------------------------+--------------------------</span>
<span class="mi">1</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000299</span>
<span class="mi">3</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000314</span>
<span class="mi">2</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000295</span>
<span class="p">(</span><span class="mi">3</span> <span class="k">rows</span><span class="p">)</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT tenant_attribute,
read_count_in_this_period,
query_count_in_this_period,
cpu_usage_in_this_period
FROM citus_stat_tenants;
tenant_attribute | read_count_in_this_period | query_count_in_this_period | cpu_usage_in_this_period
------------------+---------------------------+----------------------------+--------------------------
1 | 1 | 5 | 0.000299
3 | 0 | 4 | 0.000314
2 | 1 | 3 | 0.000295
(3 rows)
">Copy</button>
</div>
<p>Now you have insight on your tenants’ activities.</p>
<h2 id="monitoring-in-real-time">Monitoring tenants in real time</h2>
<p>Activity trends of your tenants might change in time. A new tenant could be much more active than a year old one. Instead of tracking the activity of your tenants since they entered the database, <code>citus_stat_tenants</code> monitors them within time buckets. Each time period’s query and CPU statistics are counted separately. Once a period ends, that period’s numbers are finalized and only stored for one more period. That means <code>citus_stat_tenants</code> only shows the current and the last period’s statistics.</p>
<p>A period is 60 seconds by default, and you can change it with <code>citus.stat_tenants_period</code> parameter.</p>
<p>Let us say you waited 60 seconds (1 period) and ran new queries.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">companies</span><span class="p">.</span><span class="n">name</span> <span class="n">company</span><span class="p">,</span> <span class="n">campaigns</span><span class="p">.</span><span class="n">name</span> <span class="n">campaign</span>
<span class="k">FROM</span> <span class="n">companies</span> <span class="k">JOIN</span> <span class="n">campaigns</span> <span class="k">ON</span> <span class="n">companies</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">campaigns</span><span class="p">.</span><span class="n">company_id</span>
<span class="k">WHERE</span> <span class="n">companies</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">AND</span> <span class="n">campaigns</span><span class="p">.</span><span class="n">name</span> <span class="k">LIKE</span> <span class="s1">'%zy%'</span><span class="p">;</span>
<span class="n">company</span> <span class="o">|</span> <span class="n">campaign</span>
<span class="c1">------------+--------------------</span>
<span class="n">GigaMarket</span> <span class="o">|</span> <span class="n">Crazy</span> <span class="n">Wednesday</span>
<span class="n">GigaMarket</span> <span class="o">|</span> <span class="n">Frozen</span> <span class="n">Food</span> <span class="n">Frenzy</span>
<span class="n">GigaMarket</span> <span class="o">|</span> <span class="n">Lazy</span> <span class="n">Lunch</span>
<span class="p">(</span><span class="mi">3</span> <span class="k">rows</span><span class="p">)</span>
<span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">campaigns</span> <span class="k">WHERE</span> <span class="n">company_id</span> <span class="o">=</span> <span class="mi">2</span> <span class="k">AND</span> <span class="n">name</span> <span class="o">=</span> <span class="s1">'April Selection'</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT companies.name company, campaigns.name campaign
FROM companies JOIN campaigns ON companies.id = campaigns.company_id
WHERE companies.id = 1 AND campaigns.name LIKE '%zy%';
company | campaign
------------+--------------------
GigaMarket | Crazy Wednesday
GigaMarket | Frozen Food Frenzy
GigaMarket | Lazy Lunch
(3 rows)
DELETE FROM campaigns WHERE company_id = 2 AND name = 'April Selection';
">Copy</button>
</div>
<p>If you check the <code>citus_stat_tenants</code> view you can see something like:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">citus_stat_tenants</span> <span class="k">WHERE</span> <span class="n">tenant_attribute</span> <span class="o">=</span> <span class="s1">'1'</span> <span class="k">OR</span> <span class="n">tenant_attribute</span> <span class="o">=</span> <span class="s1">'2'</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="c1">--------------+---------</span>
<span class="n">nodeid</span> <span class="o">|</span> <span class="mi">2</span>
<span class="n">colocation_id</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">tenant_attribute</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">read_count_in_this_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">read_count_in_last_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_this_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_last_period</span> <span class="o">|</span> <span class="mi">5</span>
<span class="n">cpu_usage_in_this_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000054</span>
<span class="n">cpu_usage_in_last_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000299</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">2</span> <span class="p">]</span><span class="c1">--------------+---------</span>
<span class="n">nodeid</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">colocation_id</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">tenant_attribute</span> <span class="o">|</span> <span class="mi">2</span>
<span class="n">read_count_in_this_period</span> <span class="o">|</span> <span class="mi">0</span>
<span class="n">read_count_in_last_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_this_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_last_period</span> <span class="o">|</span> <span class="mi">3</span>
<span class="n">cpu_usage_in_this_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000132</span>
<span class="n">cpu_usage_in_last_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000295</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT * FROM citus_stat_tenants WHERE tenant_attribute = '1' OR tenant_attribute = '2';
-[ RECORD 1 ]--------------+---------
nodeid | 2
colocation_id | 1
tenant_attribute | 1
read_count_in_this_period | 1
read_count_in_last_period | 1
query_count_in_this_period | 1
query_count_in_last_period | 5
cpu_usage_in_this_period | 0.000054
cpu_usage_in_last_period | 0.000299
-[ RECORD 2 ]--------------+---------
nodeid | 1
colocation_id | 1
tenant_attribute | 2
read_count_in_this_period | 0
read_count_in_last_period | 1
query_count_in_this_period | 1
query_count_in_last_period | 3
cpu_usage_in_this_period | 0.000132
cpu_usage_in_last_period | 0.000295
">Copy</button>
</div>
<p>Note that in addition to query and CPU usage, <code>citus_stat_tenants</code> also includes these columns:</p>
<ul>
<li><strong>nodeid</strong>: id of the Citus node the tenant lives in, can also be seen in <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#worker-node-table">pg_dist_node</a></li>
<li><strong>colocation_id</strong>: <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#co-location-group-table">Colocation group id</a> for the distributed table the tenant is from</li>
<li><strong>tenant_attribute</strong>: id of the tenant (the partition key value)</li>
</ul>
<p>If you want to create graphs and dashboards with <code>citus_stat_tenants</code>, you can use the “last period” columns for every period. Because “last period” columns have values that cannot be changed, the values in the “last period” columns will be removed after the next period is over.</p>
<h2 id="insights-on-active-tenants">Insights on the most active tenants (versus on all tenants)</h2>
<p>The new tenant monitoring feature of Citus 11.3 is not meant for investigating each and every tenant in your Citus database.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p>Rather, the new <code>citus_stat_tenants</code> is designed to help you gather information on <em>your most active tenants</em>. The tenant monitoring view will store and show the tenants that ran the most queries the most recently.</p>
</blockquote>
<p>If a tenant in your Citus database stops running queries on your distributed Postgres tables, that tenant’s statistics will eventually be removed from the monitor to open space for the more active tenants.</p>
<p>By default, <code>citus_stat_tenants</code> will show 100 tenants. You can change the number of tenants stored with <code>citus.stat_tenants_limit</code> parameter.</p>
<h2 id="tenant-level-statistics">Citus stores extra data to ensure consistency of tenant-level statistics</h2>
<p>The tenant monitor will show the top <code>citus.stat_tenants_limit</code> number of tenants and only for the current and the last period. But to make the monitor and the statistics consistent and more useful, in the background, <code>citus_stat_tenants</code> stores some more tenants and some extra data related to recency and the number of queries. Citus stores extra data is to make sure scenarios like “an active tenant dropping out of the monitor list, just because the tenant didn’t run queries during one period” does not happen.</p>
<h2 id="local-stats">Local stats from Citus worker nodes</h2>
<p>You can query <code>citus_stat_tenants</code> from any node and the view will show the statistics from all the cluster. However, the tracking of the data is done locally on the Citus node where the tenant resides in.</p>
<p>You might not be interested in the whole Citus cluster’s data, if:</p>
<ul>
<li>you plan to create a set of graphs for each of your Citus nodes,</li>
<li>you used the <a href="https://docs.citusdata.com/en/stable/admin_guide/cluster_management.html#tenant-isolation">tenant isolation feature</a> and are not particularly interested in the isolated tenants’ usages, or</li>
<li>you have a specific Citus node which you are trying to optimize</li>
</ul>
<p>If you just want to get the monitoring data from the <a href="/updates/v11-0/#metadata-sync">node you are connected to</a>, you can use the <code>citus_stat_tenants_local</code> view.</p>
<p>From Citus worker node with node id 1:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">citus_stat_tenants_local</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="c1">--------------+--</span>
<span class="n">colocation_id</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">tenant_attribute</span> <span class="o">|</span> <span class="mi">2</span>
<span class="n">read_count_in_this_period</span> <span class="o">|</span> <span class="mi">0</span>
<span class="n">read_count_in_last_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_this_period</span> <span class="o">|</span> <span class="mi">1</span>
<span class="n">query_count_in_last_period</span> <span class="o">|</span> <span class="mi">3</span>
<span class="n">cpu_usage_in_this_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000132</span>
<span class="n">cpu_usage_in_last_period</span> <span class="o">|</span> <span class="mi">0</span><span class="p">.</span><span class="mi">000295</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT * FROM citus_stat_tenants_local;
-[ RECORD 1 ]--------------+--
colocation_id | 1
tenant_attribute | 2
read_count_in_this_period | 0
read_count_in_last_period | 1
query_count_in_this_period | 1
query_count_in_last_period | 3
cpu_usage_in_this_period | 0.000132
cpu_usage_in_last_period | 0.000295
">Copy</button>
</div>
<p>Note that you can only see the tenants whose <a href="https://docs.citusdata.com/en/stable/get_started/concepts.html#shards">shards</a> are in the Citus node with node id 1 and you no longer have the nodeid column that you had in <code>citus_stat_tenants</code> view.</p>
<h2 id="clear-statistics-history">Optional choice to clear tenant statistics history</h2>
<p>The <code>citus_stat_tenants</code> shows the statistics for one period. But if you want to reset the monitor and clear all the data you can use the <code>citus_stat_tenants_reset</code> function.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_stat_tenants_reset</span><span class="p">();</span>
<span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">citus_stat_tenants</span><span class="p">;</span>
<span class="k">count</span>
<span class="c1">-------</span>
<span class="mi">0</span>
<span class="p">(</span><span class="mi">1</span> <span class="k">row</span><span class="p">)</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_stat_tenants_reset();
SELECT COUNT(*) FROM citus_stat_tenants;
count
-------
0
(1 row)
">Copy</button>
</div>
<p>There is also a function for cleaning local monitor data of a single Citus node, <code>citus_stat_tenants_reset_local()</code>. Keep in mind that resetting the data for only one Citus node might create inconsistent results across the cluster, so for most cases you should not need the local reset function.</p>
<h2 id="getting-started">Getting started with Citus & Citus tenant monitoring for multi-tenant SaaS</h2>
<p>At Citus we try make your database faster every day, which includes giving you the tools to optimize your database performance yourself.</p>
<p>Tenant monitoring is created with this thought in mind. Now that you have the tools to learn more about the activity and usage statistics for the top tenants in your SaaS application, you can make more-informed decisions on how to optimize your Citus database and cluster.</p>
<p>To learn more about Citus 11.3 and tenant monitoring:</p>
<ul>
<li><a href="/blog/2023/05/05/whats-new-in-citus-11-3-multi-tenant-saas/">What’s new in Citus 11.3 for multi-tenant SaaS apps</a></li>
<li><a href="https://www.youtube.com/live/ihgr_QEX6IU">Livestream of the Citus 11.3 Release Party, with demos</a> – Will include a demo of tenant monitoring, happening on Mon 15 May @ 9:00am PDT (<a href="https://www.addevent.com/event/Dt16932785">mark your calendar</a>)</li>
<li><a href="/updates/v11-3/#citus_stat_tenants">Tenant monitoring section of the 11.3 Updates / Release notes page</a></li>
</ul>
<p>And to get started with Citus:</p>
<ul>
<li><a href="https://github.com/citusdata/citus">Citus open source repo on GitHub</a></li>
<li><a href="/getting-started/">Getting started with Citus</a></li>
<li><a href="/download/">Download & install Citus open source</a></li>
<li><a href="https://docs.citusdata.com/">Citus docs</a></li>
<li><a href="https://slack.citusdata.com/">Join the Citus Public Slack</a></li>
</ul>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2023/05/12/tenant-monitoring-in-citus-and-postgres-with-citus-stat-tenants/'>citusdata.com</a>.</em></p>Monitor distributed Postgres activity with citus_stat_activity & citus_lock_waitshttps://www.citusdata.com/blog/2022/07/21/citus-stat-activity-views-for-distributed-postgres/2022-07-21T16:51:00+00:002022-07-21T16:51:00+00:00Halil Ozan Akgul<p>We released <a href="/updates/v11-0/">Citus 11</a> in the previous weeks and it is packed. Citus went full open source, so now previously enterprise features like the non-blocking aspect of the shard rebalancer—and multi-user support—are all open source for everyone to enjoy. One other huge change in Citus 11 is now you can query your distributed Postgres tables from any Citus node, by default.</p>
<p>When using <a href="https://github.com/citusdata/citus">Citus</a> to distribute Postgres before Citus 11, the coordinator node was your application’s only point of contact. Your application needed to connect to the coordinator to query your distributed Postgres tables. Coordinator node can handle high query throughput, about 100K per second but your application might need even more processing power. Thanks to our work in Citus 11 you can now query from any node in the Citus database cluster you want. In Citus 11 we sync the metadata to all nodes by default, so you can connect to any node and run queries on your tables.</p>
<p>Running queries from any node is awesome but you also need to be able to monitor and manage your queries from any node. Before, when you only connected the coordinator, using Postgres’ monitoring tools was enough but this is not the case anymore. So in Citus 11 we added some ways to observe your queries similar to you would do in a single Postgres instance. </p>
<p>In this blogpost you’ll learn about some new monitoring tools introduced in Citus 11 that’ll help you track and take control of your distributed queries, including:</p>
<ul>
<li><a href="#gpid">New global PID for Citus</a></li>
<li><a href="#citus-activity">New citus_stat_activity view</a></li>
<li><a href="#dist-activity">Updated citus_dist_stat_activity view</a></li>
<li><a href="#citus-locks">Updated citus_lock_waits view</a></li>
<li><a href="#pg-cancel">Cancel a query from any node with pg_cancel_backend</a></li>
<li><a href="#more-functions">Even more helper functions</a></li>
</ul>
<h2 id="gpid">New identifier for Citus processes: Global PID</h2>
<p>You can now use the Citus global process id, global PID for short, which is new to Citus 11. The Global PID is just like Postgres’ process id, but this new value is unique across a Citus cluster. We call the new value <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#citus-backend-gpid">global process identifier</a> and we use global PID or GPID for short.</p>
<p>To find the global PID of your current backend, you can use our new <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#citus-backend-gpid">citus_backend_gpid</a> function:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_backend_gpid</span><span class="p">();</span>
<span class="n">citus_backend_gpid</span>
<span class="c1">--------------------</span>
<span class="mi">110000000123</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_backend_gpid();
citus_backend_gpid
--------------------
110000000123
">Copy</button>
</div>
<p>We tried to make GPIDs human readable, so they consist of the Postgres PID of your current backend and the node id of your current node. Smallest 10 digits of global PID are the Postgres process id—which you can find with <code>pg_backend_pid</code> function—and rest of the digits are the node id, which you can find in the <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#worker-node-table">pg_dist_node</a> table.</p>
<figure>
<img src="/assets/images/blog/gpid.png" width="500" height="229" alt="global PID" loading="lazy" />
<figcaption><strong>Figure 1:</strong> Global PID, the new identifier number for queries in Citus database clusters, consists of the Citus node id (of the node the query started in) followed by the Postgres PID of the backend of the query.</figcaption>
</figure>
<p>Global PIDs are unique in a Citus cluster. Also, remember that a distributed query might need to run some queries on the shards. Those shard query executions also get the same GPID. In other words, all the activity of a distributed query can be traced via the same GPID. For example, if you run a <code>SELECT</code> query that has the GPID <code>110000000123</code>, the queries that will <code>SELECT</code> from the shards will also have <code>110000000123</code> as global PID.</p>
<p>Note that global PIDs are big integers where Postgres PIDs are 4 byte integers.</p>
<h2 id="citus-activity">New citus_stat_activity view to give you pg_stat_activity views across a Citus cluster</h2>
<p>To find the Citus global PIDs and more information about your Postgres queries in a Citus cluster, you can use our new <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#distributed-query-activity">citus_stat_activity</a> view. <code>citus_stat_activity</code> is a collection of <a href="https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW">pg_stat_activity</a> views from all nodes in the Citus cluster. When you query the <code>citus_stat_activity</code> it goes to every node and gathers <code>pg_stat_activity</code> views. <code>citus_stat_activity</code> includes all the columns from <code>pg_stat_activity</code> and we added three extra columns:</p>
<ul>
<li><strong>global_pid</strong>: Citus global process id associated with the query.</li>
<li><strong>nodeid</strong>: Citus node id of the node the <code>citus_stat_activity</code> row comes from</li>
<li><strong>is_worker_query</strong>: Boolean value, showing if the row is from one of the queries that run on the shards.</li>
</ul>
<p>Let’s say you have a <a href="https://docs.citusdata.com/en/stable/get_started/concepts.html#type-1-distributed-tables">distributed table</a> <code>tbl</code> with 4 shards and you run an update query on it in a node:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">BEGIN</span><span class="p">;</span>
<span class="k">UPDATE</span> <span class="n">tbl</span> <span class="k">SET</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">100</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="BEGIN;
UPDATE tbl SET b = 100 WHERE a = 0;
">Copy</button>
</div>
<p>You can connect to any node and use <a href="https://docs.citusdata.com/en/stable/admin_guide/diagnostic_queries.html#active-queries">citus_stat_activity</a> to find info about the <code>UPDATE</code> query.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">global_pid</span><span class="p">,</span> <span class="n">nodeid</span><span class="p">,</span> <span class="n">is_worker_query</span><span class="p">,</span> <span class="n">query</span>
<span class="k">FROM</span> <span class="n">citus_stat_activity</span>
<span class="k">WHERE</span> <span class="n">global_pid</span> <span class="o">=</span> <span class="mi">110000000123</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="c1">---+----------------------------------------------------------------------------</span>
<span class="n">global_pid</span> <span class="o">|</span> <span class="mi">110000000123</span>
<span class="n">nodeid</span> <span class="o">|</span> <span class="mi">11</span>
<span class="n">is_worker_query</span> <span class="o">|</span> <span class="n">f</span>
<span class="n">query</span> <span class="o">|</span> <span class="k">UPDATE</span> <span class="n">tbl</span> <span class="k">SET</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">100</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">2</span> <span class="p">]</span><span class="c1">---+----------------------------------------------------------------------------</span>
<span class="n">global_pid</span> <span class="o">|</span> <span class="mi">110000000123</span>
<span class="n">nodeid</span> <span class="o">|</span> <span class="mi">2</span>
<span class="n">is_worker_query</span> <span class="o">|</span> <span class="n">t</span>
<span class="n">query</span> <span class="o">|</span> <span class="k">UPDATE</span> <span class="k">public</span><span class="p">.</span><span class="n">tbl_102009</span> <span class="n">tbl</span> <span class="k">SET</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">100</span> <span class="k">WHERE</span> <span class="p">(</span><span class="n">a</span> <span class="k">OPERATOR</span><span class="p">(</span><span class="n">pg_catalog</span><span class="p">.</span><span class="o">=</span><span class="p">)</span> <span class="mi">0</span><span class="p">)</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT global_pid, nodeid, is_worker_query, query
FROM citus_stat_activity
WHERE global_pid = 110000000123;
-[ RECORD 1 ]---+----------------------------------------------------------------------------
global_pid | 110000000123
nodeid | 11
is_worker_query | f
query | UPDATE tbl SET b = 100 WHERE a = 0;
-[ RECORD 2 ]---+----------------------------------------------------------------------------
global_pid | 110000000123
nodeid | 2
is_worker_query | t
query | UPDATE public.tbl_102009 tbl SET b = 100 WHERE (a OPERATOR(pg_catalog.=) 0)
">Copy</button>
</div>
<p>In the output above, you can see:</p>
<ul>
<li>record 1 is the original query that we ran on node with <code>nodeid</code> 11</li>
<li>record 2 is the query that runs on the shard from node with <code>nodeid</code> 2</li>
<li>both records have the same <code>global_pid</code></li>
<li><code>is_worker_query</code> column is <code>true</code> for record 2 and <code>false</code>for the original query, record 1.</li>
</ul>
<p>Don’t forget, <code>citus_stat_activity</code> includes all the columns of <code>pg_stat_activity</code>, not just the ones we filter for demonstrating here. So, you can find much more information in <code>citus_stat_activity</code> view.</p>
<h2 id="dist-activity">Use citus_dist_stat_activity view to get summarized info on your queries</h2>
<p>If you are not interested in every single query from all the nodes in the Citus cluster and only care about the original distributed queries you can use <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#distributed-query-activity">citus_dist_stat_activity</a> view.</p>
<p><code>citus_dist_stat_activity</code> hides the queries that run on the shards from the <code>citus_stat_activity</code> view, so you can find some high level information about your Postgres queries.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">global_pid</span><span class="p">,</span> <span class="n">nodeid</span><span class="p">,</span> <span class="n">is_worker_query</span><span class="p">,</span> <span class="n">query</span>
<span class="k">FROM</span> <span class="n">citus_dist_stat_activity</span>
<span class="k">WHERE</span> <span class="n">global_pid</span> <span class="o">=</span> <span class="mi">110000000123</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="c1">---+--------------------------</span>
<span class="n">global_pid</span> <span class="o">|</span> <span class="mi">110000000123</span>
<span class="n">nodeid</span> <span class="o">|</span> <span class="mi">11</span>
<span class="n">is_worker_query</span> <span class="o">|</span> <span class="n">f</span>
<span class="n">query</span> <span class="o">|</span> <span class="k">UPDATE</span> <span class="n">tbl</span> <span class="k">SET</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">100</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT global_pid, nodeid, is_worker_query, query
FROM citus_dist_stat_activity
WHERE global_pid = 110000000123;
-[ RECORD 1 ]---+--------------------------
global_pid | 110000000123
nodeid | 11
is_worker_query | f
query | UPDATE tbl SET b = 100 WHERE a = 0;
">Copy</button>
</div>
<p>As, you might have guessed <code>citus_dist_stat_activity</code> is <code>citus_stat_activity</code> filtered with <code>is_worker_query = false</code>. We created <code>citus_dist_stat_activity</code> because when you are interested in the process as a whole and not each and every process on the shards, then the general information that <code>citus_dist_stat_activity</code> provides about the initial queries should be enough.</p>
<h2 id="citus-locks">Find blocking processes with citus_lock_waits view</h2>
<p>When something in your Postgres database is blocked you are in the need of monitoring the most. Citus 11 has you covered when your cluster is blocked too. The newly updated <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#distributed-query-activity">citus_lock_waits</a> shows the queries in your cluster that are waiting for some lock on another query.</p>
<p>Let’s say you run a <code>DELETE</code> query on the <code>tbl</code> that will be blocked on the previous <code>UPDATE</code> query:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">tbl</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="DELETE FROM tbl WHERE a = 0;
">Copy</button>
</div>
<p>You can connect to any node and use <code>citus_lock_waits</code> to find out which query is blocking your new query:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">citus_lock_waits</span><span class="p">;</span>
<span class="o">-</span><span class="p">[</span> <span class="n">RECORD</span> <span class="mi">1</span> <span class="p">]</span><span class="c1">-------------------------+-----------------------------</span>
<span class="n">waiting_gpid</span> <span class="o">|</span> <span class="mi">20000000345</span>
<span class="n">blocking_gpid</span> <span class="o">|</span> <span class="mi">110000000123</span>
<span class="n">blocked_statement</span> <span class="o">|</span> <span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">tbl</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">current_statement_in_blocking_process</span> <span class="o">|</span> <span class="k">UPDATE</span> <span class="n">tbl</span> <span class="k">SET</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">100</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">waiting_nodeid</span> <span class="o">|</span> <span class="mi">2</span>
<span class="n">blocking_nodeid</span> <span class="o">|</span> <span class="mi">11</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT * FROM citus_lock_waits;
-[ RECORD 1 ]-------------------------+-----------------------------
waiting_gpid | 20000000345
blocking_gpid | 110000000123
blocked_statement | DELETE FROM tbl WHERE a = 0;
current_statement_in_blocking_process | UPDATE tbl SET b = 100 WHERE a = 0;
waiting_nodeid | 2
blocking_nodeid | 11
">Copy</button>
</div>
<p>The result above shows the <code>UPDATE</code> statement is blocking the <code>DELETE</code> statement.</p>
<p>Once you find the blocking queries you can use <code>citus_stat_activity</code> and <code>citus_dist_stat_activity</code> with the global PIDs from <a href="https://docs.citusdata.com/en/stable/admin_guide/diagnostic_queries.html#detecting-locks">citus_lock_waits</a> to gather more insight.</p>
<h2 id="pg-cancel">Cancel a Postgres query from any Citus node with pg_cancel_backend</h2>
<p>After you find out and get more information about the blocking and blocked queries you might decide you need to cancel one of them. Before Citus 11 you needed to go to the node that the query is being run on, and then use <code>pg_cancel_backend</code> with the process id to cancel.</p>
<p>Now in Citus 11 we override the <a href="https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL">pg_cancel_backend</a> function to accept global PIDs too.</p>
<p>So, good news, things are now easier, you can cancel queries on your Citus clusters from any node:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">pg_cancel_backend</span><span class="p">(</span><span class="mi">20000000345</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT pg_cancel_backend(20000000345);
">Copy</button>
</div>
<p>will cause:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">tbl</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">ERROR</span><span class="p">:</span> <span class="n">canceling</span> <span class="k">statement</span> <span class="n">due</span> <span class="k">to</span> <span class="k">user</span> <span class="n">request</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="DELETE FROM tbl WHERE a = 0;
ERROR: canceling statement due to user request
">Copy</button>
</div>
<p>Remember that global PIDs are always big integers and Postgres PIDs are 4-byte integers. The difference in size is how <code>pg_cancel_backend</code> differentiates between a PID and a GPID.</p>
<p>Also, like <code>pg_cancel_backend</code>, Citus 11 overrides <code>pg_terminate_backend</code> to accept global PIDs too. So, you can also terminate queries from different nodes using global PIDs.</p>
<h2 id="more-functions">More helper functions</h2>
<p>In addition to all the Citus activity and lock views mentioned, we added some more smaller functions to help you monitor your database cluster. The new functions try to make it easier for you to get some info that can be useful when writing monitoring queries, including:</p>
<h3>Get nodename and nodeport information</h3>
<p>You can use <code>citus_nodename_for_nodeid</code> and <code>citus_nodeport_for_nodeid</code> to get info about the node with a node id:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_nodename_for_nodeid</span><span class="p">(</span><span class="mi">11</span><span class="p">);</span>
<span class="n">citus_nodename_for_nodeid</span>
<span class="c1">---------------------------</span>
<span class="n">localhost</span>
<span class="k">SELECT</span> <span class="n">citus_nodeport_for_nodeid</span><span class="p">(</span><span class="mi">11</span><span class="p">);</span>
<span class="n">citus_nodeport_for_nodeid</span>
<span class="c1">---------------------------</span>
<span class="mi">9701</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_nodename_for_nodeid(11);
citus_nodename_for_nodeid
---------------------------
localhost
SELECT citus_nodeport_for_nodeid(11);
citus_nodeport_for_nodeid
---------------------------
9701
">Copy</button>
</div>
<p>You can find both info above in the <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#worker-node-table">pg_dist_node</a> table too.</p>
<h3>Parse the GPID into nodeid and Postgres pid components</h3>
<p>You can use <code>citus_nodeid_for_gpid</code> and <code>citus_pid_for_gpid</code> to parse a GPID.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_nodeid_for_gpid</span><span class="p">(</span><span class="mi">110000000123</span><span class="p">);</span>
<span class="n">citus_nodeid_for_gpid</span>
<span class="c1">-----------------------</span>
<span class="mi">11</span>
<span class="k">SELECT</span> <span class="n">citus_pid_for_gpid</span><span class="p">(</span><span class="mi">110000000123</span><span class="p">);</span>
<span class="n">citus_pid_for_gpid</span>
<span class="c1">--------------------</span>
<span class="mi">123</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_nodeid_for_gpid(110000000123);
citus_nodeid_for_gpid
-----------------------
11
SELECT citus_pid_for_gpid(110000000123);
citus_pid_for_gpid
--------------------
123
">Copy</button>
</div>
<p>As I mentioned earlier, we tried to make GPIDs human readable and with the two functions above they are also easily machine readable too.</p>
<p>With the functions above you can find out about a node from a GPID like this:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_nodename_for_nodeid</span><span class="p">(</span><span class="n">citus_nodeid_for_gpid</span><span class="p">(</span><span class="mi">110000000123</span><span class="p">)),</span>
<span class="n">citus_nodeport_for_nodeid</span><span class="p">(</span><span class="n">citus_nodeid_for_gpid</span><span class="p">(</span><span class="mi">110000000123</span><span class="p">));</span>
<span class="n">citus_nodename_for_nodeid</span> <span class="o">|</span> <span class="n">citus_nodeport_for_nodeid</span>
<span class="c1">---------------------------+---------------------------</span>
<span class="n">localhost</span> <span class="o">|</span> <span class="mi">9701</span>
<span class="p">(</span><span class="mi">1</span> <span class="k">row</span><span class="p">)</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_nodename_for_nodeid(citus_nodeid_for_gpid(110000000123)),
citus_nodeport_for_nodeid(citus_nodeid_for_gpid(110000000123));
citus_nodename_for_nodeid | citus_nodeport_for_nodeid
---------------------------+---------------------------
localhost | 9701
(1 row)
">Copy</button>
</div>
<h3>Use citus_coordinator_nodeid to find the coordinator’s node id</h3>
<p>Finally, you can use <code>citus_coordinator_nodeid</code> to find the node id of the coordinator node.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">citus_coordinator_nodeid</span><span class="p">();</span>
<span class="n">citus_coordinator_nodeid</span>
<span class="c1">--------------------------</span>
<span class="mi">3</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT citus_coordinator_nodeid();
citus_coordinator_nodeid
--------------------------
3
">Copy</button>
</div>
<h2>With Citus 11 you can monitor from any node in a Citus database cluster</h2>
<p>With Citus 11 you can query your distributed Postgres tables from any node by default. And with the tools you learned about in this blog post, you know how to monitor and manage your Citus cluster from any node, like you would do with a single Postgres instance.</p>
<p>If you’re interested in all that changed in Citus 11 check out:</p>
<ul>
<li><a href="/blog/2022/06/17/citus-11-goes-fully-open-source/">Marco’s Citus 11 blog post, titled</a>: Citus 11 for Postgres goes fully open source, with query from any node</li>
<li><a href="/updates/v11-0/">Citus 11.0 updates page</a></li>
<li><a href="https://youtu.be/JDUIiCYEaFw">Recording of the Citus 11 release party</a>, our first livestream release party with demos of several of the new features</li>
</ul>
<p>And if you want to download Citus you can always find the <a href="/download">latest download instructions</a> on the website; we’ve put together lots of useful resources on the <a href="/getting-started/">Getting Started page</a>; you can file issues in the <a href="https://github.com/citusdata/citus">GitHub repo</a>; and if you have questions please join us (and other users in the community) in the <a href="https://slack.citusdata.com/">Citus Public Slack</a>.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2022/07/21/citus-stat-activity-views-for-distributed-postgres/'>citusdata.com</a>.</em></p>Citus Tips for Postgres: How to alter distribution key, shard count, & morehttps://www.citusdata.com/blog/2021/05/03/citus-tips-for-postgres-how-to-alter-distribution-key-shard-count/2021-05-03T16:22:00+00:002021-05-03T16:22:00+00:00Halil Ozan Akgul<p>Citus is an extension to Postgres that lets you distribute your application’s workload across multiple nodes. Whether you are using <a href="https://github.com/citusdata/citus">Citus open source</a> or using Citus as part of a <a href="https://learn.microsoft.com/azure/cosmos-db/postgresql/overview">managed Postgres service</a> in the cloud, one of the first things you do when you start using Citus is to <a href="https://docs.citusdata.com/en/stable/develop/reference_ddl.html#creating-and-distributing-tables">distribute your tables</a>. While distributing your Postgres tables you need to decide on some properties such as <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html">distribution column</a>, <a href="https://docs.citusdata.com/en/stable/admin_guide/cluster_management.html#shard-count">shard count</a>, <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html#table-co-location">colocation</a>. And even before you decide on your distribution column (sometimes called a distribution key, or a sharding key), when you create a Postgres table, your table is created with an <a href="https://www.postgresql.org/docs/current/tableam.html">access method</a>.</p>
<p>Previously you had to decide on these table properties up front, and then you went with your decision. Or if you really wanted to change your decision, you needed to start over. The good news is that <a href="/blog/2021/03/05/citus-10-release-open-source-rebalancer-and-columnar-for-postgres/">in Citus 10</a>, we introduced 2 new user-defined functions (UDFs) to make it easier for you to make changes to your distributed Postgres tables. </p>
<p><a href="/blog/2020/11/14/citus-9-5-whats-new/">Before Citus 9.5</a>, if you wanted to change any of the properties of the distributed table, you would have to create a new table with the desired properties and move everything to this new table. But in Citus 9.5 we introduced a new function, <a href="/blog/2021/02/06/citus-tips-how-to-undistribute-a-distributed-postgres-table/">undistribute_table</a>. With the <code>undistribute_table</code> function you can convert your distributed Citus tables back to local Postgres tables and then distribute them again with the properties you wish. But undistributing and then distributing again is… 2 steps. In addition to the inconvenience of having to write 2 commands, undistributing and then distributing again has some more problems:</p>
<ol>
<li>Moving the data of a big table can take a long time, undistribution and distribution both require to move all the data of the table. So, you must move the data twice, which is much longer.</li>
<li>Undistributing moves all the data of a table to the Citus coordinator node. If your coordinator node isn’t big enough, and coordinator nodes typically don’t have to be, you might not be able to fit the table into your coordinator node.</li>
</ol>
<p>So, in Citus 10, we introduced 2 new functions to reduce the steps you need to make changes to your tables:</p>
<ul>
<li><code>alter_distributed_table</code></li>
<li><code>alter_table_set_access_method</code></li>
</ul>
<p>In this post you’ll find some tips about how to use the <code>alter_distributed_table</code> function to change the shard count, distribution column, and the colocation of a distributed Citus table. And we’ll show how to use the <code>alter_table_set_access_method</code> function to change, well, the access method. <strong>An important note: you may not ever need to change your Citus table properties</strong>. We just want you to know, if you ever do, you have the flexibility. And with these Citus tips, you will know how to make the changes.</p>
<h2>Altering the properties of distributed Postgres tables in Citus</h2>
<p>When you distribute a Postgres table with the <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#create-distributed-table">create_distributed_table</a> function, you must pick a distribution column and set the <code>distribution_column</code> parameter. During the distribution, Citus uses a configuration variable called <a href="https://docs.citusdata.com/en/stable/develop/api_guc.html#citus-shard-count-integer">shard_count</a> for deciding the shard count of the table. You can also provide <code>colocate_with</code> parameter to pick a table to colocate with (or colocation will be done automatically, if possible).</p>
<p>However, after the distribution if you decide you need to have a different configuration, starting from Citus 10, you can use the <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#alter-distributed-table">alter_distributed_table</a> function.</p>
<p><code>alter_distributed_table</code> has three parameters you can change:</p>
<ul>
<li>distribution column</li>
<li>shard count</li>
<li>colocation properties</li>
</ul>
<h2>How to change the distribution column (aka the sharding key)</h2>
<p>Citus divides your table into shards based on the distribution column you select while distributing. <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html">Picking the right distribution column</a> is crucial for having a good distributed database experience. A good distribution column will help you parallelize your data and workload better by dividing your data evenly and keeping related data points close to each other.However, choosing the distribution column might be a bit tricky when you’re first getting started. Or perhaps later as you make changes in your application, you might need to pick another distribution column.</p>
<p>With the <code>distribution_column</code> parameter of the new <code>alter_distributed_table</code> function, Citus 10 gives you an easy way to change the distribution column.</p>
<p>Let’s say you have customers and orders that your customers make. You will create some Postgres tables like these:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">customers</span> <span class="p">(</span><span class="n">customer_id</span> <span class="nb">BIGINT</span><span class="p">,</span> <span class="n">name</span> <span class="nb">TEXT</span><span class="p">,</span> <span class="n">address</span> <span class="nb">TEXT</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">orders</span> <span class="p">(</span><span class="n">order_id</span> <span class="nb">BIGINT</span><span class="p">,</span> <span class="n">customer_id</span> <span class="nb">BIGINT</span><span class="p">,</span> <span class="n">products</span> <span class="nb">BIGINT</span><span class="p">[]);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="CREATE TABLE customers (customer_id BIGINT, name TEXT, address TEXT);
CREATE TABLE orders (order_id BIGINT, customer_id BIGINT, products BIGINT[]);
">Copy</button>
</div>
<p>When first distributing your Postgres tables with Citus, let’s say that you decided to distribute the <code>customers</code> table on <code>customer_id</code> and the <code>orders</code> table on <code>order_id</code>.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'customers'</span><span class="p">,</span> <span class="s1">'customer_id'</span><span class="p">);</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'orders'</span><span class="p">,</span> <span class="s1">'order_id'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT create_distributed_table ('customers', 'customer_id');
SELECT create_distributed_table ('orders', 'order_id');
">Copy</button>
</div>
<p>Later you might realize distributing the <code>orders</code> table by the <code>order_id</code> column might not be the best idea. Even though <code>order_id</code> could be a good column to evenly distribute your data, it is not a good choice if you frequently need to join the <code>orders</code> table with the <code>customers</code> table on the <code>customer_id</code>. When both tables are distributed by <code>customer_id</code> you can use <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html#table-co-location">colocated joins</a>, which are very efficient compared to joins on other columns.</p>
<p>So, if you decide to change the distribution column of <code>orders</code> table into <code>customer_id</code> here is how you do it:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">alter_distributed_table</span> <span class="p">(</span><span class="s1">'orders'</span><span class="p">,</span> <span class="n">distribution_column</span> <span class="p">:</span><span class="o">=</span> <span class="s1">'customer_id'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT alter_distributed_table ('orders', distribution_column := 'customer_id');
">Copy</button>
</div>
<p>Now the <code>orders</code> table is distributed by <code>customer_id</code>. So, the customers and the orders of the customers are in the same node and close to each other, and you can have fast joins and foreign keys that include the <code>customer_id</code>.</p>
<p>You can see the new distribution column on the <a href="https://docs.citusdata.com/en/stable/develop/api_metadata.html#citus-tables-view">citus_tables view</a>:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">distribution_column</span> <span class="k">FROM</span> <span class="n">citus_tables</span> <span class="k">WHERE</span> <span class="k">table_name</span><span class="p">::</span><span class="nb">text</span> <span class="o">=</span> <span class="s1">'orders'</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT distribution_column FROM citus_tables WHERE table_name::text = 'orders';
">Copy</button>
</div>
<h2>How to increase (or decrease) the shard count in Citus</h2>
<p>Shard count of a distributed Citus table is the number of pieces the distributed table is divided into. <a href="https://docs.citusdata.com/en/stable/admin_guide/cluster_management.html#shard-count">Choosing the shard count</a> is a balance between the flexibility of having more shards, and the overhead for query planning and execution across the shards. Like distribution column, the shard count is also set while distributing the table. If you want to pick a different shard count than the default for a table, during the distribution process you can use the <a href="https://docs.citusdata.com/en/stable/develop/api_guc.html#citus-shard-count-integer">citus.shard_count</a> configuration variable, like this:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">products</span> <span class="p">(</span><span class="n">id</span> <span class="nb">BIGINT</span><span class="p">,</span> <span class="n">name</span> <span class="nb">TEXT</span><span class="p">);</span>
<span class="k">SET</span> <span class="n">citus</span><span class="p">.</span><span class="n">shard_count</span> <span class="k">TO</span> <span class="mi">20</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'products'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="CREATE TABLE products (id BIGINT, name TEXT);
SET citus.shard_count TO 20;
SELECT create_distributed_table ('products', 'id');
">Copy</button>
</div>
<p>After distributing your table, you might decide the shard count you set was not the best option. Or your first decision on the shard count might be good for a while but your application might grow in time, you might add new nodes to your Citus cluster, and you might need more shards. The <code>alter_distributed_table</code> function has you covered in the cases that you want to change the shard count too.</p>
<p>To change the shard count you just use the <code>shard_count</code> parameter:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">alter_distributed_table</span> <span class="p">(</span><span class="s1">'products'</span><span class="p">,</span> <span class="n">shard_count</span> <span class="p">:</span><span class="o">=</span> <span class="mi">30</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT alter_distributed_table ('products', shard_count := 30);
">Copy</button>
</div>
<p>After the query above, your table will have 30 shards. You can see your table’s shard count on the <code>citus_tables</code> view:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">shard_count</span> <span class="k">FROM</span> <span class="n">citus_tables</span> <span class="k">WHERE</span> <span class="k">table_name</span><span class="p">::</span><span class="nb">text</span> <span class="o">=</span> <span class="s1">'products'</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT shard_count FROM citus_tables WHERE table_name::text = 'products';
">Copy</button>
</div>
<h2>How to colocate with a different Citus distributed table</h2>
<p>When two Postgres tables are colocated in Citus, the rows of the tables that have the same value in the distribution column will be on the same Citus node. <a href="https://docs.citusdata.com/en/stable/develop/reference_ddl.html#co-locating-tables">Colocating the right tables</a> will help you with better relational operations. Like the shard count and the distribution column, the colocation is also set while distributing your tables. You can use the <code>colocate_with</code> parameter to change the colocation.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">alter_distributed_table</span> <span class="p">(</span><span class="s1">'products'</span><span class="p">,</span> <span class="n">colocate_with</span> <span class="p">:</span><span class="o">=</span> <span class="s1">'customers'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT alter_distributed_table ('products', colocate_with := 'customers');
">Copy</button>
</div>
<p>Again, like the distribution column and shard count, you can find information about your tables’ colocation groups on the <code>citus_tables</code> view:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">colocation_id</span> <span class="k">FROM</span> <span class="n">citus_tables</span> <span class="k">WHERE</span> <span class="k">table_name</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'products'</span><span class="p">,</span> <span class="s1">'customers'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT colocation_id FROM citus_tables WHERE table_name IN ('products', 'customers');
">Copy</button>
</div>
<p>You can also use <code>default</code> and <code>none</code> keywords with <code>colocate_with</code> parameter to change the colocation group of the table to default, or to break any colocation your table has.</p>
<p>To colocate distributed Citus tables, the distributed tables need to have the same shard counts. But if the tables you want to colocate don’t have the same shard count, worry not, because <code>alter_distributed_table</code> will automatically understand this. Then your table’s shard count will also be updated to match the new colocation group’s shard count.</p>
<h2>How to change more than one Citus table property at a time</h2>
<p>Here is a tip! If you want to change multiple properties of your distributed Citus tables at the same time, you can simply use multiple parameters of the <code>alter_distributed_table</code> function.</p>
<p>For example, if you want to change both the shard count and the distribution column of a table here’s how you do it:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">alter_distributed_table</span> <span class="p">(</span><span class="s1">'products'</span><span class="p">,</span> <span class="n">distribution_column</span> <span class="p">:</span><span class="o">=</span> <span class="s1">'name'</span><span class="p">,</span> <span class="n">shard_count</span> <span class="p">:</span><span class="o">=</span> <span class="mi">35</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT alter_distributed_table ('products', distribution_column := 'name', shard_count := 35);
">Copy</button>
</div>
<h2>How to alter the Citus colocation group</h2>
<p>If your Postgres table is colocated with some other tables and you want to change the shard count of all of the tables to keep the colocation, you might be wondering if you have to alter them one by one… which is multiple steps.</p>
<p>Yes (you can see a pattern here) the Citus tip is that you can use the <code>alter_distributed_table</code> function to change the properties of all of the colocation group.</p>
<p>If you decide the change you make with the <code>alter_distributed_table</code> function needs to be done to all the tables that are colocated with the table you are changing, you can use the <code>cascade_to_colocated</code> parameter:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SET</span> <span class="n">citus</span><span class="p">.</span><span class="n">shard_count</span> <span class="k">TO</span> <span class="mi">10</span><span class="p">;</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'customers'</span><span class="p">,</span> <span class="s1">'customer_id'</span><span class="p">);</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'orders'</span><span class="p">,</span> <span class="s1">'customer_id'</span><span class="p">,</span> <span class="n">colocate_with</span> <span class="p">:</span><span class="o">=</span> <span class="s1">'customers'</span><span class="p">);</span>
<span class="c1">-- when you decide to change the shard count</span>
<span class="c1">-- of all of the colocation group</span>
<span class="k">SELECT</span> <span class="n">alter_distributed_table</span> <span class="p">(</span><span class="s1">'customers'</span><span class="p">,</span> <span class="n">shard_count</span> <span class="p">:</span><span class="o">=</span> <span class="mi">20</span><span class="p">,</span> <span class="n">cascade_to_colocated</span> <span class="p">:</span><span class="o">=</span> <span class="k">true</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SET citus.shard_count TO 10;
SELECT create_distributed_table ('customers', 'customer_id');
SELECT create_distributed_table ('orders', 'customer_id', colocate_with := 'customers');
-- when you decide to change the shard count
-- of all of the colocation group
SELECT alter_distributed_table ('customers', shard_count := 20, cascade_to_colocated := true);
">Copy</button>
</div>
<p>You can see the updated shard count of both tables on the <code>citus_tables</code> view:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">shard_count</span> <span class="k">FROM</span> <span class="n">citus_tables</span> <span class="k">WHERE</span> <span class="k">table_name</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'customers'</span><span class="p">,</span> <span class="s1">'orders'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT shard_count FROM citus_tables WHERE table_name IN ('customers', 'orders');
">Copy</button>
</div>
<h2>How to change your Postgres table’s access method in Citus</h2>
<p>Another amazing feature introduced in Citus 10 is <a href="https://docs.citusdata.com/en/stable/admin_guide/table_management.html#columnar-storage">columnar</a> storage. This <a href="/blog/2021/03/06/citus-10-columnar-compression-for-postgres/">Citus 10 columnar blog post</a> walks you through how it works and how to use columnar tables (or partitions) with Citus—complete with a Quickstart. Oh, and Jeff made a short <a href="https://youtu.be/SS7jcq9fTnw">video demo about the new Citus 10 columnar functionality</a> too—it’s worth the 13 minutes to watch IMHO.</p>
<p>With Citus columnar, you can optionally choose to store your tables grouped by columns—which gives you the benefits of compression, too. Of course, you don’t have to use the new columnar access method—the default access method is "heap" and if you don’t specify an access method, then your tables will be row-based tables (with the heap access method.)</p>
<p>It would not be fair to introduce this cool new Citus columnar access method without also giving you a way to convert your tables to columnar. So Citus 10 also introduced a way to change the access method of tables.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">SELECT</span> <span class="n">alter_table_set_access_method</span><span class="p">(</span><span class="s1">'orders'</span><span class="p">,</span> <span class="s1">'columnar'</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="SELECT alter_table_set_access_method('orders', 'columnar');
">Copy</button>
</div>
<p>You can use <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#alter-table-set-access-method">alter_table_set_access_method</a> to convert your table to any other access method too, such as <code>heap</code>, Postgres’s default access method. Also, your table doesn’t even need to be a distributed Citus table. You can also use <code>alter_table_set_access_method</code> with Citus reference tables as well as regular Postgres tables. You can even change the access method of a Postgres partition with <code>alter_table_set_access_method</code>.</p>
<h2>Under the hood: How do these new Citus functions work?</h2>
<p>If you’ve read the <a href="/blog/2021/02/06/citus-tips-how-to-undistribute-a-distributed-postgres-table/">blog post about undistribute_table</a>, the function Citus 9.5 introduced for turning distributed Citus tables back to local Postgres tables, you mostly know how the <code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> functions work. Because we use the same underlying methodology as the <code>undistribute_table</code> function. Well, we improved upon it.</p>
<p>The <code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> functions:</p>
<ol>
<li>Create a new table in the way you want (with the new shard count or access method etc.)</li>
<li>Move everything from your old table to the new table</li>
<li>Drop the old table and rename the new one</li>
</ol>
<p>Dropping a table for the purpose of re-creating the same table with different properties is not a simple task. Dropping the table will also drop many things that depend on the table.</p>
<p>Just like the <code>undistribute_table</code> function, the <code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> functions do a lot to preserve the properties of the table you didn’t want to change. The functions will handle indexes, sequences, views, constraints, table owner, partitions and more—just like <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#undistribute-table">undistribute_table</a>.</p>
<p><code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> will also recreate the foreign keys on your tables whenever possible. For example, if you change the shard count of a table with the <code>alter_distributed_table</code> function and use <code>cascade_to_colocated := true</code> to change the shard count of all the colocated tables, then foreign keys within the colocation group <em>and</em> foreign keys from the distributed tables of the colocation group to Citus reference tables will be recreated.</p>
<h2>Making it easier to experiment with Citus—and to adapt as your needs change</h2>
<p>If you want to learn more about our previous work which we build on for <code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> functions go check out <a href="/blog/2021/02/06/citus-tips-how-to-undistribute-a-distributed-postgres-table/">our blog post on undistribute_table</a>.</p>
<p>In Citus 10 we worked to give you more tools and more capabilities for making changes to your distributed database. When you’re just starting to use Citus, the new <code>alter_distributed_table</code> and <code>alter_table_set_access_method</code> functions—along with the <code>undistribute_table</code> function—are all here to help you experiment and find the database configuration that works the best for your application. And in the future, if and when your application evolves, these three Citus functions will be ready to help you evolve your Citus database, too.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2021/05/03/citus-tips-for-postgres-how-to-alter-distribution-key-shard-count/'>citusdata.com</a>.</em></p>Citus Tips: How to undistribute a distributed Postgres tablehttps://www.citusdata.com/blog/2021/02/06/citus-tips-how-to-undistribute-a-distributed-postgres-table/2021-02-06T17:35:00+00:002021-02-06T17:35:00+00:00Halil Ozan Akgul<p>Once you start using the Citus extension to distribute your Postgres database, you may never want to <em>go back</em>. But what if you just want to experiment with Citus and want to have the comfort of knowing you can go back? Well, as of <a href="/blog/2020/11/14/citus-9-5-whats-new/">Citus 9.5</a>, now there is a new <code>undistribute_table()</code> function to make it easy for you to, well, to revert a distributed table back to being a regular Postgres table.</p>
<p>If you are familiar with <a href="https://github.com/citusdata/citus">Citus</a>, you know that Citus is an open source extension to <a href="https://www.postgresql.org/">Postgres</a> that distributes your data (and queries) to multiple machines in a cluster—thereby parallelizing your workload and scaling your Postgres database horizontally. When you start using Citus—whether you’re using Citus open source or whether you’re using <a href="https://learn.microsoft.com/azure/cosmos-db/postgresql/overview">Citus as part of a managed service</a> in the cloud—usually the first thing you need to do is distribute your Postgres tables across the cluster. </p>
<h2>What is undistribute_table()?</h2>
<p>To distribute your Postgres tables with the <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#create-distributed-table">create_distributed_table()</a> function of Citus, you first need to make some decisions, such as: which column to choose as the <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html">distribution column</a>, how many shards you need, and which Postgres tables you need to distribute.</p>
<p>If you just want to try different settings and go back when you want to, you're now in luck. Our Citus team introduced the <code>undistribute_table()</code> function in the <a href="/blog/2020/11/14/citus-9-5-whats-new/">Citus 9.5 release</a>—enabling you to turn distributed Citus tables back into regular Postgres tables.</p>
<p>If you are one of the Citus users who has asked for the ability to undistribute your Citus tables—like in the request below from <a href="https://stackify.com/sql-percentile-aggregates-and-rollups-with-postgresql-and-t-digest/">Matt Watson of Stackify</a>—we hope this new feature will help you.</p>
<blockquote>
<p>Also, is there a way to convert a distributed table to not being distributed? I could then change it back to distributed and fix my colocate… without having to drop the table.</p>
</blockquote>
<p>The new <code>undistribute_table()</code> function will:</p>
<ul>
<li>return all the data of a distributed table from the Citus worker nodes back to the Citus coordinator node,</li>
<li>remove all the shards of the distributed table from the Citus workers,</li>
<li>make the previously distributed table a local Postgres table on the Citus coordinator node</li>
</ul>
<p>Here is the simplest code example of going distributed with Citus and coming back:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- First distribute your table</span>
<span class="k">SELECT</span> <span class="n">create_distributed_table</span> <span class="p">(</span><span class="s1">'my_table'</span><span class="p">,</span> <span class="s1">'id'</span><span class="p">);</span>
<span class="c1">-- Now your table has shards on the worker nodes and any data that was in the table is distributed to those shards.</span>
<span class="c1">-- To go back to local, just call the undistribute_table function with your table as parameter</span>
<span class="k">SELECT</span> <span class="n">undistribute_table</span><span class="p">(</span><span class="s1">'my_table'</span><span class="p">);</span>
<span class="c1">-- Now your table is only on the coordinator node just like before you distributed.</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- First distribute your table
SELECT create_distributed_table ('my_table', 'id');
-- Now your table has shards on the worker nodes and any data that was in the table is distributed to those shards.
-- To go back to local, just call the undistribute_table function with your table as parameter
SELECT undistribute_table('my_table');
-- Now your table is only on the coordinator node just like before you distributed.
">Copy</button>
</div>
<p>Undistributing a Citus table is as simple as the one line of SQL code in the code block above.</p>
<p>Note that when you distribute a Postgres table with Citus you need to pass the <a href="https://docs.citusdata.com/en/stable/sharding/data_modeling.html">distribution column</a> into the <code>create_distributed_table()</code> function—but when undistributing, the only parameter you need to pass into the <code>undistribute_table()</code> function is the table name itself.</p>
<p>After undistributing, the distribution column becomes a regular column. If in the future, you want to distribute your Postgres table again, you can just pick another distribution column (or use the same one).</p>
<p>In the past, before we introduced the <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#undistribute-table">undistribute_table()</a> function in Citus 9.5, if you wanted to turn a distributed table back into a local table, you would have had to create a new Postgres table on your coordinator node. Then, you would have needed to move all the data from the distributed table to this new local table. However, Citus did not have an easy way to move data from distributed Citus tables to local Postgres tables so you would have had to do some workarounds. Let me explain:</p>
<h2>The usefulness of INSERT INTO local SELECT .. FROM distributed</h2>
<p>To undistribute a table, distributed data needs to be moved back to the Citus coordinator from all the shards in the cluster. But prior to the <a href="/blog/2020/09/05/citus-9-4-whats-new/">Citus 9.4 release</a>, Citus did not support queries that SELECT from distributed tables and INSERT into local tables. So, there was a need to implement support for:</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">local_table</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">distributed_table</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="INSERT INTO local_table SELECT * FROM distributed_table;
">Copy</button>
</div>
<p>In fact, the <code>INSERT INTO local SELECT .. FROM distributed</code> feature was introduced in Citus 9.4 to make the <code>undistribute_table()</code> function possible.</p>
<p>Other than being necessary for undistributing tables, inserting distributed data into local tables has some more beneficial use cases.</p>
<h3>Rollup Tables</h3>
<p>A <a href="https://docs.citusdata.com/en/stable/develop/reference_dml.html#caching-aggregations-with-rollups">rollup table</a> in Postgres is a table that you pre-aggregate your data into. Before we introduced <code>INSERT INTO local SELECT .. FROM distributed</code> in Citus 9.4, you could still have rollup tables. (And many of you did!) But your rollup tables had to be distributed tables, which may not have been the best option in every case. Especially if your rollup table was a very small table.</p>
<p>Let me give you an example.</p>
<p>Let's say you have a distributed table and a graph that shows some daily statistics of the data on that table. Instead of calculating the statistics from scratch every time you open the graph, you can now create a local Postgres table on the Citus coordinator that you will rollup into. Every night, you can calculate the statistics value for the day and insert the result of the calculations to the rollup table. When you open the graph, the data will be readily available.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Every midnight</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">rollup_table</span> <span class="k">SELECT</span> <span class="n">your_analysis_function</span><span class="p">(</span><span class="n">statistics_column</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">distributed_table</span> <span class="k">WHERE</span> <span class="nb">date</span> <span class="o">=</span> <span class="k">CURRENT_DATE</span><span class="p">;</span>
<span class="c1">-- When you need the graph</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">rollup_table</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Every midnight
INSERT INTO rollup_table SELECT your_analysis_function(statistics_column) FROM distributed_table WHERE date = CURRENT_DATE;
-- When you need the graph
SELECT * FROM rollup_table;
">Copy</button>
</div>
<h3>ETL in the Database</h3>
<p>ETL (Extract, Transform, Load) is the process of gathering data from a data source, transforming the data into a more meaningful form, and then storing the transformed data. Imagine running an online store, and imagine you have a distributed table for customer data and another distributed table for purchases the customers made. What if you need to find the best 100 customers and send them e-mails about a special discount for the top customers?</p>
<p>With the new <code>INSERT INTO local SELECT .. FROM distributed</code> feature and the ETL logic, you can create a local Postgres table for your best customers.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Create the table for the top customers</span>
<span class="k">CREATE</span> <span class="k">TEMP</span> <span class="k">TABLE</span> <span class="n">top_customers</span> <span class="p">(</span><span class="n">customer_id</span> <span class="nb">bigint</span> <span class="k">primary</span> <span class="k">key</span><span class="p">,</span> <span class="n">email</span> <span class="nb">text</span><span class="p">,</span> <span class="n">total_purchase</span> <span class="n">money</span><span class="p">);</span>
<span class="c1">-- Find the best customers and put their data into the top_customers table</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">top_customers</span>
<span class="k">SELECT</span> <span class="n">customer_id</span><span class="p">,</span> <span class="n">email</span><span class="p">,</span> <span class="n">total_purchase</span>
<span class="k">FROM</span> <span class="n">customers</span> <span class="k">JOIN</span>
<span class="p">(</span>
<span class="k">SELECT</span> <span class="k">sum</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span> <span class="k">AS</span> <span class="n">total_purchase</span><span class="p">,</span> <span class="n">customer_id</span>
<span class="k">FROM</span> <span class="n">purchases</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">customer_id</span>
<span class="p">)</span> <span class="n">total_purchases</span> <span class="k">ON</span> <span class="n">customers</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">total_purchases</span><span class="p">.</span><span class="n">customer_id</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">total_purchase</span> <span class="k">DESC</span>
<span class="k">LIMIT</span> <span class="mi">100</span><span class="p">;</span>
<span class="c1">-- Load the top customer IDs back into the distributed table</span>
<span class="k">UPDATE</span> <span class="n">customers</span> <span class="k">SET</span> <span class="n">is_top_customer</span> <span class="o">=</span> <span class="k">true</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="k">IN</span> <span class="p">(</span><span class="k">SELECT</span> <span class="n">customer_id</span> <span class="k">FROM</span> <span class="n">top_customers</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Create the table for the top customers
CREATE TEMP TABLE top_customers (customer_id bigint primary key, email text, total_purchase money);
-- Find the best customers and put their data into the top_customers table
INSERT INTO top_customers
SELECT customer_id, email, total_purchase
FROM customers JOIN
(
SELECT sum(amount) AS total_purchase, customer_id
FROM purchases
GROUP BY customer_id
) total_purchases ON customers.id = total_purchases.customer_id
ORDER BY total_purchase DESC
LIMIT 100;
-- Load the top customer IDs back into the distributed table
UPDATE customers SET is_top_customer = true WHERE id IN (SELECT customer_id FROM top_customers);
">Copy</button>
</div>
<h3>Increased support for INSERT SELECT in Citus</h3>
<p>As of Citus 9.4 any INSERT SELECT command works!</p>
<p>The logic for <code>INSERT INTO local SELECT .. FROM distributed</code> queries is quite similar to the logic for <code>SELECT .. FROM distributed</code>. When you just want to get the distributed data with SELECT, Citus will:</p>
<ul>
<li>gather data from the Citus distributed worker nodes</li>
<li>combine the data, on the Citus coordinator node</li>
<li>return the combined data back to you</li>
</ul>
<p>If you want to <code>INSERT INTO local SELECT .. FROM distributed</code>, Citus does all the steps the same way, except for the last one. In the last step, instead of returning the combined data to you, Citus inserts the data to the local Postgres table on the Citus coordinator node.</p>
<p>After all the engineering effort, it would be selfish to keep the <code>INSERT INTO local SELECT .. FROM distributed</code> feature just for internal use. So, we added support for the feature in <a href="/blog/2020/09/05/citus-9-4-whats-new/">Citus 9.4</a>.</p>
<h2>What does the Citus undistribute_table() function do, under the hood?</h2>
<p>So as of Citus 9.4, with help from the new <code>INSERT INTO local SELECT .. FROM distributed</code> feature, you could undistribute your tables manually, if you needed to revert. To undistribute Citus tables manually, you used to have to:</p>
<ol>
<li>Create a new Postgres table</li>
<li>Insert Select everything from your old, distributed table into the new Postgres table</li>
<li>Drop the old table and rename the new table.</li>
</ol>
<p>That might seem easy enough, but that's not all. Some of the things you might have also had to deal with:</p>
<ol>
<li>keep the Postgres indexes you had on the old distributed Citus table in mind…</li>
<li>create partitions again, if your table was a partitioned table…</li>
<li>deal with the fact that while dropping your table, you also dropped any views that depended on your distributed table—and the views that depended on those views, too.</li>
</ol>
<p>The good news is that as of <a href="/blog/2020/11/14/citus-9-5-whats-new/">Citus 9.5</a> or later—you can now use the new <code>undistribute_table()</code> function and let Citus seamlessly handle everything. Specifically, when you use the <code>undistribute_table()</code> function, Citus automatically:</p>
<ul>
<li>creates the indexes you had for the distributed table,</li>
<li>handles sequences owned by the table so they continue from where you left off,</li>
<li>recursively finds the views that directly or indirectly depend on your table and moves them to the new Postgres table,</li>
<li>preserves constraints, and the table owner,</li>
<li>if your table was a partitioned table, does all these steps for the partitions,</li>
<li>and more…</li>
</ul>
<h2>Bottom line: undistribute_table() makes it easier to experiment with Citus distributed tables</h2>
<p>Hopefully it's interesting to know a bit more about why our Citus team introduced the <code>INSERT INTO local SELECT .. FROM distributed</code> feature in Citus 9.4—and the <a href="https://docs.citusdata.com/en/stable/develop/api_udf.html#undistribute-table">undistribute_table()</a> function in Citus 9.5.</p>
<p>The most important thing to know is that distributing a Postgres table with Citus is not a one-way street. It's easy to go back and to undistribute a Citus table. So if you want to <a href="/getting-started/">get started with Citus</a>, it's now easier to experiment—<em>as long as you're running Citus 9.5 or later</em>. After <a href="/download/">downloading</a> the Citus open source packages—or <a href="https://learn.microsoft.com/azure/cosmos-db/postgresql/quickstart-create-portal">provisioning a Hyperscale (Citus)</a> server group on Azure—you can distribute your tables or make your tables reference tables and then undistribute back to local Postgres tables—and find what data model works best for you and your application. And if you change your mind later, you can just undistribute again.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2021/02/06/citus-tips-how-to-undistribute-a-distributed-postgres-table/'>citusdata.com</a>.</em></p>