Citus Data BlogCitus DataScaling data and analytics with Postgreshttps://www.citusdata.com/blog/2024-03-01T20:58:00+00:00What’s in a name? Hello POSETTE: An Event for Postgres 2024https://www.citusdata.com/blog/2024/03/01/whats-in-a-name-hello-posette-an-event-for-postgres-2024/2024-03-01T20:58:00+00:002024-03-01T20:58:00+00:00Claire Giordano<p>When I think about naming something—like a feature or product or even an event—this quote always comes to mind. </p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p>What’s in a name? That which we call a rose<br>
By any other name would smell as sweet;</p>
<p>–William Shakespeare</p>
</blockquote>
<p>What’s in a name, after all? I’m no expert on Romeo and Juliet, but friends tell me Shakespeare’s point was that names don’t matter. The thing itself is the thing itself, regardless of the name.</p>
<p>My parents named my sister “Helen” at birth but never actually called her that. They always called her by a nickname, “Lyena”. So my sister’s sense of self became intertwined with her nickname: she “felt” like a Lyena. And the only people that ever called her Helen were officious school principals, gate-check agents looking at her passport—and our paternal grandfather. It made her so mad. Whenever my grandfather insisted on calling her Helen, you could almost see the steam coming out of my sister’s ears.</p>
<p>My husband told me about a thing I’ve unconsciously done for years: whenever we drive through Suisun City en route to the mountains, I say the name of the city out loud to myself. Not just once but several times, like I’m chewing on the word. Turns out I really like the way it feels when I say “Suh-soon-si-tee” out loud.</p>
<p>Names carry meaning. They trigger emotions. The phonetic sound of a word affects whether you can remember it. And some words just “roll off the tongue” in a way that makes it easy to say <em>and</em> easy to remember. Bottom line, names matter.</p>
<p>Which is why we decided to give “Citus Con: An Event for Postgres” a new name. People had told us that when they heard the event’s nickname of “Citus Con” they thought it was <em>only</em> about Citus—and did not realize that over 66% of <a href="https://aka.ms/cituscon-playlist-2023">last year’s Citus Con talks</a> were about Postgres, and not about Citus.</p>
<p>Say hello to <strong><a href="/posette/">POSETTE: An Event for Postgres</a></strong>, now in its 3rd year. A free and virtual developer event brought to you with 🧡 by the Postgres team here at Microsoft. </p>
<h2>What does the “POSETTE” in “POSETTE: An Event for Postgres” stand for?</h2>
<p><a href="/posette">POSETTE: An Event for Postgres</a> is a name inspired by developer acronyms. More specifically, the inspiration for the name POSETTE came from FOSDEM.</p>
<p>FOSDEM is an open source developer conference that happens every year in the deep cold winter in Brussels. But in the developer world, some people don’t realize that FOSDEM was initially OSDEM. And many FOSDEM attendees probably can’t tell you what all the letters in the acronym stand for. I mean, it’s sort of obvious that FOSDEM stands for “Free” and “Open Source” and probably “Developer” too—but after that, who knows what the “E” and the “M” stand for. Am I right?</p>
<p>POSETTE is pronounced /Pō-zet/ and stands for <strong><u>P</u></strong>ostgres <strong><u>O</u></strong>pen <strong><u>S</u></strong>ource <strong><u>E</u></strong>cosystem <strong><u>T</u></strong>alks <strong><u>T</u></strong>raining & <strong><u>E</u></strong>ducation.</p>
<p>The words behind the acronym are intentional. We aim to have the <strong><u>T</u></strong>alks focus on <strong><u>P</u></strong>ostgres and the entire <strong><u>E</u></strong>cosystem of tooling and extensions and community initiatives—to help <strong><u>E</u></strong>ducate and <strong><u>T</u></strong>rain the growing numbers of Postgres <strong><u>O</u></strong>pen <strong><u>S</u></strong>ource users and developers and contributors—and of course Azure Database for PostgreSQL customers too!</p>
<p>The word <strong>Ecosystem</strong> in the name POSETTE is particularly important: we welcome <a href="/posette/2024/cfp/">CFP talk proposals</a> about the entire Postgres ecosystem from the core of the database to tooling to PG extensions. And there are so many useful Postgres extensions (including Citus!)</p>
<h2>What is POSETTE (formerly Citus Con)—and why did we create it?</h2>
<p>If you’re learning about <a href="/posette/">POSETTE: An Event for Postgres</a> for the first time, the key things to know are:</p>
<ul>
<li>it is a free & virtual developer event</li>
<li>in its 3rd year</li>
<li>with 4 different & unique livestreams will happen Jun 11-13, 2024</li>
<li>livestreams will also have a live, interactive chat where you can connect with speakers and attendees</li>
<li>all talks made available on YouTube after the event is over, so you can watch at your convenience later <a href="https://youtube.com/shorts/NujyLhklIrI?feature=shared">at 2X speed even</a></li>
<li>event is brought to you with 🧡 by the Postgres team at Microsoft</li>
<li>the <a href="/posette/2024/cfp/">CFP is currently open</a> for talk proposals until Apr 07</li>
<li>there is of course a <a href="/posette/2024/coc/">Code of Conduct</a> for POSETTE</li>
<li>formerly called Citus Con</li>
<li>you can stay connected by following <code>@PosetteConf</code> on <a href="https://twitter.com/PosetteConf">X/Twitter</a>, <a href="https://mastodon.social/@posetteconf">Mastodon</a>, or <a href="https://www.threads.net/@posetteconf">Threads</a>—or popping into the <code>#posetteconf</code> channel on the <a href="https://aka.ms/open-source-discord">Microsoft Open Source Discord</a></li>
</ul>
<p>And why did we create this virtual Postgres event? Virtual, free, and global—no matter where you are in the world, as long as you have an internet connection, you can participate.</p>
<p>The appeal of a high-quality, INCLUSIVE virtual event is real. I <em>love</em> in-person events and it’s no secret that PGConfEU is my favorite PG event. And I’m so honored to be presenting at <a href="https://www.postgresql.eu/events/nordicpgday2024/schedule/">Nordic PGDay</a> on March 12, and to be giving a <a href="https://www.postgresql.eu/events/pgdayparis2024/schedule/session/5425-lightning-talks/">lightning talk at pgDay Paris</a> on March 14. But it’s a fact that many people cannot travel to in-person events, due to lack of travel budget or family responsibilities or a multitude of different reasons. So we are <a href="https://x.com/clairegiordano/status/1752772004179333280?s=20">big fans of the accessibility</a> and inclusiveness of virtual events, especially when there are great speakers with high-quality talks and well-produced videos.</p>
<p>We hope you can join us for POSETTE—and please help to spread the word, so more Postgres users and developers discover it!</p>
<p>In the meantime, +1 and <strong>gratitude</strong> to all the a-ma-zing Postgres speakers who are submitting into the <a href="/posette/2024/cfp/">CFP for POSETTE</a> and who believe in the value of <a href="/blog/2022/01/11/why-give-a-conference-talk/">giving Postgres conference talks</a>. Also, +1 and <strong>thanks</strong> to my teammates working hard behind the scenes at Microsoft to make this Postgres event happen, with special thanks to our 2024 POSETTE organizing chair, Teresa Giacomini.</p>
<figure>
<picture>
<source srcset="https://cituscdn.azureedge.net/images/blog/blog-how-to-pronounce-1200x675.webp" type="image/webp">
<img src="https://cituscdn.azureedge.net/images/blog/blog-how-to-pronounce-1200x675.jpg" alt="how to pronounce POSETTE" loading="lazy" width="850" height="478" style="box-shadow:0 0 12px rgba(0,0,0,.2)" />
</picture>
<figcaption><strong>Figure 1:</strong> This graphic makes it easy to “see” what the word POSETTE in <a href="/posette/">POSETTE: An Event for Postgres 2024</a> stands for. The CFP is open until Apr 07 2024—and the free & virtual event will happen on June 11-13, 2024.</figcaption>
</figure>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2024/03/01/whats-in-a-name-hello-posette-an-event-for-postgres-2024/'>citusdata.com</a>.</em></p>Podcast about transitioning from developer to PostgreSQL specialist, with Derk van Veenhttps://www.citusdata.com/blog/2024/02/28/podcast-about-transitioning-from-dev-to-postgres-specialist/2024-02-28T18:07:00+00:002024-02-28T18:07:00+00:00Ari Padilla<p>How do you feel when your day doesn’t go as planned? In this episode of the Path To Citus Con, the podcast for developers who love Postgres, guest <a href="https://www.linkedin.com/in/derk-van-veen-database-specialist/">Derk van Veen</a> joins co-hosts <a href="https://hachyderm.io/@clairegiordano">Claire Giordano</a> and <a href="https://www.linkedin.com/in/pinodecandia">Pino de Candia</a> to talk about his journey from Java developer to Postgres specialist.</p>
<p>What makes you feel alive at work? Is it the routine tasks, the predictable outcomes, the stable environment? Or is it the unexpected challenges, the unknown variables, the chaotic situations? If you are like Derk (and I), you thrive on the latter. Maybe you love to jump on tough problems and find beautiful solutions—or maybe you enjoy the thrill of finding the root cause of a slow system or some faulty code. You don't just follow a recipe. You ask questions, explore options, and experiment with different strategies. How do you partition a table? Why do you partition a table? What are the trade-offs of each approach? </p>
<p>In this post, you’ll find some of our favorite episode highlights and quotes. You’ll find links to where you can subscribe and listen to past episodes of the podcast at the end of the post.</p>
<figure>
<picture>
<source srcset="https://cituscdn.azureedge.net/images/blog/PathToCitusCon-Ep12-From-developer-to-PostgreSQL-youtube-1200x675.webp" type="image/webp">
<img src="https://cituscdn.azureedge.net/images/blog/PathToCitusCon-Ep12-From-developer-to-PostgreSQL-youtube-1200x675.jpg" loading="lazy" width="850" height="478" alt="Path to Citus Con Ep. 12 YouTube thumbnail" />
</picture>
<figcaption><strong>Figure 1:</strong> YouTube thumbnail for episode 12 of the Path To Citus Con, the podcast for developers who love Postgres, with (starting in the top left, listed clockwise) Derk van Veen, Claire Giordano, and Pino de Candia. The topic is “From developer to Postgres specialist.”</figcaption>
</figure>
<h2>Highlights from the podcast episode with Derk van Veen</h2>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“It always is about the why. Why do you share this? Why do you have to tell this to this audience? Why do you have this slide? Why is this visual on your slide? It's always about the why. Every part of it.”</strong> – Derk van Veen</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>When talking about conferences and public speaking, Derk shares how to sell a story, meaning that every part of a presentation and what you say needs to be intentional. You need to understand why it is there and make sure it is there for the right reasons. The idea is to guide the audience along a straight line, a smooth path, without unnecessary distractions or deviations.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“Usually first my heart rate doubles and I'm like, all right, this is not good. And I try first to take a step back. Like, just walk down the stairs, one floor, get a cup of tea, take a few deep breaths, go back up and then just start slowly and methodically [tackling] the issue.”</strong> – Derk van Veen</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>When your heart starts racing, it's like your body's alarm system is kicking in. But instead of letting the panic take over, it's about taking control. Stepping back, even just for a moment, can work wonders and this is what Derk does when things aren’t going as planned. Sometimes a small break is all you need to hit the reset button. Then, when you're ready, tackle the problem one step at a time.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“So you can do all this development work. You can see whether it works out and you don't have to worry about breaking something because you're on your own fork.”</strong> – Derk van Veen</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>Forks are freedom. Derk discusses that at first, he saw forks as a waste of effort. However, through talking to colleagues and giving it a try, he realized that forking is not just a technical skill. It’s also a social one. It is a way of collaborating with other developers, sharing your ideas, and editing the code without breaking things for everyone else. It’s a way to contribute to the software and understand if what you are creating will work, be useful, and help others.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“[The slide] has to be very easy, easy to digest, easy to understand. The easier the better. It's a contribution to the story, to you as a presenter, to what you're trying to share with your audience. If you want to share bullet points with your audience, just send them an email.”</strong> – Derk van Veen</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>According to Derk, when it comes to your presentation slides, simplicity is your secret weapon. Think about it—your slides should practically speak for themselves. If they're too complicated, your audience might miss the point entirely. Keep it easy, digestible, and clear. Your slides aren't just background noise; they're an essential part of your storytelling.</p>
<h2>Where to find Path To Citus Con podcast episodes (and transcripts)</h2>
<p>You can listen to and find the show notes online for this <a href="https://pathtocituscon.transistor.fm/episodes/from-developer-to-postgresql-specialist-with-derk-van-veen">From developer to Postgres specialist</a> episode with Derk van Veen—and the <a href="https://pathtocituscon.transistor.fm/episodes/from-developer-to-postgresql-specialist-with-derk-van-veen/transcript">transcript</a> is online too.</p>
<p>Our next podcast episode will be recorded live:</p>
<ul>
<li><strong>Where:</strong> Microsoft Open Source Discord</li>
<li><strong>When:</strong> Wed Mar 06 @ 10am PST | 1pm EST | 6pm UTC</li>
<li><strong>Guest:</strong> with the amazing <a href="https://www.linkedin.com/in/aytekinar/">Arda Aytekin</a>, to talk about</li>
<li><strong>Topic:</strong> spinning up on Postgres & AI</li>
<li><strong>Calendar invite:</strong> <a href="https://aka.ms/pathtocituscon-ep13-cal">Mark your calendar</a> to participate in the live text chat that happens in parallel to the live recording (we’re huge fans of social audio on Discord). All the instructions for joining the live show are in the cal invite.</li>
</ul>
<p>You can find all the past episodes for the Path To Citus Con podcast on:</p>
<ul>
<li><a href="https://podcasts.apple.com/us/podcast/path-to-citus-con/id1695014346">Apple Podcasts</a></li>
<li><a href="https://open.spotify.com/show/6XtjTEc4KGMK9fIGvmPLn7">Spotify</a></li>
<li><a href="https://aka.ms/PathToCitusCon-playlist">YouTube</a></li>
<li><a href="https://pathtocituscon.transistor.fm/subscribe">And many more podcast platforms</a></li>
</ul>
<p>More useful links:</p>
<ul>
<li><a href="/podcast/path-to-citus-con/">Index of all past Path To Citus Con episodes</a></li>
<li><a href="https://aka.ms/PathToCitusCon-cal">Subscribe to the calendar</a> to know when all the future shows are coming up here.</li>
</ul>
<p>Thanks for listening! We hope you enjoy the episodes of the Path To Citus Con podcast and would love that you share it with friends and the database community. Rating and reviewing us on your favorite podcasting platform will also help others discover it.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2024/02/28/podcast-about-transitioning-from-dev-to-postgres-specialist/'>citusdata.com</a>.</em></p>What’s new in the Postgres 16 query planner / optimizerhttps://www.citusdata.com/blog/2024/02/08/whats-new-in-postgres-16-query-planner-optimizer/2024-02-08T17:16:00+00:002024-02-08T17:16:00+00:00David Rowley<p>PostgreSQL 16 introduces quite a few improvements to the query planner and makes many SQL queries run faster than they did on previous versions of PostgreSQL.</p>
<p>If you look at the <a href="https://www.postgresql.org/docs/16/release-16.html">PG16 release notes</a>, you’ll see some of these planner improvements. But with the volume of changes made in each PostgreSQL release, it’s not possible to provide enough detail about each and every change. So maybe you might need a bit more detail to know what the change is about—before you understand if it’s relevant to you.</p>
<p>In this blog post, assuming you’ve already got a handle on the <a href="https://www.postgresql.org/docs/16/using-explain.html#USING-EXPLAIN-BASICS">basics of EXPLAIN</a>, you’ll get a deep dive into the 10 improvements made in the PostgreSQL 16 query planner. For each of the improvements to the PG16 planner (the planner is often called an optimizer in other relational databases), you’ll also get comparisons between PG15 and PG16 planner output—plus examples of what changed, in the form of a self-contained test you can try for yourself. </p>
<p>Let’s dive into these 10 improvements to the PostgreSQL planner in PG16:</p>
<ol>
<li><a href="#distinct-queries">Incremental sorts for DISTINCT queries</a></li>
<li><a href="#faster-orderby-distinct">Faster ORDER BY / DISTINCT aggregates</a></li>
<li><a href="#union-all">Memoize for UNION ALL queries</a></li>
<li><a href="#right-anti-join">Support Right Anti Join</a></li>
<li><a href="#parallel-hashfull-rightjoin">Parallel Hash Full and Right Joins</a></li>
<li><a href="#frame-clauses">Optimize window function frame clauses</a></li>
<li><a href="#window-functions">Optimize various window functions</a></li>
<li><a href="#join-removals">JOIN removals for partitioned tables</a></li>
<li><a href="#short-circuit">Short circuit trivial DISTINCT queries</a></li>
<li><a href="#merge-joins">Incremental Sort after Merge Join, in more cases</a></li>
</ol>
<h2 id="distinct-queries">1. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=3c6fc58209f24b959ee18f5d19ef96403d08f15c">Allow incremental sorts in more cases, including DISTINCT (David Rowley)</a></h2>
<p><a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=d2d8a229bc58a2014dce1c7a4fcdb6c5ab9fb8da">Incremental sorts</a> were first added in PostgreSQL 13. These incremental sorts reduce the effort required to get sorted results. How? By exploiting the knowledge that some given result set is already sorted by 1 or more leading columns—and only performing a sort on the remaining columns.</p>
<p>For example, if there’s a btree index on column <code>a</code> and we need the rows ordered by <code>a</code>,<code>b</code>, then we can use the btree index (which provides presorted results on column <code>a</code>) and sort the rows seen so far only when the value of <code>a</code> changes. With the quicksort algorithm used by PostgreSQL, sorting many smaller groups is more efficient than sorting one large group.</p>
<p>The PostgreSQL 16 query planner now considers performing incremental sorts for <code>SELECT DISTINCT</code> queries. Prior to PG16, when the sorting method was chosen for <code>SELECT DISTINCT</code> queries, the planner only considered performing a full sort (which is more expensive than an incremental sort.)</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">distinct_test</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">b</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">distinct_test</span>
<span class="k">SELECT</span> <span class="n">x</span><span class="p">,</span><span class="mi">1</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span><span class="n">x</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="k">on</span> <span class="n">distinct_test</span><span class="p">(</span><span class="n">a</span><span class="p">);</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">distinct_test</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="k">DISTINCT</span> <span class="n">a</span><span class="p">,</span><span class="n">b</span> <span class="k">FROM</span> <span class="n">distinct_test</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE distinct_test (a INT, b INT);
INSERT INTO distinct_test
SELECT x,1 FROM generate_series(1,1000000)x;
CREATE INDEX on distinct_test(a);
VACUUM ANALYZE distinct_test;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT DISTINCT a,b FROM distinct_test;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------
HashAggregate (actual rows=1000000 loops=1)
Group Key: a, b
Batches: 81 Memory Usage: 11153kB Disk Usage: 31288kB
-> Seq Scan on distinct_test (actual rows=1000000 loops=1)
Planning Time: 0.065 ms
Execution Time: 414.226 ms
(6 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------
HashAggregate (actual rows=1000000 loops=1)
Group Key: a, b
Batches: 81 Memory Usage: 11153kB Disk Usage: 31288kB
-> Seq Scan on distinct_test (actual rows=1000000 loops=1)
Planning Time: 0.065 ms
Execution Time: 414.226 ms
(6 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
------------------------------------------------------------------
Unique (actual rows=1000000 loops=1)
-> Incremental Sort (actual rows=1000000 loops=1)
Sort Key: a, b
Presorted Key: a
Full-sort Groups: 31250 Sort Method: quicksort Average Memory: 26kB Peak Memory: 26kB
-> Index Scan using distinct_test_a_idx on distinct_test (actual rows=1000000 loops=1)
Planning Time: 0.108 ms
Execution Time: 263.167 ms
(8 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
------------------------------------------------------------------
Unique (actual rows=1000000 loops=1)
-> Incremental Sort (actual rows=1000000 loops=1)
Sort Key: a, b
Presorted Key: a
Full-sort Groups: 31250 Sort Method: quicksort Average Memory: 26kB Peak Memory: 26kB
-> Index Scan using distinct_test_a_idx on distinct_test (actual rows=1000000 loops=1)
Planning Time: 0.108 ms
Execution Time: 263.167 ms
(8 rows)
">Copy</button>
</div>
<p>In the PostgreSQL 16 <code>EXPLAIN</code> output above, you can see the planner chose to use the <code>distinct_test_a_idx</code> index on the <code>a</code> column and then performed an <code>Incremental Sort</code> to sort all of the equal values of <code>a</code> by <code>b</code>. The <code>Presorted Key: a</code> indicates this. Because the <code>INSERT</code> statements above only added a single value of <code>b</code> for each value of <code>a</code>, each batch of tuples sorted by incremental sort only contains a single row.</p>
<p>The <code>EXPLAIN</code> output for PostgreSQL 16 above shows that the <code>Peak Memory</code> for the <code>Incremental Sort</code> was just 26 kilobytes, whereas the hashing method used by PostgreSQL 15 needed much memory, so much so that it needed to spill about 30 megabytes of data to disk. <strong>The query executed 63% faster on PostgreSQL 16</strong>.</p>
<h2 id="faster-orderby-distinct">2. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=1349d2790bf48a4de072931c722f39337e72055e">Add the ability for aggregates having ORDER BY or DISTINCT to use pre-sorted data (David Rowley)</a></h2>
<p>In PostgreSQL 15 and earlier, aggregate functions containing an <code>ORDER BY</code> or <code>DISTINCT</code> clause would result in the executor always performing a sort inside the <code>Aggregate</code> node of the plan. Because the sort was always performed, the planner would never try to form a plan to provide presorted input to aggregate the rows in order.</p>
<p>The PostgreSQL 16 query planner now tries to form a plan which feeds the rows to the plan’s <code>Aggregate</code> node in the correct order. And the executor is now smart enough to recognize this and forego performing the sort itself when the rows are already pre-sorted in the correct order.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">aggtest</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">b</span> <span class="nb">text</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">aggtest</span> <span class="k">SELECT</span> <span class="n">a</span><span class="p">,</span><span class="n">md5</span><span class="p">((</span><span class="n">b</span><span class="o">%</span><span class="mi">100</span><span class="p">)::</span><span class="nb">text</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">)</span> <span class="n">a</span><span class="p">,</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">100000</span><span class="p">)</span><span class="n">b</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="k">ON</span> <span class="n">aggtest</span><span class="p">(</span><span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">);</span>
<span class="k">VACUUM</span> <span class="k">FREEZE</span> <span class="k">ANALYZE</span> <span class="n">aggtest</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">BUFFERS</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">a</span><span class="p">,</span><span class="k">COUNT</span><span class="p">(</span><span class="k">DISTINCT</span> <span class="n">b</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">aggtest</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">a</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE aggtest (a INT, b text);
INSERT INTO aggtest SELECT a,md5((b%100)::text) FROM generate_series(1,10) a, generate_series(1,100000)b;
CREATE INDEX ON aggtest(a,b);
VACUUM FREEZE ANALYZE aggtest;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, BUFFERS)
SELECT a,COUNT(DISTINCT b) FROM aggtest GROUP BY a;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------
GroupAggregate (actual rows=10 loops=1)
Group Key: a
Buffers: shared hit=892, temp read=4540 written=4560
-> Index Only Scan using aggtest_a_b_idx on aggtest (actual rows=1000000 loops=1)
Heap Fetches: 0
Buffers: shared hit=892
Planning Time: 0.122 ms
Execution Time: 302.693 ms
(8 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------
GroupAggregate (actual rows=10 loops=1)
Group Key: a
Buffers: shared hit=892, temp read=4540 written=4560
-> Index Only Scan using aggtest_a_b_idx on aggtest (actual rows=1000000 loops=1)
Heap Fetches: 0
Buffers: shared hit=892
Planning Time: 0.122 ms
Execution Time: 302.693 ms
(8 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------
GroupAggregate (actual rows=10 loops=1)
Group Key: a
Buffers: shared hit=892
-> Index Only Scan using aggtest_a_b_idx on aggtest (actual rows=1000000 loops=1)
Heap Fetches: 0
Buffers: shared hit=892
Planning Time: 0.061 ms
Execution Time: 115.534 ms
(8 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------
GroupAggregate (actual rows=10 loops=1)
Group Key: a
Buffers: shared hit=892
-> Index Only Scan using aggtest_a_b_idx on aggtest (actual rows=1000000 loops=1)
Heap Fetches: 0
Buffers: shared hit=892
Planning Time: 0.061 ms
Execution Time: 115.534 ms
(8 rows)
">Copy</button>
</div>
<p>Aside from PostgreSQL 16 executing the query over twice as fast as in PG15, the only indication of this change in the <code>EXPLAIN ANALYZE</code> output above is from the <code>temp read=4540 written=4560</code> that’s not present in the PostgreSQL 16 output. In PG15, this is caused by the implicit sort spilling to disk.</p>
<h2 id="union-all">3. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=9bfd2822b3201f6b0de1e87305b11ee3885b36d9">Allow memoize atop a UNION ALL (Richard Guo)</a></h2>
<p><code>Memoize</code> plan nodes were first introduced in PostgreSQL 14. The <code>Memoize</code> plan node acts as a cache layer between a parameterized <code>Nested Loop</code> and the Nested Loop’s inner side. When the same value needs to be looked up several times, Memoize can give a nice performance boost as it can skip executing its subnode when the required rows have been queried already and are cached.</p>
<p>The PostgreSQL 16 query planner will now consider using <code>Memoize</code> when a <code>UNION ALL</code> query appears on the inner side of a parameterized <code>Nested Loop</code>.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">t1</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">t2</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">lookup</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">t1</span> <span class="k">SELECT</span> <span class="n">x</span> <span class="k">FROM</span> <span class="n">generate_Series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">10000</span><span class="p">)</span> <span class="n">x</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">t2</span> <span class="k">SELECT</span> <span class="n">x</span> <span class="k">FROM</span> <span class="n">generate_Series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">10000</span><span class="p">)</span> <span class="n">x</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">lookup</span> <span class="k">SELECT</span> <span class="n">x</span><span class="o">%</span><span class="mi">10</span><span class="o">+</span><span class="mi">1</span> <span class="k">FROM</span> <span class="n">generate_Series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span><span class="n">x</span><span class="p">;</span>
<span class="k">ANALYZE</span> <span class="n">t1</span><span class="p">,</span><span class="n">t2</span><span class="p">,</span><span class="n">lookup</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="p">(</span><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">t1</span> <span class="k">UNION</span> <span class="k">ALL</span> <span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">t2</span><span class="p">)</span> <span class="n">t</span>
<span class="k">INNER</span> <span class="k">JOIN</span> <span class="n">lookup</span> <span class="n">l</span> <span class="k">ON</span> <span class="n">l</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">t</span><span class="p">.</span><span class="n">a</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE t1 (a INT PRIMARY KEY);
CREATE TABLE t2 (a INT PRIMARY KEY);
CREATE TABLE lookup (a INT);
INSERT INTO t1 SELECT x FROM generate_Series(1,10000) x;
INSERT INTO t2 SELECT x FROM generate_Series(1,10000) x;
INSERT INTO lookup SELECT x%10+1 FROM generate_Series(1,1000000)x;
ANALYZE t1,t2,lookup;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM (SELECT * FROM t1 UNION ALL SELECT * FROM t2) t
INNER JOIN lookup l ON l.a = t.a;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-------------------------------------------------------------------------------
Nested Loop (actual rows=2000000 loops=1)
-> Seq Scan on lookup l (actual rows=1000000 loops=1)
-> Append (actual rows=2 loops=1000000)
-> Index Only Scan using t1_pkey on t1 (actual rows=1 loops=1000000)
Index Cond: (a = l.a)
Heap Fetches: 1000000
-> Index Only Scan using t2_pkey on t2 (actual rows=1 loops=1000000)
Index Cond: (a = l.a)
Heap Fetches: 1000000
Planning Time: 0.223 ms
Execution Time: 1926.151 ms
(11 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-------------------------------------------------------------------------------
Nested Loop (actual rows=2000000 loops=1)
-> Seq Scan on lookup l (actual rows=1000000 loops=1)
-> Append (actual rows=2 loops=1000000)
-> Index Only Scan using t1_pkey on t1 (actual rows=1 loops=1000000)
Index Cond: (a = l.a)
Heap Fetches: 1000000
-> Index Only Scan using t2_pkey on t2 (actual rows=1 loops=1000000)
Index Cond: (a = l.a)
Heap Fetches: 1000000
Planning Time: 0.223 ms
Execution Time: 1926.151 ms
(11 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------------------------
Nested Loop (actual rows=2000000 loops=1)
-> Seq Scan on lookup l (actual rows=1000000 loops=1)
-> Memoize (actual rows=2 loops=1000000)
Cache Key: l.a
Cache Mode: logical
Hits: 999990 Misses: 10 Evictions: 0 Overflows: 0 Memory Usage: 2kB
-> Append (actual rows=2 loops=10)
-> Index Only Scan using t1_pkey on t1 (actual rows=1 loops=10)
Index Cond: (a = l.a)
Heap Fetches: 10
-> Index Only Scan using t2_pkey on t2 (actual rows=1 loops=10)
Index Cond: (a = l.a)
Heap Fetches: 10
Planning Time: 0.229 ms
Execution Time: 282.120 ms
(15 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------------------------
Nested Loop (actual rows=2000000 loops=1)
-> Seq Scan on lookup l (actual rows=1000000 loops=1)
-> Memoize (actual rows=2 loops=1000000)
Cache Key: l.a
Cache Mode: logical
Hits: 999990 Misses: 10 Evictions: 0 Overflows: 0 Memory Usage: 2kB
-> Append (actual rows=2 loops=10)
-> Index Only Scan using t1_pkey on t1 (actual rows=1 loops=10)
Index Cond: (a = l.a)
Heap Fetches: 10
-> Index Only Scan using t2_pkey on t2 (actual rows=1 loops=10)
Index Cond: (a = l.a)
Heap Fetches: 10
Planning Time: 0.229 ms
Execution Time: 282.120 ms
(15 rows)
">Copy</button>
</div>
<p>In the PostgreSQL 16 EXPLAIN output above, you can see the <code>Memoize</code> node is put atop of the <code>Append</code> node—which caused a reduction in the number of <code>loops</code> in the <code>Append</code> from 1 million in PG15 down to 10 in PG16. Each time the <code>Memoize</code> node has a cache hit, there’s no need to execute the <code>Append</code> to fetch records. This results in the <strong>query running around 6 times faster on PostgreSQL 16</strong>.</p>
<h2 id="right-anti-join">4. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=16dc2703c5413534d4989e08253e8f4fcb0e2aab">Allow anti-joins to be performed with the non-nullable input as the inner relation (Richard Guo)</a></h2>
<p>When performing a <code>Hash Join</code> for an <code>INNER JOIN</code>, PostgreSQL prefers to build the hash table on the smaller of the two tables. Smaller hash tables are better as it’s less work to build them. Smaller tables are also better as they’re more cache-friendly for the CPU, and it’s less likely that the CPU will stall while waiting for data to arrive from main memory.</p>
<p>In PostgreSQL versions before 16, an <code>Anti Join</code>—as you might see if you use <code>NOT EXISTS</code> in your queries—would always put the table mentioned in the <code>NOT EXISTS</code> part on the inner side of the join. This meant there was no flexibility to hash the smaller of the two tables, resulting in possibly having to build a hash table on the larger table.</p>
<p>The PostgreSQL 16 query planner can now choose to hash the smaller of the two tables. This can now be done because PostgreSQL 16 supports <code>Right Anti Join</code>.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">small</span><span class="p">(</span><span class="n">a</span> <span class="nb">int</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="k">large</span><span class="p">(</span><span class="n">a</span> <span class="nb">int</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">small</span>
<span class="k">SELECT</span> <span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">100</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="k">large</span>
<span class="k">SELECT</span> <span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">small</span><span class="p">,</span><span class="k">large</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">small</span> <span class="n">s</span>
<span class="k">WHERE</span> <span class="k">NOT</span> <span class="k">EXISTS</span><span class="p">(</span><span class="k">SELECT</span> <span class="mi">1</span> <span class="k">FROM</span> <span class="k">large</span> <span class="n">l</span> <span class="k">WHERE</span> <span class="n">s</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">l</span><span class="p">.</span><span class="n">a</span><span class="p">);</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE small(a int);
CREATE TABLE large(a int);
INSERT INTO small
SELECT a FROM generate_series(1,100) a;
INSERT INTO large
SELECT a FROM generate_series(1,1000000) a;
VACUUM ANALYZE small,large;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM small s
WHERE NOT EXISTS(SELECT 1 FROM large l WHERE s.a = l.a);
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------
Hash Anti Join (actual rows=0 loops=1)
Hash Cond: (s.a = l.a)
-> Seq Scan on small s (actual rows=100 loops=1)
-> Hash (actual rows=1000000 loops=1)
Buckets: 262144 Batches: 8 Memory Usage: 6446kB
-> Seq Scan on large l (actual rows=1000000 loops=1)
Planning Time: 0.103 ms
Execution Time: 139.023 ms
(8 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------
Hash Anti Join (actual rows=0 loops=1)
Hash Cond: (s.a = l.a)
-> Seq Scan on small s (actual rows=100 loops=1)
-> Hash (actual rows=1000000 loops=1)
Buckets: 262144 Batches: 8 Memory Usage: 6446kB
-> Seq Scan on large l (actual rows=1000000 loops=1)
Planning Time: 0.103 ms
Execution Time: 139.023 ms
(8 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-----------------------------------------------------------
Hash Right Anti Join (actual rows=0 loops=1)
Hash Cond: (l.a = s.a)
-> Seq Scan on large l (actual rows=1000000 loops=1)
-> Hash (actual rows=100 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on small s (actual rows=100 loops=1)
Planning Time: 0.094 ms
Execution Time: 77.076 ms
(8 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-----------------------------------------------------------
Hash Right Anti Join (actual rows=0 loops=1)
Hash Cond: (l.a = s.a)
-> Seq Scan on large l (actual rows=1000000 loops=1)
-> Hash (actual rows=100 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 12kB
-> Seq Scan on small s (actual rows=100 loops=1)
Planning Time: 0.094 ms
Execution Time: 77.076 ms
(8 rows)
">Copy</button>
</div>
<p>You can see from the <code>EXPLAIN ANALYZE</code> output above that due to PG16’s planner opting to use a <code>Hash Right Anti Join</code>, the <code>Memory Usage</code> in PostgreSQL 16 was much less than in PostgreSQL 15 and the <code>Execution Time</code> was almost halved.</p>
<h2 id="parallel-hashfull-rightjoin">5. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=11c2d6fdf5af1aacec9ca2005543f1b0fc4cc364">Allow parallelization of FULL and internal right OUTER hash joins (Melanie Plageman, Thomas Munro)</a></h2>
<p>PostgreSQL 11 saw the introduction of <code>Parallel Hash Join</code>. This allows multiple parallel workers in a parallel query to assist in the building of a single hash table. In versions prior to 11, each worker would have built its own identical hash table, resulting in additional memory overheads.</p>
<p>In PostgreSQL 16, <code>Parallel Hash Join</code> has been improved and now supports <code>FULL</code> and <code>RIGHT</code> join types. This allows queries that have a <code>FULL OUTER JOIN</code> to be executed in parallel and also allows <code>Right Joins</code> plans to execute in parallel.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">odd</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">even</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">odd</span>
<span class="k">SELECT</span> <span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">even</span>
<span class="k">SELECT</span> <span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1000000</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">odd</span><span class="p">,</span> <span class="n">even</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="k">COUNT</span><span class="p">(</span><span class="n">o</span><span class="p">.</span><span class="n">a</span><span class="p">),</span><span class="k">COUNT</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">a</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">odd</span> <span class="n">o</span> <span class="k">FULL</span> <span class="k">JOIN</span> <span class="n">even</span> <span class="n">e</span> <span class="k">ON</span> <span class="n">o</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">e</span><span class="p">.</span><span class="n">a</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE odd (a INT);
CREATE TABLE even (a INT);
INSERT INTO odd
SELECT a FROM generate_series(1,1000000,2) a;
INSERT INTO even
SELECT a FROM generate_series(2,1000000,2) a;
VACUUM ANALYZE odd, even;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT COUNT(o.a),COUNT(e.a) FROM odd o FULL JOIN even e ON o.a = e.a;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-------------------------------------------------------------------
Aggregate (actual rows=1 loops=1)
-> Hash Full Join (actual rows=1000000 loops=1)
Hash Cond: (o.a = e.a)
-> Seq Scan on odd o (actual rows=500000 loops=1)
-> Hash (actual rows=500000 loops=1)
Buckets: 262144 Batches: 4 Memory Usage: 6439kB
-> Seq Scan on even e (actual rows=500000 loops=1)
Planning Time: 0.079 ms
Execution Time: 220.677 ms
(9 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-------------------------------------------------------------------
Aggregate (actual rows=1 loops=1)
-> Hash Full Join (actual rows=1000000 loops=1)
Hash Cond: (o.a = e.a)
-> Seq Scan on odd o (actual rows=500000 loops=1)
-> Hash (actual rows=500000 loops=1)
Buckets: 262144 Batches: 4 Memory Usage: 6439kB
-> Seq Scan on even e (actual rows=500000 loops=1)
Planning Time: 0.079 ms
Execution Time: 220.677 ms
(9 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
--------------------------------------------------------------------------------
Finalize Aggregate (actual rows=1 loops=1)
-> Gather (actual rows=2 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Partial Aggregate (actual rows=1 loops=2)
-> Parallel Hash Full Join (actual rows=500000 loops=2)
Hash Cond: (o.a = e.a)
-> Parallel Seq Scan on odd o (actual rows=250000 loops=2)
-> Parallel Hash (actual rows=250000 loops=2)
Buckets: 262144 Batches: 4 Memory Usage: 6976kB
-> Parallel Seq Scan on even e (actual rows=250000 loops=2)
Planning Time: 0.161 ms
Execution Time: 129.769 ms
(13 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
--------------------------------------------------------------------------------
Finalize Aggregate (actual rows=1 loops=1)
-> Gather (actual rows=2 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Partial Aggregate (actual rows=1 loops=2)
-> Parallel Hash Full Join (actual rows=500000 loops=2)
Hash Cond: (o.a = e.a)
-> Parallel Seq Scan on odd o (actual rows=250000 loops=2)
-> Parallel Hash (actual rows=250000 loops=2)
Buckets: 262144 Batches: 4 Memory Usage: 6976kB
-> Parallel Seq Scan on even e (actual rows=250000 loops=2)
Planning Time: 0.161 ms
Execution Time: 129.769 ms
(13 rows)
">Copy</button>
</div>
<p>The <code>EXPLAIN</code> output shows that PostgreSQL 16 was able to perform the join in parallel and this resulted in a significant reduction in the query’s <code>Execution Time</code>.</p>
<h2 id="frame-clauses">6. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=ed1a88ddaccfe883e4cf74d30319accfeae6cfe5">Allow window functions to use faster ROWS mode when RANGE mode active but unnecessary (David Rowley)</a></h2>
<p>When a query contains a window function such as <code>row_number()</code>, <code>rank()</code>, <code>dense_rank()</code>, <code>percent_rank()</code>, <code>cume_dist()</code> and <code>ntile()</code>, if the <a href="https://www.postgresql.org/docs/16/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS">window clause</a> did not specify the <code>ROWS</code>option, then PostgreSQL would always use the default <code>RANGE</code> option. The <code>RANGE</code> option causes the executor to look ahead until it finds the first “non-peer” row. A peer row is a row in the window frame which compares equally according to the window clause’s <code>ORDER BY</code> clause. When there is no <code>ORDER BY</code> clause, all rows within the window frame are peers. When processing records which have many rows which sort equally according to the window clause’s <code>ORDER BY</code> clause, the additional processing to identify these peer rows could be costly.</p>
<p>The window functions mentioned above don’t behave any differently whether <code>ROWS</code> or <code>RANGE</code> is specified in the query’s window clause. However, the executor in PostgreSQL versions prior to 16 didn’t know that, and because <strong>some</strong> window functions do care about the <code>ROWS</code>/<code>RANGE</code> option, the executor had to perform checks for peer rows in all cases.</p>
<p>The PostgreSQL 16 query planner knows which window functions care about the <code>ROWS</code>/<code>RANGE</code> option and it passes this information along to the executor so that it can skip the needless additional processing.</p>
<p>This optimization works particularly well when <code>row_number()</code> is being used to limit the number of results in the query as shown in the example below.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">scores</span> <span class="p">(</span><span class="n">id</span> <span class="nb">INT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">score</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">scores</span> <span class="k">SELECT</span> <span class="n">s</span><span class="p">,</span><span class="n">random</span><span class="p">()</span><span class="o">*</span><span class="mi">10</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span><span class="n">s</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="k">ON</span> <span class="n">scores</span><span class="p">(</span><span class="n">score</span><span class="p">);</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">scores</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span><span class="n">ROW_NUMBER</span><span class="p">()</span> <span class="n">OVER</span> <span class="p">(</span><span class="k">ORDER</span> <span class="k">BY</span> <span class="n">score</span><span class="p">)</span> <span class="n">rn</span><span class="p">,</span><span class="n">score</span>
<span class="k">FROM</span> <span class="n">scores</span>
<span class="p">)</span> <span class="n">m</span> <span class="k">WHERE</span> <span class="n">rn</span> <span class="o"><=</span> <span class="mi">10</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE scores (id INT PRIMARY KEY, score INT);
INSERT INTO scores SELECT s,random()*10 FROM generate_series(1,1000000)s;
CREATE INDEX ON scores(score);
VACUUM ANALYZE scores;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM (
SELECT id,ROW_NUMBER() OVER (ORDER BY score) rn,score
FROM scores
) m WHERE rn <= 10;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-------------------------------------------------------------------------------
WindowAgg (actual rows=10 loops=1)
Run Condition: (row_number() OVER (?) <= 10)
-> Index Scan using scores_score_idx on scores (actual rows=50410 loops=1)
Planning Time: 0.096 ms
Execution Time: 29.775 ms
(5 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-------------------------------------------------------------------------------
WindowAgg (actual rows=10 loops=1)
Run Condition: (row_number() OVER (?) <= 10)
-> Index Scan using scores_score_idx on scores (actual rows=50410 loops=1)
Planning Time: 0.096 ms
Execution Time: 29.775 ms
(5 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
----------------------------------------------------------------------------
WindowAgg (actual rows=10 loops=1)
Run Condition: (row_number() OVER (?) <= 10)
-> Index Scan using scores_score_idx on scores (actual rows=11 loops=1)
Planning Time: 0.191 ms
Execution Time: 0.058 ms
(5 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
----------------------------------------------------------------------------
WindowAgg (actual rows=10 loops=1)
Run Condition: (row_number() OVER (?) <= 10)
-> Index Scan using scores_score_idx on scores (actual rows=11 loops=1)
Planning Time: 0.191 ms
Execution Time: 0.058 ms
(5 rows)
">Copy</button>
</div>
<p>The <code>Index Scan</code> node in the <code>EXPLAIN</code> output for PG15 above shows that 50410 rows had to be read from the <code>scores_score_idx</code> index before execution stopped. While in PostgreSQL 16, only 11 rows were read as the executor realized that once the row_number got to 11, there’d be no more rows matching the <code><= 10</code> condition. Both this and the executor using the <code>ROWS</code> window clause option led to this query running over 500 times faster on PostgreSQL 16.</p>
<h2 id="window-functions">7. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=456fa635a909ee36f73ca84d340521bd730f265f">Optimize always-increasing window functions ntile(), cume_dist() and percent_rank() (David Rowley)</a></h2>
<p>This change expands on work done in PostgreSQL 15. In PG15 the query planner was modified to allow the executor to stop processing <code>WindowAgg</code> executor nodes early. This can be done when an item in the <code>WHERE</code> clause filters a window function in a way that once the condition becomes false, it will never be true again.</p>
<p><code>row_number()</code> is an example of a function which can offer such guarantees as it’s a monotonically increasing function, i.e. subsequent rows in the same partition will never have a row_number lower than the previous row.</p>
<p>The PostgreSQL 16 query planner expands the coverage of this optimization to also cover <code>ntile()</code>, <code>cume_dist()</code> and <code>percent_rank()</code>. In PostgreSQL 15, this only worked for <code>row_number()</code>, <code>rank()</code>, <code>dense_rank()</code>, <code>count()</code> and <code>count(*)</code>.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">marathon</span> <span class="p">(</span><span class="n">id</span> <span class="nb">INT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="nb">time</span> <span class="n">INTERVAL</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">marathon</span>
<span class="k">SELECT</span> <span class="n">id</span><span class="p">,</span><span class="s1">'03:00:00'</span><span class="p">::</span><span class="n">interval</span> <span class="o">+</span> <span class="p">(</span><span class="k">CAST</span><span class="p">(</span><span class="n">RANDOM</span><span class="p">()</span> <span class="o">*</span> <span class="mi">3600</span> <span class="k">AS</span> <span class="nb">INT</span><span class="p">)</span> <span class="o">||</span> <span class="s1">'secs'</span><span class="p">)::</span><span class="n">INTERVAL</span> <span class="o">-</span> <span class="p">(</span><span class="k">CAST</span><span class="p">(</span><span class="n">RANDOM</span><span class="p">()</span> <span class="o">*</span> <span class="mi">3600</span> <span class="k">AS</span> <span class="nb">INT</span><span class="p">)</span> <span class="o">||</span> <span class="s1">' secs'</span><span class="p">)::</span><span class="n">INTERVAL</span>
<span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">50000</span><span class="p">)</span> <span class="n">id</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="k">ON</span> <span class="n">marathon</span> <span class="p">(</span><span class="nb">time</span><span class="p">);</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">marathon</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="p">(</span><span class="k">SELECT</span> <span class="o">*</span><span class="p">,</span><span class="n">percent_rank</span><span class="p">()</span> <span class="n">OVER</span> <span class="p">(</span><span class="k">ORDER</span> <span class="k">BY</span> <span class="nb">time</span><span class="p">)</span> <span class="n">pr</span>
<span class="k">FROM</span> <span class="n">marathon</span><span class="p">)</span> <span class="n">m</span> <span class="k">WHERE</span> <span class="n">pr</span> <span class="o"><=</span> <span class="mi">0</span><span class="p">.</span><span class="mi">01</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE marathon (id INT PRIMARY KEY, time INTERVAL NOT NULL);
INSERT INTO marathon
SELECT id,'03:00:00'::interval + (CAST(RANDOM() * 3600 AS INT) || 'secs')::INTERVAL - (CAST(RANDOM() * 3600 AS INT) || ' secs')::INTERVAL
FROM generate_series(1,50000) id;
CREATE INDEX ON marathon (time);
VACUUM ANALYZE marathon;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT * FROM (SELECT *,percent_rank() OVER (ORDER BY time) pr
FROM marathon) m WHERE pr <= 0.01;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-----------------------------------------------------------------------
Subquery Scan on m (actual rows=500 loops=1)
Filter: (m.pr <= '0.01'::double precision)
Rows Removed by Filter: 49500
-> WindowAgg (actual rows=50000 loops=1)
-> Index Scan using marathon_time_idx on marathon (actual rows=50000 loops=1)
Planning Time: 0.108 ms
Execution Time: 84.358 ms
(7 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-----------------------------------------------------------------------
Subquery Scan on m (actual rows=500 loops=1)
Filter: (m.pr <= '0.01'::double precision)
Rows Removed by Filter: 49500
-> WindowAgg (actual rows=50000 loops=1)
-> Index Scan using marathon_time_idx on marathon (actual rows=50000 loops=1)
Planning Time: 0.108 ms
Execution Time: 84.358 ms
(7 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-----------------------------------------------------------------------
WindowAgg (actual rows=500 loops=1)
Run Condition: (percent_rank() OVER (?) <= '0.01'::double precision)
-> Index Scan using marathon_time_idx on marathon (actual rows=50000 loops=1)
Planning Time: 0.180 ms
Execution Time: 19.454 ms
(5 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-----------------------------------------------------------------------
WindowAgg (actual rows=500 loops=1)
Run Condition: (percent_rank() OVER (?) <= '0.01'::double precision)
-> Index Scan using marathon_time_idx on marathon (actual rows=50000 loops=1)
Planning Time: 0.180 ms
Execution Time: 19.454 ms
(5 rows)
">Copy</button>
</div>
<p>From the PostgreSQL 16 <code>EXPLAIN</code> output above, you can see that the planner was able to use the <code>pr <= 0.01</code> condition as a <code>Run Condition</code>, whereas, with PostgreSQL 15 this clause appeared as a <code>Filter</code> on the subquery. In PG16, the run condition was used to abort the execution of the <code>WindowAgg</code> node early. This resulted in the <code>Execution Time</code> in PG16 being more than 4 times faster than in PG15.</p>
<h2 id="join-removals">8. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=3c569049b7b502bb4952483d19ce622ff0af5fd6">Allow left join removals and unique joins on partitioned tables (Arne Roland)</a></h2>
<p>For a long time now, PostgreSQL has been able to remove a <code>LEFT JOIN</code> where no column from the left joined table was required in the query and the join could not possibly duplicate any rows.</p>
<p>However, in versions prior to PostgreSQL 16, there was no support for left join removals on partitioned tables. Why? Because the proofs that the planner uses to determine if there’s any possibility any inner-side row could duplicate any outer-side row were not present for partitioned tables.</p>
<p>The PostgreSQL 16 query planner now allows the <code>LEFT JOIN</code> removal optimization with partitioned tables.</p>
<p>This join elimination optimization is more likely to help with views, as it’s common that not all columns that exist in a view will always be queried.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">part_tab</span> <span class="p">(</span><span class="n">id</span> <span class="nb">BIGINT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span> <span class="n">payload</span> <span class="nb">TEXT</span><span class="p">)</span> <span class="n">PARTITION</span> <span class="k">BY</span> <span class="n">HASH</span><span class="p">(</span><span class="n">id</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">part_tab_p0</span> <span class="n">PARTITION</span> <span class="k">OF</span> <span class="n">part_tab</span> <span class="k">FOR</span> <span class="k">VALUES</span> <span class="k">WITH</span> <span class="p">(</span><span class="n">modulus</span> <span class="mi">2</span><span class="p">,</span> <span class="n">remainder</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">part_tab_p1</span> <span class="n">PARTITION</span> <span class="k">OF</span> <span class="n">part_tab</span> <span class="k">FOR</span> <span class="k">VALUES</span> <span class="k">WITH</span> <span class="p">(</span><span class="n">modulus</span> <span class="mi">2</span><span class="p">,</span> <span class="n">remainder</span> <span class="mi">1</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">normal_table</span> <span class="p">(</span><span class="n">id</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">part_tab_id</span> <span class="nb">BIGINT</span><span class="p">);</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">nt</span><span class="p">.</span><span class="o">*</span> <span class="k">FROM</span> <span class="n">normal_table</span> <span class="n">nt</span> <span class="k">LEFT</span> <span class="k">JOIN</span> <span class="n">part_tab</span> <span class="n">pt</span> <span class="k">ON</span> <span class="n">nt</span><span class="p">.</span><span class="n">part_tab_id</span> <span class="o">=</span> <span class="n">pt</span><span class="p">.</span><span class="n">id</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE part_tab (id BIGINT PRIMARY KEY, payload TEXT) PARTITION BY HASH(id);
CREATE TABLE part_tab_p0 PARTITION OF part_tab FOR VALUES WITH (modulus 2, remainder 0);
CREATE TABLE part_tab_p1 PARTITION OF part_tab FOR VALUES WITH (modulus 2, remainder 1);
CREATE TABLE normal_table (id INT, part_tab_id BIGINT);
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT nt.* FROM normal_table nt LEFT JOIN part_tab pt ON nt.part_tab_id = pt.id;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-------------------------------------------------------------------
Merge Right Join (actual rows=0 loops=1)
Merge Cond: (pt.id = nt.part_tab_id)
-> Merge Append (actual rows=0 loops=1)
Sort Key: pt.id
-> Index Only Scan using part_tab_p0_pkey on part_tab_p0 pt_1 (actual rows=0 loops=1)
Heap Fetches: 0
-> Index Only Scan using part_tab_p1_pkey on part_tab_p1 pt_2 (actual rows=0 loops=1)
Heap Fetches: 0
-> Sort (actual rows=0 loops=1)
Sort Key: nt.part_tab_id
Sort Method: quicksort Memory: 25kB
-> Seq Scan on normal_table nt (actual rows=0 loops=1)
Planning Time: 0.325 ms
Execution Time: 0.037 ms
(14 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-------------------------------------------------------------------
Merge Right Join (actual rows=0 loops=1)
Merge Cond: (pt.id = nt.part_tab_id)
-> Merge Append (actual rows=0 loops=1)
Sort Key: pt.id
-> Index Only Scan using part_tab_p0_pkey on part_tab_p0 pt_1 (actual rows=0 loops=1)
Heap Fetches: 0
-> Index Only Scan using part_tab_p1_pkey on part_tab_p1 pt_2 (actual rows=0 loops=1)
Heap Fetches: 0
-> Sort (actual rows=0 loops=1)
Sort Key: nt.part_tab_id
Sort Method: quicksort Memory: 25kB
-> Seq Scan on normal_table nt (actual rows=0 loops=1)
Planning Time: 0.325 ms
Execution Time: 0.037 ms
(14 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
-----------------------------------------------------
Seq Scan on normal_table nt (actual rows=0 loops=1)
Planning Time: 0.244 ms
Execution Time: 0.015 ms
(3 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
-----------------------------------------------------
Seq Scan on normal_table nt (actual rows=0 loops=1)
Planning Time: 0.244 ms
Execution Time: 0.015 ms
(3 rows)
">Copy</button>
</div>
<p>The important thing to notice here is that the PostgreSQL 16 plan does not include a join to <code>part_tab</code> meaning all there is to do is scan <code>normal_table</code>.</p>
<h2 id="short-circuit">9. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5543677ec90a15c73dab5ed4f0902b3b920f0b87">Use Limit instead of Unique to implement DISTINCT, when possible (David Rowley)</a></h2>
<p>The PostgreSQL query planner is able to avoid including plan nodes to de-duplicate the results when it can detect that all rows contain the same value. Detecting this is trivial and when the optimization can be applied it can result in huge performance gains.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">abc</span> <span class="p">(</span><span class="n">a</span> <span class="nb">int</span><span class="p">,</span> <span class="n">b</span> <span class="nb">int</span><span class="p">,</span> <span class="k">c</span> <span class="nb">int</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">abc</span> <span class="k">SELECT</span> <span class="n">a</span><span class="o">%</span><span class="mi">10</span><span class="p">,</span><span class="n">a</span><span class="o">%</span><span class="mi">10</span><span class="p">,</span><span class="n">a</span><span class="o">%</span><span class="mi">10</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span><span class="n">a</span><span class="p">;</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">abc</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="k">DISTINCT</span> <span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="k">c</span> <span class="k">FROM</span> <span class="n">abc</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="k">AND</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">5</span> <span class="k">AND</span> <span class="k">c</span> <span class="o">=</span> <span class="mi">5</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE abc (a int, b int, c int);
INSERT INTO abc SELECT a%10,a%10,a%10 FROM generate_series(1,1000000)a;
VACUUM ANALYZE abc;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT DISTINCT a,b,c FROM abc WHERE a = 5 AND b = 5 AND c = 5;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
------------------------------------------------------------------------
Unique (actual rows=1 loops=1)
-> Gather (actual rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Unique (actual rows=1 loops=3)
-> Parallel Seq Scan on abc (actual rows=33333 loops=3)
Filter: ((a = 5) AND (b = 5) AND (c = 5))
Rows Removed by Filter: 300000
Planning Time: 0.114 ms
Execution Time: 30.381 ms
(10 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
------------------------------------------------------------------------
Unique (actual rows=1 loops=1)
-> Gather (actual rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Unique (actual rows=1 loops=3)
-> Parallel Seq Scan on abc (actual rows=33333 loops=3)
Filter: ((a = 5) AND (b = 5) AND (c = 5))
Rows Removed by Filter: 300000
Planning Time: 0.114 ms
Execution Time: 30.381 ms
(10 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------
Limit (actual rows=1 loops=1)
-> Seq Scan on abc (actual rows=1 loops=1)
Filter: ((a = 5) AND (b = 5) AND (c = 5))
Rows Removed by Filter: 4
Planning Time: 0.109 ms
Execution Time: 0.025 ms
(6 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------
Limit (actual rows=1 loops=1)
-> Seq Scan on abc (actual rows=1 loops=1)
Filter: ((a = 5) AND (b = 5) AND (c = 5))
Rows Removed by Filter: 4
Planning Time: 0.109 ms
Execution Time: 0.025 ms
(6 rows)
">Copy</button>
</div>
<p>If you look carefully at the SQL query, you’ll notice that each column in the <code>DISTINCT</code> clause also has an equality condition in the <code>WHERE</code> clause. This means that all the output rows in the query will have the same values in each column. The PostgreSQL 16 query planner is able to take advantage of this knowledge and simply <code>LIMIT</code> the query results to 1 row. PostgreSQL 15 produced the same query result by reading the entire results and using the <code>Unique</code> operator to reduce all the rows down to a single row. The <code>Execution Time</code> for PostgreSQL 16 was more than 1200 times faster than PostgreSQL 15.</p>
<h2 id="merge-joins">10. <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=b592422095655a64d638f541df784b19b8ecf8ad">Relax overly strict rules in select_<wbr>outer_<wbr>pathkeys_<wbr>for_<wbr>merge() (David Rowley)</a></h2>
<p>Before PostgreSQL 16, when the query planner considered performing a <code>Merge Join</code>, it would check if the sort order of the merge would suit any upper-level plan operation (such as <code>DISTINCT</code>, <code>GROUP BY</code> or <code>ORDER BY</code>) and only use that order if it matched the requirements for the upper-level exactly. This choice was a little outdated as <code>Incremental Sorts</code> can be used for these upper-level operations and incremental sorts can take advantage of results that are presorted by only some of the leading columns that the results need to be sorted by.</p>
<p>The PostgreSQL 16 query planner adjusted the rule used when considering <code>Merge Join</code> orders from “the order of the rows must match exactly” to “there must be at least 1 leading column correctly ordered”. This allows the planner to use <code>Incremental Sorts</code> to get the rows into the correct order for the upper-level operation. We know from earlier in this blog that incremental sorts, when they’re possible, require less work than a complete sort as incremental sorts are able to exploit partially sorted inputs and perform sorts in smaller batches, resulting in less memory consumption and fewer sort comparisons overall.</p>
<div class="highlight">
<pre class="highlight sql"><code><span class="c1">-- Setup</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">a</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">b</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">b</span> <span class="p">(</span><span class="n">x</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">y</span> <span class="nb">INT</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">a</span> <span class="k">SELECT</span> <span class="n">a</span><span class="p">,</span><span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">b</span> <span class="k">SELECT</span> <span class="n">a</span><span class="p">,</span><span class="n">a</span> <span class="k">FROM</span> <span class="n">generate_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1000000</span><span class="p">)</span> <span class="n">a</span><span class="p">;</span>
<span class="k">VACUUM</span> <span class="k">ANALYZE</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">;</span>
<span class="k">SET</span> <span class="n">enable_hashjoin</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span>
<span class="k">SET</span> <span class="n">max_parallel_workers_per_gather</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span>
<span class="k">EXPLAIN</span> <span class="p">(</span><span class="k">ANALYZE</span><span class="p">,</span> <span class="n">COSTS</span> <span class="k">OFF</span><span class="p">,</span> <span class="n">TIMING</span> <span class="k">OFF</span><span class="p">)</span>
<span class="k">SELECT</span> <span class="n">a</span><span class="p">,</span><span class="n">b</span><span class="p">,</span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">a</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="n">b</span> <span class="k">ON</span> <span class="n">a</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">b</span><span class="p">.</span><span class="n">x</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="n">a</span><span class="p">,</span><span class="n">b</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="n">a</span> <span class="k">DESC</span><span class="p">,</span> <span class="n">b</span><span class="p">;</span>
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text="-- Setup
CREATE TABLE a (a INT, b INT);
CREATE TABLE b (x INT, y INT);
INSERT INTO a SELECT a,a FROM generate_series(1,1000000) a;
INSERT INTO b SELECT a,a FROM generate_series(1,1000000) a;
VACUUM ANALYZE a, b;
SET enable_hashjoin=0;
SET max_parallel_workers_per_gather=0;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF)
SELECT a,b,count(*) FROM a INNER JOIN b ON a.a = b.x GROUP BY a,b ORDER BY a DESC, b;
">Copy</button>
</div>
<h3>PG15 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------------------
GroupAggregate (actual rows=1000000 loops=1)
Group Key: a.a, a.b
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC, a.b
Sort Method: external merge Disk: 17664kB
-> Merge Join (actual rows=1000000 loops=1)
Merge Cond: (a.a = b.x)
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a
Sort Method: external merge Disk: 17664kB
-> Seq Scan on a (actual rows=1000000 loops=1)
-> Materialize (actual rows=1000000 loops=1)
-> Sort (actual rows=1000000 loops=1)
Sort Key: b.x
Sort Method: external merge Disk: 11768kB
-> Seq Scan on b (actual rows=1000000 loops=1)
Planning Time: 0.175 ms
Execution Time: 1010.738 ms
(18 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------------------
GroupAggregate (actual rows=1000000 loops=1)
Group Key: a.a, a.b
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC, a.b
Sort Method: external merge Disk: 17664kB
-> Merge Join (actual rows=1000000 loops=1)
Merge Cond: (a.a = b.x)
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a
Sort Method: external merge Disk: 17664kB
-> Seq Scan on a (actual rows=1000000 loops=1)
-> Materialize (actual rows=1000000 loops=1)
-> Sort (actual rows=1000000 loops=1)
Sort Key: b.x
Sort Method: external merge Disk: 11768kB
-> Seq Scan on b (actual rows=1000000 loops=1)
Planning Time: 0.175 ms
Execution Time: 1010.738 ms
(18 rows)
">Copy</button>
</div>
<h3>PG16 EXPLAIN output</h3>
<div class="highlight">
<pre class="highlight "><code> QUERY PLAN
---------------------------------------------------------------------------
GroupAggregate (actual rows=1000000 loops=1)
Group Key: a.a, a.b
-> Incremental Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC, a.b
Presorted Key: a.a
Full-sort Groups: 31250 Sort Method: quicksort Average Memory: 26kB Peak Memory: 26kB
-> Merge Join (actual rows=1000000 loops=1)
Merge Cond: (a.a = b.x)
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC
Sort Method: external merge Disk: 17672kB
-> Seq Scan on a (actual rows=1000000 loops=1)
-> Materialize (actual rows=1000000 loops=1)
-> Sort (actual rows=1000000 loops=1)
Sort Key: b.x DESC
Sort Method: external merge Disk: 11768kB
-> Seq Scan on b (actual rows=1000000 loops=1)
Planning Time: 0.140 ms
Execution Time: 915.589 ms
(19 rows)
</code></pre>
<button class="copy-button" data-clipboard-action="copy" data-clipboard-text=" QUERY PLAN
---------------------------------------------------------------------------
GroupAggregate (actual rows=1000000 loops=1)
Group Key: a.a, a.b
-> Incremental Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC, a.b
Presorted Key: a.a
Full-sort Groups: 31250 Sort Method: quicksort Average Memory: 26kB Peak Memory: 26kB
-> Merge Join (actual rows=1000000 loops=1)
Merge Cond: (a.a = b.x)
-> Sort (actual rows=1000000 loops=1)
Sort Key: a.a DESC
Sort Method: external merge Disk: 17672kB
-> Seq Scan on a (actual rows=1000000 loops=1)
-> Materialize (actual rows=1000000 loops=1)
-> Sort (actual rows=1000000 loops=1)
Sort Key: b.x DESC
Sort Method: external merge Disk: 11768kB
-> Seq Scan on b (actual rows=1000000 loops=1)
Planning Time: 0.140 ms
Execution Time: 915.589 ms
(19 rows)
">Copy</button>
</div>
<p>In the PG16 EXPLAIN output above, you can see that an <code>Incremental Sort</code> was used (compared to PG15 which instead used a <code>Sort</code>) and that resulted in a small reduction of the query’s <code>Execution Time</code> in PG16 and a large reduction in the memory used to perform the sort.</p>
<h2>Conclusion</h2>
<p>A lot of engineering work was done in PostgreSQL 16 to improve the query planner by many engineers from all around the world. I’d like to thank all the people who helped by reviewing the pieces that I worked on and all the people who gave feedback on the changes.</p>
<p>Each of the 10 improvements to the PostgreSQL 16 planner above are enabled by default—and are either applied in all cases where the optimization is possible, or are selectively applied by the query planner when it thinks the optimization will help.</p>
<p>If you’re running an older version of PostgreSQL, I encourage you to try your workload on PostgreSQL 16 to see which of your queries are faster. And as always, feedback about real-world usage of PostgreSQL in the wild is welcome on the <a href="mailto:pgsql-general@postgresql.org?subject=PG16%20query%20planner%20feedback">pgsql-general@postgresql.org</a> mailing list—you don’t have to just file issues, you can always share the positive experiences too. So please drop us a line about your experience with the PostgreSQL 16 planner.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2024/02/08/whats-new-in-postgres-16-query-planner-optimizer/'>citusdata.com</a>.</em></p>Podcast highlights on benchmarking Postgres performance, from Path To Citus Con Episode 11https://www.citusdata.com/blog/2024/02/02/podcast-highlights-on-benchmarking-postgres-performance/2024-02-02T17:41:00+00:002024-02-02T17:41:00+00:00Aaron Wislang<p>Episode 11 of <a href="/podcast/path-to-citus-con/">Path To Citus Con</a>—the monthly podcast for developers who love Postgres—is now out. This episode featured guests <a href="https://twitter.com/JelteF">Jelte Fennema-Nio</a> and <a href="https://twitter.com/marcoslot">Marco Slot</a> who joined us (along with co-hosts <a href="https://hachyderm.io/@clairegiordano">Claire Giordano</a> and <a href="https://www.linkedin.com/in/pinodecandia">Pino de Candia</a>) to talk about performance benchmarking, specifically benchmarking databases and Postgres.</p>
<p>The official title of Episode 11 with Jelte and Marco is <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-performance-benchmarking-with-jelte-fennema-nio-marco-slot">“My Journey into Performance Benchmarking”</a>.</p>
<p>I love hearing how people find themselves doing what they do, so it was fascinating to hear how they got started. One of the most interesting insights was the impact something like benchmarking can have on your career—even if you don’t particularly enjoy it! </p>
<p>This episode was also packed with benchmarking tools and techniques you can pick up and use today—take a peek at some of the resources in the <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-performance-benchmarking-with-jelte-fennema-nio-marco-slot">show notes for this episode</a>—along with helpful insights resulting from Jelte’s and Marco’s years of combined experience in the space.</p>
<p>And once you have the data, how do you answer the right questions, or tell the story effectively? Perhaps, like Marco, you’re also a fan of <a href="https://www.brendangregg.com/flamegraphs.html">flame graphs</a>. Or you haven’t used flame graphs yet but are about to discover a new favorite thing. Measuring things can be fun! The tooling continues to span tried-and-true Linux tools to fast-moving technology such as eBPF.</p>
<p>Not a database benchmarking fan, yet? I’m willing to bet you’ll come away with a fresh perspective on how impactful it can be, as well as some ideas on how you can apply benchmarking to your own workflows at the right moment.</p>
<p>These insights are broadly applicable and extend beyond Postgres, too. I found myself reflecting on the time I’ve spent hacking on similar efforts in the Linux & Cloud Native world, and how I can import some of these ideas into application workloads.</p>
<p>This was Marco’s second time on the podcast, after kicking off our very first episode alongside Simon Willison with <a href="https://pathtocituscon.transistor.fm/episodes/working-in-public-on-open-source">Working in public on open source</a>. And Jelte was a speaker at Citus Con: An Event for Postgres last year discussing <a href="https://www.youtube.com/watch?v=g8lzx0BABf0">Postgres without SQL: Natural language queries using GPT-3 & Rust</a>. After this excellent deep-dive into benchmarking Postgres, I’m hoping Jelte and Marco will join us again on the podcast in the future!</p>
<h2>Where to find Path To Citus Con podcast episodes (and transcripts)</h2>
<p>You can listen to this podcast episode on almost every podcast platform! You can also find the show notes online for <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-performance-benchmarking-with-jelte-fennema-nio-marco-slot">My Journey into Postgres Benchmarking</a> with Jelte and Marco—plus also the <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-performance-benchmarking-with-jelte-fennema-nio-marco-slot/transcript">written transcript</a>.</p>
<p>Our next podcast episode (Ep12!) will be recorded live:</p>
<ul>
<li>Where: Microsoft Open Source Discord </li>
<li>When: <strong>Wed Feb 07 @ 10am PST | 1pm EST | 6pm UTC</strong> </li>
<li>Guest: with the amazing Derk van Veen, to talk about </li>
<li>Topic: <strong>from developer to PostgreSQL specialist</strong></li>
<li><strong>Calendar invite</strong>: <a href="https://aka.ms/PathToCitusCon-Ep12-cal">Mark your calendar</a> to participate in the live text chat that happens in parallel to the live recording (we’re huge fans of social audio on Discord). All the instructions for joining the live show are in the cal invite.</li>
</ul>
<p>You may have seen Derk talk about his Postgres work at Adyen on the PG conference circuit—he’s a brilliant speaker, and this promises to be a challenging conversation!</p>
<p>You can find all the past episodes for the Path To Citus Con podcast on:</p>
<ul>
<li><a href="https://podcasts.apple.com/us/podcast/path-to-citus-con/id1695014346">Apple Podcasts</a></li>
<li><a href="https://open.spotify.com/show/6XtjTEc4KGMK9fIGvmPLn7">Spotify</a></li>
<li><a href="https://aka.ms/PathToCitusCon-playlist">YouTube</a></li>
<li><a href="https://pathtocituscon.transistor.fm/subscribe">And many more podcast platforms</a></li>
</ul>
<p>More useful links:</p>
<ul>
<li><a href="/podcast/path-to-citus-con/">Index of all past Path To Citus Con episodes</a></li>
<li><a href="https://aka.ms/PathToCitusCon-cal">Subscribe to the calendar</a> to know when all the future shows are coming up here.</li>
</ul>
<p>We hope you enjoy the episodes of the Path To Citus Con podcast as much as we enjoy creating them. If you do, please share with friends and the database community, and perhaps join us live for our next episode. You might even be our next guest! Rating and reviewing us on your favorite podcasting platform will also help others discover it.</p>
<figure>
<picture>
<source srcset="https://cituscdn.azureedge.net/images/blog/PathToCitusCon-Ep11-My-Journey-into-Performance-Benchmarking-youtube-1000x563.webp" type="image/webp">
<img src="https://cituscdn.azureedge.net/images/blog/PathToCitusCon-Ep11-My-Journey-into-Performance-Benchmarking-youtube-1000x563.jpg" width="850" height="479" alt="YouTube thumbnail of Path to Citus Con episode 11" loading="lazy" />
</picture>
<figcaption><strong>Figure 1:</strong> YouTube thumbnail for Path to Citus Con episode 11, My Journey Into Performance Benchmarking, with profile photos (clockwise from top left) of Jelte Fennema-Nio, Claire Giordano, Marco Slot, and Pino de Candia.</figcaption>
</figure>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2024/02/02/podcast-highlights-on-benchmarking-postgres-performance/'>citusdata.com</a>.</em></p>You should submit a Postgres talk to the CFP for PGConf.devhttps://www.citusdata.com/blog/2024/01/11/you-should-submit-a-postgres-talk-to-cfp-for-pgconf-dev/2024-01-11T17:15:00+00:002024-01-11T17:15:00+00:00Melanie Plageman<p><strong>The PGConf.dev CFP closes on Monday, January 15 at 11:59pm PST</strong>, so if you want to speak at the inaugural PGConf.dev, submit a proposal!</p>
<p><a href="https://2024.pgconf.dev/">PGConf.dev</a> is the new PostgreSQL Development Conference, the successor to PGCon, a Postgres contribution-focused conference that took place every year in Ottawa. Pronounced "Pee-gee-conf-dot-dev", the inaugural year of PGConf.dev will take place in beautiful Vancouver, Canada, on May 28-31, 2024—with many of the same conference features that made PGCon so great:</p>
<ul>
<li>sessions covering Postgres hacking and contribution</li>
<li>full-day Unconference geared toward collaboration and impromptu creativity</li>
<li>opportunities to brainstorm with others interested in Postgres development in the “hallway track”</li>
</ul>
<p><strong>What type of talks is PGConf.dev looking for?</strong> The <a href="https://2024.pgconf.dev/cfp/">CFP page for PGConf.dev</a> has more details, but, in short: session proposals on all topics related to contributing to Postgres and how to ensure Postgres continues to be the best open source relational database on the planet. </p>
<h2>What’s new and different at PGConf.dev?</h2>
<p>Part of our vision for PGConf.dev is that it becomes an explicit destination for learning how to become a Postgres contributor.</p>
<p>In addition to the advanced PostgreSQL developer topics you might expect—the hope is for PGConf.dev to include multiple introductory sessions, beginner-friendly hallway track conversations, and hands-on workshops that will help you begin (or continue) your Postgres contributor journey. </p>
<p>If this sounds like something you can help with, well, then <a href="https://2024.pgconf.dev/cfp/">you should submit a talk</a>.</p>
<p>And there will be "workshops" as well as regular-length talks. The CFP solicits both:</p>
<ul>
<li>40-50 minute regular session proposals, and</li>
<li>workshops which are twice the length of a regular session</li>
</ul>
<p>From the <a href="https://2024.pgconf.dev/cfp/">PGConf.dev CFP page</a>:</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p>Workshop sessions will run for 110 minutes, which is double the time of a regular session. Workshops are meant to facilitate educational content that is more in-depth or interactive than the 50 minute format allows.</p>
</blockquote>
<h2>My first PGCon conference talk submission</h2>
<p>In 2018, I attended my very first PGCon. I had limited experience with Postgres development and had only been a professional software engineer for a year. I was interested in any opportunity to learn how to become a Postgres hacker, and I heard that PGCon was the best place for this.</p>
<p>Most of what I remember about PGCon 2018 was struggling to understand the sessions I attended and the hallway track conversations I lurked on. Everything was fascinating… but also far beyond my comprehension at that point.</p>
<p>Eight months later, the CFP opened up for PGCon 2019. I had learned so much in the intervening time, I decided to submit my first conference talk. I assumed it wouldn't be accepted. After all, I was hardly an expert. I submitted a talk titled <a href="https://github.com/melanieplageman/debugging_planner">An Intro to Hacking on the Postgres Planner</a>. I knew there were other people who could speak with more authority on hacking on the Postgres planner, but I felt well-positioned to speak on <em>learning</em> to hack on the Postgres planner. To my surprise, my talk was accepted.</p>
<p>I spent months developing the content for my presentation. Developing this talk was the first time I had really tried to explain query optimization or the Postgres planner to anyone. And as you might imagine, the process was a fantastic learning experience.</p>
<p>That year, my PGCon experience was different. I understood more of what people were talking about. I specifically sought out sessions with an introductory focus and understood the content. Being a speaker, people approached me to ask questions about my talk and get my opinion on their own Postgres planner ideas. I realized you don’t have to be an expert to contribute to the Postgres community.</p>
<h2>Who is PGConf.dev for? Who is the audience?</h2>
<ul>
<li><p><strong><a href="https://2024.pgconf.dev/">PGConf.dev</a> is not just for Postgres hackers.</strong> If you are a conference volunteer, a PUG meetup organizer, a podcast host, a blog author, or a Postgres community volunteer—please, <a href="https://2024.pgconf.dev/cfp/">you should submit a talk</a>.</p>
<p>You can think of PGConf.dev as a conference for learning about and improving Postgres contributions of all kinds (not just code!)</p></li>
<li><p><strong>PGConf.dev's development track is not just about Postgres core code development.</strong> There are so many projects and products which make Postgres work for thousands of unique use cases. If you maintain an extension, a client, a driver, library, or other Postgres ecosystem project, we need your perspective. <a href="https://2024.pgconf.dev/cfp/">You should submit a talk</a>. :)</p></li>
<li><p><strong>PGConf.dev is also a place for new ideas and perspectives.</strong> Are you a database internals academic? We would love to hear about the latest research and how it could apply to Postgres. Are you a member of another open source community? We would love to hear about tough problems you solved. <a href="https://2024.pgconf.dev/cfp/">You should submit a talk</a>. :)</p></li>
</ul>
<p><strong>If you have questions about the CFP and whether or not your talk proposal is a fit</strong>, feel free to reach out to me at mplageman at microsoft dot com. Or you can read Claire’s post on <a href="/blog/2022/01/11/why-give-a-conference-talk/">why give a conference talk</a>. I hope I will read your proposal soon.</p>
<p>📅 <strong>The deadline for the <a href="https://2024.pgconf.dev/cfp/">PGConf.dev CFP</a> is Monday, January 15th @ 11:59pm PST.</strong> So, now is your chance. If you have ideas, expertise, and experiences to share with the Postgres community, you should submit a talk.</p>
<hr>
<figure>
<picture>
<source srcset="https://cituscdn.azureedge.net/images/blog/pgconfdev_screenshot.webp" type="image/webp">
<img src="https://cituscdn.azureedge.net/images/pgconfdev_screenshot.jpg" width="850" height="622" alt="screenshot of PGConf.dev event page" loading="lazy" />
</picture>
<figcaption><strong>Figure 1:</strong> Screenshot of the PGConf.dev conference home page, with beautiful Vancouver imagery in the background. If you’re interested in sharing your perspectives on Postgres, you should submit a talk!</figcaption>
</figure>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2024/01/11/you-should-submit-a-postgres-talk-to-cfp-for-pgconf-dev/'>citusdata.com</a>.</em></p>Highlights from podcast episode about Postgres monitoring with Lukas Fittl and Rob Treathttps://www.citusdata.com/blog/2023/12/15/highlights-from-podcast-episode-about-postgres-w-lukas-fittl-and-rob-treat/2023-12-15T16:31:00+00:002023-12-15T16:31:00+00:00Ari Padilla<p>The latest episode of <a href="/podcast/path-to-citus-con/">Path To Citus Con</a>—the monthly podcast for developers who love Postgres—is now out. This episode featured guests <a href="https://mastodon.social/@lukas@hachyderm.io">Lukas Fittl</a> (founder of <a href="https://pganalyze.com/">pganalyze</a>) and <a href="https://twitter.com/robtreat2">Rob Treat</a> (an early Circonus developer) on the topic <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-postgres-monitoring-with-lukas-fittl-rob-treat">“My Journey into Postgres Monitoring”</a> along with co-hosts <a href="https://hachyderm.io/@clairegiordano">Claire Giordano</a> and <a href="https://www.linkedin.com/in/pinodecandia">Pino de Candia</a>.</p>
<p>Have you ever asked yourself: “Why is my query so slow?” Or had to figure out which query is slowing things down? Or why your database server is at 90% CPU? According to Lukas, you might find these and many more answers by reviewing your error logs.</p>
<p>If you’re running Postgres on a managed service, what kinds of things do you need to monitor & optimize for? Versus what will your cloud service provider do? There is a discussion on this as well as a segue onto monitoring vs. observability: what’s the difference? </p>
<h2>My Journey into Postgres Monitoring</h2>
<p>Lukas Fittl and Rob Treat, joined by co-hosts Claire Giordano and Pino de Candia, had a broad conversation about all things monitoring: ways or tools to monitor Postgres (pganalyze, pgMustard, pgBadger, pgDash, your cloud provider’s Query Performance Insights, pg_stat_statements, pg_stat_io, & more), access to log files, pain points that people are trying to solve, and the role that AI might play in monitoring databases of the future.</p>
<p>Let’s dive into some interesting bits from the episode…</p>
<figure>
<picture>
<source srcset="https://cituscdn.azureedge.net/images/blog/PathToCitusCon-Ep10-My-Journey-into-Postgres-Monitoring-youtube-1200x675.webp" type="image/webp">
<img src="https://img.youtube.com/vi/BPqFMLeTINQ/maxresdefault.jpg" alt="YouTube thumbnail for Path to Citus Con Ep. 10" loading="lazy" width="850" height="478" />
</picture>
<figcaption><strong>Figure 1:</strong> YouTube thumbnail for episode 10 of the Path To Citus Con podcast for developers who love Postgres, with (starting in the top left, listed clockwise) Pino de Candia, Claire Giordano, Rob Treat, and Lukas Fittl. The topic = “My Journey into Postgres Monitoring.”</figcaption>
</figure>
<h2>Highlights from the podcast episode with Lukas & Rob</h2>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“I think [the biggest pain point] is definitely slow queries, right? Because that's when you get yelled at as a DBA. Is when the queries are slow. And then I think there's that next level, which is like CPU and memory and disk IO… But the thing that causes somebody to come to your desk (as if we still did that) and say, Hey, there's a problem, right? That's slow queries.”</strong> – Rob Treat</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>Queries are used to get data from a database. When you are online shopping and using filters while searching for products, an application is running a query to get you the results. If a filter takes longer than expected to give you the results, you might be bothered or go to a different website. Nobody likes slow query responses.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“The thing that still surprises me is most people don't look at their logs for Postgres or at the error logs. And there's so much useful information in there. And if you talk to the typical hacker, they would say, yeah, sure, I'll put a log message right when this happens, but most people just don't look.”</strong> – Lukas Fittl</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>It’s often useful to be reminded of things that are obvious. Because obvious things can be overlooked. In this episode, Lukas reminds us to look at the logs when investigating issues with the database. When you are investigating a problem, the error logs might contain the answer to your questions.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“You generally are going to need external systems in order to trend data over time. So be aware that whatever your journey is going to look like, probably is going to involve external tools. Whatever those might be because you're going to need some way to trend that data over time in a way to make it easy to understand and analyze.”</strong> – Rob Treat</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>Since I started working on Postgres, one of the things that I quickly learned was about this rich ecosystem not only of extensions but also tooling. Given that Postgres is an open-source project and easy to get started in, and also because developers love working on Postgres—a lot of useful tools have been created. Including several cool tools that will help you visualize the performance of your database over time.</p>
<div class="normal-quote" aria-hidden="true"></div>
<blockquote>
<p><strong>“So that kind of sparked that initial idea that what if, we just made a dashboard that showed all the queries that have run on your database in, like the last hour, the last 24 hours, just giving you a really clear overview of the database's view of the world, because as an application engineer, you're oftentimes not really seeing that, right?”</strong> – Lukas Fittl</p>
</blockquote>
<div class="response" aria-hidden="true"></div>
<p>Some of you have told us that you like the origin stories we cover in the Path To Citus Con podcast. For example, there are a couple of episodes about how people got started as a developer (and in Postgres). In the case of Lukas and pganalyze, it was interesting to hear what motivated Lukas to start this bootstrapped monitoring company in order to solve his own use case.</p>
<h2>Where to find Path To Citus Con podcast episodes (and transcripts)</h2>
<p>You can listen to and find the show notes online for this <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-postgres-monitoring-with-lukas-fittl-rob-treat">My Journey into Postgres Monitoring</a> episode with Lukas Fittl and Rob Treat—and the <a href="https://pathtocituscon.transistor.fm/episodes/my-journey-into-postgres-monitoring-with-lukas-fittl-rob-treat/transcript">transcript</a> is online too.</p>
<p>Our next podcast episode will be recorded live on Discord on <strong>Wed Jan 10 @ 10am PST | 1pm EST | 6pm UTC</strong></p>
<ul>
<li><strong>Calendar invite</strong>: To participate in the live text chat that happens in parallel to the live recording (it’s fun), <a href="https://aka.ms/PathToCitusCon-Ep11-cal">mark your calendar</a>. All the instructions for joining the live show on Discord are in the calendar invite.</li>
</ul>
<p>You can find all the past episodes for the Path To Citus Con podcast on:</p>
<ul>
<li><a href="https://podcasts.apple.com/us/podcast/path-to-citus-con/id1695014346">Apple Podcasts</a></li>
<li><a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy50cmFuc2lzdG9yLmZtL3BhdGgtdG8tY2l0dXMtY29u">Google Podcasts</a></li>
<li><a href="https://open.spotify.com/show/6XtjTEc4KGMK9fIGvmPLn7">Spotify</a></li>
<li><a href="https://aka.ms/PathToCitusCon-playlist">YouTube</a></li>
<li><a href="https://pathtocituscon.transistor.fm/subscribe">And many more podcast platforms</a></li>
</ul>
<p>More useful links:</p>
<ul>
<li><a href="/podcast/path-to-citus-con/">Index of all past Path To Citus Con episodes</a></li>
<li><a href="https://aka.ms/PathToCitusCon-cal">Subscribe to the calendar</a> to know when all the future shows are coming up here.</li>
</ul>
<p>Thanks for listening! And if you enjoy the episodes of the Path To Citus Con podcast for developers who love Postgres, please tell your teammates. Also, we really appreciate ratings and reviews, so more people will discover the podcast, and hopefully be as delighted as you.</p>
<p><em>This article was originally published on <a href='https://www.citusdata.com/blog/2023/12/15/highlights-from-podcast-episode-about-postgres-w-lukas-fittl-and-rob-treat/'>citusdata.com</a>.</em></p>