If you want to learn more about Citus on Microsoft Azure, read this post about Hyperscale (Citus) on Azure Database for PostgreSQL.
Intrinio Scales Out Postgres for Real-Time Financial Data API with Citus Cloud
Intrinio is a financial markets data API providing access to more than 300 data feeds. Intrinio is dedicated to the idea that innovation can only succeed if financial data is easy to work with and priced to move. Co-founded in 2012, the team has grown rapidly over the last few years and works tirelessly to deliver high-quality financial data at disruptively low prices.
Intrinio’s Amazon RDS database was a ticking time bomb
Being a data provider, Intrinio’s database is the lifeblood of the business. Intrinio’s database must scale, deliver high performance, and be available 24/7. Intrinio launched with a single instance of Amazon RDS Postgres. However, as the Intrinio API neared RDS Postgres size restrictions, the API began to encounter problems.
“Our RDS Postgres database was growing and AWS was making it difficult for us to upgrade and make it bigger. It was a ticking time bomb,” says Alex Solo, CTO for Intrinio. “I was anticipating continued data growth, so something had to give. We needed to make a fundamental shift.”
When it came time to scale, the team at Intrinio found it difficult to obtain the enterprise-grade support and capabilities they needed in RDS Postgres. “You can hit one button in AWS and your database will go down. Business grinds to a halt. We’re a data company. The database is the lifeblood of the business. It’s not acceptable for the database to be down,” says Solo. “We were looking for something with more of a personal touch, but also more advanced technology that could adjust to our needs.”
The process of moving from RDS Postgres to Citus Cloud was completely seamless. We had no lapse of service. Nobody even noticed. My developers said, ‘We changed database providers without downtime? What kind of black magic is this?’Alex Solo, CTO, Intrinio
Counting down to doomsday—Intrinio’s database was too big to be vacuumed
In addition to support and scalability issues, Intrinio’s database was too big to be cleaned or vacuumed. “We were adding and changing data faster than the database could keep up. We were approaching a point where the database would start to corrupt itself. It was like watching a doomsday clock,” says Solo. “I was staying up nights trying to figure out how to solve this problem, because if the database goes down or becomes corrupt, I could pretty much say goodbye to my job.”
Solo determined that sharding at the application layer—splitting the existing Postgres table into separate databases on separate nodes, and re-architecting the application to know about multiple different databases—would solve these problems. Unfortunately, manually sharding the database like this would take at least a month of dedicated work and it wasn’t without risk.
“I wasn’t about to manually shard the database myself because it would have been a headache at least, and really dangerous at most. I also didn’t want to maintain multiple databases by hand. I want to be focused on the application and creating business value,” says Solo.
Intrinio discovered the Citus extension to Postgres—open source, relied upon by large enterprises, & available as a database as a service
While researching options, Solo came across Citus, an extension to Postgres that transforms Postgres into a distributed database. Solo did his due diligence to ensure that migrating the entire business over to the Citus Cloud database was a safe move. “I wanted to make sure that Citus Data wasn’t four people working in a garage. I found that much bigger enterprises with much bigger databases than ours use Citus to scale out Postgres, which told me that Citus is trustworthy.”
Solo was also reassured that Citus Data has an open source repo on GitHub. “Unless a provider has massive resources, like Amazon, I am wary of a closed proprietary platform. With Citus, it was a relief to see that the technology is open source. I could theoretically use the Citus extension myself if I needed to. That gives me peace of mind as a recovery option,” says Solo.
Performance was acceptable on RDS Postgres, but I wanted it to be stellar. By moving to Citus, our API response time has improved by 20%. And while speed is important, the real winner for me is that with Citus, the queries almost never timeout or fail.Alex Solo, CTO, Intrinio
Warp migration feature delivered a seamless migration from RDS Postgres to Citus Cloud
Migrating a database can be risky, especially when the database is business critical and must remain available. Solo migrated the Intrinio database from Amazon RDS Postgres to Citus Cloud using the Citus warp migration feature. The warp migration feature read data continuously from Intrinio’s single-node RDS Postgres database, and streamed data to the new Citus database cluster—without any pauses to writes or reads against the existing cloud database.
“The process of moving from our old RDS Postgres database to Citus Cloud was completely seamless,” says Solo. “We had no lapse of service. Nobody even noticed. My developers said, ‘We changed database providers without downtime? What kind of black magic is this?’”
Solo was assisted by a team of Citus Data engineers throughout the migration. “It was a fantastic experience. During the onboarding process, I was just blown away by the Citus database team’s involvement. They stayed up with me until 4am to make sure the migration went well. It was great to realize I wasn’t doing the migration alone,” says Solo.
On Citus, Intrinio’s queries never fail and response time has improved 20%
It’s been smooth sailing since the migration. “We haven’t had any problems on Citus Cloud. Everything just works,” says Solo. “The Citus database can scale easily, so I’m no longer worried about the doomsday clock that I used to have on my dashboard. Because Citus has distributed the database, formerly massive tables are now small, and the database maintenance workers are more capable of keeping up with our loads. We can scale out infinitely.”
By scaling out Postgres with Citus Cloud, Intrinio is also providing better performance to their customers. “Performance was acceptable on AWS, but I wanted it to be stellar. By moving to Citus, our API response time has improved by 20%. But while speed is important, the real winner for me is that the queries almost never timeout or fail,” says Solo. “Because of the Citus database, I can make a challenging query on a cold database and Citus will return the results within a reasonable time. Before, Amazon RDS would simply fail. After 25 seconds RDS Postgres would time out (even with proper indexing). That would happen a lot to our users, and that was unacceptable.”
When I think about our database now, it’s with warm, fuzzy feelings—not anxiety. I still watch it carefully, but it’s more performant and reliable than ever. When the business team comes to me about onboarding a new data set, I’m not freaking out in the back of my mind, wondering how we’re going to scale our database to handle it. With Citus Cloud, we have a superior database infrastructure, and an amazing technology platform that scales out Postgres.Alex Solo, CTO, Intrinio
With Citus, when Solo thinks about Intrinio’s database, it’s with warm, fuzzy feelings
Working with Citus Data has changed the way Solo thinks about the database. “I have peace of mind because I know there’s a team of competent experts who are willing to help me. Our database is so extremely important. Working with a provider that delivers a personal touch and expertise is amazing,” says Solo.
On Citus, Solo no longer worries about the database being available. “Minutes are painful when something goes wrong, right? We’re shooting for 100 percent uptime. We’re ingesting new data all the time. The database has to be up. Our users must have access to real-time data because they’re making investments. There’s a lot on the line. The Citus team demonstrated that they’re willing and capable to be there for us.”
Solo added: “When I think about our database it’s with warm, fuzzy feelings—not anxiety. I still watch it carefully, but it’s more performant and reliable than ever. When the business team comes to me about onboarding a new data set, I’m not freaking out in the back of my mind, wondering how we’re going to scale our database to handle it. With Citus Cloud, we have a superior database infrastructure, and an amazing technology platform that scales out Postgres.”
Alex Solo, CTO, Intrinio
When I think about our database it’s with warm, fuzzy feelings—not anxiety. I still watch it carefully, but it’s more performant and reliable than ever. When the business team comes to me about onboarding a new data set, I’m not freaking out in the back of my mind, wondering how we’re going to scale our database to handle it. With Citus Cloud, we have a superior database infrastructure, and an amazing technology platform that scales out Postgres.
Intrinio believes that financial innovation can only happen if data is easy to work with and priced to move, but the traditional method of data sharing wasn’t designed to encourage the development of game changing financial analytics applications. The Intrinio Fintech Marketplace upends tradition, removing financial and technical roadblocks so developers can get to work.