How One Open Source Project Derived from Another’s Limits
One of the great things about open source is that its communities can be hothouses of innovation. Work on one project and you may find that it generates others, as contributors find ways to solve the new problems that emerge when you’re working together to build and fine-tune a solution to an existing problem.
In this episode of The New Stack Makers, Vini Jaiswal, principal developer advocate at ByteDance (the parent company of TikTok), talked with Alex Williams, publisher and founder of TNS, about her journey through open source and her work on ByConity.
The open source project grew out of ByteDance’s use of Clickhouse, an open source, column-oriented database management system.
ByConity, which is built on Clickhouse, “emerged when we wanted to solve our business problem, when our business actually grew and data volume was spiking a lot,” Jaiswal said. “So we saw an opportunity to enhance the separation of compute and storage. And why we did that is to improve multitenancy support and optimize the query performance in cloud native environments. This inspired us to embark on the journey by creating by community.”
Because compute and storage are separated in ByConity, she added, it prevents problems with using data lakes if the pipeline is corrupted. ByConity allows for read-and-write separation, elastics and scalability.
“By adopting this architecture,” Jaiswal said, “we dynamically were able to expand and contract computing resources based on demand ensuring optimal utilization and performance.”
Fixing Bugs, Together
Jaiswal, who oversees ByteDance’s open source program office, started working on open source projects in college but got more deeply involved about a decade ago while working on digital transformation initiatives at CitiBank.
“As I worked with cloud technologies and data science tools, I found that open source was a very powerful enabler for innovation, and it was also helping us accelerate the transformations,” she told Williams. “So it allowed me to collaborate with a global community, address bugs and vulnerabilities.
“And it would be solved very quickly, because, you know, I would throw out question there, I would throw out like, ‘Oh, this is the exact error I’m getting’ and we then would customize the solution to fit our needs.”
That community, and sense of putting heads together to solve a problem, is a strength of open source, she said.
She urged podcast listeners who are intrigued by open source but shy about contributing to projects to give it a try.
“For those who are embarking on this journey, I recommend them starting identifying a pain point and just start contributing,” Jaiswal said. “A small contribution can make a very big impact over time.”
Check out the entire episode to learn more about her open source journey, including work on the machine learning tool PyTorch, and how open source and generative AI will intersect.
