Dragonfly Brings Peer-to-Peer Image Sharing to Kubernetes
Dragonfly, a peer-to-peer image and file-sharing technology developed by Alibaba Cloud, is now an incubation-level hosted project of the Cloud Native Computing Foundation. The software provides a way to quickly distribute images across large cloud native deployments, eliminating the dependency on a single registry to distribute all the copies of an image.
“Dragonfly is one of the backbone technologies for container platforms within Alibaba’s ecosystem, supporting billions of application deliveries each year, and in use by many enterprise customers around the world,” said Li Yi, Alibaba senior staff engineer said, in a statement.
Alibaba Cloud created Dragonfly in 2015, originally to ease file distribution. By 2017, when it was adopted to share containers within Kubernetes environments, it was being used by the Chinese cloud service to share 3.4PB each month. It was originally accepted into the CNCF Sandbox program in October 2018.
A peer-to-peer file sharing mechanism could help cloud native deployments in distributing new and updated container images across a cluster more quickly and with less stress to the upstream network. P2P, first popularized by music sharing programs such as Napster two decades ago, can make full use of the cluster’s bandwidth while eliminating the possible bottleneck of having a single server respond to all the requests for a new image. In a P2P network, each node, or “peer” can share files with each other rather than all the nodes saturating the bandwidth to the image server by downloading identical copies of a single image.
Dragonfly’s performance improves as more peers share the same material.
Dragonfly is not a pure P2P technology It still requires a supernode, to schedule and control distribution within the peer network. An agent on each node, dfget, downloads the file pieces. Another component, the dfdaemon proxy, intercepts image downloading requests from a container engine to dfget.
The latest version, Dragonfly 1.0.0, was completely rewritten in Golang to improve performance and can be integrated with the OCI (Open Container Initiative). It can work with CNCF’s Harbor to speed delivery further through image pre-fetching. Users can collect Dragonfly metrics within Prometheus and display them on a Grafana dashboard. Helm can be used to install Dragonfly within a Kubernetes cluster.
Project maintainers come from Alibaba, ByteDance, eBay, and Meitu. Overall it has 67 contributors from 21 organizations. It has been downloaded over 100,000 times from Docker Hub and has massed 6,000 GitHub stars.
Learn more about Dragonfly, visit https://github.com/dragonflyoss/Dragonfly.
The Cloud Native Computing Foundation is a sponsor of The New Stack
Feature image by liggraphy from Pixabay.
