DBaaS - Kafka
Problem: Our scaling users want to use a managed service for their Kafka (Apache Kafka is a distributed event triggers and streaming platform used to handle large amounts of real-time data that is used by more than 80% of Fortune 100 companies) clusters so they can have more time to focus on their goals instead of their architecture. Self-managed Kafka is a huge learning curve and challenge for users and takes several months to get a production-ready system.
Solution: Integrate Kafka into our Managed Database engine solutions and provide a simple experience to setup Kafka-specific inputs. This allows users to not worry about the provisioning, managing, scaling, updating, and security of Kafka clusters and help mitigate human-error of self-managing a multi-node cluster that could lead to operational downtime and data loss.
Goals: Ensure Apache Kafka is fully integrated into our Managed Database engines with high availability and reliability with an easy-to-use experience that breaks down Kafka’s complex “Topics”.
Results
eCommerce Conversion rate: An average increase of 0.11% was seen with our Beta clients, contributing to their rising revenue and orders
Role
Product Designer Lead for DBaaS
Collaboration
PM, Technical PM, Back-end Eng, Front-end Eng, Insights, Marketing, Pdocs
Research
As always, competitors are researched to evaluate how their managed Kafka solutions are presented to their users and how the complexity is broken down.
NPS and CSAT surveys were used to view past feedback that mentioned Kafka. Additionally an initial interest survey was sent to existing DBaaS users and answered by 60+ users about:
- Their knowledge of event streaming technologies
- What they currently use or have used
- What challenges they face with self-managed Kafka
- What pain points they expect to solve with Managed Kafka
- How they plan to use Managed Kafka for testing and production
- What information or tutorials would be helpful
Research Findings
- Average 4/5 rating knowledge of streaming technologies. Users are generally knowledgeable
- Users are looking for a managed solution and to simplify their architecture
- Users are wanting uptime reliability and High Availability
- Users are concerned about pricing and the ease of Topic management
Beta Feedback
While the UI or experience wasn’t brought up by Beta users (I consider that a good thing), Beta users requested 2 graphs be added into Insights, messages per second and incoming messages. Most users use Kafka through the API.
Price was the largest concern during Beta. We were able to work through configurations to offer lower cost Kafka clusters that are still stable and are highly available.
Primary Experience
Cluster Create
Kafka has different creation and configuration requirements than any of DO’s other Managed Database engines. Kafka is unique in the fact that it absolutely requires 3 nodes instead of the usual per node configuration that is offered. This is due to the fact that Kafka requires High Availability as well as its Replica condition. In order to properly convey this in our UI, I designed an info box describing that Kafka requires 3 nodes in a cluster and that pricing is based on that. Additionally, our engines usually have read-only and standby nodes so I made sure to make users aware of that in case they find themselves wondering why those options are not visible.
Since we are packaging the Kafka configurations much differently than other databases, I made sure to label the total review section to thoroughly explain to users that they are in fact priced for 3 nodes and I broke down the specs for each node. This breakdown is not available or needed for the other DBaaS engines.
Onboarding & Connection Details
To quickly guide users on setting up their Kafka clusters after provisioning, I have created an onboarding experience in which users can quickly set up their first Kafka Topic and connect to their architecture. To aid users in creating a Highly Available Topic, the defaulted values are optimal for stability so users can simply name their Topic and quickly move on without being bogged down by making sure they input the right values.
Users are quickly able to copy and paste their connection details. I reinforced security messaging by a persistent VPC info box explaining why users should connect via VPC. Additionally, I collaborated with the back-end engineering and product documentation team to create a connection video for Kafka to help users who may be stuck connecting the Kafka cluster.
Topic Creation
Topics are what make Kafka specifically unique in DO’s DBaaS experience. As mentioned above in Onboarding, I’ve added prefilled values into the inputs so users are quickly able to create Highly Available Topics by simply just naming them. To help guide users and educate them about fields that may not understand, I have created helpful tooltips that not only explain what the item does, but talk about the limitations. The limitations for both the Retention fields are so incredibly high that users should not realistically encounter the max limit.
Table Overview & User Roles
DO’s other database tables typically show Users & Databases but Kafka is a bit different. For Kafka, I have split the table overviews into Topics & Users. Topics is the most important aspect of Kafka while Users have roles in which they are able to access Topics. I designed a modal for the permissions editor that clearly describe what those roles are and how those roles apply to Topics. For this first launch, all permission types are applied to all Topics.
Check Out the Official DO Video on the Kafka Experience
🤔 Challenges
- This was my very first task when I first joined DO and I’ve never worked on such a technical product before so there were some learning curves that I had to overcome in order to provide the best experience for DBaaS users
- The ability to assign specific users to Topics was removed out of this first launch Kafka but allows us to see if this feature is highly requested by production Kafka users
✨ Opportunities
- Beta Users requested several Kafka features that they would like to see: Kafka Connect, Kafka MirrorMaker, and ZooKeeper
- This first GA launch of Kafka doesn’t have the ability to assign users to a specific Topic but it is something the DBaaS team and users would love the ability to do to increase security in their architecture
👨💻 User Feedback
DigitalOcean Managed Kafka simplified and sped up our management of Kafka. Self-managing Kafka, it would take about eight weeks to go from initial development to a production-ready system. With DigitalOcean Managed Kafka, we cut that timeline to just one week.
-Daniel, CTO of a sports tech company
The ability to create topics using a UI is super nice. It used to be a CLI job.
-unknown survey user
✍️ Peer Feedback
