[ad_1]
Ana Codallo, CTO at Key Opinion Leaders (KOLs).
getty
The past two years have been tumultuous, highlighted by a global pandemic and an international conflict that has touched millions of lives and entire economies.
While these events unfolded, at Key Opinion Leaders, we were taking on a major challenge: to build a technology stack that could address a problem on a global scale without the resources of an international company.
Here are the six things I saw that made the difference in our platform’s rapid growth that your company can also use without breaking the bank.
1. Be (really) Cloud-Native
Cloud technologies have one major advantage: you don’t have to bear the initial cost of developing the technology. In other words, one gains access to technologies that have cost (Google, Amazon, Azure) billions of dollars to build, at a cost of cents per hour. But not all cloud technologies are created equal. some experts put them in two buckets:
• Cloud-native: Technologies where the consumer has no operational burden and can focus on building functionality on top of the technology (eg Google Appengine, Amazon Lambda, Bigquery, Redshift).
• Cloud-hosted: These are bare-metal technologies that run in the cloud using virtual machines (eg Google CloudSQL, Elastic search hosted in a cloud service, Hadoop running on AWS).
We learned that by designing our stack to be cloud-native, our operational overhead turned out to be very low. Of course, this comes at a price: “Vendor download”, but that’s a topic for a different article.
2. Start small
If it’s possible in your problem space, it might be worthwhile to create a sample version of the problem so your teams can iterate on their early/alpha builds more quickly.
For example, Key Opinion Leaders’ bread and butter is data, and data is huge—in the hundreds of terabytes. The massive data sets we were working with made it obvious that the only way we could iterate quickly on data quality and pipelines was to create short iteration cycles.
We spent a lot of time building a sampling engine that could produce small development datasets for developers to iterate on quickly, but without biasing the models during development/tuning.
3. Do it yourself First
In order to get a good understanding of the technical challenges (and shortcomings) of your technology stack and teams, it’s probably best to get first-hand experience with how the stack works. For example, all of our senior technical leaders managed the data pipelines (as they were in development) and trained the models themselves (end-to-end). this was a critical exercise so that all of us could experience the pain points first hand.
This process made us very clear on specific pain points and, in a way, told us where we needed to invest time and energy so that the team could move forward faster.
4. Witness The Developer’s Experience
Engineers are tough—they can handle many different challenges. Sometimes, when faced with suboptimal processes, your engineers will either endure the pain or find ways around the shortcomings. It’s worth it to watch how the engineers and developers iterate and be a silent witness to this journey.
For example, on our journey watching engineer iterations, we learned things that needed to be improved or clever ways we could avoid deficiencies altogether, saving time and resources.
5. First data
If you’ve ever had the opportunity to work in startup environments, you’ve probably seen teams focused on complex infrastructure or shiny user interfaces. As infrastructure becomes a commodity (eg, almost everything is available in the cloud as a service), you’ll likely find better and more frequent opportunities to add value to the data side of the stack. We found that, in our case, it was much more efficient to iterate on the accuracy and recall of our data first, and then move on to infrastructure challenges like how to serve such a large data set with millisecond latency.
6. If you need an army of (Humans), it’s probably not a good solution
While doing all of the above, it’s also important to maintain the perspective you need to build a platform that your company can sustain and continue to improve with a relatively low headcount. For example, during all of our R&D iterations, we came across several solutions or designs that, while beautiful, would have placed a functional or conservative burden on the team, making them unfeasible. We have learned to resist the temptation to adopt solutions because of their beauty and lean more towards cost effectiveness and operational efficiency.
Bottom line, while all companies and teams are different, you’ll likely get better mileage out of your technical teams and systems if you embrace the idea of being cost-conscious from the early stages and be clear about what costs are taking many different forms, some of them being very thin.
For example, cold storage costs and service costs would be traditional metrics to consider when making design decisions. However, there are other more fine-grained metrics such as cost of operation, cost per query, number of heads required to run a feature, development cost per new feature, or average time to market for a new feature to have multiplier effect on productivity if you focus on them (and understand the concept of their importance) early on.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Am I eligible?
[ad_2]
Source link
