Apache Spot… the end of the road
While I did not start the project that became Apache Spot I would like to believe that my agitation against the Intel project ONI in 2016 helped move it from “Maybe OSS” inside Intel to donated to Apache. I do believe today that this project is ready for the attic but to do so is worth recognizing the state of play changes in the field within the last 8 years.
Friends, Contributors, Moochers I come to bury Spot, and also to praise it.
Before I explain I would like to give credit to a number of crucial contributors to the project without whom we would have never made it to Apache…
While I know that there are many I am missing (at the time of donation intel had a team of almost 40 not listed here working on the project) I would be remiss to state that this project would never have existed without important vision, strategy or contributions from the following individuals.
Following on to Spot’s incubation the following contributors gave significant time to the Spot Operational Data Models that have become the basis for a number of modern security products:
This project has had a number of internal mentors rotate threw but I would be remiss if I did not appreciate the guidance certain members who were always ready to jump on the phone when times were tough in particular:
One should not mention those who contributed to the project without mentioning with love those who inadvertently sabotaged it… in the early 2010’s before we even got started
wrote a blog post that was distributed by Cloudera about how to implement Apache Flume in the absolutely worst possible way. It quickly became for no reason I can understand the number one google response for “how to implement flume”. I personally believe that the misdirect of this blog post led almost entirely to the modern popularity of Kafka which might realistically be completely unnecessary when it comes to true security Big Data ingestion at scale. Anyone with Confluent stock now you know who to thank.
Finally without taking credit for other’s work I like to hope and believe that my personal obsession with the data models for ONI and Spot during my time at eBay helped in some small way inspire the work that our former colleagues at eBay have contributed to the Open Cybersecurity Schema Framework (OCSF) namely:
Now that acknowledgements are complete, I would like to explain why I believe we failed to make true progress on this project over the years since my team first became engaged with it.
TL;DR the companies that require a solution (hopefully more robust) in the style of Spot are on a dollars and cents basis a tiny piece of the market.
It’s important to understand why security organizations (in 2016 and today) at large companies tend to work to rely on vague telemetry monitoring. Namely engineering organizations prefer to maintain their distance and black box their solutions facing InfoSec (and I say what comes next after 8 years of work in security where the previous 10 years was part of data science and core platform engineering) out of a desire to punt solving problems correctly to anyone not associated with their Jira backlog.
I can say with 100% certainty that as the final say in our orgs leadership since our team left eBay that every time we have had the dollar buffer to contribute to our code bases we never seemed to make progress on Spot instead we seemed to consistently make progress for our cloud customers on what we would now call SMFSD.
State Management For Secure Deployment
While cloud was not new at the advent of Spot it has in the intervening time it has eaten the middle market in the best possible way. Cloud services have provided a powerful new surface of transparency for security organizations willing to take the plunge. The ability to on demand audit for example the configuration of an entire API surface that would have been previously have been at best part of logging as “nginx server errors/exceptions” creates a unique opportunity for security organizations to move from a on your heels reactive poster to an on your toes preventative posture. While this change doesn’t discount the need for mature security organizations to focus on monitoring, it does change the order of operations for smaller organizations when it comes to the question “what does it mean to be more secure?”
We today (contributors with whom I collaborate) chose to focus on this middle path as we believe it is the most effective way to raise all ships with our limited resources. Within that we will do all we can to bolster projects like Open Cybersecurity Schema Framework which we believe that in the long tail will achieve what we set out to do here….
Apache will always have a Spot in our Heart
