The Agency Eating Machine

Despite a sensory information stream of 1 gigabit per second, humans are only capable of thinking at 10 bits per second.

So even though we can see, feel, and experience a very many things at any moment, we are only capable of consciously thinking in a single thought stream at a time, and only at a rate of 10 bits per second.

This means every person’s daily budget for logical thinking is only around 576,000 bits of unique thought, assuming they’re awake for 16 hours out of the day, and in complete control of their consciousness.

At best, our maximum capacity for thought in a day is a little more than half of the sensory information we receive every single second.

Focused, conscious attention is a very valuable, fragile, and limited resource.

But for most of my life, I’ve been expected to conjure it up on demand, every working day.

How Knowledge Work… Works

That’s because, for the past 20+ years I’ve written software systems.

This has proven challenging, as creating software requires bringing up a special sort of focused attention that can be exhausting to create and maintain, but also very easy to lose.

Successful software development requires consistently blocking distractions, mentally loading problems, and systematically planning a solution, without getting distracted.

Fortunately, in software you can usually tell very quickly whether or not your work has “solved” the problem.

But notably, without a minimum threshold of focused attention, it’s impossible to write any software.

Most other knowledge work is also like this. Without the ability to focus and load up problems into our mental framework, we can’t get anything done. Think of it as a hill. Without the ability to get over the hill of required initial attention, it’s impossible to get any actual knowledge work done.

But, now that’s changing!

Enter the Thinking Machines

Because now, we’ve got a new alien intelligence to work with and against, called LLMs.

Despite being trained on massive, internet scale dataset, LLMs are like us, limited in their ability to hold a finite set of symbols in their mind when solving a given problem. We call these symbols “tokens”.

Each time we interact with them we have a limited window within which to explain and set the stage for our unique problems, using the language of their tokens. This is called a “context window”.

Each LLM is diferent, but for example, Claude has a context window size of 200,000 tokens for Pro users.

This means if we want to extract answers and explore ideas with Claude, we must do so within a 200,000 token context window.

How LLMs Load Context for Your Problems

This context for describing and navigating through our problem is so critical to the peformance of LLMs, that Anthropic has released a whole new protocol called “Model Context Protocol”, to allow developers to empower the LLM to autonomously decide when it needs to to grab more context for any given query.

This is because LLM model weights are “frozen” in production. The LLM doesn’t know of, and can’t reason on things that have changed since it was trained.

Model Context Protocol lets us change that, by exposing Tools, Prompts, and Resources.

The LLM models can use these resources to autonomously decide to search the web, access a database, or create new things like web pages. Again, just as long as each of these actions and their results fit within the Context Window.

This makes the LLMs much more powerful at solving novel problems autonomously.

The Human Model Context Protocol of Media

Similarly, we humans generally rely on media platforms to load new context into our own thinking spaces.

What we read, see, and watch largely explains the context we have for understanding and relating to the world.

So how do we decide what we load into our own personal context?

In 1976, a man named Wilbur Schramm created a theory for how people decide which media to consume:

Expected value divided by the required effort.

At the time, Schramm saw television as a promising new medium.

Television could possibly educate an entire population at once, delivering sensory rich content (critical for effective learning), at a fixed production cost.

This content could be built by world experts, designed to have maximum educational impact, and shape a shared social context and value system for an entire population at a time.

Programs like Sesame Street tried to achieve this goal, and educate the young population.

This dream came with a trade-off: the viewers’ focused attention for 30-minute blocks, interrupted by commercials, in exchange for polished entertainment.

For decades, this agreement worked.

But then social media arrived, and the math on our collective attention spans changed.

Attention at 30 Seconds or Less

Where television asked for thirty minute units of our attention, TikTok and Instagram demanded just thirty seconds for a novel experience.

Better still, once on these platforms, they they elinimate the need for conscious choice. Instead their algorithm can usually find a better piece of content than we’d be able to find ourselves, without any conscious effort on our end.

This might seem like a simple shift in duration and cost of decision, but it fundamentally breaks Schramm’s ratio and lowers our ability to choose media experiences outside of these algorithms.

Traditional media has to create shows and films to target a very large audience, in order for the cost of creation to pay itself back. Social media flipped this approach, making the audience fund their own entertainment, allowing the audience to discover and create much more niche content than previsouyl feasible in mass media.

When the required effort of a media platform approaches zero, and content is infinitely personalized, the other media choices become even less relevant.

Why invest attention elsewhere, for a lower possible reward on average?

TikTok’s unique breakthrough of steering recommendations based upon whether or not we swipe “next” on a video has created an algorithm that now has the average teenager spending 3.5 hours per day, locked in a feed.

(For reference, teenage television viewership peaked in 1995 at around 2.95 hours per day.)

Of course, we should mention there are positive associated benefits:

A majority of adolescents report that social media helps them feel more accepted (58%), like they have people who can support them through tough times (67%), like they have a place to show their creative side (71%), and more connected to what’s going on in their friends’ lives (80%)

But there’s also something fundamentally different about how social media affects their mental health.

Around ⅓ of teens report using social media “almost constantly”, and:

adolescents who spent more than 3 hours per day on social media faced double the risk of experiencing poor mental health outcomes including symptoms of depression and anxiety

(There’s plenty more from the Youth Mental Health Social Media Advisory.)

So what could be responsible for this rapid increase in misery and isolation?

Schramm’s Model of How we Communicate With Each Other

To answer this, we can look another model Schramm created for explaining how communication occurs between people.

Critically, when we want to send a message to someone, we must first encode it, using a our shared experience as a medium.

Platforms like television and books give us a common set of myths, used as a shorthand to tell our inner stories, and relate to one another.

But through this lens, social media seems to be a shared experience lowering tool.

Social media is an isolating system by design, as the feeds become increasingly personalized. The ideal feed shrinks to fit our unique needs, and minimizes the size of our shared collective experience, pushing each us into isolated, self-reinforcing idea bubbles.

Social media customizes itself using our non-verbal, non-thinking cues. Again, these happen quicker than our verbal, conscious thinking process. They continously steer us further away from a shared cultural reference point, and further split us into a set of feedback loops inducing emotional responses and increasing stickiness.

Calling the current social feeds “algorithms” doesn’t really cut it anymore. The recommender systems that being built are advanced forms of artificial intelligence, trained to addict us in exchange for the opportunity to sell some ad space.

Better Living Through LLMs

Because once again, AI has an unequal footing with our biological systems. We can only consciously perceive deliberate thoughts at our 10 bits / second when we given them our full attention, but the software systems built around us operate at a much faster pace.

LLMs and future media could increasingly short circuit our more basic processes and turn us into media addicted zombies if left unsupervised.

We’ve all seen people in ideal sensory zones– the beach, concerts, on dates… choosing to be on their phone, rather than in their current physical space.

But what if we wanted to build a media platform that challenged us, challenged each other to be better, instead?

What might that look like? How might we be pulled away from the endless loop of personal gratification and distraction addiction?

How could we take the positive things from social media, and add them to the emerging, lower cost of thinking powers that LLMs give us?

LLMs are Already a New Media Platform

For me, the video above shows social media at its best.

When we work together to create something new, opening each of us to participate in something absurd and fun.

Although we’re still in the early innings with LLMs, they do seem to be actively rewriting large portions of how we interact with and build a new media.

We can see this when we ask LLMs to do work for us that seems tedious, or when we feel stuck on a problem. They lower the costs of asking a “dumb” question to zero.

This means instead of the avoiding the normal anxiety associated with avoiding a problem, we can ask an LLM to take a first stab at a problems. We can then step in to correct the places that feel wrong.

And LLMs at their best allow us to be more ambitious in our work, writing software we might not have otherwise written, or thinking ideas that would otherwise be inaccessible.

LLMs lower the cost of thinking new ideas.

So what’s the problem?

LLMs by default have the same social problems that we see with social media.

They are fundamentally shared context shrinking.

Again, added to this is the fact that they have a knowledge cutoff date, and are currently “frozen” in time.

Some models can incorporate search results at prompt time, but they cannot “learn” from their conversations with you in real time, and they cannot “learn” from the current cultural climate.

But they have made the process of exploring and linking new ideas together cheaper. LLMs allow us to explore new ideas with a fundamentally lower cost of required attention investment. We can ask an LLM to link together two disparate ideas, have them translated into our knowledge domain, and see whether or not the beginning results match what we’d hoped for.

If we give LLMs the ability to better fill up their context windows with information about us, and about the people we care about, they don’t have to be socially isolating.

This will allow us to collectively think a lot of new thoughts that previously would have had too high of an attention cost to explore effectively, while minimizing the social costs.

I dream of making the tool that would combine the fun of the video above with the power and flexibility of LLMs to explore new ideas. In fact, that’s what I’ve been working on.

Drawing the Owl of the Next Media Platform

Can a better media platform emerge from the best parts of social media and LLMs?

I’ve spent the past year and a half focused on this very idea, trying to come up with something that could be fundamentally better.

One of the last social media platforms I built started with video editing.

It’s become apparent that using LLMs to search through piles of video for us is useful. We can use Model Context Protocol and custom Tools to suss out the interesting bits, or all the times when something related to our ideas happens.

We can also isolate things in video now too. This combined with the idea exploration LLMs are so good at makes for a much more dynamic, interactive, experimental platform.

I’ve started to put these ideas together, but it’s still incredibly early, with a lot of unanswered questions.

For example, what is the LLM medium equivalent of the TikTok recommendation algorithm for ideas?

On the day I published this post, almost every top app in the app store was an LLM chat tool, or some media platform. (Temu and a VPN app were the only two non media platforms.)

LLMs are already a new media platform, as proven by their spots in the app store. The question is how we will decide to use them going forward.

If you’re interested in an alternative, you can sign up for the waitlist for some new tools at video-jungle.com.

You can also reach out, and follow me on the existing media platforms. Because they are where everyone is, for now.

In the meantime, in order to reach people we will need to continue to perform, speak, and act in the ways the addiction algorithms reward.

Until we don’t.

Thanks to Erica Dohring for reading an early version of this blog post.

All the animations on this blog post were made with the help of Claude and Manim, and the source code for them lives on Github.

Last year I was let go after just 6 months in a new role.

I had left a great company and boss to take a chance on a startup, and before I’d even begun, it was over.

I decided to take the event as an opportunity, and explore what was now becoming possible in video with LLMs, Diffusion models, and the growing number of other open models.

See, years ago I’d helped build a generative video editor that became a unicorn, and had ideas from then I’d wanted to see built.

These ideas were mostly unreasonable back in 2015, but given LLM and computer vision model progress, were now becoming possible.

The GPU Crunch and Local First, Multi-Modal Generative AI

So I initially focused on building a local video editor improved with multi-modal artificial intelligence. It used computer vision to detect, extract, and track objects in video, combined with Diffusion models to add and animate new objects in to videos.

I’d previously done daily video sketches 5 years ago, using Mask-RCNN, experimenting with skateboard videos:

These video sketches previously allowed me to explore the medium of AI-assisted video editing, without any strong expectations.

I assumed building a tool to continue exploring this work would prove fruitful:

And indeed it did! I was soon playing with video as a more fluid medium, one that felt a bit more editable. I began to understand how the new vision models worked, and how the GPU could be used to speed up rendering, inference, and video.

By using a combination of models I was able to create a prompt for adding unique, diffusion generated objects into videos, already masked off.

You can read about that process in a previous blog post.

A Sidequest for Safety (Not the AI Kind)

But as I was building out computer vision pipelines and prototypes for the video editor, I experienced a string of tragic local deaths.

Bicyclists and pedestrians kept getting hit by cars.

So as a side project, I started researching cyclist safety, and soon discovered just how terrible the pedestrian infrastructure is in the United States. I wondered if there wasn’t maybe a technical solution to reduce or eliminate these deaths, as the statistics showed they’re rapidly increasing.

So on a whim I put together a proposal to address this using artificial intelligence and robotics, and submitted it to the NSF’s SBIR program.

To my surprise, they invited me to submit a formal, Phase I proposal.

If accepted, this meant I could get up to $2 million to pursue and develop my technology, without the government taking any equity.

So of course, I did that.

It took a few months worth of work, and brought me far out of my comfort zone.

But! One of the conditions of submitting my proposal was that I had to pause all of my open source work related to the project, as the government couldn’t give me a grant for work already done.

This frustrated me, as pedestrians continued to die in my town. I felt guilty for each additional person who got hit.

As a consolation, since I’d already done the tedious paperwork to form and qualify a company to accept government grants like the SBIR (via SAM.gov), most of the labor to create another proposal was already taken care of.

So I also submitted an SBIR proposal to the Department of Transportation for Complete Streets AI.

This proposal imagined using smartphones and computer vision to help fill in gaps of pedestrian infrastructure knowledge at the DOT.

6+ months later, I’d found out the final answer from both the NSF and DOT:

“No.”

(On a positive note, this means I’m once again able to be public about this work, and solicit help.)

Taking a Step Back From the Obvious

And after six months of working on the local video editor, I also hit a wall.

It became apparent that AI as a layer “on top of” the existing video editor workflows didn’t make much sense, given how different ML workflows incorporating large language models had become, and how much engineering had gone into everything else around modern, flagship desktop video editors.

More powerful vision and audio models could of course be used to add features that reduce toil in existing workflows, but the underlying assumptions behind the user interface of the video editors seemed to be constraining the discovery of potential new methods for video creation, and more importantly, the evolution of video as a medium.

It seemed the process of video creation itself had to be rethought, using the power and possibility of LLMs, multi-modal embeddings / search, and computer vision / diffusion models as a collaborator.

Which led me to a thought:

What if video was more personal? More maleable? More collaborative?

This meant rethinking current video editing workflows according to the strengths of these models.

Building A Generative Video Platform

So I went back to the drawing board, and rethought the whole concept of a video editor. When I did, a thought occured to me:

What if instead of a single video out of a video editing process we had a video generator out of the video editing process?

What if instead of a single, static artifact at the end of editing, we had a video generator, capable of rendering each video tailored to the viewer? Created on demand, according to unique the specific needs of the viewer?

What if we allowed for the user to collaborate in the experience of their video?

I’d imagined videos would no longer be static outputs, and would instead be like code: dynamic and generated specifically for the viewer, or a specific audience, video becoming more of a medium for play and interaction, rather than the typical, passive consumption model.

Creating a Dynamic Video Generation Pipeline with Promptflow

With that, I started building a new prototype of generative video using Microsoft’s LLM framework Promptflow. It allows mixing calls to an LLM with code, building up whole graph based pipelines for generative AI workflows.

You define these workflows via a yaml file, in which you can express your variables you’d like passed in to your prompts to your ChatGPT tool. The results from these prompts can then be passed back to Python, or used to gnerate more LLM calls.

With this tool as a basis, I built an initial Horoscope video generator, using the most basic generative approaches to video generation. It took a prompt, injected the user’s variables, called out to an LLM to generate a video script, and then generated images, transformed them into videos, added a voice narrator, subtitles, and finally, put together a video edit.

The flow looks something like this:

In this generator, the Video Generator takes in an Astrological Sign, a Date, and a Random Seed. (LLMs are known to have difficulty in generating random numbers.)

These are all used to run a pipeline that generates a unique, on demand video horoscope reading for the user.

From this initial prototype, I immediately ran into a few limitations.

The Promptflow design expected users to build wrappers on a service like ChatGPT, with mostly static flows. Think of things like customer service bots, using RAG to fill in dynamic information necessary to answer a query.

This didn’t match the level of dynamic video generation and editing processes I envisioned getting to. The design and writing of these static flows didn’t feel like the right layer of abstraction.

So I switched my Promptflow over to a different workflow execution engine, Temporal.

Generative Workflows with Temporal

Temporal allowed me to restructure the Generative processes I was building with durable execution as a primitive.

Rather than building for static graphs of execution, I could build out individual tools / processes, and later allow for the user to decide how to link and execute the tools together for their specific process. (In Temporal, these are called “Workflows”, and come with automatic retries and more).

With these as a base, writing ML generation workflows becomes a bit more straightforward. I could define Activities as discreet units with retry, and synchronize the execution of these Activites via Workflows.

To give an example, let’s say we want to analyze a video file that has been uploaded. (We want to know what’s in the video as, as well as generate embeddings, text, and tags.)

There are many places where a failure could occur. A web request may fail to download a file, a file may have an unsupported encoding, or a GPU machine may not be schedulable. Each of these events would normally require logic to handle failure, have a set number of retries, and decide how to gracefully fail.

With Temporal, we instead set our number of retries and failure conditions across each activity. The Temporal platform automatically handles retries when things go wrong:

This is especially useful during development, where if an error occurs, I can usually spot and fix it, and then restart the worker.

The workflow then retries from the last succesful execution, and usually completes.

This substantially speeds up the development flow for my graph execution workflows.

The Challenges of Building with (sometimes unpredictable) LLMs

Creating language model prompts for software workflows is a fuzzy process.

Designing a prompt requires significant time investment, to understand how your chosen model performs for your specific use case. And even once you’ve decided on a prompt, each model seems to have it’s own quirks about how it interprets and decides whether and how to to follow instructions.

This means one prompt may end up being more appropriate for one specific model, versus another.

It’s tough to tell ahead of time if one model may be more appropriate for your task than another.

People try to address this by writing evals.

Evals are tests to see whether or not an LLM comes up with an appropriate answer. You can either write evals using code, or by asking the LLM to determine whether or not its answers are correct.

To help with this, Anthropic now has what it calls a “Workbench”, from which you can use Claude to generate and analyze specific prompts:

Using Workbench allows you to get a feel for how you can approach your chosen prompt task, and what sort of outputs you can expect while developing.

You can quickly evaluate the performance of these generated prompts against one another within the user interface.

Thanks to the example prompts from Workbench, I added a process for generating prompts into my Video Generator, using Anthropic’s metaprompting example from Github.

These metaprompts are a great starting point for helping users get started with a prompt template to build from.

Amazingly, asking an advanced model to generate its own prompts seems to mostly work.

Embeddings Might Not Be the Solution You Think They Are

Vicky Boykis has written an amazing, free book on building embeddings models.

Prior to reading it, I assumed vector databases would dominate search and retrieval for anyone working with LLMs. The hype sold in 2023 was that vector databases would be the future of information retrieval.

But as I began working with embeddings and vector databases, the results didn’t seem to match to the hype.

As I dug in, I discovered this is because embeddings are fundamentally a compressing technology, squashing the unique features in your dataset into a fixed length vector output, across the embedding space.

How well these dimensions map to the data related to your business use case depends on how well you’ve built your embedding space.

But! Most people getting started aren’t training their own embeddings models for their specific use case, and are instead relying on off the shelf, generic embeddings models to apply to their business problems.

This blind application of generalized embeddings models over traditional search can lead to worse results, and less easily debuggable systems.

Let’s take a concrete example.

A Song Search Example

Years ago I built a search engine for songs.

One of the challanges I faced was bootstrapping relevant results, for very generic search terms.

See, it turns out most song titles aren’t very unique, and so naive text search is terrible on song names, albums, and artists.

Because of this, a generic text embeddings model would be especially challenged to give decent results.

Say our user is searching for the term “stop”:

There may be tens of thousands of songs, albums, and artists with the word “stop” in it.

How do you begin to determine which ones should be most relevant?

To solve the problem, I turned to music top charts. These have been published since the 40’s, and contain some of the most important songs, culturally.

By adding a weight or bias score to the songs previously in the top charts, I could help bootstrap an initial search system.

If I had instead started with an embeddings model, I’m not sure I would have as easily built a solution. Maybe an off the shelf embedding model already has partial knowledge of the top charts, but how much?

Similarly, in building an automatic video editor, I’ve discovered it’s necessary to have a mix of embeddings models, along with traditional search ideas, and a bit of domain specific ideas / experimentation.

The Wonderful, Totally Great Process of Building Something New

Any time I see someone finish a thing, I try to go out of my way to congratulate them.

Getting anything out the door always includes an unseen number of challenges, and entropy works against all of us.

So of course, as I’ve worked towards a vision of building something new the past year, I’ve taken a few detours.

There is a great quote from Jenson Huang, when asked about whether or not he’d start NVIDIA again, he says he wouldn’t, because he now knows how difficult it is:

Similarly, over the past year I’ve wondered if I’ve been too selfish, too naive in attempting to build something new on my own, rather than build off my existing success and luck, and playing it safe with a full time job.

I don’t know the answer yet, but I am grateful for the chance to find out.

In the last post, I introduced how I took my idea for a self-driving skatepark, and turned it into an invitation to submit an SBIR Phase I proposal.

In this post, I’ll go through the process of preparing and submitting that Phase I proposal from scratch, as someone who had never written a grant proposal before.

If you’re not familiar, the SBIR program gives up to $2,000,000 (with zero dilution!) to small businesses based in the United States to develop and bring to market potentially breakthrough technology.

It starts with a Project Pitch, where someone from the NSF will take your idea, and see whether or not it will be a fit for their currently open programs. My initial proposal was a self-driving robot to protect cyclists, under the AI solicitation, and it was accepted.

When I received that acception email, I immediately became intimidated.

I assumed submitting a formal proposal would mean writing something very similar to an academic paper, something I’d never done before. (I didn’t attend university, instead opted to live in central america and farm for a few years.)

A Stroke of Luck, Meeting an Expert

In a stroke of luck, I met someone who had received two Phase I proposals a few weeks later. This person was an amazing help, and shared their full, successful proposals. This gave me a giant leg up, boosted my confidence, and important context for just how high the standards are for acceptance.

When I read his proposals, it became clear successful proposals do read like an academic papers.

But beyond being academic, these papers also had make a convincing argument for a potential technical breakthrough, show a sustainable business from said breakthrough, show a market need for the technology, and build a realistic budget and team to execute the vision.

This is a lot to take on!

My industry experience has taught me that ideas and strategies must first be tested in the real world. So I set out to start building the physical pieces of what would eventually become my strategy with the SBIR application.

Researching the Problem Space

So I started with my initial goal: How do we protect cyclists?

As I dove in to the data around cyclist safety, I found that cyclists and pedestrians tend to get hit at night. Given that computer vision doesn’t work at night (low light sensors are expensive, and infrared models need high powered lights and to be custom trained), my original computer vision only approach wouldn’t work.

Instead, I’d need a sensor that could augment the computer vision pipeline I’d originally applied with in order to truly be effective.

It turns out there’s a sensor that works in the dark, and is capable of augmenting computer vision.

mmWave radar allows for accurate tracking of objects that are occluded, and allows for more accurate speed estimation.

Using a computer vision pipeline augmented with mmWave radar would allow for an extremely resilient approach to tracking objects around a cyclist.

Luckily, the investment in self-driving vehicles has created a market for low-cost mmWave sensors. This means we can take off the shelf sensors meant for tracking vehicles while driving, and repurpose them for bicycle protection.

Applying the Problem to the State of the Art in Artificial Intelligence

Of all the industries I know of with rapid change, artificial intelligence in late 2023 and early 2024 have to be in the lead.

Over the course of the month’s research, state of the art architectures for route planning and object tracking moved from transformers, to diffusion, back to diffusion transformers. I continuously updated what my technical approach would be, given the improvements I read from the papers. I tried out a set of models that would work together to target my approach.

As I worked through it, a part of my solution would involve training a custom model. This meant I had to have a realistic approach for how I’d gather my data, train my model, and test that my model would indeed perform well in the expected environment.

Bootstrapping A Data Pipeline

How do we bootstrap a training set to build a model (or ensemble of models) to protect cyclists?

I decided that I’d need to collect data from a fleet of cyclists in the real world. It seemed a bicycle mounted device would be best way to collect a dataset.

So I built a prototype using an NVIDIA Jetson Orin Nano, a depth camera, a DeWalt 20v battery, and a mmWave radar. I built a pipeline for recording bicyclists, and scoped out a computer vision pipeline for live tracking of hazards.

As I was writing this up for my proposal, I ran into a cyclist who had an interesting device attached to his bicycle. I asked him about it, and it was a radar device from Garmin.

It seemed my idea for mmWave radar was on the right path.

Doing the Paperwork

But as the technical details were progressing, the necessary paperwork was also there, waiting.

The NSF has a few requirements for the company who applies for funding. In the extremely helpful webinars they give, they go over all of the needs for your company.

In particular, one of most frustrating parts of the paperwork process is getting a UEI from Sam.gov. You need to provide a set of documents that identify your company, and then follow up in the web application to ensure the UEI is actually created. I almost missed this, because it’s not clear in the UI which stage you’re in, and how you progress!

Additionally, you need to come up with a budget for your proposal. In the webinars (again, extremely helpful!), they emphasized that they’d help with creating a realistic proposal if they were interested in funding the approach.

Despite this, I took the time to try and find what I thought would be a realistic approach. Thinking through the process with the budget to accomplish my Phase I goals actually helped a lot with my overall strategy.

My Biggest Challenge, Industry Support

A key part of a successful Phase I proposal is having industry recommendations. They’re not required, but highly encouraged. This can take the form of a letter from another company saying there is a market for the product, should it come to market.

This is one area where I really struggled. I’d say I failed here if completely honest.

I started by emailing the person in charge of cyclist safety in Florida. It turns out there’s an elected official who’s entire job is to minimize cyclist deaths.

After five emails and five phone calls to his office and assistants, I got nothing back from this person.

Frustrated, I turned to my contact who’d already received two Phase I proposals. He said it would be hard to get a politician to say something on the record, and recommended I reach out to a local cycling shop.

The first cycling shop I went to, they immediately went into a diatribe about how cyclists getting hit was their own fault. When I presented my approach for reducing cyclist deaths, they simply said “we don’t do technology”.

I went to another bicycle shop, and this time they immediately “got” it. They’d ridden earlier in the day with their Garmin radar device, and knew about the cyclists who’d been hit in the past month.

It seemed after a frustrating few first interactions I’d finally found my letter of support.

This turned out to not be the case.

Despite meeting multiple times, they never followed up. As the deadline window was approaching, I submitted without their letter of recommendation.

In hindsight, I should have been more aggressive about this key component. I wanted to wait until I had a clear vision of what the device would look like, and by the time that vision was there, it was too late in the process to really have time to court a company.

If I could do it again, I’d start with a “less big” ask for a few companies at the beginning. I’d try to get them to publicly commit to public safety, and build a relationship from there. It seemed it was a bridge too far to cold ask for support for such an alien process.

Pressing the Submit Button

By this point, I had what I hoped was a great proposal that matched the style and rigor of the proposal that had been shared with me. I was proud, and confident it was as good as I’d be able to muster.

When I went to submit my proposal in the actual portal for submissions, it became apparent I’d written it in the wrong format. It turned out that the grants.gov site expects the proposal to be broken up into specific PDF pieces. I rewrote what I’d written again, and made it fit the format they’d expected.

But once I’d finished it up, I still had a few days remaining. I tried once more to get a letter from a business, but just didn’t get one in time.

I submitted my proposal despite this, we’ll see whether or not it matters.

Was it worth it?

The SBIR process requires a person be in a unique situation. They must have a small business, and be willing to commit to working on the proposed project at least 50% of the time for the next 6 months.

They also need a market opportunity, and a team that seems capable of both building a breakthrough technology and bringing it to market.

It’s also a non-zero amount of work to put together this proposal. Given the proposal must be ambitious, it can feel as though you’re not being realistic, and that the proposal itself is mostly a pie in the sky idea.

I certainly struggled with that feeling through the process, and realistically, there was a non-zero cost to attempting the proposal.

But the process of taking an initially scary idea to a full proposal was an amazing learning experience. It helped me form a holistic approach to thinking about a disruptive robot technology, and what market conditions must be in place for it to succeed. It also forced me to put myself out there, despite the lack of buy in from the industry.

I hope my story inspires you to pursue ideas that scare you, no matter the outcome.

If you’re interested in doing a proposal too and have questions, don’t hesitate to reach out. Please reach out via Twitter.

In the meantime, I’ll share my updates as my proposal gets reviewed.

Last year a question fell into my brain that I haven’t been able to get rid of since:

What if we’d invested in safety instead of autonomy?

According to McKinsey, more than $100 billion has been spent so far on attempting to build fully autonomous vehicles.

What if we’d invested that money on vehicle safety instead? What if rather than autonomy, we attempted to get the risk of death for drivers, cyclists, and pedestrians to zero? What if we attempted to do it in a way that still allowed for the feeling of 99% autonomy for drivers?

I became obsessed with this idea last year. Could a technical solution exist to eliminate these deaths? Would it be closer, or further away than autonomous driving?

I decided to go looking.

How are people dying from vehicles?

Safety advances have mostly benefitted people riding in modern, luxury vehicles.

If you are lucky enough to drive a BMW X3 4WD, Nissan Pathfinder 2WD, or a Lexus ES 350, the NHTSA has a driver death rate of 0 for you.

Conversely, if you’re driving a Ram 3500 Crew Cab long bed 4WD, you are more than 5x more likely to kill someone else with your car, rather than yourself!

It seems the vehicles most likely to kill you while behind the wheel are big pickup trucks, muscle cars with RWD, or small cars.

Sidnote, as a sports car owner: Two wheel drive cars with high horsepower can lose control if you step on the pedal hard. The high horsepower muscle cars with AWD aren’t nearly as deadly for drivers.

But most alarmingly in all this data, between 2010 and 2021, bicycle fatalities in traffic increased by 58%.

Pedestrians made up 17% (!) of all traffic fatalities in 2021, with 77% of those deaths occurring in the dark.

Clearly we need to do a better job protecting cyclists and pedestrians from cars, and reverse the trend of deaths.

What are we doing to prevent deaths?

Thankfully, new cars and trucks are getting a layer of protection for pedestrians and cyclists.

It’s not yet standard across all vehicles, but Pedestrian Autonomous Emergency Braking allows for automatic braking when a collision is detected as iminent with a pedestrian. There’s even a whole set of tests for cars, included in the PDF linked above.

Unfortunately, the average age of vehicles on the road is 12.2 years, so even if we did add the technology to all vehicles tomorrow, we’re still facing at another 12 years to catch up.

Another possibility is building roads to give a larger physical buffer of protection to cyclists.

But building safer roads is prohibitively expensive in the United States. There are some programs looking to bring the cost down, but they cost an average of $133k per mile, and take two years to build if rushed.

What can we scale quickly to prevent these deaths?

Given the costs and time frames of the existing approaches to safety, we aren’t realistically going to reverse the trends of death.

What if we had a low-cost, widely distributable way to protect cyclists and pedestrians?

How High Risk, High Impact Technology Gets Funding in the United States

Last year I found out about America’s Seed Fund.

The SBIR program seeks to fund research and development of high risk, deep tech companies in the United States. Given I’d been interested in cyclist and pedestrian protection, I applied on a whim.

I created a proposal for a self-driving robot platform to protect cyclists and pedestrians.

I imagined a self-driving robot which followed a target person, creating a physical buffer between them and vehicles on the road. The idea scared me, because it’s a very ambitious goal, and not something straightforward to implement.

Ignoring that fear, I submitted my pitch.

Getting into Just Enough Trouble

Of course, two weeks later I was formally invited to submit a Phase I proposal.

I made up my mind to pursue this idea futher. I needed to think through what an actual approach to eliminating cyclist and pedestrian deaths would look like, and how I’d get it done.

With that, I jumped in to the problem. At first, I didn’t really believe it was possible. But over time, I started to see a path to making the first steps of the project work.

Jumping In to the Deep End

It turns out there’s a lot of data about how and where vehicles are crashing in the United States. Every crash is logged into a database with the NHTSA, among other places. A great paper showed how we could supplement the existing database with auxilary data to build an even richer dataset for training.

Imagine if you could get an alert that you were about to ride into a high risk zone on your bicycle. How might your change your cycling approach?

We already have weather applications to predict the weather, why not have cycling predictions too?

I started building a platform to ingest mmWave radar, depth, and camera data from a cyclist. I used the latest NVIDIA Jetson Orin Nano as my platform, with a DeWalt 20v battery as my power supply.

I built a data ingestion pipeline using rerun, and got a computer vision model up and running with live feedback over a bicycle network. You can read more about some of the process at my Github repo.

But people in my city kept getting hit by vehicles in broad daylight.

The Stakes Get Higher

Since I started work on this project, there’s been a steady stream of pedestrians hit by vehicles in my town. In one day, a child was hit by a school bus and two people were critically injured.

Frustrated, I reached out to the person responsible for cyclist safety in Florida for help with my proposal. I figured nobody would be more interested in helping me out than the person elected to eliminate cyclist deaths.

In my research, I discovered Florida has the highest rate of cyclist fatalities in the entire United States, despite a goal of zero incidents. Given the rapid population growth of the state, it appears to only be set to get worse.

After four emails without any response and three phone calls to the office, I gave up. I wasn’t going to get a response from my elected official who was responsible for safety, and so I needed to find help elsewhere.

I ended up finding my first help where I least expected it.

Continued in Part 2 where I go in to detail about how to submit a Phase I SBIR proposal.

I’d also love to hear from you if you have any ideas or want to help. Please reach out via Twitter.

Six months ago, I started working on a Generative AI video editor. I began with the assumption that the newest machine learning models would unlock a new sort of software that wasn’t previously possible.

Of course, I didn’t know what that new software would look like, so I decided to start building with the smallest idea I had. Since then, the number of models applicable to video, audio, and text has exploded. It seems every week there is a new, more efficient model I should rewrite my application for.

Despite 15 years of experience developing backend applications, the past 6 months of working in the AI space has shown me that it’s fundamentally different from the rise of cloud and mobile. There are still very many open questions about what the next generation of applications built using these models will look like, and how they’ll be deployed.

But, I want to share some of the things that have surprised me so far about building AI applications.

Morality is a Top Business and Product Concern When Building AI Products

Given I’m working on a video editor, there are many moral and ethical questions which need to be addressed when I’m choosing what to build.

Models can now clone voices, replace faces, and move bodies in whatever way the user wants. Giving users the tools to do this on their own is not just a question of morality and consent, but also of long term business feasibility.

What happens if a lawsuit blames you for abuse done by users? It’s easy enough to do simplistic blocks for nudity, but what else? And beyond the legal reprocussions, can you live with the potential mob of people who distrust the technology, and believe it’s enabling a loss of personal consent?

There’s no better example for the pontential long term consequences of Generative AI than what’s been happening with the cryptocurrency space. Companies previously operated on fuzzy moral and legal grounds for a long time. The legal system eventually caught up with the ecosystem, and the things which were assumed legal because of a lack of action, weren’t. I expect we’ll see the same with AI products eventually.

The Models Don’t Actually Matter, Because Better Ones Will be Here Next Week

This is a wild insight, because six months ago I would have laughed at the possibility. But the models don’t really matter much in practice.

Building any model is a race to the bottom and a Red Queen’s Race. At this point a lot of incredibly intelligent people are building models with billions of dollars worth of compute and resources. These mega models will be made largely without input from the many GPU poors, and will largely be heavily censored, opaque black boxes for end users.

What remains for the rest of us is to use an ever improving ensemble of smaller models, and to build tools and interfaces atop of them.

And building these interfaces is really where the value lies. Open models will continue to improve, and the state of the art will continue to be pushed. The models themselves must be interchangable, as a better model will come about sooner than you think.

But these models as they exist are currently unpredictable and difficult to understand. Bringing understanding, or at least predictability to models shows potential as a moat. (But of course, if AGI comes, there will be no moats, anywhere, unless you have a source of energy, compute, and water larger than your competitors. Oh and your model is better at stealing their model’s weights and…)

The Immediate Future is Going to Be Weird

At this point, the internet as a source of training data is being filled with generated text and images, and the noise from this model generated output will only continue to grow.

We can assume eventually the output will become so good humans won’t be able to tell the difference from human generated content and not.

What does this mean for us as humans, if the place where we currently have most of our social interactions occur is no longer primarily human content?

Depending on your viewpoint, we may already have the answer. The algorithms for social media platforms are already incredibly sticky and good at capturing our attention.

What if a combination of the algorithms and Generative AI builds the perfect skinner box? Then the algorithms will compete to see who can give the best, most tailored emotional experience currently desired by the user. We then have a carefully orchestrated algorithm to subtly shape the behavior of humanity.

Open Source is in a Fundamentally Weaker Position

When I started out building software, Open Source gave me the tools that I couldn’t otherwise afford as a young person. A compiler was hundreds of dollars, but a Linux CD gave me access to all the tools I’d need to start building, right away.

Fast forward the decade or two I’ve been in software development, and Open Source powered the cloud. Trillions of dollars in economic value were generated off the back of Open Source contributions.

But for machine learning, there are two fundamental constraints which require access to capital to scale. Building a large model can cost millions of dollars at the low end, and grow from there for a state of the art model. Building a home scale computer for training smaller models can cost thousands, especially if you need multiple high end GPUs.

Additionally, the datasets to train on are large! In the past six months I’ve been working, I’ve hit bandwidth caps for my ISP multiple times, just using models. If I were to take in training data from the internet, things would look even worse.

Given the high costs of participation, the barrier to entry for Open Source is higher than it’s ever been. There are only so many deep learning labs capable of working with these large models. This means Open Source models will have less eyes and be smaller, until the hardware becomes cheaper, the data is more evenly distributed, or development is subsidized, either by venture capital or governments.

Incumbents Have Strong Advantages

One of the conventional narratives about startups is that they are small and nimble, and can build things faster than big companies.

But with Generative AI, this just isn’t true.

Building a better model currently requires access to capital and data. Large companies have both.

Building a good end user experience means having tooling around your models for easy exploration and better human understanding of the model’s behavior. Again, incumbent companies already have interfaces built up over years, which can be used to augment data before and after inference.

But here there is a real weakness in how the best of developers have been treated over the past two years.

With layoffs making the rounds, companies have parted ways with some of the best, most talented developers in their rosters. Without them to navigate the boundary between the existing product and new possibilites created by these models, large companies will lose, despite these inherent advantages.

We Don’t Know Where the Moat will Come From

If you look at AI right now, there appears to be relatively few moats.

NVIDIA, of course, seems to have the biggest one. They’ve built the GPUs, but more importantly, also the libraries and software to support researchers and builders. And they’ve been building the infrastructure for them for over a decade.

No other company was making as deep of an investment in the tooling to build accelerated computing, with as consistent of a vision.

Since then, of course, there is the story of OpenAI, who built a model that brought them a billion dollars in revenue in a year.

But how defensible is OpenAI’s moat? Open Source models are catching up, and at OpenAI’s last demo day they showcased products like LaundryBuddy, a far cry from the next step after GPT-4 to AGI.

The truth is, we don’t know where the moat will come from with Generative AI. In the meantime, the pickaxe and shovel companies will do well. Platforms like Modal and Replicate will make ML tooling approachable for developers, and we’ll soon see what the Uber of machine learning looks like.

Robotics Are Probably the Next Moat

Building and testing robots is expensive, as the real world is much more difficult than software to model. A basic robot for automation can start at $20k+, and the development iteration loop can be extremely slow, when you factor in having to test each software change in the real world, and hardware which can break unexpectedly.

To address this, NVIDIA has been building and pushing its next generation platform, Omniverse.

Omniverse is a platform to model and simulate environments. As an example, you could use digital twins, to recreate and test your drone’s performance in a high resolution scan of Seattle.

Using ray-tracing and digital envionments, you can model, test, and more importantly generate realistic training data for your robots virtually, allowing you to run tens of thousands of simulated tests in a digital environment.

Between this and the growth of model capabilities, a sharp team who can navigate the boundaries of physical, cloud, and models should be able to build an iPhone like technical coordination moat. It remains to be seen whether this is a bipedal robot, or something else.

Nobody Can Keep Up with Progress

Even the most intelligent and deliberate of my peers can’t seem to keep up with the speed of advancements in the space. It seems every week we get a new breakthrough, one which may have applications, or contribute to a breakthrough in the current problem we’re solving.

Because of this, it’s easy to develop an underlying unease about our chosen problem spaces. Is it a dead end? Is there somewhere else that might have better results? Is there a completely different architecture I should be chasing?

Being in technology, there has always been an unease about the pace of learning the latest technology. But in the AI space, this feels faster than anything I’ve ever experienced. How you manage to stay focused, while not getting locked into dead ends is a core part of navigating the space effectively.

There Are More Vibes than Hard Data at the Edges

How do you measure the performance of a Large Language Model?

More importantly, how do you measure it against another language model?

Right now there are tradeoffs across the available models, and there are tools to try them all out using the same prompt, to see the difference in results.

But largely, these opinions on the “correctness” of an output for the highest performing models is mostly a gut opinion. And over time, people have opinions that they’ve changed for the worse, while black box model providers insist nothing has changed. (There are, of course, formal tests of a model’s capabilities, but most experts agree these are flawed.)

Because of the relative gap between a “correct” answer, and the one a person personally deems correct, there won’t really be an absolute measure of what the “correct” answer is. For instance, if a street level drug dealer asked your language model questions about strategies for growing their market presence, what response should be deemed “correct”?

Secrecy and Security Matter in Ways They Don’t Normally

What do you think the market value for the raw weights of GPT-4 is?

If someone leaked them as a Torrent (like LLama), how soon would it be before it was optimized to run on consumer hardware?

The current moat of language model companies revolves around the premise of their weights never being leaked. This means they have to trust their cloud providers, their employees, and the security of their systems to protect each layer of their infrastructure, and the entirety of their business.

I’m certain intelligence agencies from all over the world are interested in the applications these of advanced language models and their employees. I also don’t expect these companies to be defending their technology on their own.

Between this and hundred million dollar plus training runs, the upper echelons of machine learning are a bit scary! Throw into the mix people who are convinced the training runs are potentially catastrophic for humanity’s future, and there’s surely to be a bit of intrigue in these companies.

Finding Your Position in the Coming AI Landscape

I was recently interviewed on a podcast, and asked about the future of machine learning.

At the time, I felt uncertain about offering any sort of advice. After six months of working in the space, I didn’t really have any purely optimistic, encouraging advice for people entering the space. There are genuine traps here for builders, and incumbents really do have non-trivial edges in the space, in ways they didn’t for cloud or mobile.

Despite this, I still want to build, and encourage others to do the same. When software began taking over the world, it had the potential to alienate people who didn’t understand how it was built, and thus couldn’t model how it misbehaved.

But AI threatens to do the same to everyone else. Except for a few thousand engineers and researchers, the rest of humanity will be captive to the decisions made about what to prioritize, censor, and mark as the correct answer for these giant models. That’s too important of a collective decision to be left to so few.

Although Open Source and the GPU Poors may not have the same advantages, I believe we must try.

all the artwork in this post by magritte

The current iteration of of Generative AI doesn’t feel built for the benefit of artists.

Instead, the focus seems to be on maximizing shareholder value. Training the models used in Generative AI costs large amounts of capital, data, electricity, and water.

The models powering this generation are trained on a giant corpus of art, but with very few exceptions, aren’t compensating the source artists at all.

But it doesn’t have to be this way. Generative AI can be used instead to empower artists and make art more valuable.

Artists and their Relationship with Disruptive Technology

When streaming rose to prominence, the creatives who had created shows and music were mostly left out of streaming deals. Dave Chappelle famously had his show streamed on Netflix without ever seeing compensation, and fought hard to convince executives to take down the show, on the principle of fairness. Taylor Swift also had to negotiate hard to establish fair royalties, pulling her catalog off Spotify completely until Spotify agreed to restructure rates paid to her.

Given this context, generative tools for creatives have to thread a very thin needle. If they are to be embraced by artists they need to show that they can be used to enhance the value of art, and grow the market for independent artists.

Capital tends to push artists where ever possible. Any new technology will inevitably be used to try to put even more financial pressure on artists (who mostly lack capital to defend themselves legally), and maximize larger returns to existing pools of capital.

Storytelling, music, and the visual arts enrich our collective human experience. Generative AI designed poorly has the potential to muddy our main collective commons (the internet) with bland secondary generated content, created en masse for pennies.

Giving Artists a Fighting Chance with Generative AI

Given such a daunting set of challenges coming for artists, where do you begin? If Generative AI is going to disrupt creative processes, how do we ensure artists get a seat at the table?

I don’t have any answers.

But, I’m willing to explore.

I’ve found that when I want to learn about something, it helps to just start builiding a thing. And when it comes to building something new, it’s best to start building with the tiniest possible idea.

Rather than building a giant machine to empower artists, what if we used the existing models to enable new methods of creativity?

My experience is mostly with computer vision, so I started there. I’ve always been a fan of artists like cyriak, and would love to build a tool to make the world a bit more cyriakish.

As a start, I saw a style I was impressed by:

Facebook’s Segment Anything model would be a great tool for making this sort of effect easier.

The original artist made in After Effects, with a lot of patience and manual masking of himself. Of course, Clay is an incredibly talented dancer, and made his own story line to fit the effect too. But(!) the manipulation of the outlines of people and masking is a chore.

A tool to help you creatively explore the segments of your video with better masking would be enough of a start.

So with that, I was off.. (ish)

Building an Open Source AI Video Editor in… Python?

Given the progress of machine learning models, it seemed the shortest path to building a piece of software capable of helping an artist make a video like Clay’s was to try to make as much of it in Python. Python is a lot of things, but I’ve mostly used it for backend infrastructure development, not desktop applications.

So I started by looking for a toolkit to build a video editor in Python, to see whether or not it would even be possible.

It turns out, ModernGL along with ModernGL-Window make for a great way to get an OpenGL interface across Mac, Windows, and Linux. NVIDIA also has a Python library for hardware decoding and encoding of videos when using its video cards. Given the two, it seemed like enough to get started.

Of course, I’d need some way to interact with my videos I’m editing. For that, I used PyImgui to build a user interface on top of my OpenGL window.

Building a Prototype Interface with Imgui

With moderngl-window, you’re given a few functions to define and build your render loop:

import moderngl_window
from moderngl_window.text.bitmapped import TextWriter2D


class App(moderngl_window.WindowConfig):
    title = "Text"
    aspect_ratio = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.writer = TextWriter2D()
        self.writer.text = "Hello ModernGL!"

    def render(self, time, frame_time):
        self.writer.draw((240, 380), size=120)


App.run()

In your __init__ you can set the resolution of your window, along with any other configurations you may need.

Moderngl-window also comes with an imgui integration. Adding it is easy enough:

import moderngl_window
import imgui
from moderngl_window.integrations.imgui import ModernglWindowRenderer
from moderngl_window.text.bitmapped import TextWriter2D


class App(moderngl_window.WindowConfig):
    title = "Text"
    aspect_ratio = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        imgui.create_context()
        self.imgui = ModernglWindowRenderer(self.wnd)
        self.writer = TextWriter2D()
        self.writer.text = "Hello ModernGL!"

    def render(self, time, frame_time):
        self.writer.draw((240, 380), size=120)
        imgui.new_frame()
        imgui.begin("Custom window")
        imgui.text("hello world")
        imgui.end()
        imgui.render()
        self.imgui.render(imgui.get_draw_data())

    def resize(self, width: int, height: int):
        self.imgui.resize(width, height)

    def key_event(self, key, action, modifiers):
        self.imgui.key_event(key, action, modifiers)

    def mouse_position_event(self, x, y, dx, dy):
        self.imgui.mouse_position_event(x, y, dx, dy)

    def mouse_drag_event(self, x, y, dx, dy):
        self.imgui.mouse_drag_event(x, y, dx, dy)

    def mouse_scroll_event(self, x_offset, y_offset):
        self.imgui.mouse_scroll_event(x_offset, y_offset)

    def mouse_press_event(self, x, y, button):
        self.imgui.mouse_press_event(x, y, button)

    def mouse_release_event(self, x: int, y: int, button: int):
        self.imgui.mouse_release_event(x, y, button)

    def unicode_char_entered(self, char):
        self.imgui.unicode_char_entered(char)

App.run()

We’ve added a lot of code, but it’s mostly just passing events through to our imgui instance, so when we click in places imgui knows when and where we clicked.

Imgui uses the immediate mode, allowing you to define your UI directly within a loop, which makes for a bit quicker iteration speed when building a prototype. With it you can build and test new ideas very rapidly.

Loading and Scrubbing Video in OpenGL

Of course, any creative tool is really only useful if it feels real time.

To do this effectively, I used the hardware accelerated decoder built into most recent NVIDIA graphics cards.

This will allow us to use a dedicated chip on the GPU to do the decoding of each video frame, allowing us to seek and play with less latency, leading to better, lower latency playback, usually without having to do conversions of loaded files.

On NVIDIA hardware, you can just use the great VideoProcessingFramework. It even comes with example code, showcasing how to decode a video from the input colorspace to either a Pytorch tensor or an OpenGL texture:

# to seek to frame
src_surface = nvDec.DecodeSingleSurface(nvc.SeekContext(frame_no))

# convert to rgb color space using pipeline
rgb_pln = to_rgb.run(src_surface)

# convert to pytorch tensor
src_tensor = surface_to_tensor(rgb_pln)

# push to CPU, as a numpy array
b = src_tensor.cpu().numpy()

This gives us something we can then manipulate in Pillow or PyTorch / Numpy while prototyping, and save out as frames.

With this, we can then hook up Segment Anything, and use imgui to pick our points within the OpenGL frame. But before we can do that, ModernGL-Window supports MacOS. But MacOS doesn’t have any NVDIA hardware at all.

Building Hardware Video Decoding and Stable Diffusion XL in MacOS

Despite looking for a while, I couldn’t seem to be able to find a way to do hardware decoding on MacOS, despite there being hardware support for it implemented on the M1 and M2 series processors.

So I decided to try and write a Python hardware playback library for MacOS. Apple has extensions written in Objective-C to do hardware decoding, but unfortunately hasn’t released a Python interface too these APIs.

In order to do this, I used Cython, and took from OpenFramework’s implementation of their video player. This allowed me to convert the Objective-C code into C++, and then into a Python API that could play back the video.

After a lot of trial and error, I eventually had a library I could import and run on my MacOS machine, with a reasonable API that mostly matched NVIDIA’s VideoProcessingFramework:

import videoplayback

player = videoplayback.AVFPlayer()
player.load("filename")

# unfortunately haven't made loading a file synchronous yet
time.sleep(.3)

numFrames = player.length_in_frames()
w = player.width()
h = player.height()

destination_frame = 1
# get a frame 
player.seek(destination_frame)
image = np.asarray(player.imageframe())
image = image.view(np.uint8).reshape(image.shape + (-1,))
# shape (1080, 1920, 4)
image = image[:,:,0:3]
b[:,:,[0,2]] = b[:,:,[2,0]]
# shape (1080, 1920, 3)
image = image.copy(order='C')

As for Stable Diffusion XL, Apple has released a repository with some optimizations, making it possible to generate images using hardware acceleration on M1 and M2 hardware.

Unfortunately, these optimizations bring the time to generate a single image to around 2 minutes on my M1 Macbook Pro with 64GB of memory. Not a great feedback loop for creatives. For reference, generating a Stable Diffusion XL image on my desktop computer with a 4090 takes a few seconds.

However, using a service like Modal, I was able to get inference down to a few seconds by using a serverless GPU instance.

Getting Early Feedback

When I showed a version of the editor prototype to a friend, he was excited about the ability to segment objects out of a video, while moving.

So I built out the tools to be able to do this, using the De-AOT model. This allows for predicting up to around the next 20 frames of a video at a time, and mostly works as well as you could ask.

As I built this out, I started seeing some potential uses of Stable Diffusion that could be a tool to empower creatives, rather than imitating their work whole cloth.

Building a Model Pipeline for Asset Generation

See normally, Diffusion models aren’t built to live within an existing frame or context. Instead, they’re built to create an image whole cloth from a token prompt, imagining from noise what a picture may look like, iteratively.

In order to generate a transparent asset for a video (like a UFO or an arrow, or…), we’d need to be able to isolate and segment what we want out of an image, hopefully automatically.

With this, we can now see and generate assets:

def generate_segmented_diffusion(object, prompt, negative_prompt="", auto=True, seed=None):
    if seed is not None:
        generator = torch.Generator(device="cuda").manual_seed(seed) 
        pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
            "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True,
            cache_dir="/app/.cache", generator=generator
            ).to("cuda")
    else:
        pipeline_text2image = AutoPipelineForText2Image.from_pretrained(
            "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True,
            cache_dir="/app/.cache"
            ).to("cuda")

    img = pipeline_text2image(prompt=prompt,negative_prompt=negative_prompt).images[0]

    if auto: # just take first mask from SAM
        model = get_dino_model()
        img_ground = transform_pil_image_for_grounding(img)
        b, p = get_grounding_output(model=model, image=img_ground, caption=object, box_threshold=.35, text_threshold=.25)
        boxes = b * 1024 # 1024 x 1024 for sdxl images
        boxes = box_convert(boxes=boxes, in_fmt="cxcywh", out_fmt="xyxy").numpy()
        predictor.set_image(np.asarray(img))
        masks, _, _ = predictor.predict(box=boxes[0]) # just take first for auto
        masks = np.where(masks, 255, 0)
        masks = masks.copy(order='C')
        img_cutout = Image.new('RGBA', (1024, 1024), color=(0,0,0,0))
        mask = Image.fromarray(masks[0].astype('uint8'))
        img_cutout.paste(img, (0,0), mask=mask)
        img_cutout = img_cutout.transpose(Image.Transpose.FLIP_TOP_BOTTOM)
        return img_cutout
    else: # return image with masks
        model = get_dino_model()
        img_ground = transform_pil_image_for_grounding(img)
        b, p = get_grounding_output(model=model, image=img_ground, caption=object, box_threshold=.35, text_threshold=.25)
        boxes = b * 1024 # 1024 x 1024 for sdxl images
        boxes = box_convert(boxes=boxes, in_fmt="cxcywh", out_fmt="xyxy").numpy()
        predictor.set_image(np.asarray(img))
        masks, _, _ = predictor.predict(box=boxes[0]) # just take first for auto
        masks = np.where(masks, 255, 0)
        masks = masks.copy(order='C')
        return img, masks

Rather than creating a final image whole cloth, we can now explore our existing artistic vision with this. (Sidenote, I mentioned at the beginning skepticism with Stable Diffusion and its training data set. I think (I might be wrong!) that incorporating feedback with it like is a better approach than just generating things whole cloth. I’m still not sure here.)

Using a ControlNet to Collaborate with your Image

ControlNets are a way to steer and direct the diffusion process as it occurs. You can train a model to steer output using your own input, so there’s a bit more control of how your diffusion model generates.

For example, you can use a model called OpenPose to show how the people in your images should be posed. You can use ControlNet to show exactly how many people should be in your image, and how they should oriented.

Alternatively, if you’ve got an object you’d like to transform into a diffusion, you can use something like Canny Edge detection to get an outline of your object as part of the steering of the generation.

We can use these models to change assets within our video, while retaining their original proportions. In the video above I’ve turned an orange barrier into a concrete barrier.

But if you look at it closely, there’s still a few issues. Ideally I’d have a virtual 3D asset I could move with the camera’s movement.

Using a Diffusion Network for Video with TokenFlow

But using a ControlNet on it’s own doesn’t work very well for video. If you look at the video on the right, you’ll see you get consistently flickering frames, as the diffusion model guesses a different outcome based upon the input noise. Even if you keep the same initial seed image consistent across frames, the way Diffusion models work means you’ll get flicker.

There are techniques to improve the flicker, a commonly used technique is AnimateDiff, which adds a temporal layer in the middle of the Diffuser. But these techniques aren’t perfect.

The best implementation for temporal stability I’ve seen so far is TokenFlow, which uses a different method called Plug and Play to steer the generation of an image while retaining the spatial meaning. You can see a snipped of the above video with a prompt of “a man surfing a wave”.

Plug and Play uses an initial image which is inverted to noise via DDIM, and then run through the diffusion process, allowing the features relevant to generating the image to be extracted. These features are then injected in the self-attention layers of the diffusion model, while using the same inverted noise of the original image.

By combining this technique with frame to frame correlation, you can then have a temporally consistent video, relatively free of flicker, as shown above. But still, both of these techniques appear very artificial, and have the stigma associated with “AI video”, that sort of artificially generated artifacts.

Building a Model Exporter for ebsynth

Ebsynth is a non-deep learning method for replacing the textures of your video with another art style. It works by taking an input series of video frames, along with some keyframes that have been painted to match the style you’d like your video to be replaced by.

We can use ControlNet along with Stable Diffusion to take our video, and explore different text based ideas for textures.

In the example here, I used “red hot lava volcano fire flames” as my prompt, as I wanted to create a glowing effect to myself.

For the ControlNet, I detect whether or not there are people present in the image, and if so, add a weighted ControlNet for OpenPose, in addition to the Canny Image ControlNet.

Given all the other tools we’ve already built, adding ebsynth generation is straightforward enough. We need to select a subset of frames, and then run Stable Diffusion on them. This allows us to ensure they all come from the same pathway generated by the diffusion model. I’ve also added a mask mode to the video, allowing us to isolate a specific person for texture creation, versus applying it to the whole video.

Again, this allows for us to generate special effects, ensuring we have a single effect applied to a single subject in our video.

With this, we can put in our generated versions of our characters back in to our videos.

Building AI for Artists

After having spent a few months using computer vision models and generative AI to build an editor, what have I learned?

Video is inherently multi-modal. You have a sequence of images, audio, and a narrative. Some machine learning models are now capable of working with multi-modal inputs. But the day to day work of video creation doesn’t fit well into a single model just yet. The most interesting results I’ve gotten have come from a mixture of traditional video editing, along with an orchestration of models.

But these tools don’t replace creativity! Creativity is mostly just continuing to show up for work, every day. Some days you show up and the results seem to come easily, and others it feels impossible to do anything.

We’re still so early in the process of Generative AI, that I still can’t really tell what the future will look like. I can see the tools starting to take shape, but it’s not clear yet what will win, or how things will work.

I have hope that we’ll be able to beat the massive pools of capital seeking to replace creatives, and that the creatives will win.

If you want to follow along as I keep exploring, I encourage you to share this article, and sign up below for early access to the video editor.

I’d also love to hear from you if you have any ideas, please reach out via Twitter.

In late 2022, there was a ton of Azure capacity issues reported in the media. At the time, the lack of capacity was blamed on “supply chain” problems.

But a quarter or two later, OpenAI released GPT-4, which showcased new capabilities previously unseen in Large Language Models (LLMs), and apparently required a massive amount of computation from Azure to train.

The novelty and usefulness of GPT-4 were apparent immediately. An incredible early paper from Microsoft showcased how GPT-4 could be the beginning of Artificial General Intelligence, a multi-modal, general purpose reasoning machine with a mostly human level understanding of the world.

GPT-4 was so disruptive, in fact, that OpenAI is now on track to generate $1 billion in annual revenue from its ChatGPT product less than a year after the launch.

Other companies are now waking up.

The High Costs of Doing Machine Learning

If you’re not actively following the AI space, you may have missed NVIDIA’s Q2 2024 results. They were unbelievably good, given the size and scale of the company. Revenue was up 141% from the previous quarter, and up 171% from a year ago.

It seems every large company is now scrambling to catch up in the AI game, and in the process are spending outlandish sums of money to build datacenters filled with the latest NVIDIA GPUs necessary for training.

To be clear, each of NVIDIA’s latest GPUs (the H100) cost around $34,000, but generally is used in machines designed for clusters of 8 in a single machine. These machines cost between $300k-400k each.

For reference, a single training run for a 70B parameter language model (LLama2 in this example) uses 1,720,320 GPU hours worth of compute at 400W of energy usage each.

(In this case, LLama2 used A100s, the prior generation to train. But the hours / investment are comparable for training a given model.)

This would take 10,240 GPUs, running 24/7 (at a cost of around $350 mil in GPUs alone!) to train the model in a week.

If we wanted to avoid CapEx and instead use AWS, a p4d.24xlarge costs $32.77 per hour, and comes with 8 A100 GPUs each. That’s an affordable $7 million to train one model run, ignoring the costs of data transfer, debugging, and setting up of data pipelines. (If there are any available).

Given the extremely high costs associated with developing these state of the art models, SemiAnalysis has coined the term “GPU rich” vs “GPU poor”. Companies which have invested heavily in GPU infrastructure prior to the GPT-4 explosion are considered GPU rich, and capable of building state of the art models, while everyone else is considered GPU poor and scrambling to catch up, but currently locked out of building these models.

This is a wild experience, because it’s the first time in my life it’s been impossible to develop a kind of software without access to a giant pool of capital and proprietary data.

The thing that made software interesting to me as a young person was the lack of costs and low barrier to entry. Anyone anywhere in the world could build software and contribute to the global conversation of software.

A compiler to build programs was free, and an operating system to run them was free, thanks to GNU and Linux. Everyone working in or contributing to open source could take and give freely to the overal value available in Open Source. This open ecosystem led to the amazing growth of the cloud and software in general.

Trillions of dollars in tech company valuations and returns were only possible because of the open source ecosystem.

AI Success Is a Perfect Storm for Capital Concentration and Increased Inequality

Given the massive costs associated with building or renting a GPU cluster, the potential moat available to companies with access to the giant pools of capital necessary for GPU clusters seems unparalleled.

And indeed, this is why we’re seeing startups that are pre-product raising hundreds of millions of dollars, again because the cost of entry into the state of the art machine learning model business is literally hundreds of millions in compute costs.

For someone used to the previous, cloud based and open source software paradigm, there couldn’t be a wider difference.

Suddenly the costs of building a product aren’t concentrated in the humans necessary to build the software itself, but rather in the raw costs of materials, energy, and data necessary to participate.

Which sets artificial intelligence up for a socially dangerous feedback loop.

There will only be so many superclusters in the world capable of training these giant models, and there will be only a relative few engineers who are using and gaining experience with these superclusters.

The talent pool will concentrate and shrink, and the pressure to deliver will grow on the people running and training these models.

As models like GPT-4 show the potential to automate and destroy the vast majority of the very best paying of jobs for workers available, (that is, knowledge work in general), we’re looking at a future where existing capital structures have the potential to become permanently self-perpetrating, largely off training data off the back of the public’s work (which comprises the majority of training data used on the GPUs).

As AI’s capabilities grow, a substanial amount of previously well employed knowledge workers may actually be permanently captured by these models, trained on the data taken from them, without compensating them for their efforts.

The Open Source Machine Learning Hero is… Meta?!

Given the costs associated with training these models, the moat for OpenAI’s GPT-4 appeared to be enormous and durable. But then Meta (prev Facebook) released Llama.

Llama was originally supposed to be a large language model released just to researchers, on a case by case basis. Inevitably though, the model leaked, and showed up on Torrent websites. Soon enough, a model that cost millions to create was in the hands of everyone who wanted to try it out, and knew how to use a torrent link.

This led to a rapid explosion in the public progress around large language models. Researchers fine tuned the leaked model on GPT-4 output and made it public. Shortly afterwards a reasonably useful model was made to run on consumer level hardware, thanks to projects like llama.cpp.

With that rapid public progress, Meta has continued to release powerful models, focused on computer vision, language, and more.

Better still, they’ve released some of their models in a way that allows for commercial use. And most recently, they released a completely free large language model, Llama 2.

The moat inherent in machine learning’s costs is becoming less apparent. Especially when you consider a trained model could always leak.

The Moat for AI is Now Secrecy and Paranoia

In an interview the CEO of Anthropic, Dario Amodei spoke a bit about his approach to cyber security with training his models.

He mentioned that some of the edge for his company relies on what he calls “compute multipliers”. The core idea is that they have discovered training optimizations that are the equivalent of having more compute.

At his organization, he’s implemented a compartamentalization strategy, similar to a spy agency, where no one person could leak the overall approaches necessary to train the Antropic Claude models.

What’s interesting is that in the same interview, he tends to beat around the bush of how much the business model of AI relies on the model never being leaked or hacked. There is an acknowledgement that governments will be able to hack and steal any model they like, given the model in question shows high enough value. But there isn’t an equivalent belief that someone will leak the same.

Similarly, for now there is a relatively minor moat in larger language models. It may require multiple, $34,000 GPUs to do execution after training. But again, the open source work of projects like llama.cpp, huggingface, and others is chipping away at the moat.

The Vibes of Training AI Models are Kinda Wack

So let’s recap the dynamics of privately funded AI:

• Extremely high costs
• Secrecy concerns / Model Leaks
• Defense / Spy considerations
• Alignment risks
• Source training data liabilities

All of this is happening behind closed doors, with giant pools of capital building super machines, with the usage of the humanity’s collective knowledge and knowledge labor as input.

To capital, AI seems to be the ultimate sort of privatization of public knowledge, along with the ability to steer the source of what will certainly become “the reference” for an “unbiased” source of information for a substantial portion of humanity.

Control of the knowledge base from which humanity has its reference is of course, slightly interesting to intelligence agencies and governments.

Given the vibes and given the costs, why even try to participate? Why not leave the AI game to the “grown ups”, and let them tell you the future?

Open Source And the Battle Against Techno-Disillusionment

I’ve been lucky enough over the past decade’s worth of technological improvements to have had the tiniest seat at the table. I’ve been able to steer at least a part of the conversation for what the future of technology and software will look like.

My peers outside of tech, however, have mostly not had a voice. For an increasingly online and software driven world, most people living within it don’t have a say for how it should work.

Instead, it’s mostly been venture funded organizations seeking massive growth who have built the digital worlds everyone inhabits.

This mostly worked out fine for the past 10 years because VC was mostly in the business of giving money away. Apps like Uber were famously money losing, scaling in an effort to build out market dominance. In the meantime, humanity got a sweet discount on rides.

But this new generation of technological improvments threatens to turn the disillusionment dials up to 11. Imagine only a few thousand people steering all of the most important software used by the rest of the world. Imagine the sorts of power games and pressure with such a limited number of people working in the space. Imagine the immense pressure to return capital on investment for billions.

And add the lack of accountability inherent in a machine learning model that is incomprehensible to humans.

The stakes are generally too high for all of us to have artificial intelligence not be developed in the open.

The GPU Poors are our best hope.

Let’s build a better future together, in the open.

Ever since I first played Paperboy as a kid, I’ve wanted a way to transform the streets of suburbia into something more exciting. My current neighborhood has a sidewalk along the entire street, and I often take my dog on runs, where he pulls me on my skateboard.

When we ride together, I’ve often wished there were ramps sprinkled at houses along the way, like in Paperboy. On one of these runs, the thought occurred to me that I could possibly build a skate ramp on a motorized platform to ride with us. This idea grew, and eventually I came up with the goal to make an entire self-driving skatepark that could journey with you to your destination, ramps taking turns stopping and getting back in front of you along the way.

In today’s blog post we’ll step through the process so far, from idea to a few prototypes, and how I’ve built a platform for remote controlled skate ramps. At the end there will be a Github repository if you want to help out, or try building your own.

With that, let’s get started!

Architecture of a Self-Driving Skate Ramp

When I first got the idea for a self-driving skate ramp, I immediately thought of a golf cart being repurposed into a ramp on wheels. There was something very appealing about the absurdity of a gigantic skate ramp driving down the street.

But as I looked for prior art, I quickly realized that the linear actuator and mechanics of the steering system seemed like a bit too much of an investment to get to a prototype to test an idea. So instead, I focused on a mix between an electric skateboard platform and the NVIDIA JetRacer.

The NVIDIA JetRacer is a Jetson Nano powered self-driving RC car. It runs on ROS, and follows a basic track around a loop, using computer vision. It seemed promising as prior art.

Really, the JetRacer was a backup to prove that if I could get the hardware down, the software should be implementable, given the performance of JetRacers. (It also helped that I have two Jetson Nanos laying around, with the shortage of chips everywhere.)

With that decision, it was time to figure out how to get started. So I went on eBay and bought a broken electric skateboard to begin my journey.

Controlling Electric Skateboard Motors in Software

I spent some time last year building FPV drones, but when I began this project didn’t have any familiarity with electric skateboards. Luckily, it seems there’s an open source movement for their motor control systems, similar to the one found on hobbyist drones.

Electric skateboards mostly use BDLC motors to provide an enormous amount of power in combination with LiPo batteries. The trio of motor controller software, motor, and LiPo batteries is largely what powers the eBike, eScooter, eSkate, and drone markets. So the architecture across platforms is awfully similar.

Getting the motors controlled via USB in Python was straightforward enough. I hooked up my benchtop power supply, strapped down my skateboard trucks, and was off to the races with the pyVESC library:

import pyvesc

serial_port = '/dev/ttyACM1'
front_motor = pyvesc.VESC(serial_port=serial_port)

while True:
  front_motor.set_rpm(3000)
  sleep(10)
  front_motor.set_rpm(0)

My original plan for steering the platform was to use two different motors, mounted on opposite sides of my skateboard. Depending on which direction I needed to turn, I’d spin them at different speeds.

This really didn’t work well in practice at all. When I built a first version, I had problems with the inconsistency of grip with my wheels, couldn’t use it to reliably steer at all. Instead, my ramp kind of hopped and spun out in either direction. That wouldn’t work for driving in and out of sidewalks.

Another problem I had was with the synchronization of my motors. When I first tried out my platform, I hooked both motors up via USB. This led to a bit of a delay in activation for the motors, and a slight difference in speeds. To fix this, I wired the controllers in a master / slave configuration with CANBUS.

This proved to be a bit tricky to do over USB, as it wasn’t very well documented, and was completely unsupported by PyVesc. So I ended up searching a bit, and eventually found the command to send a CANBUS slave message over USB (and control the second controller via a first controller):

class SetRPM(metaclass=pyvesc.VESCMessage):
    """
    Sets the RPM on a CANBUS connected device. 
    Messages have to be sent repeatedly, as CANBUS has a timeout that's configurable
    within the VESC application. In my case, I set it to 5 seconds, which means the
    motor only works for 5 seconds after sending this signal.
    """
    id = 34
    fields = [
        ('motor_id', 'B'), # my slave is set to 1
        ('command', 'B'), # 8, thanks to this page: https://www.vesc-project.com/node/774
        ('rpm', 'i') # because we're assuming RPM setting
    ]

Because of the requirements for rider safety, you can’t just send a single command via CANBUS on the VESC platform. Instead, you need to continuously send the message.

In order to do this, I had to make some updates to the way my Python program ran. I added a thread to send a heartbeat of the current message:

def send_target_message():
    myData = threading.local()
    myData.drive_speed = 0
    while True:
        try:
            val = queue.get(block=False)
            myData.drive_speed = val
        except Exception as e:
            time.sleep(.001)
        front_motor.write(encode(SetRPM(1, 8, myData.drive_speed)))
        time.sleep(.001)
        front_motor.set_rpm(myData.drive_speed)

controller = RampController(interface="/dev/input/js0", connecting_using_ds4drv=False)
t = threading.Thread(target=send_target_message)
t.start()
controller.listen()

With this, I’m able to send messages to both controllers via a single USB connection.

Learning from the First Iteration Prototype

Despite the obvious shortcomings, I took my ramp for a spin. In practice, it ended up being much scarier than I’d anticipated.

Normal skate ramps don’t move at all. In this case, when I hit the ramp, it moved on two axis. One towards me, and the other side to side. Between this and the lack of steering, it meant I didn’t really have a ramp capable of steering itself. At best, I’d have a ramp I could use to pull me to a place, and try to steer by leaning on.

My next version had to have some sort of way to put the ramp on the ground completely for when I skated it, and also be reasonably steerable. Both of these led me to my first metalworking projects.

Bringing Steering and Ramp Lifting / Lowering to the Platform

Given the failures, I didn’t want to give up the ecosystem of parts available with electric skateboards with the second prototype. So I settled on trying to attach a linear actuator to the trucks on the skateboard for steering, using a motor mount.

This worked surprisingly well, but built up some serious forces on my screws, and ripped them right out. So I realized I needed to make some bolts go through the plywood base, to keep the linear actuator from ripping the mounts right off.

This led me to the next problem, how to raise and lower the ramp itself. It seemed like overkill to get another linear actuator, and try to fabricate another mount. Instead, I went looking, and found an interesting electric jack on Amazon.

I ordered the jack, and decided to fabricate a metal mount between the skateboard and the ramp.

This was my first welding attempt ever, and also my first time using a metal brake. Both mostly went off without a hitch for the first round. YouTube helped with learning MIG welding, along with a low-cost Harbor Freight flux-core welder. (I’d always wanted to learn welding, but assumed it would be thousands of dollars to get started.)

Once the parts were fabricated, the only thing was left to build the software to control a linear actuator and an electric jack.

Luckily, both of these are basically the same, simple circuit. You put electricity in one way for up, down for the other. Two relays wired correctly are enough to control the linear actuator going out or in, and the jack going up or down. We can control the relays with a Teensy microcontroller and setting two pins either HIGH or LOW for each actuator:

#include <Packet.h>
#include <PacketCRC.h>
#include <SerialTransfer.h>

SerialTransfer steerTransfer;

const long interval = 1000;
unsigned long previousMillis = 0;
int ledState = LOW;

struct __attribute__((packed)) STRUCT {
  char d;
  int x;
} steerStruct;

void setup() {
  Serial.begin(115200);
  steerTransfer.begin(Serial);
  // left and right
  pinMode(A9, OUTPUT);
  pinMode(A8, OUTPUT);
  // up and down
  pinMode(A7, OUTPUT);
  pinMode(A6, OUTPUT);
  pinMode(13, OUTPUT);
  digitalWrite(A9, LOW);
  digitalWrite(A8, LOW);
  digitalWrite(A7, LOW);
  digitalWrite(A6, LOW);
}

void loop() {
 unsigned long currentMillis = millis();

 if (currentMillis - previousMillis >= interval) {
  previousMillis = currentMillis; 
  // blink if teensy has power
  if (ledState == LOW) {
    ledState = HIGH;
  } else {
    ledState = LOW;
  }
  digitalWrite(13, ledState);
 }
 
 if(steerTransfer.available()) {
  uint16_t recSize = 0;
  recSize = steerTransfer.rxObj(steerStruct, recSize);
  if (steerStruct.d == 'R') {
    if (steerStruct.x == 1) {
      digitalWrite(A9, HIGH);
    }
    else {
      digitalWrite(A9, LOW);
      digitalWrite(A8, LOW);
    }
  }
  else if (steerStruct.d == 'L') {
    if (steerStruct.x == 1) {
      digitalWrite(A8, HIGH);
    }
    else {
      digitalWrite(A9, LOW);
      digitalWrite(A8, LOW);
    }
  }
    else if (steerStruct.d == 'U') {
      if (steerStruct.x == 1) {
        digitalWrite(A6, HIGH);
      }
      else {
        digitalWrite(A7, LOW);
        digitalWrite(A6, LOW);
      }
    }
    else if (steerStruct.d == 'D') {
      if (steerStruct.x == 1) {
        digitalWrite(A7, HIGH);
      }
      else {
        digitalWrite(A7, LOW);
        digitalWrite(A6, LOW);
      }
    }
 }
}

If you read the code, you’ll notice I have commands that I’m sending to Arduino from Python, U or D for up or down, and L or R for left and right. To send these custom commands from Python, I used one of my new favorite libraries, SerialTransfer.

It allows you to specify a custom command to send, and it just works. Sending my message from Python is literally this easy:

from pySerialTransfer import pySerialTransfer

link = pySerialTransfer.SerialTransfer('/dev/ttyACM0')
link.open()

# define our data structure (direction and on or off)
class dataStruct(object):
    d = 'L'
    x = 1

# send the up command on 
testStruct = dataStruct
testStruct.d = 'U'
testStruct.x = 1
sendSize = 0
sendSize = link.tx_obj(testStruct.d, start_pos=sendSize)
sendSize = link.tx_obj(testStruct.x, start_pos=sendSize)
link.send(sendSize)

On the bench, this setup works great. The ramp platform is super responsive, and has about as great of performance as I could ask for. But in practice, once I throw a ramp on top of the platform, there’s still so much to do.

Debugging a Remote Control Ramp

Once a ramp is placed on the robot, the platform itself becomes largely inaccessible. This means debugging can be especially tricky, given the wireless setup. Sometimes there seems to be a delay in bluetooth controlling the robot itself. Again, of right now, the platform is controlled with a PS4 controller via a Bluetooth connection to the Jetson Nano.

This ended up being a challenge on my maiden voyage with the new platform. As I drove the ramp into place, I pressed the button on the PS4 controller to lower the ramp. Nothing happened, so I pressed it again. Still nothing. Curious, I started walking to the ramp, only to have it start getting lowered, and then continue lowering itself until the jack ripped the threadscrew out of itself entirely and broke.

Disaster! This meant I had to completely refabricate the jack and mount, which really is the most work of any component on the ramp. This time, I added limited switches to my Arduino code. (But if you look close, they still don’t work!)

Choosing a PS4 controller means I have a limited range for communication with the platform, especially when the ramp is on it. One of my next moves will be to put a real radio controller on it, along with a proper RC controller.

Initially, I had my Jetson connect via Wifi to my home network. I’ve since added Tailscale to the Jetson Nano, and made it a permanent machine. This allows me to remotely access it from any network. I then added a long lived ephemeral key to Gitpod as a secret key. With this, I’m able to remotely log in and debug from a Wifi hotspot on my iPhone. This means I can remotely debug, and run longer distances with my ramp, away from the home as long as there’s cellular reception. (And my cell phone is around.)

A Second, Successful Voyage

Finally, I rebuilt my ramp jack, and wired up limit switches to prevent disaster. And, last Friday I was finally able to raise, drive, lower, and skate my ramp! There’s still a lot to be done before we get to self-driving, but it hey, let’s celebrate this milestone.

As always, the repository for the project is on Github, and has some more technical details. You should be able to mostly run with the project, as the code itself is pretty straightforward for now, and launchable within Gitpod. (You obviously won’t be able to connect to my Tailscale network.)

Just in case, the steps to get it running are: connect power for the Jetson and linear actuators, connect the PS4 controller via Bluetooth, connect the 10s battery power to the ESCs, and then run the loop_with_canbus.py Python script. Then the pressing R1 on the PS4 disables the safety switch, and allows you to steer / control the platform.

If you’d like to be notified of the next blog post, enter your info below:

Guitarists have this idea: “tone” is in the fingertips.

The argument is, the distinctive sound of your favorite musician isn’t because of some expensive guitar, fancy pedals, or boutique amplifier.

Instead, their talent comes from years of practice, and no amount of expensive equipment is a shortcut to sounding great. So for new practitioners, it’s best to focus on your competency and technique, and forget about the tools.

Regardless of the tools you use, you’ll become great with practice, and that greatness will translate regardless of tools with mastery.

But Tools Can Improve the Quality of Practice

Believing this to be the case, I spent the first 7 years of playing electric guitar with a decent, but not great amplifier.

Why invest in an expensive amplifier and guitar when your skills don’t justify it?

But then the pandemic hit, and I got bored. Many of the “reasonable” guitars I wanted were sold out, and weren’t expected to be in stock any time soon. So I started looking at more expensive guitars and amplifiers, as they were the only ones left.

Out of desperation I bought a substantially more expensive setup than I ever dreamed of– an ESP E-II Horizon FR and a Mesa Boogie Mark V:35. When I plugged in and started playing, my mind was blown.

Why didn’t anyone tell me the difference was so big?!

I could now sound just like the musicians I grew up with, and my practice routine went from being a grind to a pleasurable experience. Thanks to my improved tools, I also started picking up on subtle sloppiness in my playing.

I literally bought myself a better practice routine, and a substantially better sound. Tone may be in the fingertips, but it helps to be able to hear what’s happening in those fingertips!

The Best Says He Can’t Work without His Tools

Given this personal discovery, I was again surprised while watching Tom Morello’s Masterclass on guitar playing. For those of you who aren’t familiar, Tom is widely considered to be one of the greatest guitar players of all time.

In his Masterclass session, Tom recounts a time when he and the rest of Rage Against the Machine went to South America, separated from their specific instruments, pedals, and amplifiers. They went to practice on rented instruments, and by his account, they sounded terrible.

If you’re one of the best guitarists in the world, and tools don’t matter, how could a guitar and amplifier make you sound terrible?

Choosing Tools as Creative Constraints

The answer in Tom’s case is that he has deliberately chosen his tools as his constraints. Tom’s sounds are unique to his tools, built through their limitations. Those constraints are what have determined his creative boundaries and possibilities.

Tom made a bold choice early on, deciding not to fiddle with settings on his amplifier, chasing the “perfect” tone.

Instead, he’s got a few specific guitar pedals, and importantly, a killswitch (which lets him mute with a press) built into his guitars that he’s decided as the constraints of what will shape his sound.

Early on, he decided to stop bouncing back and forth between tweaking and optimizing tools, and instead focused on what was possible using a deliberate, static set of tools. He dialed in a specific setting for his amplifier, and left it at that.

By minimizing his toolset, he determined the boundaries he’d explore within his creative world.

Tools Shape Creative Possibilities

I’ve since become increasingly convinced that a few deliberately selected tools as constraints, pushed to mastery, is what pushes people into creative breakthroughs and outlier performance.

And again, developers specifically have wildly different, deeply held beliefs about all of their tools.

These beliefs about tools are often in conflict with other, competent developers. A given tool is seen as either terrible or great, depending on the person.

With Python, I feel as though I’m supposed to explore my way to a solution. Rather than understand an entire problem space on a first go. I feel as though I can stumble along, and discover what my data structures should be as I write.

Python allows me to create a rough sketch for what my program should do, and iterate my way to an understanding of the solution.

But importantly, very little of the code I’m writing nowadays lives in any kind of production systems with tight requirements for performance or collaboration.

Someone who is working in a different problem space, with different constraints might find my tools quite terrible, or even impossible for the job.

Tool Switching Has a Deceptively High Cost (and we’re doing it all the time)

What has only recently become apparent to me is how often we developers are expected to substantially change our tools and processes. Tom Morello’s thousands of hours with his limited tool set empowered him to create his own sound of excellence for his chosen tools.

It seems every new engineering role comes with a new build process, release methods, and methods of working.

And those jobs we’re usually working within legacy systems, attempting to navigate our way around an old set of constraints, while introducing new constraints closer to the ways we think we’ll be most effective.

The way people are gated for these roles is usually via a coding challenge, in an interactive notebook, with minimal context. (We’re expected to solve a problem with an algorithm we’ve got memorized on the spot.)

Of course, the distance to the role itself has been a topic of debate, and the transferable nature of the work doubly so.

But what of the tools of the role? What about that specific knowledge?

The Musician and the Music

There is a difference between being able to play a piece of music, and being able to write one.

Some people achieve technical proficiency at playing music, without the context for writing their own. They master the restatement, but none of the creation.

What is the role we’re optimizing for? Is it obedience and adherence to the planning? Or is it something else?

Ideally, a person is capable of building the music to be played. Creating new music to be played, writing new things, is the role itself,. And the way you build up context for building new things is by building things.

And in an interview, we should be able to “jam” just like in a musical try out for a band. We’re seeing whether or not we can make music together.

We’re Imprecise When We Talk About and Evaluate Tools

In my current role, I often get the chance to see people experiment with new tech stacks.

And without fail, the people trying out new tech stacks underestimate the secondary knowledge costs associated with changing their stack. Beyond the programming language changes, the secondary knowledge of How the new language’s package manager works, which libraries are the best, what types of build and release process are most effective, and how to debug, all show up and end up destroying productivity.

And when we start paying attention to tools as constraints, it becomes apparent that each software development role we get into truly requires discovering its own@abut set of tools and approaches.

Your tools should narrow the fences of possibility, so you can focus on the immediate problem at hand.

Hypothetically, individual tools are fungible, and with enough talent and time, all of them are interchangeable. You could write great software in Python and Javascript and Rust and everything else. You could spend the time to relearn the intricacies of how Python’s pip, and Node’s NPM, and Rust’s Cargo all behave, how their debuggers work best for you workflow, and everything else.

But to truly break out creatively, you need to close doors aggressively. You need to cut until there is only the right amount of context, and nothing else.

At the very least, be deliberate about the tools you decide to pursue excellence in.

Modern management largely has its genesis in the work of Frederick Winslow Taylor, who created the idea of Scientific Management.

His main insight was that industrial labor wasn’t being nearly as rigorously designed as the work being done by manufacturing machines.

In his mind, all human labor should be deliberately measured, studied, and orchestrated by expert observers.

The goal of these observers should be to optimize the overall system of human labor for higher quality and volume output, with lower labor costs. These newly designed, higher output systems should then justify higher wages for fewer, vastly more effective laborers than non-designed labor. (He recommended shooting for 60% more pay for these deliberately orchestrated workers than the industry average).

His main tool for implementing these designs was accurate time study, watching and observing the most effective people as they went about their day. In his book Shop Management, he shows a case study giving a 70% increase in wages to the workers, and by a partial re-design of their work, having each worker moving 3.5x the regular worker’s tonnage of work through the factory.

So, if we look at modern management, the genesis of the work of the manager is the design of the work to be done. Management focused on the choreography of work, so work could more effectively be accomplished.

The Problem with Applying Scientific Management to Software: We Don’t Know how to Quantifiably Measure Output

But there is a major problem with this model in software development: we are not moving a measurable amount of coal from one part of the factory to another.

An experienced software developer anticipates the secondary effects of every change, and weighs tradeoffs between a multitude of approaches in solving problems. By continuously modeling software systems, the organization they work for, and customer behavior, the most effective developers readjust their approach to problems daily.

And that daily work tends to be unique and vague, with a large search space for solutions. This means it’s difficult to observe and optimize the system of software development itself. Despite years of effort, and a market value for individual software companies of over trillions of dollars, we’ve yet to formalize a way of measuring how effective a developer is or isn’t. So we can’t measure our way to higher team performance.

But if we can’t use an adaptation of scientific management for improving the effectiveness of software creation, what can we use? How do we optimize the system of work for something like high wages and low costs, when we can’t even determine whether the work is good or not, done reasonably quickly, or not?

It’s (Sometimes) Worse than Not Being Able to Measure Output

Of course, one of the other fundamental assumptions of scientific management is that minimizing costs and maximizing profits and wages is our goal. This assumes that we work in a business which needs profitability to survive, and that imperative must be held higher than any other value.

In practice, this isn’t the case for many venture funded technology companies.

Instead, depending on the maturity of the company, the main goal may be: capturing market share, gaining users, increasing revenue, or absurdly, building headcount. All of these approaches smell of unsustainability, but importantly, can become the actual goal of management. Absurdly, some of the people who make it into management at companies losing billions per year later position themselves as thought leaders for effective engineering management.

So if profitably sometimes doesn’t matter, and we can’t measure engineers, how does someone in management build a framework for effective engineering management? And what the hell is that management doing all day anyways?

Managers Should Gather and Push Context to the Edges

Because measurement for effective creative work is such an obvious, big problem, somebody has thought about it before. In 1967, Peter Drucker created the idea of “knowledge work”, and more importantly, the “knowledge worker” in The Effective Executive. In it, he redefined the role of the worker, and the supervisor’s relation to work.

Even in 1967, humans had begun working on technologies where it just wasn’t possible for the people directing and managing the work to have enough context to orchestrate it’s entire implementation. Looking ahead, the complexity inherent in modern engineering projects was only going to grow.

The solution, he decided, is that knowledge workers had to become their own managers (in the book referred to as executives), both designing the work to be done, while thinking through the tradeoffs and minding the overall business context.

Critically, he defined what the goal of effective knowledge work was in this world devoid of absolutes:

Knowledge work is not defined by quantity. Neither is defined by its costs. It is defined by its results.

Results. That’s an odd and fuzzy definition for an output for a manager to optimize for. But it seems closer to the reality both engineering managers and their subordinates find themselves in. Leadership expects management to deliver “results”, but the definition of “results” changes constantly.

It is then management’s job to ensure there is enough context for everyone on the team to be able to pursue the correct results. They do this by gathering, and then distributing context for the team. This ideally empowers the team to lead themselves, directing their efforts to the most effective outputs, given the current situation at hand. (For a good example of this in practice, check out Turn the Ship Around)

But how do managers identify the right “results”? How do they know what to build, given the multitude of options of things to build? And after things are built, how do we evaluate how effectively (or not!) those results were delivered?

Help the Team Identify and Pursue “Objectives and Key Results”

Depending on where you work, many tech organizations are steered by quarterly objectives. These are known as OKRs, and are a method for communicating what quantifiable results you are going to chase after, and achieve as a group for the next three months. They’re also famously a part of High Output Management, often referred to as one of the best modern engineering management books, written by the ex-CEO of Intel.

OKRs are meant to be ambitious, and by default are meant to have a significant portion of them fail every quarter. They are a medium for continuously identifying new results for your business unit to achieve. Part of leadership’s job is to evaluate these and ensure the new results being pursued will be effective for the current strategy of the company. But really, the process of thinking through what would be the most effective result for your group is the point of the exercise.

At the end of the quarter, results are “graded”, and the manager (and team) are evaluated on how well they accomplished the results they set out to do. There is an unwritten, somewhat arbitrary rule that an effective team should be making 70% of their objectives. (This hypothetically helps ensure the team is being ambitious enough in their goal setting.)

So we’ve seen the role of a software engineering manager is to build context, push context to the edges, and create objectives for the team. Finally, the team is kept in line by being held to account for what was promised in the agreed upon objectives.

But what about that failure rate for OKRs? In practice, since the graded OKRs are measurable (you hit a certain percentage of your goals), we’ve transformed the unmeasurable productivity of a developer into a measurable thing. You told us you’d accomplish something, and you either hit, or didn’t hit the things you’d promised. The grade you get from your OKRs gives a definite number for your team, in a space that is fundamentally not measurable.

Is this good? Does this lead to increased effectiveness?

But What’s Missing from the Objectives and Key Results Loop?

Taylor focused on more effective physical labor, with the goal of lower overall cost, whereas Drucker focused on more effective knowledge work, with the goal of more effective results.

But without an overall strategy, those results don’t matter.

For the overall organization, the user growth must lead to another round of investment, or the new features shipped must lead to better user churn, or the latest marketing copy must improve the cost of acquisition. So even with perfectly delivered results, there is no guarantee of effectiveness for the overall organization.

This brings up another few problems:

As an organization grows, some of the results we deliver might be the wrong ones for the effectiveness of the overall organization. They may also not get delivered, for reasons outside of our control. Maybe we needed to coordinate with another team who didn’t have bandwidth, or maybe we had a critical team member quit.

Worse still, if two teams must coordinate together for a goal, and each team targets an average of 70% of overall goal accomplishment, we should expect a 49% chance of accomplishment. Increase the number of teams who must coordinate, and you rapidly diminish the chances of the goal ever being hit. Obviously, as an organization grows, the number of teams coordinating tends to grow too.

Again, even if we do get past the coordination costs with the right goal, the business environment may change, rendering our results useless. Luckily, in these large enough organizations, our larger organization’s general effectiveness can help smooth over a missed quarter.

The Arena of Bullshit

We can do all the right things well, and still be wrong in creative work. The uncertainty of success is part of doing truly creative work.

There are, of course, ways to minimize the risk of doing the wrong things. But what happens when your team delivers the “wrong” results two quarters in a row? Three? Your focus becomes the narrative of the work, and why externalities are conspiring against you. Or about resources you need that you don’t have.

Because, as we’ve seen, there is no way of measuring creative output well and consistently. So creating narratives about why things work, or what went wrong, despite the obvious results in the short term is really the only thing that matters when everyone else is executing well, or if your organization has built itself a monopoly position. Or if you have access to a ton of cheap capital and losses don’t matter. Or…

Consider two teams, one responsible for delivering a critical component for the company. Midway through the quarter, it becomes apparent they need the support of an adjacent team, and none of the work was captured in that team’s OKRs.

Should that adjacent team help? Should they drop their committed results for the health of the overall organization? Or should they protect their department and keep the undelivered results contained in the other team? Should that other team have planned better?

The “Effective” Rent Seekers

My favorite book about the rationality of “ineffective” management is Barbarians at the Gate. It’s the story of how F. Ross Johnson became the CEO of RJ Reynolds through a merger with Nabisco, and then proceeded to massively enrich himself and his friends at the expense of the company itself.

The book goes through the details of how, after failing to deliver results to the stock price, he decided to do a leveraged buyout to take RJ Reynolds private.

His original deal would net him over $100 million for the transaction. (Potentially worth over a billion if revenue targets were hit in the next few years.) By the end of the book, it’s apparent he attempted to extract as much as possible for himself, his friends, and the bankers involved in the deal. His flagrant greed led to the board working on other deals to sell the company, and eventually, nearly doubling his original offer for a purchase price of the company. After the board’s efforts, Johnson walked away with a mere $23 million instead of his potential billion.

For those of us who expect to contribute a net positive with our labor, the story of Barbarians at the Gate flips the model on its head.

Seeing how pure self interest to the detriment of the whole can work in practice was an eye opener for me. How do you effectively defend against hiring these kinds of people?

Larger organizations tend to accumulate these types of people on a smaller scale. Rather than focus on delivering results (which again is difficult(!), might be wrong, and often not easily identified by leadership), these people spend time making alliances and crafting narratives about outcomes instead of delivering. Title acquisition becomes the goal, seeking to maximize the expected organizational rewards, at the expense of direct results, all while working to minimize accountability.

Importantly for all managers, this game of bullshit management is a game to be played and defended against, regardless of your overall strategy and values.

You may genuinely care about delivering results, and they may be the right ones, but without a compelling narrative and web of support, your influence and overall effectiveness are stripped from you. Eventually, without your fellow teams continuing to deliver results, the larger organization suffers, and worst case, eventually dies.

So, Is Engineering Management Bullshit?

We’ve seen that portions of engineering management’s responsibilities are unquestionably bullshit. In organizations of a large enough size, the “ineffective” management persona occupies a certain percentage of the organization. Their efforts and jockeying for political capital and stolen results inflict an increasing cost from the “effective” managers. Operating to mitigate their impact becomes operational overhead. Without leadership holding management accountable, the problem only grows.

So if you’ve wondered why so much of management seems to be occupied by sociopaths, it’s because of this very pattern. The minority of people who deliver results are drowned out by the people focusing on controlling the narratives. And for those who stick around, the operational overhead of the ineffective management only grows.

But there is a flip side. If we allow the sociopaths to capture all the organizational authority and power, there’s nobody else left. We’re stuck in dysfunctional organizations everywhere, with no ability to effect change, other than dropping out of society.

Increasingly, the organizations themselves become rent seeking, rather than value producing.

Keeping Bad Leadership at Bay

It feels as though the costs of bullshit leadership continue to extract growing damages from society, while dodging any real consequences for said “bad” behavior. So how do we minimize their impact, and effect change here in a way that improves the world we live in?

As consumers, we can’t see the costs of good or bad management in the price of our products. We can’t look at two different boxes of cereal and decide to pay an extra dollar for the one with management that doesn’t blame and inflict psychological violence on their employees.

As someone who cares (you do, right?!), it’s your job to run towards accountability and responsibility. It’s also your duty to be aware of the methods and strategies of sociopaths, along with their strategies for influence and blame.

Minimizing your personal costs while maximizing your personal effectiveness allows you optionality. Being the person who does the work this time, gives you the ability to accomplish it better next time. Eventually, you end up a key person, capable of choosing where to invest your specific talents.

Be deliberate about these investments, and they will have a very real impact.

So what is Engineering Managment’s Job?

We’ve touched upon a vague set of goals, but haven’t really settled on what an effective manager’s daily job really is.

Going back to Drucker, a manager’s job is to prepare people to perform, and to give them freedom to do so.

Creative people need psychological safety in order to perform. An effective manager provides this, along with coaching to help reports become more effective versions of themselves. This should then allow them to reap long term rewards.

An effective manager builds trust and space for their reports to execute, along with guidance for where their efforts will have the largest impact, helping them to continue growing in their career.

An effective manager cares for their team, their work, and its effect beyond the immediate organization. They are stewards first, realizing that much is beyond their control.

The ones who care are just trying to keep the bullshit at bay.