A mage on the Hero’s Journey: a fantasy epic on how a startup rose from the ashes

May 31, 2023 · 15 minute read

Tommy Dang

Engineering

TLDR

Mage

pivoted from an AI platform to an

open-source data pipeline tool

and is making a huge impact on the lives of data engineers around the world.

Outline

  1. Prologue

  2. Adventure, trials, and tribulations

  3. Death, rebirth, and transformation

  4. Battle, freedom, and victory

  5. Epilogue

Prologue

Once upon a time, there lived a promising young mage that left the magic academy early, journeying off on her own to make a difference in the world.

The young mage journeying off on her own to make a difference in the world.

She believed that helping townsfolk harness the power of AI would create an economic boom across villages around the world.

Adventure, trials, and tribulations

The young mage created an AI tool to help developers at small companies build, train, and deploy AI models. Initially, the villagers were excited and had lots of interest in using the tool. Many of them lined up for days outside the village just to schedule demos and paid trials.

The young mage showing off the magical AI tool to the villagers.

However, when the day approached to implement the tool, villagers kept giving the young mage reasons for why they weren’t ready. Some of these reasons included:

  • “We need a data warehouse first.”

  • “We need to make our first data hire.”

  • “We need a data pipeline management tool first.”

  • “We need to organize and clean our data first.”

After many sleepless nights, the young mage sensed that an evil dark presence had secretly infiltrated the village. This dark force came to be known as the Harbinger of Unnecessary Tools.

The Harbinger of Unnecessary Tools secretly infiltrates villages and companies.

In that moment, the young mage realized that the AI tool was not a necessity; the villagers had more urgent problems that needed a remedy immediately. This realization was devastating because the AI tool had been worked on for over a year.

The young mage was defeated by the Harbinger of Unnecessary Tools. In pain and despair, she was driven out of the town and went into hiding; uncertain of her future.

The young mage defeated… but not giving up yet!

Here are a few lessons inscribed in her tome:

  • Pay attention to why users don’t give a resounding “yes”. Avoid “maybe” like the black plague. They can be a secret poison because it gives hope that it’ll eventually be a “yes” when in fact it’s a “no”.

  • Fail fast. Find ways to prove the product wrong as fast as possible. The quicker it fails, the more chances there are to try something different and succeed.

Death, rebirth, and transformation

The young mage began doubting herself, questioning whether she left the magic academy too early. Injured and depleted of mana (energy that powers magic), she began wandering aimlessly through the abyss of the multiverse. Along the way, she spoke with nearly a thousand data professionals and asked them this question: what was the most boring part of your work?

The young mage traveling through the multiverse.

Enlightened by the responses, the young mage meditated on the reasons why the villagers weren’t ready to implement the AI tool. After meditating and deciphering the arcane knowledge of responses gathered throughout the multiverse, the young mage had a revelation: companies need urgent help moving their data and preparing it for usage.

The young mage began rebuilding and leveling up her powers. She trained day and night for what seemed like an eternity. The young mage took some of the technology she used in the previous AI tool, infused it with power-ups, and open-sourced it. Legend has it that her reborn powers are known as the Data Pipeline Tool.

The young mage training, leveling up, and upgrading her magic attributes.

However, this was no ordinary tool; it has 3 major differentiators:

  1. It’s designed to have the easiest developer experience by providing a user interface; enabling developers to build data pipelines visually, quickly, and intuitively.

  2. It combines 3 use cases that have strong synergy and affinity for one another: batch processing pipelines, data integration pipelines, and streaming pipelines.

  3. Engineering best practices are built-in. The tool enables modular design of data pipelines; making each step in your pipeline easily reusable and simple to test with data validations.

At full power, the mage was ready to return and defend the village from the Harbinger of Unnecessary Tools!

Mage at full power.

Battle, freedom, and victory

The mage walked across astral planes and arrived at the village that was being oppressed by the Harbinger of Unnecessary Tools. She summoned all her powers and open-sourced the data pipeline tool.

The mage getting ready to battle the Harbinger of Unnecessary Tools.

After releasing the tool, spells of fire, water, wind, and lightning were cast at the dark force. With every passing moment, the open-source tool grew more powerful. As the battle raged on, bugs were eliminated, scalability issues were banished, powerful new features were added, and chromatic color began returning to the village.

Mage casting spells at the dark force.

After a fortnight of intense dueling, the Harbinger of Unnecessary Tools was finally defeated! The darkness of an unnecessary product, that had previously haunted the people, was lifted once and for all.

The Harbinger of Unnecessary Tools defeated.

Everyone was liberated from the dark force’s grip and joy overflowed in the streets. The entire village praised the mage for saving them from the pain of using data tools with a dreadful developer experience. Countless villagers expressed their gratitude and gave many thanks to the mage.

The darkness of dreadful developer experiences have finally lifted.

Here are some of the testimonies from developers in that village:

“I was awestruck when I used Mage for the 1st time. It’s super clean and user friendly.” —

Ajith Shetty, Data Engineer

“Recently tried Mage 🧙 and I must say I’m amazed by its developer centric usability.” —

Salman Ahmed, Data Engineer

“Mage is such a refreshing orchestrator compared to Airflow.” —

Anil Kulkarni, Senior Data Engineer

“All throughout this Slack space, you guys are quick, resourceful, and have an open mind. It really separates you from other orchestrators.” —

Greg Lenane, Senior Analytics Engineer

“It took minimal work to start understanding and building pipelines as opposed to Airflow.” —

Pedro Dellazzari, Data Scientist

“I would like to express my love for using Mage. My experience with it has been fantastic so far.” — Fabián Sepúlveda, Data Engineer

“I truly appreciate being part of this amazing community and am honored to have had the opportunity to contribute to its success.” —

Dhia Gharsallaoui, Data Architect

“Just wanted to say that I’m incredibly impressed with what Mage is capable of. It’s incredibly powerful and user friendly.” —

Matt Pegler, SVP of Innovation

“Hey! Transferring all our stuff to Mage from Airflow. We have around 80 pipelines running (and will be growing), managed by a team of 4.” —

Nazari Goudin, Head of Data

“Pushed existing dbt repo into Mage repo as sub module. Oh Man, Mage is 🔥” —

Vijayasarathy Muthu, Data Engineer

“To be honest, I am really loving Mage.” —

Alexander Bolaño, Senior Data Engineer

“Congrats on creating one helluva DX. Night and day for all other tools we’ve been testing.” — Tomas Roaldsnes

“I have never seen such a friendly place to ask questions, I love the openness of it!” —

Davis Vance, Data Engineer

“I finished the tutorial and my reaction was… DAMMMMM! It is a really nice platform! OMG” —

Paulo Mota, Data Engineer

“I love the tool and see potential for not only being a standard tool but also a standard user experience. This is an amazing product y’all.” —

Farman Pirzada, Senior Software Engineer

“Just deleted Cloud Composer yesterday and fully moved to Mage.” —

Le Minh Nguyen, Data Scientist

“I’ve been using Mage for 2 months now and I must say that I’m really impressed with the work that has been done so far.” —

Dion Salomon, Data Engineer

“It massively lowers the barrier for entry on data engineering, which has had to turn into its own specialized profession (Airflow isn’t exactly fun to debug).” —

Nicolas Essipova, CTO

“I think this is one of the most beautiful pieces of software I’ve ever used. There is powerful sorcery at work here.” —

Patrick Clark, Data Engineer

Since the release of the

open-source data pipeline tool

in June 2022, the project has received over 4.7 thousand stars on

GitHub

, over 2.2 thousand community members on

Slack

, and over a hundred teams using the tool in production.

The tool has traction in the village.

In return for defeating the Harbinger of Unnecessary Tools, the Magic Council of Venture Capital decided to award the mage with an additional investment of $5 million. This investment was led again by

Gradient Ventures

, included previous investors (

Essence VC

,

Designer Fund

,

Mana Ventures

), and added an amazing group of strategic angel investors:

Gold coins invested to continue powering up data engineers!

In addition to acquiring more gold, the mage gathered a powerful and wise group of advisors:

Powerful and wise Mage advisors.

Epilogue

Mage

is on a mission to make data engineering more accessible so that developers can harness the power of data to create magical experiences.

Mage making advanced technology more accessible to the world.

That’s why

Mage

is recruiting other fellow mages who want to go on an adventure and achieve this mission together. Here are the roles in the party that are currently open:

In the coming years,

Mage

will create a cooperative experience so that developers can build data pipelines with their team and level up together. After that journey, Mage will go on an epic quest to create the 1st open world community experience in the data universe.

Open World mode, unlocked.

Thank you so much for your relentless support of the mission and continued belief in the century long vision!

It’s never the end. Mage will be the 1st millenia long company.

Start building for free

No need for a credit card to get started.
Trying out Mage to build ranking models won’t cost a cent.

No need for a credit card to get started. Trying out Mage to build ranking models won’t cost a cent.

 2024 Mage Technologies, Inc.
 2024 Mage Technologies, Inc.