An Interview With Will Squires from Macrocosmos

“I think we’ve mastered a lot of the art of building robust, well designed incentives that scale well. Also, we’ve gone through more trial and error than anyone, and I say that with a positive attitude. We’ve been exploited left, right and centre, and know very well how to build systems that are defensible and give you what you need from your miners. It’s not exactly an intuitive skill that people can simply come to the ecosystem with. We see quite a few builders express their ideas using suboptimal incentive mechanism designs, whereas game theoretic AI is something of an obsession for us. We spend a lot of time discussing and debating the best ways to get the most out of our Subnets and we’ve learned to red team everything we build. We’re also very scientific in our approach; we run lots of experiments and have sophisticated monitoring, benchmarking and testing infrastructure. It’s a big part of our culture, which means when we say something, we stand behind it.”

“Our first year was about validating and prototyping core ideas across the AI stack, from data collection and indexing to pre- and post-training and agentic inference. In our journey, we learned where strategic advantages (cost, speed, scale) could be created by leveraging Bittensor. We’ve also learned to iterate through ideas quickly and discard the ones which don’t work, because this ecosystem moves lightning fast.

In the last 6 months we’ve really been trying to evolve from experimenting with what Bittensor CAN do (proof of concept) to what Bittensor SHOULD do (where its unique and powerful qualities represent real-world scaling potential and competitive advantage).

Prior to IOTA, we spent around a year refining a pretraining approach on SN9 which was based on model training competitions. We trained some respectable models, in fact some of the earliest decentralized 7B and 14B models ever made were ours. However, we were eventually confronted by a rather stark reality: our design had reached full maturity but fell short of our ambitious goals to compete with centralized labs. There was an economic barrier to entry and other inefficiencies which stood between us and our goal to train the first large decentralized models. In other words, the scaling laws that ground model training were not in our favour.

When we re-conceived IOTA, we tried to think two or three steps ahead. How could we create a truly collaborative system for training? How could we create a system that aligned its strengths with Bittensor’s (the ability to organise thousands of nodes and co-train a model) and gave us a path to overcome the scaling laws? Working on this has been challenging, but really rewarding. It feels like we’ve come back to our roots, armed to the teeth with more mature ideas and a much deeper understanding of what is technically possible. Our other Subnets (1 & 13) are now oriented towards supporting our pretraining efforts so there is much greater cohesion and focus in our work.

If we project IOTA forward, we will have a system that we could actually train a frontier model on, that supplies compute for the world’s most valuable task at a fraction of the cost.”

“Our current north star for IOTA is to make our system indistinguishable for training purposes from centralised alternatives. This means just as fast (hence our focus on throughput and algorithmic design), just as large (hence our push to scale network size from 256 nodes to much, much larger - more on that very soon), and we have a lot of research ongoing in numerical instability to resolve this.

Ultimately, we should just be able to monetise training FLOPS. This could be done by the network, by combining intelligence from model designers, data from other systems, and training compute from IOTA to create phenomenal models, or we could lease the network and swarm to other participants to train their own models. We are doing a lot of work on making the model weights inherently inconstructible to help unlock this, as privacy preserving is critical for enterprise customers. 

We want to be the first and only distributed compute layer for model training that is just as good, unlocking the path for organisations, whether decentralised or otherwise, to train brilliant models cheaply with distributed resources.”

“Firstly, none of these teams have a live mechanism, or a live token. Coming to Bittensor, you are forced to comprehend both from Day One, and we have more war wounds than anyone to help us learn from this. We view ourselves as experts in game theoretic AI, and these battle scars help us to build better, more defensible, and more performant systems. 

Secondarily, the miner community on Bittensor is second to none. Most competitor teams have effectively done permissioned, friends and family runs - this limits the available compute at max scale, and means it is in effect hand offs between friends. Without unlocking the permissionless scaling that Bittensor provides, you cannot achieve the upsides that make decentralised training economically competitive, which is a critical proof point in the whole thesis. Our vision is that anyone, anywhere, can train at any time. This means the system must be adversarially resistant – if you don’t solve this issue there will always be a ceiling and you’ll always be fighting gravity. 

Finally, working on Bittensor can feel like being part of a group of investors and other teams, something like a start up incubator. We get live feedback on our design, our systems, our metrics, we have advisors with deep experience in AI like Jacob Steeves supporting us, and we have fellow comrades in arms like Templar. We have our own bubble of talent to help us succeed. ”

“IOTA is currently working in production. We’re iterating relentlessly to drive the speed of the network up. We now believe we have the world’s best implementation of pipeline parallelism in speed terms (the crucial metric), we are the only team in the world that has built a ground-up orchestration layer and scheduling system for training multiple models at one point (all other teams build on Hivemind, an open source framework), and we have a huge release coming this month that we think will excite the community, and really change the perception of decentralised training in the market.”

“It’s a diverse group. We have professional Bittensor mining teams with relationships to the scale data centre providers funding the UK Sovereign AI Stargate programme, we have nameless crypto pirates in shades, we have AI engineers moonlighting in their spare time, and we have dedicated experts. Compute and talent come from all places, and it’s what makes IOTA so strong.”

“Current SOTA methods in decentralised pretraining use a methodology called Data Parallel, where each node hosts a full copy of the weights. This means that as model size scales, the requirement for any individual node scales to the point it becomes just as expensive as centralised training. Our research into compression allowed us to create a novel bottleneck architecture that allows us to “fracture” the model, splitting it up into blocks of much smaller size while reducing the amount of data that must be communicated between them by orders of magnitude. For participants, this means that the hardware requirements and economic barrier to entry is low and independent of model size. In other words, our system is able to perform global cost arbitrage in order to train large models cheaply. We also developed a suite of techniques which make the system adversarially robust, which is a key factor for hitting critical scale thresholds.

The only other team working in this space is Pluralis Research, who we have immense respect for, but we believe we have edges with our orchestration system and beyond, on top of building on Bittensor that means we will win.”

“You can’t look anywhere in the AI sector without looking at compute build out, the insane GPU deals, or the CAPEX associated. This represents two key theses we have - one is that compute will continue to be constrained throughout the 2020s, and the second is that the economic barrier to participate in the AI race will increase if CAPEX is necessary. Keith Rush and Arthur Douillard of DeepMind are working on distributed training as they believe that it will become a critical sustaining innovation to allow Google to keep training bigger and better models.

We view it as a disruptive innovation, that whilst many thought initially as impossible, it will come to be seen as inevitable. Analogous to cloud computing, we think in the 2030s, many, many more models will be trained in a fully distributed way.”