I want to raise a concern about the direction TIG’s benchmarking and algorithm-adoption system is heading in.
This is not intended as an attack on TigPool, and it isn’t a complaint that one pool has more compute than another. TigPool has built a large operation and there is nothing inherently wrong with being successful, submitting algorithms, or iterating on open-source work.
The concern is what happens when benchmarking influence, algorithm submission, delegation, adoption and rewards become increasingly concentrated inside the same economic operation — particularly when the submissions being rewarded sometimes add very little new value over work that already exists.
I think there are two separate problems here which reinforce each other:
1. Who has the power to determine adoption?
2. What level of actual innovation is that adoption rewarding?
Those questions matter because TIG is supposed to incentivise algorithmic progress.
Adoption is supposed to be a signal of quality
TIG describes benchmarking as a synthetic market for algorithms.
The intended process is essentially:
innovators create better algorithms
→ benchmarkers evaluate them
→ the best algorithms attract adoption
→ adoption rewards useful innovation
That system only works properly if adoption represents meaningful evidence that a submission is genuinely better.
But adoption in TIG isn’t simply one participant, one vote.
A benchmarker’s contribution to algorithm adoption is weighted by its influence and by how much of its qualifying work it directs towards that algorithm.
That means a sufficiently powerful benchmarker can have a very significant effect on which algorithms become adopted.
That alone isn’t necessarily a problem. Large benchmarkers should have influence if they are contributing useful benchmarking information.
The problem begins when the same economic operation is also supplying the algorithms being benchmarked.
TigPool has openly acknowledged maintaining separate innovator and pool benchmarking accounts.
There is nothing inherently improper about separate accounts.
But separate wallet addresses do not make the economic interests independent.
If an affiliated innovator submits an algorithm and an affiliated high-influence benchmarker then directs a large amount of qualifying work towards it, that benchmarker is helping create the very adoption signal which determines the value of the submission.
That creates a structural conflict:
the supplier of the algorithm is closely connected to one of the largest participants deciding whether that algorithm succeeds.
The second problem is the quality of what is being rewarded
Even this concentration would be less concerning if the submissions receiving that adoption consistently represented substantial progress.
But that is not always what we are seeing.
TIG deliberately allows people to copy existing algorithms and improve them.
I think that is a good thing.
Open development and iteration should accelerate progress.
But there is a major difference between:
a genuinely new breakthrough,
a meaningful derivative improvement,
and
a new submission that adds almost nothing to what already existed.
Those should not all carry the same economic meaning.
A submission should not be rewarded simply because it has a new name and a new submission ID.
It should be rewarded because it contributes new value.
A concrete example: there_v10 and autovector_g
I previously submitted there_v10 for Vector Search.
A later submission, autovector_g, is very clearly derived from the same underlying implementation, with the principal additional change being a caching optimisation.
There is nothing wrong with trying that optimisation.
The important question is whether it actually improved anything.
In this case, there is an important technical point:
both algorithms already operate at the exact argmin.
That means there is no remaining quality improvement available.
Once the exact nearest neighbour is returned, quality cannot be made better.
The only meaningful remaining optimisation target is runtime.
That makes the comparison very straightforward.
Across the five query sizes I tested, the timings were:
7,000 queries
autovector_g: 13.04s
there_v10: 12.59s
9,000 queries
autovector_g: 11.07s
there_v10: 11.66s
11,000 queries
autovector_g: 11.68s
there_v10: 11.09s
13,000 queries
autovector_g: 9.12s
there_v10: 9.12s
15,000 queries
autovector_g: 5.07s
there_v10: 5.07s
So the derivative is:
slower on one track,
slightly faster on another,
slower again on another,
and identical on the remaining two.
Across the complete set of reported runs:
there_v10: 49.53 seconds
autovector_g: 49.98 seconds
Effectively no improvement.
And in that particular challenge, runtime is the only place an improvement can exist.
So the relevant question is not whether someone was allowed to modify the code.
Of course they were.
The question is:
what marginal value did the new submission actually contribute?
If an already-exact algorithm is copied, modified, and resubmitted, but the only remaining optimisation objective shows no meaningful improvement, why should that new submission begin competing economically with the work it was derived from?
That is the distinction TIG needs to take seriously.
The same issue appears in another form
The recent sat_hybrid submission illustrates a different version of the same problem.
Its top-level logic largely acts as a dispatcher, selecting different existing track-specific SAT solvers depending on the instance.
Again, combining strong components can be perfectly sensible engineering.
But assembling already-discovered track solvers into a new package is not necessarily the same thing as discovering a new solving method.
There is a difference between:
creating the pieces
and
selecting the best existing pieces and packaging them together.
Both may have utility.
But they should not automatically represent the same amount of innovation.
This is not really about those two specific algorithms.
They are simply examples of a much wider question:
What exactly is TIG rewarding?
Consider two strategies.
Strategy A — genuine innovation
Spend weeks researching a challenge.
Try dozens of ideas that fail.
Develop a genuinely new approach.
Optimise it.
Benchmark it.
Submit it publicly.
Everyone can now inspect the breakthrough.
Then compare that with:
Strategy B — derivative capture
Wait for successful algorithms to become public.
Take the strongest pieces.
Modify or recombine them.
Add a small implementation change.
Submit another version.
Then use a vastly larger benchmarking operation to generate substantial adoption for that new version.
If Strategy B becomes economically more effective than Strategy A, TIG has a serious incentive problem.
Because the rational strategy becomes:
wait for somebody else to discover the breakthrough, then compete for ownership of the adoption.
That is almost the opposite of what TIG is supposed to encourage.
Concentration makes this much worse
If a small independent innovator submits a weak derivative, the market can simply ignore it.
Other benchmarkers can test it, decide it adds little or nothing, and continue running the original or superior algorithm.
That is how the adoption mechanism is supposed to protect the system.
But what happens when the innovator is affiliated with one of the largest benchmarkers?
Then the normal safeguard is weakened.
The derivative does not necessarily need to convince a broad independent market.
A significant part of its adoption can potentially be generated by the affiliated benchmarking operation itself.
Instead of:
“this derivative has to prove itself sufficiently good that independent benchmarkers choose it”
we risk moving towards:
“this derivative already has substantial adoption because the organisation behind it controls substantial benchmarking influence.”
Those are fundamentally different things.
This is where submission quality and benchmarking influence combine
A derivative that produces a genuine 20% improvement is valuable.
A derivative that materially reduces runtime is valuable.
A derivative that solves tracks its predecessor could not solve is valuable.
A genuinely better combination of methods can also be valuable.
Those things should be rewarded.
But a derivative of an already-exact algorithm that fails to materially improve the only remaining optimisation objective should not obtain the same economic significance simply because it can attract adoption.
Likewise, combining existing track winners into a wrapper may be useful engineering, but that is not automatically equivalent to creating the underlying breakthroughs.
The problem is not code reuse.
The problem is separating economic reward from marginal contribution.
Delegation adds another layer
TIG’s influence system also incorporates delegated deposits.
That creates the possibility of a reinforcing economic loop:
affiliated innovator submits code
→ affiliated benchmarker directs compute towards it
→ benchmarking influence helps create adoption
→ adoption generates innovator rewards
→ deposited TIG can be delegated back towards benchmarker influence
→ greater influence gives even more power over future adoption
Again, none of those individual actions necessarily break a rule.
The concern is the combined incentive.
A vertically integrated participant can potentially benefit at every stage:
innovation account
→ benchmarking account
→ adoption
→ rewards
→ delegation
→ greater influence
That is what I think TIG needs to examine.
Why this is harmful even if nobody is acting maliciously
I am deliberately not suggesting that TigPool has to be acting maliciously for this to be damaging.
In fact, that is the more important point.
Every individual decision can be rational.
Run the algorithms that maximise your return.
Submit changes where you find them.
Delegate where it benefits you.
Use your compute efficiently.
But if those individually rational decisions create a system where a dominant operation can repeatedly obtain adoption for affiliated submissions that contribute little new value, then the protocol incentives are producing the wrong outcome.
The damage is not just financial.
It changes what kind of behaviour TIG rewards.
Large compute should increase exploration, not reduce it
A large benchmarker’s compute could be one of TIG’s greatest assets.
It could be used to explore:
-
new algorithms;
-
under-tested submissions;
-
alternative parameter regions;
-
genuinely novel approaches;
-
competing implementations;
-
and promising work from smaller innovators.
That would make the entire network stronger.
But if an increasingly large share of that compute is concentrated on affiliated submissions, TIG loses something important:
independent exploration.
Independent algorithms receive fewer samples.
Potential breakthroughs receive less attention.
Smaller benchmarkers cannot realistically compensate for the imbalance.
And innovators cannot force the network to properly evaluate their work.
That means the problem can become worse than merely rewarding the wrong submission.
The network may fail to discover that a better approach exists at all.
This becomes an innovator-retention problem
Put yourself in the position of an independent developer.
You spend weeks producing a breakthrough.
You publish it.
A much larger economic operation can immediately inspect it.
A derivative appears.
The measured improvement is tiny, nonexistent, or limited to a minor implementation detail.
Yet the derivative receives substantial adoption because its affiliated benchmarker has enormous compute.
What incentive do you have to produce the next breakthrough?
Eventually the rational conclusion becomes:
the difficult part is creating the innovation, but the profitable part is controlling the adoption.
If enough developers reach that conclusion, TIG loses exactly the people it was designed to attract.
The network may still look active.
There may still be lots of submissions.
There may still be lots of benchmarking.
There may still be lots of adoption.
But:
submission count is not innovation.
adoption is not automatically proof of improvement.
And:
benchmarking volume is not the same thing as independent validation.
The irony around adoption rewards
This issue is made even more interesting by the previous discussion around rewarding hyperparameter adoption.
TigPool’s developer suggested an adoption-based reward mechanism, something else that would directly benefit them in the current state.
Daniel immediately raised the problem of somebody taking another participant’s configuration, changing it by a tiny amount, and potentially capturing the adoption reward.
The wider discussion also recognised that benchmarkers may follow a configuration simply because a large or successful benchmarker is already using it.
That is essentially the same problem.
Scale itself starts creating adoption.
Once that happens, adoption becomes reflexive:
the leader runs it
→ others assume it must be good
→ adoption increases
→ higher adoption makes it look even more validated
That is dangerous when adoption is merely a social signal.
It becomes much more important when adoption determines economic rewards.
I don’t think the answer is to attack TigPool
I don’t want TigPool prevented from submitting algorithms.
I don’t want their compute capped.
I don’t want reuse of open-source TIG algorithms prohibited.
And I don’t think somebody should be penalised merely for becoming successful.
The protocol should instead become better at distinguishing three things:
quality, provenance and independent validation.
For any derivative submission, we should be asking:
Quality:
What measurable improvement does this submission actually provide over the work it was derived from?
Provenance:
How much of the resulting performance comes from previously published work, and what did the new contributor actually add?
Independent validation:
Did unrelated benchmarkers independently adopt it because it was demonstrably better, or was a large proportion of its adoption generated by an affiliated benchmarker?
Those three questions together give a much better indication of genuine progress.
What I think TIG should consider
Derivative algorithms should be judged much more explicitly on their marginal contribution.
If somebody takes an existing algorithm and improves runtime by 20%, reward them.
If somebody introduces a genuinely new method, reward them.
If somebody combines previous methods in a way that demonstrably produces a substantial new performance gain, reward them.
But if a derivative of an already submitted algorithm fails to materially improve runtime or quality, what exactly is the network paying for?
That is not hostility towards iteration.
It is simply recognising that:
a new submission is not automatically new innovation.
The principle TIG needs to protect
Open-source iteration is one of TIG’s strengths.
Anyone should be able to build on anybody else’s work.
But the economic reward should correspond to what was actually added.
The original algorithm and a near-zero-impact derivative should not automatically compete on equal economic terms merely because both have different submission names.
And when the derivative is backed by a benchmarker powerful enough to materially generate its own adoption, the distortion becomes greater again.
That is why I think this issue is bigger than TigPool.
TigPool’s current position is exposing a weakness in the incentive design.
If my pool, or any other pool, eventually occupied the same position and behaved the same way, I would have exactly the same concern.
The objective should not be to make TigPool weaker.
It should be to make TIG’s market signal stronger.
TIG should reward genuine marginal innovation, not merely marginal submissions.
Adoption should validate improvement, not substitute for it.
And compute should be used to discover which innovation is best.
It should never become powerful enough to decide which innovation gets to count.