A concern about benchmarking concentration, submission quality, and whether TIG is rewarding the right behaviour

[Rootz]

I want to raise a concern about the direction TIG’s benchmarking and algorithm-adoption system is heading in.

This is not intended as an attack on TigPool, and it isn’t a complaint that one pool has more compute than another. TigPool has built a large operation and there is nothing inherently wrong with being successful, submitting algorithms, or iterating on open-source work.

The concern is what happens when benchmarking influence, algorithm submission, delegation, adoption and rewards become increasingly concentrated inside the same economic operation — particularly when the submissions being rewarded sometimes add very little new value over work that already exists.

I think there are two separate problems here which reinforce each other:

1. Who has the power to determine adoption?
2. What level of actual innovation is that adoption rewarding?

Those questions matter because TIG is supposed to incentivise algorithmic progress.

Adoption is supposed to be a signal of quality

TIG describes benchmarking as a synthetic market for algorithms.

The intended process is essentially:

innovators create better algorithms
→ benchmarkers evaluate them
→ the best algorithms attract adoption
→ adoption rewards useful innovation

That system only works properly if adoption represents meaningful evidence that a submission is genuinely better.

But adoption in TIG isn’t simply one participant, one vote.

A benchmarker’s contribution to algorithm adoption is weighted by its influence and by how much of its qualifying work it directs towards that algorithm.

That means a sufficiently powerful benchmarker can have a very significant effect on which algorithms become adopted.

That alone isn’t necessarily a problem. Large benchmarkers should have influence if they are contributing useful benchmarking information.

The problem begins when the same economic operation is also supplying the algorithms being benchmarked.

TigPool has openly acknowledged maintaining separate innovator and pool benchmarking accounts.

There is nothing inherently improper about separate accounts.

But separate wallet addresses do not make the economic interests independent.

If an affiliated innovator submits an algorithm and an affiliated high-influence benchmarker then directs a large amount of qualifying work towards it, that benchmarker is helping create the very adoption signal which determines the value of the submission.

That creates a structural conflict:

the supplier of the algorithm is closely connected to one of the largest participants deciding whether that algorithm succeeds.

The second problem is the quality of what is being rewarded

Even this concentration would be less concerning if the submissions receiving that adoption consistently represented substantial progress.

But that is not always what we are seeing.

TIG deliberately allows people to copy existing algorithms and improve them.

I think that is a good thing.

Open development and iteration should accelerate progress.

But there is a major difference between:

a genuinely new breakthrough,

a meaningful derivative improvement,

and

a new submission that adds almost nothing to what already existed.

Those should not all carry the same economic meaning.

A submission should not be rewarded simply because it has a new name and a new submission ID.

It should be rewarded because it contributes new value.

A concrete example: there_v10 and autovector_g

I previously submitted there_v10 for Vector Search.

A later submission, autovector_g, is very clearly derived from the same underlying implementation, with the principal additional change being a caching optimisation.

There is nothing wrong with trying that optimisation.

The important question is whether it actually improved anything.

In this case, there is an important technical point:

both algorithms already operate at the exact argmin.

That means there is no remaining quality improvement available.

Once the exact nearest neighbour is returned, quality cannot be made better.

The only meaningful remaining optimisation target is runtime.

That makes the comparison very straightforward.

Across the five query sizes I tested, the timings were:

7,000 queries
autovector_g: 13.04s
there_v10: 12.59s

9,000 queries
autovector_g: 11.07s
there_v10: 11.66s

11,000 queries
autovector_g: 11.68s
there_v10: 11.09s

13,000 queries
autovector_g: 9.12s
there_v10: 9.12s

15,000 queries
autovector_g: 5.07s
there_v10: 5.07s

So the derivative is:

slower on one track,
slightly faster on another,
slower again on another,
and identical on the remaining two.

Across the complete set of reported runs:

there_v10: 49.53 seconds
autovector_g: 49.98 seconds

Effectively no improvement.

And in that particular challenge, runtime is the only place an improvement can exist.

So the relevant question is not whether someone was allowed to modify the code.

Of course they were.

The question is:

what marginal value did the new submission actually contribute?

If an already-exact algorithm is copied, modified, and resubmitted, but the only remaining optimisation objective shows no meaningful improvement, why should that new submission begin competing economically with the work it was derived from?

That is the distinction TIG needs to take seriously.

The same issue appears in another form

The recent sat_hybrid submission illustrates a different version of the same problem.

Its top-level logic largely acts as a dispatcher, selecting different existing track-specific SAT solvers depending on the instance.

Again, combining strong components can be perfectly sensible engineering.

But assembling already-discovered track solvers into a new package is not necessarily the same thing as discovering a new solving method.

There is a difference between:

creating the pieces

and

selecting the best existing pieces and packaging them together.

Both may have utility.

But they should not automatically represent the same amount of innovation.

This is not really about those two specific algorithms.

They are simply examples of a much wider question:

What exactly is TIG rewarding?

Consider two strategies.

Strategy A — genuine innovation

Spend weeks researching a challenge.

Try dozens of ideas that fail.

Develop a genuinely new approach.

Optimise it.

Benchmark it.

Submit it publicly.

Everyone can now inspect the breakthrough.

Then compare that with:

Strategy B — derivative capture

Wait for successful algorithms to become public.

Take the strongest pieces.

Modify or recombine them.

Add a small implementation change.

Submit another version.

Then use a vastly larger benchmarking operation to generate substantial adoption for that new version.

If Strategy B becomes economically more effective than Strategy A, TIG has a serious incentive problem.

Because the rational strategy becomes:

wait for somebody else to discover the breakthrough, then compete for ownership of the adoption.

That is almost the opposite of what TIG is supposed to encourage.

Concentration makes this much worse

If a small independent innovator submits a weak derivative, the market can simply ignore it.

Other benchmarkers can test it, decide it adds little or nothing, and continue running the original or superior algorithm.

That is how the adoption mechanism is supposed to protect the system.

But what happens when the innovator is affiliated with one of the largest benchmarkers?

Then the normal safeguard is weakened.

The derivative does not necessarily need to convince a broad independent market.

A significant part of its adoption can potentially be generated by the affiliated benchmarking operation itself.

Instead of:

“this derivative has to prove itself sufficiently good that independent benchmarkers choose it”

we risk moving towards:

“this derivative already has substantial adoption because the organisation behind it controls substantial benchmarking influence.”

Those are fundamentally different things.

This is where submission quality and benchmarking influence combine

A derivative that produces a genuine 20% improvement is valuable.

A derivative that materially reduces runtime is valuable.

A derivative that solves tracks its predecessor could not solve is valuable.

A genuinely better combination of methods can also be valuable.

Those things should be rewarded.

But a derivative of an already-exact algorithm that fails to materially improve the only remaining optimisation objective should not obtain the same economic significance simply because it can attract adoption.

Likewise, combining existing track winners into a wrapper may be useful engineering, but that is not automatically equivalent to creating the underlying breakthroughs.

The problem is not code reuse.

The problem is separating economic reward from marginal contribution.

Delegation adds another layer

TIG’s influence system also incorporates delegated deposits.

That creates the possibility of a reinforcing economic loop:

affiliated innovator submits code

→ affiliated benchmarker directs compute towards it

→ benchmarking influence helps create adoption

→ adoption generates innovator rewards

→ deposited TIG can be delegated back towards benchmarker influence

→ greater influence gives even more power over future adoption

Again, none of those individual actions necessarily break a rule.

The concern is the combined incentive.

A vertically integrated participant can potentially benefit at every stage:

innovation account
→ benchmarking account
→ adoption
→ rewards
→ delegation
→ greater influence

That is what I think TIG needs to examine.

Why this is harmful even if nobody is acting maliciously

I am deliberately not suggesting that TigPool has to be acting maliciously for this to be damaging.

In fact, that is the more important point.

Every individual decision can be rational.

Run the algorithms that maximise your return.

Submit changes where you find them.

Delegate where it benefits you.

Use your compute efficiently.

But if those individually rational decisions create a system where a dominant operation can repeatedly obtain adoption for affiliated submissions that contribute little new value, then the protocol incentives are producing the wrong outcome.

The damage is not just financial.

It changes what kind of behaviour TIG rewards.

Large compute should increase exploration, not reduce it

A large benchmarker’s compute could be one of TIG’s greatest assets.

It could be used to explore:

• new algorithms;

• under-tested submissions;

• alternative parameter regions;

• genuinely novel approaches;

• competing implementations;

• and promising work from smaller innovators.

That would make the entire network stronger.

But if an increasingly large share of that compute is concentrated on affiliated submissions, TIG loses something important:

independent exploration.

Independent algorithms receive fewer samples.

Potential breakthroughs receive less attention.

Smaller benchmarkers cannot realistically compensate for the imbalance.

And innovators cannot force the network to properly evaluate their work.

That means the problem can become worse than merely rewarding the wrong submission.

The network may fail to discover that a better approach exists at all.

This becomes an innovator-retention problem

Put yourself in the position of an independent developer.

You spend weeks producing a breakthrough.

You publish it.

A much larger economic operation can immediately inspect it.

A derivative appears.

The measured improvement is tiny, nonexistent, or limited to a minor implementation detail.

Yet the derivative receives substantial adoption because its affiliated benchmarker has enormous compute.

What incentive do you have to produce the next breakthrough?

Eventually the rational conclusion becomes:

the difficult part is creating the innovation, but the profitable part is controlling the adoption.

If enough developers reach that conclusion, TIG loses exactly the people it was designed to attract.

The network may still look active.

There may still be lots of submissions.

There may still be lots of benchmarking.

There may still be lots of adoption.

But:

submission count is not innovation.

adoption is not automatically proof of improvement.

And:

benchmarking volume is not the same thing as independent validation.

The irony around adoption rewards

This issue is made even more interesting by the previous discussion around rewarding hyperparameter adoption.

TigPool’s developer suggested an adoption-based reward mechanism, something else that would directly benefit them in the current state.

Daniel immediately raised the problem of somebody taking another participant’s configuration, changing it by a tiny amount, and potentially capturing the adoption reward.

The wider discussion also recognised that benchmarkers may follow a configuration simply because a large or successful benchmarker is already using it.

That is essentially the same problem.

Scale itself starts creating adoption.

Once that happens, adoption becomes reflexive:

the leader runs it

→ others assume it must be good

→ adoption increases

→ higher adoption makes it look even more validated

That is dangerous when adoption is merely a social signal.

It becomes much more important when adoption determines economic rewards.

I don’t think the answer is to attack TigPool

I don’t want TigPool prevented from submitting algorithms.

I don’t want their compute capped.

I don’t want reuse of open-source TIG algorithms prohibited.

And I don’t think somebody should be penalised merely for becoming successful.

The protocol should instead become better at distinguishing three things:

quality, provenance and independent validation.

For any derivative submission, we should be asking:

Quality:
What measurable improvement does this submission actually provide over the work it was derived from?

Provenance:
How much of the resulting performance comes from previously published work, and what did the new contributor actually add?

Independent validation:
Did unrelated benchmarkers independently adopt it because it was demonstrably better, or was a large proportion of its adoption generated by an affiliated benchmarker?

Those three questions together give a much better indication of genuine progress.

What I think TIG should consider

Derivative algorithms should be judged much more explicitly on their marginal contribution.

If somebody takes an existing algorithm and improves runtime by 20%, reward them.

If somebody introduces a genuinely new method, reward them.

If somebody combines previous methods in a way that demonstrably produces a substantial new performance gain, reward them.

But if a derivative of an already submitted algorithm fails to materially improve runtime or quality, what exactly is the network paying for?

That is not hostility towards iteration.

It is simply recognising that:

a new submission is not automatically new innovation.

The principle TIG needs to protect

Open-source iteration is one of TIG’s strengths.

Anyone should be able to build on anybody else’s work.

But the economic reward should correspond to what was actually added.

The original algorithm and a near-zero-impact derivative should not automatically compete on equal economic terms merely because both have different submission names.

And when the derivative is backed by a benchmarker powerful enough to materially generate its own adoption, the distortion becomes greater again.

That is why I think this issue is bigger than TigPool.

TigPool’s current position is exposing a weakness in the incentive design.

If my pool, or any other pool, eventually occupied the same position and behaved the same way, I would have exactly the same concern.

The objective should not be to make TigPool weaker.

It should be to make TIG’s market signal stronger.

TIG should reward genuine marginal innovation, not merely marginal submissions.

Adoption should validate improvement, not substitute for it.

And compute should be used to discover which innovation is best.

It should never become powerful enough to decide which innovation gets to count.

[Rootz]

As a follow up to this post, an example of what exactly is hurting innovation and actually taking away rewards from other innovators.

hyper_sigma_v2 went live this week. If people actually checked code that TigPool is submitting they would see things like this -

10k hyperedges code Hypergraph 10k - Diffchecker
20k hyperedges code Hypergraph 20k - Diffchecker
200k hyperedges code Hypergraph 200k - Diffchecker

This is a diff between sigma_freud_v8 and hyper_sigma_v2 on 3 tracks where they literally 100% copied my code.

Why should they receive rewards for this kind of practice? Why should anyone?

66% of the code is unchanged from the original submission. On 3 tracks exactly the same performance can be gained, albeit the baked hyperparameters have simply set different settings, which can be replicated manually in the master settings.

Mixing and matching the best tracks from different algorithms and then submitting as your own code should not be rewarded. We are supposed to be accelerating algorithmic innovation here, not lazily profiting off someone else’s work.

Next round no doubt sigma_freud_opt will be copied, due to it’s huge reduction in runtime on track 10k, time will tell…

It comes back to the same question…why should innovators bother if different track code is cherry picked and re-submitted by someone else as their own?

[xnico31]

Thanks for raising this openly, and for writing it without making it personal. The questions you ask are fair ones and they deserve an answer.

We are replying because we are named specifically. But what you are describing is a question of protocol design,not a question about us — as you say yourself, it would exist for any pool that ended up in the same position.

So we are not going to go through the post point by point. Here is simply how we see it.

Code reuse on TIG isn’t new, and it isn’t rare. We compared the public source of 431 algorithms across challenges: two thirds of them reuse at least half the code of an earlier submission, and a third reuse more than 80 %. That has been the normal way this protocol works, long before this thread.

It also runs in every direction. Here are our own algorithms whose code was reused — 23 of them, by 39 later submissions from other players, several with source files byte-identical to ours. The percentage is the share of the later submission’s lines that already existed in ours:

**c001 · sat_turbo_fix** (r105) — sat_turbo_imp 78.6 %
**c001 · sat_vanguard** (r107) — sat_imp_v1 26.7 %
**c001 · sat_vanguard_v3** (r112) — prom_dusty_sat 89.0 %, prom_saffron_otter 82.9 %
**c001 · sat_vanguard_v4** (r114) — sat_tailwalk 33.0 %
**c001 · sat_vanguard_v9** (r121) — sat_valknut2 98.4 %, sat_valknut 96.5 %
**c001 · sat_hybrid** (r126) — sat_tailwalk_v6 92.7 %
**c001 · sat_hybrid_v2** (r127) — super_satis 79.0 %
**c001 · sat_hybrid_v3** (r129) — sat_imp_v5 72.4 %

**c003 · knap_one** (r45) — justatestonmyendher 100.0 %, quadkp_maximize 67.7 %
**c003 · knap_quality_opt_v9** (r115) — near_knap_v8 94.2 %, near_knap_zenith 56.0 %
**c003 · knap_quality_opt_v10** (r117) — prom_pulse_marlin_v2 73.8 %, prom_opencode_jade_s 59.2 %
**c003 · knap_quality_opt_v11** (r119) — superfast_knap_v1 52.2 %
**c003 · knap_quality_opt_v13** (r127) — combination_alg 68.6 %

**c004 · autovector_final4** (r116) — prom_kimi_vec 99.2 %
**c004 · autovector_g** (r122) — solve_hybrid 86.2 %

**c005 · hyper_opt_extrem_v11** (r124) — multitrack_bok 85.4 %

**c008 · titan** (r114) — near_arb_v1 80.9 %, iycbtjt 27.8 %
**c008 · titan_v2** (r116) — ea_frontier_router 98.1 %, codex_ea_hybrid 97.6 %, aycdicdb 75.5 %, aiwfcihq 71.0 %, delta_machine_ea_v2 31.5 %, delta_machine_ea_v1 26.6 %
**c008 · titan_v3** (r118) — prom_claw_linen_hawk 86.9 %, ic_energy_v1 80.8 %, prom_opencode_quartz 59.4 %
**c008 · titan_v4** (r120) — prom_neural_energy 81.2 %
**c008 · titan_v5** (r120) — dale_v1 80.7 %
**c008 · titan_v6** (r121) — titan_master 87.2 %, prom_inspired 62.2 %
**c008 · titan_v7** (r125) — cryp_energy 99.5 %, titan_v7_ch3b 97.8 %, titan_v8 96.4 %

We’re not asking for anything to be done about it. The protocol was built to allow it, our code is public like everyone else’s, and we knew that when we submitted it. It is not our place to question the protocol or to say how it should be designed. We have played by its rules from the start, and we will keep doing that.

That is not our interpretation, it is what the documentation says:

- **Code Submissions.** A submission can be *“Implementing an algorithmic method that is not in TIG yet”* or

*“Optimizing an existing Code Submission to improve its performance.”* Building on an existing submission isone of the two forms of submission the protocol defines. docs.tig.foundation/innovators/code-submission

- **Licensing.** The Open Data License is *“designed to promote open collaboration, transparency, and the rapid advancement of algorithmic innovation”*, and fosters a community *“who can build upon each other’s work andaccelerate the pace of discovery.”*

docs.tig.foundation/licensing

On compute: we hold only 30 % of the network’s hashrate. The other 70 % is in other hands, and those players decide for themselves which algorithms they run !

It is also worth remembering why the pool exists in the first place. It was created at a point where small solo miners could no longer mine on TIG at all. Without a pool, mining, and the tokens that come with it would have ended up entirely in the hands of a few large farms, on tig pool we split de rewards on 83 players …

The answer to being copied is to keep innovating.

The protocol also builds in a head start. Per the Code Submissions page above, a Code Submission stays in a private repo for two rounds before it is pushed public — roughly two weeks. That is two weeks to work on the next version, so that by the time the current one becomes public and can be copied, you already have something better ready to submit. If every innovator uses that window, the problem raised here largely stops existing.

That is what we do, and it is what we would advise everyone here to do. TIG may decide to change the rules of the protocol — that is a legitimate discussion to have. But as things stand, we all have to work with the rules that exist, and under those rules the answer is the same for everyone: keep innovating, constantly. If you stop, someone will pass you. That is what a competition is.

Last thing. We have been active on TIG since July 2024. We started solo, like everyone else here, and we have never jumped ship since.

[Rootz]

Thanks for responding. I think your reply actually helps clarify where we are talking past each other.

I am not arguing that code reuse should be prohibited. I said explicitly in the original post that reuse and iteration are valuable parts of TIG.

So the statistics showing that other participants reuse code — including TigPool code — don’t really address my concern.

The relevant question isn’t:

“How much existing code did somebody reuse?”

It is:

“What measurable value did the new submission add, and was its adoption independently earned because of that improvement?”

Those are very different questions.

If somebody takes 95% of your algorithm and makes it 30% faster, fantastic. They have produced a meaningful optimisation and deserve to benefit from it.

If somebody takes 95% of an existing algorithm, changes something, and the resulting implementation performs effectively identically, then I think it is perfectly reasonable to ask what new value the network is rewarding.

In fact, the documentation you quoted says exactly this. One of the defined forms of Code Submission is:

“Optimizing an existing Code Submission to improve its performance.”

The key words there, for me, are “improve its performance.”

That is precisely why I keep using there_v10 / autovector_g as an example.

Vector Search is already exact argmin. There is no quality improvement available. Runtime is the only meaningful performance dimension remaining.

Across the five tracks I tested, the total runtime was:

there_v10: 49.53s
autovector_g: 49.98s

One was faster on one track, slower on others and identical on two. There was no meaningful overall performance improvement.

So I’m genuinely asking:

What performance improvement was contributed there that justified a separate reward-bearing economic identity?

That isn’t an argument against open-source iteration. It is an argument that iteration and improvement are not synonymous.

The same distinction applies to the list you posted. Showing that 39 later submissions reused TigPool code tells us nothing by itself about whether those submissions were good or bad. I would ask exactly the same question of them: did they materially improve what they reused?

If not, I don’t think they should be economically rewarded for near-zero marginal contribution either.

This isn’t one rule for TigPool and another for everyone else.

The 30% point is actually central to my concern

You said:

“we hold only 30% of the network’s hashrate.”

Firstly, 30% of an entire network is not a trivial amount of concentration.

But more importantly, raw hashrate isn’t the metric that determines algorithm adoption. Influence is.

TIG’s published adoption formula is effectively:

share of that benchmarker’s qualifiers using algorithm × benchmarker influence

summed across benchmarkers.

And OPoW influence itself includes challenge performance, self-deposits and delegated deposits.

Code submissions become reward-eligible once they reach 25% adoption.

So I think the much more relevant transparency question is:

What is TigPool’s actual OPoW influence, and what proportion of each TigPool-affiliated algorithm’s adoption is generated by TigPool itself?

That would tell us far more than saying the pool has 30% of raw hashrate.

If an affiliated benchmarker can supply a substantial fraction of the adoption required for its own innovator submission to earn rewards, then independent adoption is weakened as a performance signal.

That is the mechanism I am questioning.

The other 70% being free to choose alternatives does not remove that issue. That 70% is fragmented amongst multiple independent participants; TigPool’s share is coordinated by one operation.

There is a fundamental information difference between:

six unrelated benchmarkers independently deciding an algorithm is best

and

one very large benchmarker providing a significant proportion of an affiliated algorithm’s adoption.

Yet the protocol can treat those adoption units similarly.

“Keep innovating” doesn’t solve this

I also don’t think:

“The answer to being copied is to keep innovating.”

really addresses the incentive problem.

Of course innovators should keep innovating.

But TIG is explicitly designed to reward them for producing useful improvements.

If the answer to someone taking a publicly released breakthrough, making a negligible modification and gaining economic adoption is simply “you should already have invented something else by then”, then we are no longer really asking whether the derivative contributed meaningful value.

We are rewarding submission velocity and adoption capacity.

That isn’t necessarily the same thing as innovation.

The two-round private period gives the original innovator a head start, but it doesn’t answer the question of what should happen when the derivative eventually becomes active.

If the derivative materially improves the algorithm: reward it.

If it doesn’t materially improve anything: why should moving quickly enough to submit it create a new economic claim over the underlying work?

And this is where TigPool’s scale matters

This would concern me far less if TigPool were simply another innovator.

A weak derivative submitted by a small innovator still has to convince the market to adopt it.

But TigPool is both an innovator operation and a very large benchmarker.

You therefore have something most innovators do not:

the ability to contribute substantial adoption to your own submissions.

That changes the economics.

And it is why your previous proposal for an Adoption Pool is relevant to this discussion.

Daniel immediately pointed out that somebody could copy another participant’s configuration, make a tiny epsilon change and create a supposedly new configuration.

Haver then described the next part of the problem: a large benchmarker or leaderboard leader could adopt that almost-identical version and other participants may follow it simply because the leader is running it.

That is almost exactly the issue I’m raising here, just applied to algorithms instead of hyperparameters.

None of this requires TigPool to be doing anything against the rules

I agree with you on one important point:

this is ultimately a protocol-design discussion.

I am not asking anyone to punish TigPool for operating according to the current rules.

I’m saying the current rules create an increasingly problematic incentive when benchmarker influence and innovator interests become vertically integrated.

TIG should want:

reuse + meaningful improvement + independent validation.

What worries me is:

reuse + negligible improvement + affiliated adoption.

Those are not economically equivalent.

And if the second combination can reliably earn rewards because the affiliated benchmarker has sufficient influence, then the protocol is rewarding something other than the marginal algorithmic progress it was designed to incentivise.

So rather than debating whether copying is permitted — we agree that it is — I think the useful questions are:

1. What measurable improvement should a derivative need to demonstrate?

2. How much of an algorithm’s adoption comes from economically affiliated benchmarkers?

3. Should self-affiliated adoption carry the same weight as independent adoption?

4. Can a benchmarker’s own influence materially contribute towards taking its affiliated submission through the 25% reward threshold?

5. Are we rewarding the discovery of better algorithms, or increasingly rewarding the ability to package existing discoveries and control their adoption?

Those are the questions I wanted this thread to raise.

I have no interest in stopping TigPool succeeding.

What I don’t want is a system where success at benchmarking gradually gives any participant — TigPool, InnoPool or anyone else — enough economic influence that adoption ceases to be reliable evidence of improvement.

That would be bad for every genuine innovator on TIG, regardless of whose code is being reused.

For clarity, I asked ChatGPT to help me come up with a more diplomatic response that gets my point across, as I let my emotions get the better of me.

[Newman]

According to the tigpool website at this moment, they received 1.74 TIG of REWARDS in the current BLOCK.
Each BLOCK gives all the REWARDS to the benchmarker, which is 3.68 TIG.
Therefore, tigpool accounts for 1.74 / 3.68 = 47.28% of the total network hashrate.

[TheDoctor]

Hi Everyone, Thanks for the detailed posts and discussion.

We’re thinking about the issues raised. I’d like to ask some clarifying questions. To be clear, I’m not accusing anyone of anything below. I think we’re all agreed on what the objective is.

@xnico31 is your position that TigPool’s choice of vector search algorithm is justified only on the basis of its benchmarking performance (when penalty from legacy multiplier is included)? Or does this choice follow from the algorithm performance/profitability when innovator rewards are included?

@Rootz do you think the issues may addressed (e.g. made significantly less likely to arise in practice) if there were to be a sufficient number of competitive mining pools? For example, lets say there were 5 pools operating on TIG, each with around 20% of the compute. What would be the effect Re the issues you’ve flagged? Which if any would remain in your view?

[Rootz]

Thanks @TheDoctor. Yes, I do think five genuinely competitive and independent pools each around 20% would make one part of the problem significantly less likely, but I don’t think it would remove the underlying issue entirely.

There is one distinction I would make first: for this discussion I think OPoW influence is more relevant than raw compute/hashrate, because influence is what weights algorithm adoption.

If we simplify your example and assume each of the five pools had roughly 20% of the network’s influence, then that would be a substantial improvement over one participant having enough influence to potentially generate the reward threshold for an affiliated algorithm largely or entirely itself.

An active Code Submission currently needs 25% adoption to earn rewards. With genuinely independent pools at approximately 20% influence each, no individual pool could simply direct all of its own qualifying work towards an affiliated algorithm and get it through that threshold alone.

It would need at least some independent adoption from elsewhere.

That is important, because it restores some market validation.

So yes, greater competition between similarly sized pools would significantly mitigate the concentration side of what I am raising.

However, I think several problems would remain.

1. The conflict between benchmarker profitability and innovator profitability would still exist

I think the question you asked xnico immediately before mine is actually at the heart of this.

If TigPool chooses autovector_g because, after accounting for the legacy multiplier, it genuinely produces the best return for the benchmarker alone, then that is the synthetic market operating as intended.

But if there_v10 would be the economically/performance-optimal choice for the benchmarker, and autovector_g becomes the preferred choice only once the innovator rewards flowing back to the same economic operation are included, then I think we have identified the structural problem very clearly.

The entity is no longer choosing:

“Which algorithm is best for benchmarking?”

It is choosing:

“Which algorithm maximises our combined return as benchmarker + innovator?”

Those can produce different answers.

And if they produce different answers, adoption stops being a clean signal of which algorithm performs best.

That problem would still exist with five pools.

It would simply be less powerful because each participant controlled less of the total signal.

2. Submission quality would still not be measured directly

The other half of my concern remains regardless of pool concentration.

The protocol currently rewards adoption. It does not directly ask:

“How much did this derivative actually improve upon what already existed?”

That is why I keep coming back to there_v10 / autovector_g.

Both already reach exact argmin, so runtime is the only meaningful improvement available.

My testing showed effectively no overall runtime improvement.

If an unrelated innovator submitted that derivative, I would still ask exactly the same question:

what new algorithmic value are we rewarding?

Five pools may make it harder for that submission to obtain adoption without convincing somebody else to run it, which is certainly better.

But the basic distinction between new submission and new innovation still exists.

3. Twenty percent is still a very large head start for an affiliated algorithm

Even in the five-pool example, imagine Pool A has 20% influence and an affiliated innovator releases Algorithm A.

If Pool A heavily benchmarks Algorithm A, it could potentially begin with something approaching a 20 percentage-point contribution to adoption before any genuinely independent benchmarker has chosen it.

It then only has to convince a comparatively small part of the remaining market to cross the reward threshold.

Compare that with an independent innovator who controls no major benchmarker.

Their submission begins with essentially zero guaranteed adoption and has to convince the market from scratch.

So the asymmetry would be greatly reduced, but it wouldn’t disappear.

4. There is also a herding effect

Five independent pools only solves the issue properly if their decisions genuinely remain independent.

That was part of the concern raised in the earlier adoption discussion.

If Pool A is successful and visibly running Algorithm A, Pools B and C may reasonably assume Pool A has done the benchmarking and begin following it.

Now adoption starts reinforcing itself:

large pool chooses algorithm → algorithm appears heavily adopted → others investigate/follow it → adoption rises further

That isn’t necessarily malicious behaviour by anybody.

But it means the initial 20% signal can have more influence than simply its numerical 20%.

5. Delegation can also disturb the neat 20/20/20/20/20 assumption

This is another reason I think we should talk about influence rather than just physical compute.

TIG’s influence calculation includes deposit factors, including delegated deposits.

So five pools with identical hardware do not necessarily have equal influence.

If token deposits and delegation start concentrating around the most successful pool, we can gradually recreate the same concentration problem even though the underlying physical compute remains relatively distributed.

What I think five competitive pools WOULD fix

I don’t want to understate it, because I think your suggestion would be a major improvement.

Five genuinely independent, similarly influential pools would:

• make it much harder for one participant to unilaterally create adoption;

• force affiliated submissions to obtain some external validation;

• increase exploration because there would be more independent algorithm-selection strategies;

• make it harder for one entity’s commercial interests to dominate the network signal;

• and give independent innovators several meaningful potential sources of adoption rather than depending heavily on one dominant operation.

All of those are positive.

So if the question is:

“Would more competitive pools significantly reduce the problem?”

My answer is absolutely yes.

If the question is:

“Would five pools at 20% mean the protocol no longer needs to consider these issues?”

I think the answer is no.

I think there are really two separate safeguards needed

One is decentralisation of benchmarking influence.

More competitive pools absolutely helps there.

The second is independence of validation.

I think TIG needs to consider whether adoption originating from an economically affiliated benchmarker should be distinguishable from genuinely independent adoption when determining Code Submission rewards.

I’m not necessarily saying affiliated adoption should count for zero.

But I do think there is an important informational difference between:

20% adoption generated by the organisation that submitted the algorithm

and

20% adoption generated independently by several unrelated benchmarkers who tested it and concluded it was better.

The second tells us considerably more about the quality of the algorithm.

And finally there is the separate question of marginal contribution.

Even perfect decentralisation wouldn’t completely answer why a derivative producing no meaningful improvement should receive a new stream of innovation rewards.

That is why I see these as three related issues:

1. Concentration — who controls the adoption signal?
2. Independence — who is validating the submission?
3. Quality — what new value did the submission actually contribute?

Five competitive pools would go a long way towards fixing 1, improve 2, but wouldn’t inherently solve 3.

For me, the ideal TIG market is one where an innovator can submit something genuinely better and have a realistic expectation that several independent benchmarkers will test it because finding the best algorithm is economically optimal for them.

If that happens, then it doesn’t matter whether the innovator is Rootz, NVX, TigPool, or someone who joined yesterday.

The best improvement should win.

That is ultimately all I am arguing for.

[xnico31]

@TheDoctor — both measurements are right. They were taken at different worker counts.

The change in autovector_g is an L2 persisting window over the fp16 database: a cache-residency hint, nothing else. Same bytes read, same arithmetic, bit-identical output. With one worker the DB tiles already stay in L2, nothing to relieve, nothing to gain. With several workers sharing the card they get evicted, and the windowstarts doing real work.

RTX 3080 Ti, 40 nonces, fixed seed, only the worker count changes. Time in seconds:

| Track | | 1w | 2w | 3w | 4w |

| T19 (7k) | there_v10 | 25.5 | 20.5 | 14.5 | 13.5 |

| T19 (7k) | autovector_g | 25.5 | **17.5** | 14.0 | **13.0** |

| T18 (15k) | there_v10 | 42.0 | 38.0 | 26.0 | 30.5 |

| T18 (15k) | autovector_g | 43.0 | 38.5 | 28.0 | **25.5** |

**1 worker:** identical on 7k, 2.4 % slower on 15k. **4 workers:** 3.7 % and 16.4 % faster.

**Scaling 1w→4w:** there_v10 ×1.89 and ×1.38 — autovector_g ×1.96 and ×1.69. On 15k there_v10 *regresses* from three workers to four, 26.0 s to 30.5 s, while autovector_g goes 28.0 s to 25.5 s.

@Rootz’s own figures confirm it: 9.12 vs 9.12 on 13k, 5.07 vs 5.07 on 15k, identical to the second decimal.
That is what a bandwidth optimisation looks like measured at one worker. His result is right for his configuration.

Caveats: single runs, one seed, one card class — the 3-4 % differences are noise, the 16 % and the regression are not. At 3 workers on 15k autovector_g is 7.7 % slower. Our fleet runs 4 to 9 workers per card depending on VRAM, so the table stops at the bottom of the range. Re-run it at four and see.

So the claim is narrower than “it is faster” and different from “it adds nothing”: **not an improvement at one worker, an improvement at the worker counts Benchmarkers actually run.**

## The influence question, answered

@Rootz, you were right that hashrate is the wrong metric and influence is the right one. Ours, from `get-opow`: **27.89 %**. The reward threshold for a Code Submission is 25 %, and adoption sums to 100 % within a challenge.

So yes — **our Benchmarker can carry one of our submissions past the threshold on its own.** That is arithmetic, not a hypothetical, and it is one API call away. For context: the top three Benchmarkers hold 61 % of influence between them, the top five 75 %, across 16 accounts.

We would rather post that than have it extracted. It is also why we proposed capping what any single Benchmarker can contribute to one algorithm’s adoption. That rule bites us first. We support it anyway.

## The same question, in both directions

The structural point applies to you as well, and I mean that as symmetry rather than counter-charge. You are also a pool operator now : your miners supply the hardware, your delegators the stake, and both are paid from what your Benchmarker earns. Neither of us escapes that, and neither should be assumed to be gaming it.

So: figures rather than suspicion. We have published our influence. Publish yours, and the share of your own submissions’ adoption that your own Benchmarker supplies, and this stops being about who suspects whom. c008 is worth putting up for the same reason, and it cuts against the picture the thread has built. We opened that challenge — `titan` and `titan_v2` have no ancestor above 25 % in the monorepo, and eighteen later submissions derive from that line. One of them, `first_energy`, carries 84.8 % of our `titan_v6` and holds 36.2 % of c008 adoption against 58.5 % for our own `titan_v9`.

**Our Benchmarker supplies 30 % of first_energy’s qualifiers, and 30 % of titan_v9’s** — the same share, to a derivative of our own code submitted by someone else, which earns us nothing and earns its author the innovator reward. We run it because it performs.

That is one challenge and I will not over-read it. But it is the kind of thing figures settle and argument cannot: whatever they show for either of us, the conflict you identified is a property of the design rather than a problem with a particular operator.

## What we would propose

Adoption tells you what is profitable to run, not what is better. Those coincide often enough to be useful and not often enough to be evidence. So take the evaluation off the Benchmarker.

**1. Measure at submission, by the protocol.**

Not a script people run at home — self-reported timings cannot be compared, and this exchange is the proof: two honest measurements, opposite conclusions, one worker against four. The benchmark runs when the algorithm is submitted, on a fixed instance class at a stated worker count, same instances and fuel budget as the incumbent, paired nonce by nonce. The submitter reports nothing.

Half of this exists already: precommits carry a `compute_type` (`aws_g4dn`, `aws_c7a`, `aws_m7a`, `aws_t3a`), so the standard machine class per challenge is defined. What is missing is who pays. Leave the 10 TIG submission fee as it is and **add a measurement fee on top — 10 or 20 TIG — covering the runs**. Nothing is diverted, nothing comes from emissions. It also prices the other problem here: a resubmission that changes two default hyperparameters costs 10 TIG today; at 30 it has to be worth the machine time it consumes.

**2. Four declared operating points per algorithm.**

Maximum quality, two intermediates, maximum speed. The harness runs those four and publishes a **quality/runtime Pareto front**, appended to the challenge’s documentation.

Finding good hyperparameters is the innovator’s job — they wrote the algorithm and know its regimes. It is not the Benchmarker’s: a Benchmarker is a miner paid for throughput, and asking it to search a parameter space is asking for unpaid research. Today it guesses. With four declared points it reads a table and picks a row — which matters most for whoever arrives next, with no reference to start from and real money burning while they find out.

It also gives the protocol something it lacks. Outside the difficulty graph on the challenge page, there is noview of how quality and runtime have moved across rounds — nobody can open a challenge and see what improved this quarter or whether it stalled. Stacked round after round, these fronts are that record, produced as a by-product.

**3. A cap on single-Benchmarker adoption.**

At 10 %, crossing the 25 % threshold needs three independent

Benchmarkers whatever anyone’s size. It requires no knowledge of who is affiliated with whom — which matters, since affiliation cannot be established on-chain. We have no view on the level; that is for the protocol to set, not the participant it constrains.

**4. A word for the Challenge Owners.**

No formula settles “how much did this improve things” — it needs judgement, and judgement needs someone with no stake in which algorithm wins. That is the Challenge Owners, and only them: Benchmarkers earn from throughput, Innovators from adoption, while their reward tracks Recent Advances. They are also the domain experts on their own challenge. So rather than propose a mechanism, I would suggest this is a question worth their attention. If a bounded coefficient on the innovator reward is a good idea, they are the ones who would know — and if it is a bad one, they are best placed to say why.

## In closing

The four points are one idea in four places: replace inference with measurement. Today the network infers quality from adoption, a derivative’s worth from a line count, and intent from who runs what. Every one of those is contestable, which is why this thread exists and why it cannot end on its own terms.

Being on the receiving end of this is not pleasant, and a public argument between two operators is not good for the protocol either. But the question underneath is real, and it has been there since round 1 — long before this thread and before either of us was in it. If a week of argument is what it took to get it moving, that is a fair price.

We have put our figures and our proposals on the table. What happens next is not ours to decide.

And we agree with @Rootz’s conclusion without reservation — the best improvement should win. The way to get there is to measure improvement, not to count lines and not to guess at affiliations.

Tigpool team

[Rootz]

Thanks @xnico31. This is a much more useful response, and I think we are now getting to the point I hoped this thread would reach.

I also want to acknowledge where your additional data changes something I previously said.

On autovector_g, my testing was all at one worker. Your results show that the L2 persisting-cache window can produce a meaningful improvement under certain levels of concurrent GPU contention. So I am happy to correct my earlier statement that it adds nothing.

The more accurate conclusion is:

autovector_g does not materially improve single-worker performance, but there is evidence that it can improve throughput at particular multi-worker operating points.

Your own results are still mixed — for example it is 7.7% slower at three workers on the 15k track but 16.4% faster at four — and, as you acknowledge, these are single runs on one seed and one GPU class. So I don’t think either of us can use our respective tests to make a universal claim about its performance.

But that actually strengthens one of the conclusions now emerging from this discussion:

performance should be measured objectively by the protocol rather than inferred from adoption or argued from benchmark results produced under different operating conditions.

I also accept the first_energy example. It is useful evidence that TigPool does not simply benchmark its own submissions exclusively, and I don’t think it would be fair for me to continue implying that.

What I think is much more important is what you have now confirmed about influence.

The influence concern is no longer hypothetical

You have published:

TigPool OPoW influence: 27.89%

and stated explicitly:

“our Benchmarker can carry one of our submissions past the threshold on its own.”

That is essentially the structural concern I was trying to raise from the beginning.

With the Code Submission reward threshold at 25%, one benchmarker currently has sufficient influence to take an affiliated submission across that threshold without requiring another benchmarker to independently validate it.

That does not prove TigPool is abusing that ability.

It proves that the protocol currently permits the possibility.

You also published that:

the top 3 Benchmarkers hold 61% of influence
the top 5 hold 75%

That concentration is worth examining regardless of who those benchmarkers are.

And I appreciate you publishing the figures openly.

My own figures, for symmetry

You asked me to publish the equivalent numbers for InnoPool, so here they are.

InnoPool OPoW influence: 2.3187%

Benchmarker address:

0x7eaf319bde33f3317e4dd54a11b9d974b94460e3

My Rootz innovator address:

0x1884f0fe1208f38a1433b706790e22be9fed1f7c

I currently benchmark three of my own algorithms:

• there_v10

• sigma_freud_opt

• adaptive_js_opt

In the API snapshot I pulled, my currently qualifying submissions showed:

Algorithm	Adoption	Total qualifiers	InnoPool qualifiers
there_v10	55.08%	230	0
sigma_freud_opt	44.98%	360	0
adaptive_js_opt	11.13%	97	28
sigma_freud_v8	11.68%	45	0
adaptive_js_v9	8.09%	14	0

To be clear, the zero qualifier entries for there_v10 and sigma_freud_opt in that snapshot do not mean I am not currently benchmarking them. I am. They simply produced no recorded qualifiers from InnoPool in that particular snapshot.

For adaptive_js_opt, however, InnoPool supplied 28 of the 97 recorded qualifiers.

So I am not arguing that pools should never benchmark their own algorithms.

I do it myself.

The distinction is scale and influence.

InnoPool has 2.32% influence.

TigPool has 27.89% influence.

Even if I heavily favour one of my own algorithms, I cannot independently take it through the 25% reward threshold.

TigPool currently can.

If InnoPool eventually grows to the point where I can do the same thing, I would want exactly the same restriction applied to me.

That is why I have always seen this as a protocol issue rather than something that should be solved by creating a TigPool-specific rule.

I think one sentence in your reply is particularly important

You said:

“Adoption tells you what is profitable to run, not what is better.”

I completely agree.

But I think that statement has a very important consequence.

If adoption tells us what is profitable to run rather than what is objectively better, then adoption cannot simultaneously be treated as definitive evidence of algorithmic improvement.

That is really at the centre of this discussion.

There may be many rational reasons why a benchmarker chooses Algorithm B over Algorithm A:

• runtime;

• quality;

• legacy multiplier;

• hardware characteristics;

• concurrency;

• hyperparameters;

• operational efficiency;

• and potentially, where benchmarker and innovator interests overlap, the economics of innovator rewards.

That is why @TheDoctor’s earlier question is so important.

If an affiliated algorithm is the best choice purely from the benchmarker’s economics and performance, that is the synthetic market doing its job.

But if a different algorithm would be optimal for the benchmarker in isolation, and the affiliated algorithm becomes optimal only once innovator rewards flowing to the same economic operation are included, then the benchmarker’s adoption decision is no longer a clean signal of algorithm superiority.

That is a structural incentive conflict, even if everybody is behaving rationally.

Measure improvement directly

I strongly agree with your proposal that TIG should measure submissions.

This thread itself demonstrates why.

My one-worker tests suggested the L2 cache optimisation added essentially nothing.

Your multi-worker tests found a meaningful improvement at certain concurrency levels.

Both measurements can be honest and both can be correct.

So the solution should not be deciding which benchmarker’s script we trust.

It should be:

standardised protocol-level measurement.

Same hardware class.
Same instances.
Same paired nonces.
Same fuel.
Same software environment.
Enough runs to separate genuine improvement from noise.

I would add one important requirement:

there should be multiple defined operating/concurrency points where that matters for the challenge.

If Vector Search were tested only at one worker, the protocol could miss the improvement you are demonstrating.

If it were tested only at four workers on one GPU architecture, it could potentially overstate how universal that improvement is.

The goal should be to measure the real operating envelope.

The Pareto-front proposal is also valuable

I like the idea of publishing quality/runtime fronts.

That would give TIG something it currently lacks: an objective historical record of actual algorithmic progress.

Instead of seeing merely:

Round 120: another algorithm was submitted

we could see:

Round 120: this submission moved the speed/quality frontier by X% in this region.

Over time, anyone could look at a challenge and see whether genuine progress was occurring or whether submissions were mostly moving sideways.

For exact-argmin Vector Search, once exactness is achieved, that frontier effectively becomes runtime/throughput.

For other challenges it may be more genuinely multi-dimensional.

That seems considerably more informative than adoption alone.

I strongly support limiting single-benchmarker adoption

Of the proposals you made, I think this is probably the most immediately relevant to the original concentration concern.

If a single benchmarker’s adoption contribution were capped at, say, 10%, then reaching 25% would require support from at least three benchmarkers.

That restores a degree of independent validation without needing the protocol to determine whether two accounts are affiliated.

And I respect the fact that you support a rule which would constrain TigPool first.

The principle I support is simple:

no single benchmarker should be capable of independently economically validating a Code Submission.

That rule should apply equally to TigPool, InnoPool and everybody else.

There is one thing that would need careful thought: account splitting.

If the cap applies purely per benchmarker address, an economic operator could theoretically divide activity between several benchmarker accounts.

I don’t have an immediate solution to that, especially because affiliation is difficult to establish objectively on-chain, but it needs to be considered when designing such a mechanism.

I am less convinced about increasing submission fees substantially

I understand the reasoning.

If measurement consumes real compute and trivial resubmissions have a cost, charging for that compute makes sense.

But I would be careful about increasing the effective submission cost from 10 TIG to 20 or 30 TIG.

A large operation can absorb that repeatedly.

An independent researcher experimenting with genuinely novel approaches may not be able to.

We should avoid solving low-value submission spam in a way that unintentionally raises the barrier to entry for exactly the independent innovators TIG wants.

Perhaps some form of refundable measurement bond could work:

• submit the measurement cost;

• protocol performs the evaluation;

• meaningful measured improvement earns some or all of it back.

I’m not claiming that is necessarily the correct mechanism, only that I think discouraging trivial submissions and discouraging small innovators are two very different things, and we should be careful not to do both at once.

Challenge Owners

I agree Challenge Owners should be involved because they understand what constitutes meaningful progress in their domain.

My preference would be for them to define the measurement framework rather than routinely judge individual algorithms subjectively.

For example:

• which hardware classes matter;

• which concurrency levels matter;

• which tracks/instances are representative;

• how many paired runs are needed;

• which metrics define improvement;

• what constitutes statistical significance;

• and how different regions of a Pareto front should be evaluated.

That preserves domain expertise while keeping the actual measurement as objective and reproducible as possible.

On submission quality

I still think this part matters independently of concentration.

Open-source reuse is absolutely a feature of TIG.

I have never argued that copying existing code should be prohibited.

But I still believe the economic reward should correspond, at least broadly, to the marginal value that was actually added.

A derivative that takes an existing algorithm and makes it 20% faster has contributed something meaningful.

A new method that improves quality materially has contributed something meaningful.

A combination of existing methods that moves the Pareto frontier has contributed something meaningful.

Those things should be rewarded.

But the existence of a new Code Submission should not itself be treated as evidence that new algorithmic value has been created.

This is why direct measurement matters so much.

It lets us move away from arguments about:

“how many lines were copied?”

and towards:

“what did this submission actually improve?”

That is a much healthier question.

Where I think this thread has ended up

I think we’ve moved quite a long way from the original disagreement.

My initial concern was that sufficiently concentrated benchmarking influence, combined with affiliated innovator activity, could distort the adoption signal and allow weak or marginal submissions to receive economic validation without broad independent support.

We now know several things:

• TigPool has 27.89% influence and can cross the 25% threshold alone.

• InnoPool has 2.3187% influence and cannot.

• Adoption measures what is profitable for benchmarkers to run, not necessarily what is technically superior.

• Individual performance measurements can disagree substantially depending on operating conditions.

• TigPool does benchmark unaffiliated algorithms where they perform well.

• And both sides now appear to agree that direct measurement and greater independence in adoption would improve the protocol.

So I don’t think the useful conclusion is:

“TigPool is the problem.”

I think the useful conclusion is:

TigPool’s scale has exposed a protocol weakness which any sufficiently dominant vertically integrated operator could eventually exploit, intentionally or otherwise.

That is a much more important thing to fix.

For me, the principles coming out of this are:

1. Measure algorithmic improvement directly rather than infer it solely from adoption.

2. Require meaningful independent validation before a submission becomes reward-bearing.

3. Prevent any one benchmarker from independently crossing the adoption threshold.

4. Make algorithm performance transparent across standardised operating points.

5. Preserve low barriers for genuinely independent innovators.

6. Apply exactly the same rules to every participant, including InnoPool.

I agree completely with your closing sentence:

the best improvement should win.

I would just add one word:

the best measured improvement should win.

If this thread leads to TIG being able to demonstrate objectively what every new submission actually contributes, while making adoption more resistant to concentration, then I think the discussion will have achieved something worthwhile.

[Rootz]

Just to follow up on autovector_g, because @xnico31 specifically suggested:

“Re-run it at four and see.”

I have now done exactly that, this time matching the 40 nonces / 4 workers setup.

The main difference is hardware: TigPool tested on an RTX 3080 Ti, while I tested on an RTX 5070 Ti.

Queries	there_v10	autovector_g	Difference
7,000	20.52s	21.73s	5.9% slower
9,000	24.55s	25.07s	2.1% slower
11,000	27.53s	30.66s	11.4% slower
13,000	29.10s	33.78s	16.1% slower
15,000	32.52s	35.17s	8.1% slower

Total elapsed across the same workloads:

there_v10: 134.22s
autovector_g: 146.41s

So on the 5070 Ti, under the same 40 nonce / 4 worker condition, autovector_g is approximately 9.1% slower overall and loses on all five tracks.

Obviously this does not mean @xnico31’s 3080 Ti figures are false. In fact, I think the opposite results are more useful than either result in isolation.

They suggest that the L2 persistence change is hardware/architecture dependent, rather than a generally superior implementation.

On TigPool’s 3080 Ti, it reportedly produces a substantial improvement under particular multi-worker conditions.

On my 5070 Ti, under the same worker count and nonce count, it produces a substantial regression.

So I don’t think the evidence supports the broad statement:

“autovector_g is an improvement at the worker counts Benchmarkers actually run.”

A more accurate statement would be:

“autovector_g can improve performance under particular hardware and concurrency conditions, while providing no benefit or materially regressing performance under others.”

That is quite a different claim.

And I think this is probably one of the clearest examples in this thread of why @xnico31’s proposal for protocol-level measurement makes sense.

We now have two people testing the same exact-argmin implementations under the same 40 nonce / 4 worker condition and reaching opposite conclusions because the hardware differs.

TIG obviously cannot realistically benchmark every submission across every GPU generation and every possible operating configuration.

So perhaps the practical answer is a single canonical reference environment per challenge, where derivatives are measured against their declared parent under identical controlled conditions.

Benchmarkers can then remain free to use whichever implementation performs best on their own hardware.

That separates two questions which I think are currently being conflated:

“Is this profitable on my hardware?”

and

“Did this submission demonstrate a measurable algorithmic improvement?”

Those do not necessarily have the same answer.

And that reinforces the wider question I’ve been raising:

before a derivative is economically rewarded as an optimisation, shouldn’t TIG establish what measurable improvement it actually contributed on a defined reference environment?

Screenshot of the test_algorithm results included to prevent any suspicion of doctored results

[Newman]

I boldly assume that… can we do the following:

Can we change the rules of the entire benchmarker.

Let the TIG official, the community, or AI make an automatic algorithm assignment (including hyperparameters).

Even the benchmarker cannot autonomously select algorithms and parameters, just like it cannot currently select TRACKs…

The purpose of the benchmarker is simply to provide machine computing power for algorithm testing for TIG.

[TheDoctor]

@Rootz

“autovector_g can improve performance under particular hardware and concurrency conditions, while providing no benefit or materially regressing performance under others.”

The sounds like a fair characterisation to me. Do you agree @xnico31 ?

@Rootz

We now have two people testing the same exact-argmin implementations under the same 40 nonce / 4 worker condition and reaching opposite conclusions because the hardware differs.

This seems to be what’s happening, however, this is what TIG was designed to do.

The central idea around TIG is to create a market for algorithms and use the resulting market signal to distribute rewards for Innovators. As with all markets, there is no need to decide upon a “single winner”. If 60% of people drink coke and 40% drink Pepsi, then Coke and Pepsi are compensated for providing their soft drinks roughly in these proportions.

It is essential that markets are not “winner takes all”, otherwise there is no incentive to create any product of service except that which is targeted at the majority group (Pepsi is less popular than Coke, so Pepsi gets zero reward and disappears). In other words, markets permit choice: the don’t compel people to accept whatever is preferred by the largest group of consumers. See attached excellent paper on this by Nobel Prize winner James Buchanan.

In the context of TIG, it’s easy to see how this can be an advantage. For example, China had a large number H900s GPUs due to export controls on Nvidia, with reduced memory bandwidth. This incentivised DeepSeek to develop a training algorithm that mitigated this limitation (it also improved the performance “regular” GPUs, albeit to a smaller degree).

Cost and availability of hardware vary significantly around the world, and its not easy to know what types of hardware are being used and in what proportions. This would ordinarily mean that its hard to know, as an algorithm designer/coder, which hardware to target. But in the case of TIG, Benchmarkers reveal their preferences through use, and development is guided by adoption.

Overall, having read through everyones feedback, it seems clear that these is a genuine issue in the sense that TigPool are at the point (or at least very close) where they could behave improperly, but there is no evidence that they have in fact done so. Do you agree @Rootz @xnico31 ?

buchanan1954 (1).pdf (1.3 MB)

[Rootz]

I agree with the broad point that TIG should not necessarily have a single universal winner, unless of course it is deserving of full rewards if it significantly improves in an area. Different algorithms can legitimately suit different hardware or operating conditions.

Where I don’t completely agree is applying that argument to autovector_g.

The Coke/Pepsi analogy assumes both products independently provide something consumers prefer.

My concern is what threshold of actual improvement should justify a new Code Submission receiving Innovator rewards when it is substantially derived from an existing one.

Initially TigPool showed a 3080 Ti result suggesting the L2 persistence change could provide a benefit at four workers. I tested the same 40-nonce / 4-worker condition on a 5070 Ti and found autovector_g approximately 9% slower overall.

I have since also tested the claimed behaviour on the same GPU model, a 3080 Ti, and again found no meaningful improvement.

So at this point I don’t think we can reasonably describe autovector_g as a demonstrated architecture-specific improvement simply because one set of self-reported results showed a gain.

It may be possible to find a particular environment where that cache hint helps. But I think there is an important distinction between:

“I can find a configuration where this implementation benchmarks better”

and

“I have contributed a meaningful algorithmic or implementation improvement that deserves a new stream of Innovator rewards.”

For benchmarkers, hardware-specific profitability absolutely matters. They should run whatever gives them the best throughput.

But Innovator rewards are supposed to incentivise advancement.

If an algorithm is almost entirely derived from an existing submission, I think it is reasonable to expect the new contribution to demonstrate a real and reproducible improvement before the protocol economically treats it as new progress.

Otherwise we create an odd incentive where increasingly narrow implementation changes can each become separately reward-bearing, even where their supposed improvement is difficult to reproduce or nonexistent across most operating conditions.

That is different from the market supporting two genuinely different products.

On your wider question about TigPool, though, I agree with the distinction you’re making.

I don’t have evidence that TigPool has broken the rules or deliberately run an inferior algorithm purely to manipulate adoption.

What this discussion has established is that they could carry an affiliated submission through the 25% reward threshold using their own 27.89% influence.

That is the part I think needs addressing.

I would also avoid describing the risk purely as “behaving improperly”, because nobody necessarily has to behave improperly.

A vertically integrated benchmarker/innovator can behave completely rationally within the current rules and still create a distorted incentive:

submit derivative → benchmark it heavily → create adoption → receive Innovator rewards

The issue is therefore not whether TigPool is cheating.

It is whether the protocol is sufficiently robust when the same economic entity can both submit an algorithm and provide enough of the market signal to make that submission reward-eligible.

And I think submission quality remains part of that question.

For me there are two separate safeguards needed:

1. A submission should demonstrate meaningful, reproducible improvement over the work it derives from.

2. No single benchmarker should be capable of independently supplying enough adoption to make an affiliated submission reward-eligible.

That still allows the market to support multiple algorithms where they genuinely serve different needs.

What I don’t think we should do is equate any measurable preference in any isolated configuration with new innovation deserving a separate economic reward.

The market should reward diversity where that diversity represents genuine value.

It shouldn’t make marginal resubmission itself the product.