Bullpen.News From Edwin Dorsey
Get Daily Recap Feedback

Thursday, October 1, 2026

Back to today

From X

@convequity

Convequity on X

Batch size sets the trade-off between speed and cost in AI inference. An AI server answers many users at once by bundling their requests into one batch. Our $CBRS note compares this to running a bus. There are two ways to run it, and each gives something up. 1. Fill the bus. Put as many passengers as possible on every run. More stops slow each passenger's trip, but the cost of the run is spread across more people, so the cost per passenger falls. Larger batches do the same: each user's answer is slower, and the cost per token falls because the system produces more tokens from the same hardware. 2. Carry fewer passengers and stop less. Each trip is faster, but fewer seats mean the cost of the run is spread less well, so the cost per passenger rises. Smaller batches are fast for one user and raise the cost per token. So in the analogy, speed for each user is how fast each passenger arrives. Cost per token is cost per passenger. A system set up for one gives up some of the other. When someone compares two inference systems, check the batch size behind each number. A speed figure or a cost-per-token figure quoted without it cannot be compared fairly.

Investor Letters

All letters

Toll roads in the sky

Asia's elevator boom favors maintenance and modernization providers, with Otis deriving nearly 90% of operating profit from after-sales…

OTIS $24B
Asian Century Stocks 1 click

Publications we follow

Every publication on this site, hand-picked.

110 publications

Newsletters 93

YouTube 4

Blogs 12

Seeking Alpha 41 of 198

Nominate a publication

X accounts we follow

Every account on this site, hand-picked.

72 accounts
Nominate an account

Investor letters we follow

189 firms

Sectors

Ordered by how often our editor reaches for them.