BesTom service

RK182X Multi-Card Cascade Inference
Layer-wise segmentation across 2 or 4 AI cards — scale on-device LLM capacity and long-prompt prefill on one host

A single RK182X AI card has a hard capacity ceiling: the model must fit in its local memory, and every token passes through one device. The cascade architecture breaks that ceiling by cutting the Transformer into consecutive decoder-layer segments, placing each segment on its own card, and letting the host orchestrate the chain. Capacity scales with the number of cards; only intermediate hidden states travel between segments — model weights never leave their card.

2 / 4 cardsDecoder-layer segmentationPCIe / PCIe Switch / USB 3.0OpenAI-compatible serving

One host, up to 4 AI cards, up to the 27B model class on-premise — no cloud round-trip, no re-architecting of your front-end devices.

Why cascade

What the cascade buys you

Three independent gains, all on the same host platform.

Break the single-card capacity ceiling

Capacity is no longer bounded by one card's local memory. Splitting the decoder stack across 2 or 4 devices moves the practical ceiling from the small-model class into the 27B class — while staying fully on-premise.

Accelerate long-prompt prefill

Long inputs are split into buckets and pipelined through the segment chain, so segment N processes bucket k while segment N−1 already works on bucket k+1. Long-prompt throughput improves instead of queueing behind a single device.

Flexible 2- or 4-card, three interconnect options

Direct PCIe lanes, a PCIe switch, or a shared USB 3.0 hub. Pick by available host ports, enclosure budget and BOM — the same segmentation and runtime work across all three.

How it works

Cascade principle

One host orchestrates; every card owns a contiguous slice of the decoder stack.

Figure 1 · Decoder-layer segmentation and cascade execution

1 · Host tokenises

Tokeniser and embedding map run on the host CPU. Text becomes token IDs and then embedding vectors.

2 · Host schedules segments

Device discovery, segment scheduling and bucket queues live on the host. It decides which segment runs where, and in what order.

3 · Segments execute in chain

Card 0 runs decoder layers 0..k, hands its intermediate hidden states to card 1, which runs k+1..m, and so on to the final segment.

4 · Final segment emits logits

The last segment carries the trailing decoder layers plus final norm and LM head. Logits return to the host for sampling, and the decode loop repeats.

Scheduling rule: stage_count = number of segments, and available devices ≥ stage_count. Intermediate hidden states are transferred host-side; model weights stay resident on their own card.

Hardware options

Three ways to attach the cards

All three carry the same segmentation model and the same runtime API — choose on ports, enclosure space and BOM.

Figure 2 · Three interconnect topologies for 2 / 4 cards

Interconnect	Uplink bandwidth	Host ports used	Best fit
Direct PCIe, one lane per card	PCIe 2.0 x1 × N	N	Host has spare PCIe lanes and you want the simplest possible topology — no switch, no hub, no shared bus.
PCIe switch expansion	PCIe 2.0 x1 / PCIe 3.0 x4	1	Host PCIe is scarce. One uplink feeds all cards, and the switch keeps per-card bandwidth independent.
USB 3.0 hub expansion	USB 3.0 × 1 (shared)	1	Fastest integration path, including retrofitting existing machines. Cards share one USB 3.0 hub.

Measured on the reference platform, PCIe and USB 3.0 land within a few percent of each other on long-prompt prefill; PCIe keeps a small edge of roughly 2–3% on time-to-first-token and prefill. Short-context workloads show very little difference, so USB 3.0 is often the pragmatic first choice.

Execution model

Prefill pipeline and decode loop

The two phases use the segment chain very differently — which is exactly where the cascade pays off.

Figure 3 · Prefill bucket pipeline vs. sequential decode loop

Prefill · bucket pipeline

Long input is chopped into buckets. Buckets are queued across the segment chain and overlap: while segment 0 works on bucket 1, segment 1 is still finishing bucket 0. This overlap is what lifts long-prompt throughput — the win grows with input length.

Decode · sequential chain

Decode produces one token at a time. Each token walks the entire chain segment by segment, the last segment emits logits, and the host samples before the next step. The chain is strictly sequential here, so decode speed tracks the slowest segment.

How the model is split

Segmentation is not a blind 50/50 cut. Weight size and memory footprint differ sharply between layers (attention blocks versus MLP blocks), so the toolkit balances quantised weight volume and per-segment memory to even out the stages. Qwen3.5-9B, for example, splits its 32 decoder layers into segments of 19 and 13 layers rather than 16 / 16.

Segment / card	Decoder layers	Artefacts
Segment 0 · card 0	Layer 0 – 18 · 19 decoder layers	`segment0.rknn + segment0.weight`
Segment 1 · card 1	Layer 19 – 31 · 13 decoder layers + final norm + LM head	`segment1.rknn + segment1.weight`

The toolkit produces the segmentation, but the parameters that matter — segment count, target context, quantisation scheme, connection type — are engineering decisions that must be made against your latency target, not defaults.

Capability envelope

Which configuration reaches which model class

Order-of-magnitude guidance, not a guaranteed benchmark. Concrete throughput depends on quantisation, context length and batch size.

Configuration	Chip & model class	Verified context
2 × RK182X	2 × RK182X · 9B	4K / 8K
4 × RK182X	4 × RK182X · 27B	4K / 8K
Host platform	RK3588 / RK3576 / x86 (Windows)	RKNN3 Runtime · rkllm3-server

Time-to-first-token, prefill throughput, decode throughput, system power and host memory footprint all vary substantially with quantisation scheme and configuration. We publish measured numbers against a specific configuration on request — tell us your model, context and card count and we will share a matching test report.

BesTom service

How BesTom delivers a cascade design

This is not an off-the-shelf box. The card stack, the switch, the hub and the enclosure are engineering decisions, and that engineering is what we take on.

BesTom is a Rockchip full-line IDH design partner. We turn the cascade architecture into a validated product for your scenario — from model feasibility through to pilot production.

Feasibility and segmentation strategy

We analyse your model structure (decoder layers, embedding, final norm, LM head, special cache or state), then fix the target: card count, connection type, context length and quantisation scheme against your latency budget. The segmentation plan comes out of this step, not from a default.

Carrier board and interface design

PCIe direct, PCIe switch or USB 3.0 hub — we design the carrier board, clock and power tree, connector placement and signal integrity for the option your enclosure allows.

Thermal and mechanical integration

Multi-card density is a thermal problem before it is a compute problem. We size heatsinking and airflow for sustained load, and integrate the stack into your mechanical envelope and mounting constraints.

BSP, driver and runtime alignment

Host BSP integration, device enumeration, runtime alignment and input/shape/sampling consistency across all segments — including API compatibility with your existing application layer.

Validation and pilot production

Prefill and decode measurement, stability and endurance runs under sustained load, power characterisation, and DFM review moving into pilot build and mass production.

What you receive

Feasibility and segmentation report · carrier-board schematic and PCB · host BSP and device-tree integration · segmentation and conversion scripts · runtime alignment notes · validation report with prefill, decode, stability and power data for your configuration · DFM review and pilot build support.

Development

Five steps from your model to cascaded silicon

The toolchain turns a HuggingFace or PyTorch checkpoint into per-segment artefacts — the work is in the structure analysis and the runtime alignment, not the conversion command.

Figure 4 · Model to cascaded silicon in five steps

Step

Step 1Structure evaluation

Step 2Segment export

Step 3RKNN conversion

Step 4Runtime alignment

Step 5Target-card validation

What an evaluation covers

Item	What it does	Output
Segment export and conversion	We confirm how your model is actually built before anything is cut — decoder layer count, embedding, final norm, LM head, and any special cache or recurrent state. Then we agree the target: model version, context length, card count and connection type, plus the prefill and decode performance you need.	`segment.rknn` `segment.weight`
Reference implementation	`rknn3-model-zoo/examples/multicard`	demo + scripts
Serving layer	`rkllm3-server`	OpenAI-compatible API

Anything that exports cleanly to ONNX with a regular decoder structure can be segmented — the pipeline is not tied to one model family. Unusual architectures with custom cache or recurrent state need extra alignment work, which is assessed in step 1.

FAQ

Cascade questions engineers actually ask

Short answers to the questions that come up in every evaluation.

Why not just use one bigger card?

The ceiling on a single card is its local memory. Cascade removes that ceiling by putting a slice of the decoder stack on each of several devices, so capacity grows with card count instead of being capped by one package. It also lets you start with 2 cards and add 2 more later without changing the architecture.

How many cards are supported?

2-card and 4-card configurations are the supported set today. In principle the scheduler only requires available devices ≥ segment count, so the design is not hard-coded to four.

How much data moves between cards?

Only intermediate hidden states — the per-layer activations at the segment boundary. Model weights stay resident on their own card and are never shipped across the link, which is why link bandwidth has a limited effect on throughput.

Do I need PCIe, or is USB 3.0 enough?

Both work. On long-prompt prefill PCIe holds roughly a 2–3% edge over USB 3.0; on short contexts the difference is small. If host PCIe lanes are scarce or you are retrofitting an existing chassis, USB 3.0 is usually the pragmatic choice.

What host do I need?

RK3588 / RK3576 running ARM64 Linux or Android, or an x86 Windows host. The host handles tokenisation, embedding, scheduling, logits and sampling; inference proper runs on the cards.

How much host CPU and memory does it consume?

Host overhead is low — it is orchestration, not inference. The exact CPU and memory footprint depends on context length and card count, and we share measured figures for your target configuration rather than quoting a single headline number.

Which models can be cascaded?

Anything that exports to ONNX with a regular Transformer decoder structure. Balancing a 32-layer decoder across 2 cards, and much larger stacks across 4, is a routine configuration; models with custom cache or recurrent state need extra alignment work.

Can it serve as an API, or is it an embedded library?

Both patterns exist. The serving layer is already adapted to expose an OpenAI-compatible API, so existing application code can point at the cascade as if it were any other model endpoint.

Edge AI Host solution

Single-card RK3576 / RK3588 + RK182X on-device LLM reference design, with HMIs and sensors as thin clients.

Rockchip SoM and core boards

RK3588 / RK3576 / RK3506 / RK182X module and development-board details.

Talk to an engineer

Bring your model, target context and card count — we will assess feasibility and segmentation strategy together.

—

Basis of the figures on this page

Architecture parameters (card counts, decoder-layer ranges, interface versions, context buckets) are properties of the cascade design and the vendor reference toolchain.

Capability tiers (9B class on 2 cards, 27B class on 4) are order-of-magnitude envelopes from reference-platform evaluation, not guarantees for a specific model, quantisation or context.

Measured performance and power figures are deliberately not published here. They depend heavily on quantisation and configuration, so we share a test report matched to your target configuration on request.

Have a model that no longer fits?

Send us the model, the context you need and your latency target. We will come back with a card count, a connection type and a segmentation strategy.

Request an assessment

tomyao@bestom.net

RK182X Multi-Card Cascade InferenceLayer-wise segmentation across 2 or 4 AI cards — scale on-device LLM capacity and long-prompt prefill on one host