Aperion Labs

Community Compute / A closer look

THE SCIENCE OF
SMALLER INTELLIGENCE

Powerful intelligence doesn't always require powerful hardware. Sometimes it requires a smarter way to use it.

Smaller numbers. Smaller specialists. More thoughtful coordination. Explore the different ways useful intelligence can fit the machines around us—and where the trade-offs begin.

Try the model-size calculator ↓

Explore the architecture. Interactive examples use illustrative inputs.

01 / Why AI models are so large

A model is a vast collection
of learned numerical relationships.

A neural network passes information through layers of mathematical operations. Its parameters are adjustable numbers; most are weights that determine how strongly signals influence one another. Training adjusts those values using examples. They encode patterns rather than a neatly indexed library of sentences.

TextSplit into tokens
Neural networkCombine context with learned weights
Next-token probabilitiesSelect a continuation and repeat

Many language models generate one token at a time. A token can represent a word, part of a word, punctuation, or another text unit. The model scores possible continuations from its vocabulary. Fluent output is a prediction—not proof that a statement is true. Transformer attention lets a model combine information from different positions in its context. [1]

Training changes the model.

Examples produce predictions; errors guide updates to the weights. Training needs working memory for intermediate values, gradients and optimizer state, as well as the weights themselves. It is a different workload from answering a question.

Inference uses the model.

Inference runs already-trained weights on new input. A normal conversation does not automatically retrain them. Conversation history, retrieved memories and temporary attention caches provide context without becoming new parameters.

One billion parameters means one billion numbers.

These bars compare ideal weight storage at 16 bits per parameter. Length is linear; 70B is 140 times the parameter count of 0.5B. Architecture and training still matter for capability.

0.5B
1 GB
7B
14 GB
14B
28 GB
32B
64 GB
70B
140 GB

Derived arithmetic, decimal GB. Weights only; actual runtime memory is higher.

Go deeper: why memory and GPUs matter

A processor must load weights and perform large matrix operations. GPUs perform many arithmetic operations in parallel and often have high memory bandwidth. Larger dense models usually move more weight data and do more arithmetic per generated token. Long contexts and multiple simultaneous requests add further work and memory. A model that spills beyond fast memory may become impractically slow even when it technically runs.

Parameter count alone does not predict speed. Architecture, batch size, prompt length, precision, kernels and how weights are placed across memory all change the result. Sparse models add another distinction: stored parameters versus parameters used for each token.

On discrete GPU systems, transfers between system RAM and GPU memory cross an interconnect such as PCIe. Available lanes, link bandwidth and transfer overhead can limit the benefit of moving work to another device. Capacity alone is not enough: measure the complete job, including data movement. NVIDIA: host/device data transfer.

02A / Making models smaller · Quantization

Keep the pattern.
Use fewer bits to describe it.

Imagine recording a measurement to fewer decimal places. Quantization similarly represents weights using a smaller set of values, often with a scale for each group of weights. That can reduce storage and memory traffic. It also introduces error. The challenge is preserving the relationships that matter for the task. [2]

Model-size calculator

3.50 GB3.26 GiB · ideal packed weights

One eighth of the 32-bit weight storage.

Weight storage is only the starting point.

Actual files may include quantization scales, metadata, padding and layers kept at higher precision. During execution, memory also holds activations, scratch buffers, runtime code and context caches.

A longer conversation or more simultaneous users can require much more context memory. CPU/GPU offloading adds placement and transfer considerations. This calculator is not a hardware compatibility test.

Formula: parameters × bits ÷ 8 = ideal weight bytes. GB uses 10⁹ bytes; GiB uses 2³⁰ bytes.

Ideal weight storage in decimal GB
Model32-bit16-bit8-bit4-bit3-bit2-bit
0.5B210.50.250.18750.125
7B281473.52.6251.75
14B56281475.253.5
32B128643216128
70B280140703526.2517.5
Compare 32, 16, 8, 4, 3 and 2 bits

32-bit floating-point formats such as FP32 provide a wide numerical representation but use four bytes per weight. 16-bit floating-point formats use two bytes; FP16 and BF16 allocate their bits differently. 8-bit methods such as INT8 use about one byte per weight before overhead. Packed 4-, 3- and 2-bit weights use roughly half, three-eighths and one-quarter byte respectively, before overhead.

Bit width does not specify the whole method: calibration, grouping, outlier handling and mixed-precision layers affect quality. At very low precision, small numerical changes can erase useful distinctions. Evaluate your own prompts and tasks, especially rare cases, rather than relying on an average benchmark alone.

Can a smaller model file run more slowly?

Yes. The hardware may lack an efficient kernel for that format, or unpacking and conversion may outweigh saved memory traffic. Some runtimes dequantize weights for arithmetic; weight precision, activation precision and accumulation precision can differ. Saving weight memory does not guarantee proportional latency or energy improvements. Formats and backends must be compatible. [6]

02B / Making models smaller · Bytes

The same words.
A different path through the model.

A conventional tokenizer maps text to IDs in a learned vocabulary. An embedding table turns those IDs into vectors; an output layer scores the next token. Larger vocabularies can shorten sequences, but their embedding and prediction tables occupy space.

UTF-8 represents text as bytes with 256 possible values. A model that predicts bytes can use a small output alphabet, usually with additional special symbols. Unicode characters may occupy several bytes. Byte-level BPE is different: it merges byte sequences into a larger token vocabulary. [3]

Conventional token path

Tiramisu

Paper Figure 2 example; special tokens omitted.

Token IDs → embedding vectors → transformer → vocabulary scores

Byte representation

UTF-8 bytes shown in hexadecimal.

Byte IDs → embedding vectors → transformer → byte scores

Animation illustrates representation, not execution speed. Non-research token splits are illustrative, not output from a production tokenizer.

Go deeper: where vocabulary memory goes

For vocabulary size V and hidden width d, one embedding matrix contains V × d weights. If input embeddings and the output head are separate, both matrices count; tied weights share a matrix. As an arithmetic illustration, 100,000 entries at width 2,048 require 204.8 million weights per matrix, compared with 524,288 for 256 entries. This is not a whole-model estimate.

Attention and feed-forward layers remain. Shrinking the alphabet does not erase those weights, and longer byte sequences can increase attention work, context memory and sequential prediction steps. Architectures that group bytes into patches address a different part of that trade-off. A token model cannot generally be converted by simply deleting its vocabulary table. [3]

Research lens / The Tiramisu example

Teaching a byte model from a token model

In Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models, “Tiramisu” is an example word, not a model name. Marathe and colleagues convert teacher token probabilities into byte-level training targets. Marginalize-It approximates later byte probabilities; End-Of-Token adds a boundary symbol to preserve probability mass. [4]

How the research connects bytes and distillation

The study compares token-based and byte-based students while holding the transformer layer size constant. This helps separate the vocabulary tables from the rest of the model. It also compares ordinary training on examples with learning from a teacher’s probability distributions.

The key idea is translating the teacher’s uncertainty into the student’s smaller alphabet. An end-of-token marker records where a teacher token ends; this avoids losing probability assigned to shorter token alternatives. That marker also adds sequence positions to process. Smaller vocabulary tables and fewer prediction steps are different goals, and a design can improve one while making the other harder. [4]

02C / Making models smaller · Distillation

Learn the useful behavior.
Leave behind unnecessary size.

Distillation trains a student using a teacher’s outputs or probability distributions. A probability distribution can communicate uncertainty and similarities between answers, beyond a single correct label. The student learns its own parameters; it does not copy every teacher weight. [5]

TeacherExamples, responses, soft targets
Student trainingLearn a narrower set of behaviors
Held-out evaluationCheck quality before choosing the student

Voice commands

Map transcribed phrases to intents such as pause or remind. Speech recognition itself requires a compatible audio model or separate stage.

Task classification

Recognize whether a request is a search, summary or action. Measure wrong-route rates, not just familiar examples.

Tool selection

Choose from allowed tools and propose structured arguments. A separate permission check decides whether execution is allowed.

Device control

Translate an approved intent into a narrow command. Unknown or ambiguous commands should ask for clarification.

Memory retrieval

Learn to rank candidate notes or rewrite a search query. The memory store and its access rules remain separate.

Everyday conversation

Practice tone and common exchanges. Unfamiliar topics and complex reasoning can still require another model.

Go deeper: what does the student fail to inherit?

A teacher may possess capabilities that never appear in the training examples. A student can reproduce mistakes, learn shortcuts, lose rare skills or be confidently wrong outside its training domain. Distillation quality depends on teacher quality, student capacity, data coverage and the learning objective. It also costs compute to generate examples and train the student.

Use separate training and evaluation data, test adversarial and ambiguous cases, and measure task completion. The ability to imitate a tool call does not confer the authority to make it. Deployment still needs permissions and validation.

02D / Making models smaller · Sparse computation

A larger collection.
A smaller active path.

A mixture-of-experts architecture routes internal representations through selected expert blocks. Shared layers still run. The router chooses numerical components; “expert” does not necessarily mean a clearly labeled human specialty.

Illustrative sparse layer: choose two of eight

Expert 1 · active
Expert 2 · active
Expert 3
Expert 4
Expert 5
Expert 6
Expert 7
Expert 8

Experts 1 and 2 active. All eight expert blocks still need storage somewhere.

For a concrete architectural example, Mixtral selects two feed-forward experts per layer per token. Other expert weights remain part of the model even when they are not selected for that token. Total parameters describe the stored collection; active parameters describe the path used for a particular computation. [7]

Go deeper: why inactive does not mean absent

Unless weights are fetched on demand, inactive experts remain resident in memory. Fetching them from storage or another device has a cost. Expert imbalance can leave some processors busy while others wait; distributed experts exchange representations over the network. Sparse activation lowers some arithmetic, but shared layers, routing and communication remain. Total storage depends on all weights and their precision.

03 / Running on your devices

Inference can happen
on the computer beside you.

A local inference runtime loads a compatible model and executes it on a CPU, GPU or supported accelerator. CPU execution uses system RAM; dedicated GPUs commonly use their own VRAM. Some systems share physical memory. Splitting a model between CPU and GPU can make it fit while adding transfer costs. [6]

Illustrative device classes—not measured benchmarks or purchase recommendations
HardwareUseful starting pointWhat to check
Inexpensive desktopSmall quantized specialists; light request volumeFree RAM, CPU instructions, memory channels and power draw
Compatible laptopShort interactions and private nearby tasksShared memory, battery drain, sustained cooling and runtime support
Mini PC or single-board computerSmall compatible models and lightweight orchestrationRAM limits, instruction support, storage and sustained cooling
Compatible mobile or gaming systemSupported runtimes, small specialists or an interface to another resourcePlatform access restrictions, memory, thermal limits and background-work policies
Refurbished desktopSuitable quantized workloads on existing hardwareDriver and runtime compatibility, reliability and whole-system power
GPU workstationLarger models or concurrent requests when memory allowsVRAM per GPU, supported kernels, interconnects and wall power

Capacity is not speed.

Enough RAM means the data can fit, not that responses will arrive quickly. Memory bandwidth measures how fast data moves. Single-request generation often spends substantial time reading weights; a faster arithmetic unit alone may not fix that bottleneck.

Speed is not one number.

Time to first token includes loading, queuing and prompt processing. Tokens per second describes generation throughput, but token definitions vary. Compare equal tasks, output lengths and quality—not raw token rates across incompatible tokenizers.

Go deeper: formats, context and energy

GGUF is a model container used by runtimes such as llama.cpp; it can hold different quantization types. The extension does not guarantee that a device supports a model architecture or a fast execution path. Check backend support and measure the actual build.

Context caches grow with sequence length and concurrent sessions. Reserve memory for the operating system and other applications. Record cold starts separately from warm runs. Measure whole-system energy over completed tasks; a lower-power device running much longer can use more energy than a higher-power device finishing quickly.

04A / The Community Compute network

Sharing work is not the same
as splitting a thought.

Task-level distribution

Request A → PC 1Request B → PC 2

Independent requests go to different machines, each running a suitable complete model. Little intermediate state crosses the network. This can suit queues of classification, extraction or other independent jobs.

Model parallelism

One layer / shard A↔ shard B

Parts of a single model execute across devices. Tensor parallelism splits operations within layers and can require frequent synchronization. Fast local interconnects are especially valuable.

Pipeline parallelism

Early layers → PC 1Later layers → PC 2

A form of model parallelism: different stages run on different devices. Intermediate activations move between stages. Multiple requests can fill a pipeline, but dependencies and empty pipeline slots add delay.

Expert distribution

RouterExpert hosts A / B

Sparse experts live on different machines. Tokens are dispatched to selected experts and results combined. Uneven routing and network exchanges can limit the benefit of reduced arithmetic.

Distant computers do not behave like one enormous GPU. A model that must wait on the internet at every layer or generation step can lose more time to communication than it saves in computation.

Go deeper: network and failure budgets

Bandwidth limits how much data can move per second. Latency is the delay before a response arrives; small frequent exchanges can be latency-bound even when bandwidth looks ample. Synchronization waits for participating devices, often including the slowest one. [8]

Nodes disconnect, owners pause participation and requests time out. Coordinators need health checks, bounded retries, cancellation, timeouts and duplicate-work handling. Retry inference separately from irreversible tool actions. Task-level routing can avoid many per-layer exchanges, but still pays for transferring input, output and any required model files.

04B / The Community Compute network · Routing

Match the task.
Respect the owner.

An intelligence router can filter resources by permission, privacy, model capability, available memory and availability, then compare eligible choices by latency, energy or cost. It should be able to say that no resource qualifies. More computing power never grants more authority.

Persistent agent identity + owner permissions + memory references
Independent of the inference provider

Explore an illustrative request

Owner permits these locations
Simulate unavailable resources

Privacy & resource filter → router

Selected: Local device

Selected model → result validation → same persistent agent

No requests leave this page. Memory budgets and preference ranks are invented teaching values, not hardware specs, prices or measurements.

Inspect the simulation’s assumptions

Assumed runtime memory: local 8 GB, Private Boost 24 GB, community 16 GB, cloud 80 GB. Command, retrieval, reasoning and batch examples need 2, 6, 20 and 12 GB respectively. Capability is a simple teaching tier, not an intelligence score. Latency, energy and cost use ordinal preference ranks. Real routing needs measured profiles for the exact model, quantization, context, concurrency and hardware.

Hard requirements are checked first. A cheap external node cannot override a local-only privacy rule or an unchecked permission. If no node qualifies, the request waits, is reduced in scope with owner agreement, or stops—it must not silently relax the rules.

Go deeper: the same agent across different models

Identity, owner permissions and persistent memories belong to the agent layer. An inference provider receives only the context needed and authorized for the request; changing that provider should not require changing the agent’s identity. Memory writes and tool actions require their own policy checks.

Encrypting data at rest or in transit does not hide plaintext from a host executing ordinary inference on it. Sensitive workloads may need to stay local or on trusted private resources. Result validation can check schemas, evidence and task-specific tests; it cannot guarantee that every answer is correct. Failed validation may justify another eligible model, subject to the same privacy and spending limits.

04C / The Community Compute network · Devices

Useful work can begin
with a machine already owned.

Desktop computers

Unused hours can suit bounded tasks if free memory and CPU support are sufficient.

Gaming PCs

A compatible GPU can contribute between gaming sessions; VRAM may be the limiting resource.

Workstations

Larger memory pools may suit heavier models, subject to software support and other workloads.

Home servers

Predictable availability can help queues, but existing home services take priority.

Compatible laptops

Plugged-in windows and temperature limits matter. A portable device is not automatically suitable for continuous load.

Dedicated nodes

A device reserved for inference can offer predictable resources while still needing maintenance and monitoring.

Not every machine runs every model. Check usable RAM and VRAM, processor instructions, backend support, cooling, electricity and reliable connectivity. Two separate memory pools do not become one seamless pool just because their capacities add up.

Go deeper: voluntary participation and resource limits

Owners can define available hours, maximum memory, power or temperature limits, concurrency and network budgets. Participation should be pausable, with the owner’s work taking precedence. Isolation must protect the host’s files and services; permission to use compute is not permission to inspect personal data.

A coordinator needs to detect when resource limits change and avoid assigning work a device cannot finish. Downloading model weights, keeping versions compatible and restarting interrupted jobs all consume resources. Evaluate useful completed work, not merely the number of connected computers.

05 / What it means

Efficiency is a result
you demonstrate.

Smaller weights can reduce memory demand. Appropriate model selection can avoid unnecessary work. Routing can improve utilization. But networking, coordination, retries, cooling and inefficient hardware can erase those gains. Idle capacity is not free energy.

Intelligence per dollar

Quality-qualified completed tasks divided by total cost: electricity, allocated hardware, network, operation and provider charges.

Intelligence per watt

Quality-qualified task throughput per watt. Over a measurement interval, tasks per joule expresses energy efficiency directly.

Intelligence per gigabyte

Defined task quality or useful throughput relative to a clearly specified memory footprint. Weight storage and peak runtime memory are different denominators.

“Intelligence” is shorthand here. Specify the task, dataset, quality threshold and latency target. There is no single universal intelligence unit that makes unlike workloads comparable.

A practical measurement protocol
  1. Choose representative tasks and define success before comparing systems.
  2. Keep prompts, output requirements, context lengths and concurrency comparable. Include difficult cases and failures.
  3. Measure wall-clock latency, accepted completions, peak memory and whole-system wall-plug energy. Separate cold and warm runs.
  4. Count coordinator and network overhead, retries and cooling within a stated boundary. Distinguish incremental power from total system power.
  5. Include electricity rates, hardware cost allocation and operational charges. Do not double-count electricity already included in a provider price.
  6. Repeat enough trials to capture variability; report the setup and uncertainty. A benefit for one task does not establish a benefit for every workload.

For example, energy in kWh × price per kWh gives electricity cost. Divide total cost by successful, quality-qualified tasks—not attempts—to estimate cost per useful inference. Environmental comparison additionally depends on electricity sources, equipment lifetime and the alternative displaced. MLPerf provides a useful example of workload-specific quality and wall-power methodology. [9]

06 / The bigger picture

HOW APERION BUILDS
MORE CAPABLE INTELLIGENCE

Smaller models. Smarter placement. The same continuing relationship.

These techniques address different bottlenecks. They can sometimes be combined, but their benefits do not simply multiply. Training choices shape the model; runtime choices determine where and how it executes.

01User requestGoal, context and boundaries
02AtlasInterpret the task
03Intelligence routerFilter eligible resources
04Suitable inferenceLocal, private, community or hosted
05Tools and contextRetrieve relevant information; authorize actions
06Atlas responseCheck the result and communicate limits

This illustrates an intended architecture, not a report of deployed capability. Quantized open-weight models, distilled specialists and sparse experts can serve different tasks. Byte-level research explores another representation choice. Retrieval supplies relevant context; supervised tools perform authorized work. Stronger hosted inference can be selected when it meets the task’s requirements and permissions.

How the pieces become a practical intelligence system

A router considers task complexity, measured model quality, hardware support, free memory, latency, cost, privacy preferences, network availability and authorized permissions. Reject ineligible routes before optimizing among eligible ones. Retrieved information must respect account and memory boundaries, and tools require their own authorization.

Atlas is Aperion’s public assistant: each user’s account history must remain isolated. A personally owned Eclipse agent has a different identity anchored to its owner’s Key. Aurora can provide an interface to either; switching inference must not mix these identities or grant additional authority.

Compression and coordination do not automatically produce frontier capability. Evaluate actual task quality, failure cases, latency, peak memory, energy and full cost. A small specialist can be excellent at a narrow job without matching a general model across every task.

OWNER → PERSISTENT IDENTITY & MEMORY → PERMISSION BOUNDARY

01 · Quantization

Use fewer bits for weights. The model needs less weight storage; runtime overhead and quality trade-offs remain.

Select any stage to inspect it. Playback is optional and pauses when the page is hidden.

VALIDATED RESULT → SAME AGENT
Different inference resources. Identity and authority stay with the owner’s agent layer.
“The future of intelligence isn't just about building bigger machines. It's about making better use of the machines we already have.”

Independence, resilience, accessibility and sustainability are goals to test through useful work: owner-controlled hardware, replaceable inference and infrastructure that can draw on suitable existing resources.

JOIN THE RESISTANCE →

Explore the proposed Aurora and Community Compute participation model. This link does not enroll a device or enable workloads.

Read the originals

Sources & technical references

Research and documentation checked October 9, 2026. Architecture diagrams and router scenarios are educational illustrations. Calculated storage values are derived arithmetic, not benchmarks.

  1. Vaswani et al. — Attention Is All You Need. Transformer architecture and attention.
  2. Frantar et al. — GPTQ. Post-training weight quantization and precision trade-offs.
  3. Xue et al. — ByT5. Byte representations and the cost of longer sequences.
  4. Marathe et al. — Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models, arXiv:2609.12303v1 (September 11, 2026). Sections 2–3 and Figure 2; byte-level distillation methods and study design.
  5. Hinton, Vinyals & Dean — Distilling the Knowledge in a Neural Network. Teacher/student learning.
  6. llama.cpp official project. CPU/GPU backends, model formats and quantized inference. Support varies by build and device.
  7. Jiang et al. — Mixtral of Experts. Total versus active parameters and sparse routing.
  8. PyTorch distributed documentation. Communication, collectives and distributed execution.
  9. MLCommons — MLPerf Inference methodology. Defined quality targets, latency constraints and whole-system power measurement.