How it works
Batch, or lot, is a group of items processed as a unit. In machine learning, the items are usually training examples, such as rows in a table, images or text snippets. In AI systems, the term also appears in batch inference, when multiple requests are gathered for processing.
During training, a batch is sent to the model to calculate predictions and measure error against the expected results. The algorithm uses that error to adjust the parameters. Instead of updating the model after each example, the system calculates an update from the batch set. A minibatch is a smaller group used within a larger data set.
Batch size is a configuration choice. Larger batches can make better use of hardware parallel processing, but require more memory. Smaller batches use less memory, but may require more steps to go through the data. The effect on training depends on the model, the infrastructure and the data set, so there is no single ideal size for all cases.
In batch inference, multiple inputs are processed together. This flow is different from responding immediately to a single interactive request. It can serve tasks that can wait, such as classifying an entire file or generating embeddings for a collection of documents.
Practical example
Imagine a set of support messages that will be classified by topic. The application splits the messages into batches, sends each group to the model and associates each result with the original message. If an input fails, the system can log the error and retry only that unit, instead of interrupting all work.
During training, data can also be shuffled and split into batches. The model processes a batch, calculates error and updates weights, then repeats the cycle with the next ones. An epoch represents a pass through the training set, although the exact way batches are organized depends on the implementation.
In an online service, batching requests can improve hardware utilization, but increases the wait for each request while the batch forms. For this reason, low latency systems may prefer smaller batches or continuous processing. The decision involves balancing response speed, processed volume and available memory.
What it's for
Batch is used to organize data processing in training, inference, import and analysis tasks. It helps control memory usage and take advantage of parallelism, in addition to making explicit how much work the system executes at each step.
When designing a pipeline, differentiate batch size from data set size. A batch is just the portion processed at a time. Also define what happens if a request in the batch fails, how results are related to inputs and whether the task can be repeated without duplicating effects.
To choose a configuration, test with representative data and observe memory, processing time and result quality. Do not automatically copy a value from another project: different models and hardware may require different adjustments.