How it works

A tensor is a generalization of vectors and matrices to N dimensions. A scalar is a 0D tensor, a vector is 1D, and a matrix is 2D. In AI, tensors represent images, text converted to numbers, and model weights.

Tensor operations such as multiplication, convolution, and attention are accelerated by GPUs and TPUs, which process thousands of operations in parallel. Broadcasting allows operations on tensors with different shapes automatically.

Shape errors are common: incompatible dimensions cause runtime failures. Understanding the shape of each tensor is essential for debugging.

Practical example

A color image of 224x224 pixels becomes a tensor with shape (3, 224, 224): 3 channels (RGB), 224 height, 224 width. A batch of 32 images has shape (32, 3, 224, 224).

Python
import torch

img = torch.randn(3, 224, 224)  # tensor 3D
batch = torch.randn(32, 3, 224, 224)  # tensor 4D
print(batch.shape)  # torch.Size([32, 3, 224, 224])

a = torch.randn(3, 1)
b = torch.randn(1, 4)
c = a + b  # shape (3, 4)

What it's for

Representing and manipulating data in AI models. Understanding tensors is essential for working with frameworks like PyTorch and TensorFlow, debugging shape errors, and optimizing memory.

Tensors are also the foundation of GPU operations: the more efficiently tensors are used, the faster training and inference become.