PCIe Address Translation Unit (ATU) in Linux

PCIe Address Translation Unit (ATU) in Linux

The PCIe Address Translation Unit (ATU) is a vital component within the PCIe bus architecture, responsible for translating virtual addresses used by the CPU into physical addresses accessible by PCIe devices. This translation process ensures proper communication between the CPU and PCIe devices, even if their address spaces overlap.
1. The Role of ATU in PCIe:
Address Translation: The ATU translates virtual addresses generated by the CPU into physical addresses used by PCIe devices. This translation is essential to prevent address conflicts between the CPU's memory space and the PCIe device's memory space.
Security: The ATU can enforce security measures by restricting access to specific PCIe devices based on defined access rights.
Performance Optimization: The ATU can optimize PCIe traffic by caching frequently accessed addresses, reducing the need for repetitive translations.
2. How ATU Works in Linux:
In Linux, the ATU is implemented within the PCIe driver framework. The following steps outline the process of address translation:
Virtual Address Generation: The CPU generates a virtual address for accessing a PCIe device.
Translation Request: The PCIe driver intercepts the virtual address and initiates a translation request to the ATU.
Translation Process: The ATU checks its internal tables (e.g., page tables) to find the corresponding physical address for the virtual address.
Address Conversion: The ATU converts the virtual address to the physical address and returns it to the PCIe driver.
Access to PCIe Device: The PCIe driver uses the physical address to communicate with the PCIe device.
In the context of PCIe, there are typically two types of address translations:

1. Inbound Translations: These translate addresses from the PCIe domain to the system memory domain. This is used when data is being written from the PCIe device into the system memory.

2. Outbound Translations: These translate addresses from the system memory domain to the PCIe domain. This is used when data is being read from the system memory and sent to the PCIe device.

For a Root-SoC, containing the PCIe Root Complex:
1] An Outbound Request travels downstream to the Endpoint.
2] An Outbound Completion returns upstream to the Root Complex.
3. An Inbound Request travels upstream to the Root Complex.
4. An Inbound Completion returns downstream to the Endpoint

For an Endpoint-SoC:
An Outbound Request travels upstream to the Root Complex.
An Outbound Completion returns downstream to the Endpoint.
An Inbound Request travels downstream to the Endpoint.
An Inbound Completion returns upstream to the Root Complex

3. Implementing ATU in Linux:
The ATU functionality is mainly implemented within the Linux kernel and is not directly accessible through user-space code. However, several system calls and configuration tools interact with the ATU:
iommu_map(): This system call maps a virtual address range to a physical address range.
iommu_unmap(): This system call unmaps a previously mapped virtual address range.
iommu_flush(): This system call flushes the ATU cache for a specific address range.
iommu_get_info(): This system call retrieves information about the available IOMMU hardware.
4. Example Code in Linux:
While direct ATU manipulation is typically handled by the kernel, we can demonstrate how to access and configure the ATU using user-space code:
#include <stdio.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <linux/ioctl.h>
#include <linux/iommu.h>

int main() {
  // Get information about available IOMMU hardware
  int iommu_fd = open("/dev/iommu", O_RDWR);
  if (iommu_fd < 0) {
    perror("Error opening IOMMU device");
    return 1;
  }
  struct iommu_info info;
  if (ioctl(iommu_fd, IOMMU_GET_INFO, &info) < 0) {
    perror("Error getting IOMMU information");
    close(iommu_fd);
    return 1;
  }
  printf("IOMMU version: %d\n", info.version);

  // Map a virtual address range to a physical address range
  struct iommu_map map;
  map.virt_addr = (unsigned long)malloc(4096);
  map.phys_addr = 0x1000;
  map.size = 4096;
  if (ioctl(iommu_fd, IOMMU_MAP, &map) < 0) {
    perror("Error mapping virtual address range");
    close(iommu_fd);
    return 1;
  }
  printf("Virtual address: 0x%lx\n", map.virt_addr);
  printf("Physical address: 0x%lx\n", map.phys_addr);

  // ... use the mapped memory space ...

  // Unmap the virtual address range
  if (ioctl(iommu_fd, IOMMU_UNMAP, &map) < 0) {
    perror("Error unmapping virtual address range");
    close(iommu_fd);
    return 1;
  }

  close(iommu_fd);

  return 0;
}

5. Limitations of ATU:
Hardware Dependency: The ATU functionality is highly dependent on the specific PCIe hardware and its capabilities.
Performance Overhead: The translation process can introduce overhead, particularly for high-bandwidth applications.
Complexity: Understanding and configuring ATU requires a thorough understanding of the PCIe bus architecture and Linux kernel internals.
Conclusion:
The PCIe ATU is an essential component for seamless communication between the CPU and PCIe devices. Its translation capabilities ensure address compatibility, security, and performance optimization. While its implementation is handled within the Linux kernel, understanding its role and interaction with system calls and configuration tools is crucial for developers working with PCIe devices in Linux.

Comments