Memory management

A crucial aspect of working with a GPU is managing the data on it. The CuArray type is the primary interface for doing so: Creating a CuArray will allocate data on the GPU, copying elements to it will upload, and converting back to an Array will download values to the CPU:

# generate some data on the CPU
cpu = rand(Float32, 1024)

# allocate on the GPU
gpu = CuArray{Float32}(undef, 1024)

# copy from the CPU to the GPU
copyto!(gpu, cpu)

# download and verify
@test cpu == Array(gpu)

A shorter way to accomplish these operations is to call the copy constructor, i.e. CuArray(cpu).

Type-preserving upload

In many cases, you might not want to convert your input data to a dense CuArray. For example, with array wrappers you will want to preserve that wrapper type on the GPU and only upload the contained data. The Adapt.jl package does exactly that, and contains a list of rules on how to unpack and reconstruct types like array wrappers so that we can preserve the type when, e.g., uploading data to the GPU:

julia> cpu = Diagonal([1,2])     # wrapped data on the CPU
2×2 Diagonal{Int64,Array{Int64,1}}:
 1  ⋅
 ⋅  2

julia> using Adapt

julia> gpu = adapt(CuArray, cpu) # upload to the GPU, keeping the wrapper intact
2×2 Diagonal{Int64,CuArray{Int64,1,Nothing}}:
 1  ⋅
 ⋅  2

Since this is a very common operation, the cu function conveniently does this for you:

julia> cu(cpu)
2×2 Diagonal{Float32,CuArray{Float32,1,Nothing}}:
 1.0   ⋅
  ⋅   2.0

Unified memory

The CuArray constructor and the cu function default to allocating device memory, which can be accessed only from the GPU. It is also possible to allocate unified memory, which is accessible from both the CPU and GPU with the driver taking care of data movement:

julia> cpu = [1,2]
2-element Vector{Int64}:
 1
 2

julia> gpu = CuVector{Int,CUDA.UnifiedMemory}(cpu)
2-element CuArray{Int64, 1, CUDA.UnifiedMemory}:
 1
 2

julia> gpu = cu(cpu; unified=true)
2-element CuArray{Int64, 1, CUDA.UnifiedMemory}:
 1
 2

On devices without concurrent managed access (Windows, and Jetson boards up to Orin), the CPU cannot access unified memory that is visible to the GPU while any kernel is running, even one that doesn't use that memory. CUDA.jl therefore keeps the unified memory it allocates attached to the host until it's used on the GPU, and attaches it to the host again when the CPU accesses it next, after waiting for the GPU to finish using it. That way, the CPU can access unified arrays while other tasks keep the GPU busy. This doesn't apply to memory used in a graph capture (which stays visible to the GPU until freed, because the graph can be launched at any time), to memory with implicit synchronization disabled, to memory last used on one of the default streams, or to memory wrapped with unsafe_wrap: accessing those on the CPU still requires the GPU to be idle. Attachment applies to an entire allocation, so using a wrapper of CUDA.jl-allocated unified memory on the GPU while the array it was taken from is attached to the host is not supported; use the original array on the GPU first.

Using unified memory has several advantages: it is possible to allocate more memory than the GPU has available, and the memory can be accessed efficiently from the CPU, either directly or by wrapping the CuArray using an Array:

julia> gpu[1]  # no scalar indexing error!
1

julia> cpu_again = unsafe_wrap(Array, gpu)
2-element Vector{Int64}:
 1
 2

This may make it significantly easier to port code to the GPU, as you can incrementally port parts of your application without having to worry about executing CPU code, or triggering an AbstractArray fallback. It may come at a cost however, as unified memory needs to be paged in and out of the GPU memory, and cannot be allocated asynchronously. To alleviate this cost, CUDA.jl automatically prefetches unified memory when passing it to a kernel.

On recent systems (CUDA 12.2 with the open-source NVIDIA driver) it is also possible to do the reverse, and access CPU memory from the GPU without having to explicitly allocate unified memory using the CuArray constructor or cu function:

julia> cpu = [1,2];

julia> gpu = unsafe_wrap(CuArray, cpu)
2-element CuArray{Int64, 1, CUDA.UnifiedMemory}:
 1
 2

julia> gpu .+= 1;

julia> cpu
2-element Vector{Int64}:
 2
 3

Right now, CUDA.jl still defaults to allocating device memory, but this may change in the future. If you want to change the default behavior, you can set the default_memory preference to unified or host instead of device.

Garbage collection

Instances of the CuArray type are managed by the Julia garbage collector. This means that they will be collected once they are unreachable, and the memory hold by it will be repurposed or freed. There is no need for manual memory management, just make sure your objects are not reachable (i.e., there are no instances or references).

Note that the garbage collector itself does not release GPU memory: that happens shortly after, as explained in the section on Releasing resources below.

Memory pool

Behind the scenes, a memory pool will hold on to your objects and cache the underlying memory to speed up future allocations. As a result, your GPU might seem to be running out of memory while it isn't. When memory pressure is high, the pool will automatically free cached objects:

julia> CUDA.pool_status()             # initial state
Effective GPU memory usage: 16.12% (2.537 GiB/15.744 GiB)
Memory pool usage: 0 bytes (0 bytes reserved)

julia> a = CuArray{Int}(undef, 1024);   # allocate 8KB

julia> CUDA.pool_status()
Effective GPU memory usage: 16.35% (2.575 GiB/15.744 GiB)
Memory pool usage: 8.000 KiB (32.000 MiB reserved)

julia> a = nothing; GC.gc(true)

julia> CUDA.pool_status()             # 8KB is now cached
Effective GPU memory usage: 16.34% (2.573 GiB/15.744 GiB)
Memory pool usage: 0 bytes (32.000 MiB reserved)

If for some reason you need all cached memory to be reclaimed, call CUDA.reclaim():

julia> CUDA.reclaim()

julia> CUDA.pool_status()
Effective GPU memory usage: 16.17% (2.546 GiB/15.744 GiB)
Memory pool usage: 0 bytes (0 bytes reserved)

Memory limits

If you're sharing a GPU with other users or applications, you might want to limit how much memory is used. By default, CUDA.jl will configure the memory pool to use all available device memory. You can change this using two environment variables:

• JULIA_CUDA_SOFT_MEMORY_LIMIT: This is an advisory limit, used to configure the memory pool. If you set this to a nonzero value, the memory pool will attempt to release cached memory until memory use falls below this limit. Note that this only happens at specific synchronization points, so memory use may temporarily exceed this limit. In addition, this limit is incompatible with JULIA_CUDA_MEMORY_POOL=none.
• JULIA_CUDA_HARD_MEMORY_LIMIT: This is a hard limit, checked before every allocation. On older versions of CUDA, before v12.2, this is a relatively expensive limit, so it is recommended to first try to use the soft limit.

The value of these variables can be formatted as a number of bytes, optionally followed by a unit, or as a percentage of the total device memory. Examples: 100M, 50%, 1.5GiB, 10000.

Avoiding GC pressure

When your application performs a lot of memory operations, the time spent during GC might increase significantly. This happens more often than it does on the CPU because GPUs tend to have smaller memories and more frequently run out of it. When that happens, CUDA invokes the Julia garbage collector, which then needs to scan objects to see if they can be freed to get back some GPU memory.

To avoid having to depend on the Julia GC to free up memory, you can directly inform CUDA.jl when an allocation can be freed (or reused) by calling the unsafe_free! method. Once you've done so, you cannot use that array anymore:

julia> a = CuArray([1])
1-element CuArray{Int64,1,Nothing}:
 1

julia> CUDA.unsafe_free!(a)

julia> a
1-element CuArray{Int64,1,Nothing}:
Error showing value of type CuArray{Int64,1,Nothing}:
ERROR: AssertionError: Use of freed memory

Batching iterator

If you are dealing with data sets that are too large to fit on the GPU all at once, you can use CuIterator to batch operations:

julia> batches = [([1], [2]), ([3], [4])]

julia> for (batch, (a,b)) in enumerate(CuIterator(batches))
         println("Batch $batch: ", a .+ b)
       end
Batch 1: [3]
Batch 2: [7]

For each batch, every argument (assumed to be an array-like) is uploaded to the GPU using the adapt mechanism from above. Afterwards, the memory is eagerly put back in the CUDA memory pool using unsafe_free! to lower GC pressure.

Releasing resources

Many CUDA operations that release resources wait for all kernels running on the device to finish, and block kernel launches from other threads in the meantime. Doing so from a finalizer could stall whichever thread the garbage collector happens to run on, or even deadlock if the running kernels depend on that thread. That is why CUDA.jl doesn't release resources from finalizers. Instead, collected resources are queued, and released later by a regular task:

• Device memory, and pinned host or unified memory allocated from a memory pool (supported by CUDA 13 and later), is freed in stream order the next time CUDA.jl allocates memory or synchronizes, or within a second. CUDA.pool_status() also does so before reporting.
• Where host or unified memory cannot be allocated from a pool (e.g., on Jetson devices), freed allocations are cached and reused once the GPU has finished using them.
• Releasing other resources may wait for the GPU, so that only happens when memory is reclaimed: when an allocation runs out of memory, or when calling CUDA.reclaim(). This includes emptying the caches mentioned above, unpinning host memory (as pinned by CUDA.pin, or when wrapping an Array with unsafe_wrap), unloading modules, destroying texture arrays, and destroying some library objects. Arrays whose memory was pinned are kept alive until then.
• Memory that a graph uses is kept alive as long as the graph, or an executable graph instantiated from it, exists, even if the array it belonged to has been freed.

As CUDA.reclaim() may wait for the GPU, don't call it while GPU work depends on the calling task to make progress. Nothing is released while a graph is being captured, and starting a capture fails while CUDA.reclaim() is releasing resources that may wait for the GPU.

Wrapping CUDA library objects

Packages that wrap objects of CUDA libraries can use CUDA.resource_finalizer instead of finalizer to destroy them the same way:

using CUDA: resource_finalizer

obj = MyLibraryObject(handle)
resource_finalizer(obj) do obj
    destroy_library_object(obj.handle)
end

By default, the destructor is only called when memory is reclaimed. If it is known not to wait for the GPU, pass blocking=false to have it called as soon as possible instead.