CUDA driver

CuError(code)

Create a CUDA error object with error code code. Internally, errors thrown by API calls are annotated with the driver's explanation of the failure, if the driver provides one.

name(err::CuError)

Gets the string representation of an error code.

julia> err = CuError(CUDACore.cudaError_enum(1))CuError(CUDA_ERROR_INVALID_VALUE)julia> name(err)"ERROR_INVALID_VALUE"

description(err::CuError)

Gets the string description of an error code.

driver_version()

Returns the latest version of CUDA supported by the loaded driver.

runtime_version()

Returns the CUDA Runtime version.

On Linux, this calls cudaRuntimeGetVersion, which returns a compile-time constant baked into the loaded libcudart. On Windows we instead read the cudart DLL's PE file version directly, because NVIDIA's cudaRuntimeGetVersion there delegates to a driver-bundled "hybrid" runtime and reports the driver's version rather than the loaded toolkit's.

compiler_version()

Returns the CUDA toolkit version that is used to provide the CUDA compiler (ptxas) and other tools. The compiler is selected independently from the CUDA runtime, although set_runtime_version! configures both to use the same version by default.

Derived by parsing ptxas --version.

CUDA.set_runtime_version!([version::VersionNumber]; [local_toolkit::Bool])

Configures the active project to use a specific CUDA runtime and compiler version from a specific source.

If local_toolkit is set, the CUDA toolkit will be used from the local system, otherwise it will be downloaded from an artifact source. In the case of a local toolkit, version informs CUDA.jl which version that is (this may be useful if auto-detection fails). In the case of artifact sources, version controls which version will be downloaded and used.

When not specifying either the version or the local_toolkit argument, the default behavior will be used, which is to use the most recent compatible artifacts. Note that this will override any Preferences that may be configured in a higher-up depot; to clear preferences nondestructively, use CUDACore.reset_runtime_version! instead.

CUDA.reset_runtime_version!()

Resets the CUDA version preferences in the active project to the default, which is to use the most recent compatible artifacts, unless a higher-up depot has configured a different preference. To force use of the default behavior for the local project, use CUDACore.set_runtime_version! with no arguments.

CuDevice(ordinal::Integer)

Get a handle to a compute device.

devices()

Get an iterator for the compute devices.

current_device()

Returns the current device.

name(dev::CuDevice)

Returns an identifier string for the device.

totalmem(dev::CuDevice)

Returns the total amount of memory (in bytes) on the device.

attribute(dev::CuDevice, code)

Returns information about the device.

attribute(X, pool::CuMemoryPool, attr)

Returns attribute attr about pool. The type of the returned value depends on the attribute, and as such must be passed as the X parameter.

attribute(X, ptr::Union{Ptr,CuPtr}, attr)

Returns attribute attr about pointer ptr. The type of the returned value depends on the attribute, and as such must be passed as the X parameter.

capability(dev::CuDevice)

Returns the compute capability of the device.

warpsize(dev::CuDevice)

Returns the warp size (in threads) of the device.

CuContext(dev::CuDevice, flags=CTX_SCHED_AUTO)
CuContext(f::Function, ...)

Create a CUDA context for device. A context on the GPU is analogous to a process on the CPU, with its own distinct address space and allocated resources. When a context is destroyed, the system cleans up the resources allocated to it.

When you are done using the context, call CUDACore.unsafe_destroy! to mark it for deletion, or use do-block syntax with this constructor.

unsafe_destroy!(ctx::CuContext)

Immediately destroy a context, freeing up all resources associated with it. This does not respect any users of the context, and might make other objects unusable.

current_context()

Returns the current context. Throws an undefined reference error if the current thread has no context bound to it, or if the bound context has been destroyed.

activate(ctx::CuContext)

Binds the specified CUDA context to the calling CPU thread.

synchronize(ctx::Context)

Block for all the operations on ctx to complete. This is a heavyweight operation, typically you only need to call synchronize() which only synchronizes the stream associated with the current task.

device_synchronize()

Block for all the operations on the device to complete. This is a heavyweight operation, typically you only need to call synchronize() which only synchronizes the stream associated with the current task.

On the device, device_synchronize acts as a synchronization point for child grids in the context of dynamic parallelism.

CuPrimaryContext(dev::CuDevice)

Create a primary CUDA context for a given device.

Each primary context is unique per device and is shared with CUDA runtime API. It is meant for interoperability with (applications using) the runtime API.

CuContext(pctx::CuPrimaryContext)

Derive a context from a primary context.

Calling this function increases the reference count of the primary context. The returned context should not be free with the unsafe_destroy! function that's used with ordinary contexts. Instead, the refcount of the primary context should be decreased by calling unsafe_release!, or set to zero by calling unsafe_reset!. The easiest way to do this is by using the do-block syntax.

isactive(pctx::CuPrimaryContext)

Query whether a primary context is active.

flags(pctx::CuPrimaryContext)

Query the flags of a primary context.

setflags!(pctx::CuPrimaryContext)

Set the flags of a primary context.

unsafe_reset!(pctx::CuPrimaryContext)

Explicitly destroys and cleans up all resources associated with a device's primary context in the current process. Note that this forcibly invalidates all contexts derived from this primary context, and as a result outstanding resources might become invalid.

CUDACore.unsafe_release!(pctx::CuPrimaryContext)

Lower the refcount of a context, possibly freeing up all resources associated with it. This does not respect any users of the context, and might make other objects unusable.

CuModule(data, options::Dict{CUjit_option,Any})
CuModuleFile(path, options::Dict{CUjit_option,Any})

Create a CUDA module from a data, or a file containing data. The data may be PTX code, a CUBIN, or a FATBIN.

The options is an optional dictionary of JIT options and their respective value.

CuFunction(mod::CuModule, name::String)

Acquires a function handle from a named function in a module.

CuGlobal{T}(mod::CuModule, name::String)

Acquires a typed global variable handle from a named global in a module.

eltype(var::CuGlobal)

Return the element type of a global variable object.

Base.getindex(var::CuGlobal)

Return the current value of a global variable.

Base.setindex(var::CuGlobal{T}, val::T)

Set the value of a global variable to val

CuLink()

Creates a pending JIT linker invocation.

add_data!(link::CuLink, name::String, code::String)

Add PTX code to a pending link operation.

add_data!(link::CuLink, name::String, data::Vector{UInt8})

Add object code to a pending link operation.

add_file!(link::CuLink, path::String, typ::CUjitInputType)

Add data from a file to a link operation. The argument typ indicates the type of the contained data.

The result of a linking operation.

This object keeps its parent linker object alive, as destroying a linker destroys linked images too.

complete(link::CuLink)

Complete a pending linker invocation, returning an output image.

CuModule(img::CuLinkImage, ...)

Create a CUDA module from a completed linking operation. Options from CuModule apply.

DeviceMemory

Device memory residing on the GPU.

alloc(DeviceMemory, bytesize::Integer;
      [async=false], [stream::CuStream], [pool::CuMemoryPool])

Allocate bytesize bytes of memory on the device. This memory is only accessible on the GPU, and requires explicit calls to unsafe_copyto!, which wraps cuMemcpy, for access on the CPU.

UnifiedMemory

Unified memory that is accessible on both the CPU and GPU.

alloc(UnifiedMemory, bytesize::Integer, [flags::CUmemAttach_flags])

Allocate bytesize bytes of unified memory. This memory is accessible from both the CPU and GPU, with the CUDA driver automatically copying upon first access.

prefetch(::UnifiedMemory, [bytes::Integer]; [device::CuDevice], [stream::CuStream])

Prefetches memory to the specified destination device.

advise(::UnifiedMemory, advice::CUDA.CUmem_advise, [bytes::Integer]; [device::CuDevice])

Advise about the usage of a given memory range.

HostMemory

Pinned memory residing on the CPU, possibly accessible on the GPU.

alloc(HostMemory, bytesize::Integer, [flags])

Allocate bytesize bytes of page-locked memory on the host. This memory is accessible from the CPU, and makes it possible to perform faster memory copies to the GPU. Furthermore, if flags is set to MEMHOSTALLOC_DEVICEMAP the memory is also accessible from the GPU. These accesses are direct, and go through the PCI bus. If flags is set to MEMHOSTALLOC_PORTABLE, the memory is considered mapped by all CUDA contexts, not just the one that created the memory, which is useful if the memory needs to be accessed from multiple devices. Multiple flags can be set at one time using a bytewise OR:

flags = MEMHOSTALLOC_PORTABLE | MEMHOSTALLOC_DEVICEMAP

register(HostMemory, ptr::Ptr, bytesize::Integer, [flags])

Page-lock the host memory pointed to by ptr. Subsequent transfers to and from devices will be faster, and can be executed asynchronously. If the MEMHOSTREGISTER_DEVICEMAP flag is specified, the buffer will also be accessible directly from the GPU. These accesses are direct, and go through the PCI bus. If the MEMHOSTREGISTER_PORTABLE flag is specified, any CUDA context can access the memory.

unregister(::HostMemory)

Unregisters a memory range that was registered with register.

ArrayMemory

Array memory residing on the GPU, possibly in a specially-formatted way.

alloc(ArrayMemory, dims::Dims)

Allocate array memory with dimensions dims. The memory is accessible on the GPU, but can only be used in conjunction with special intrinsics (e.g., texture intrinsics).

unsafe_copy2d!(dst, dstTyp, src, srcTyp, width, height=1;
               dstPos=(1,1), dstPitch=0,
               srcPos=(1,1), srcPitch=0,
               async=false, stream=nothing)

Perform a 2D memory copy between pointers src and dst, at respectively position srcPos and dstPos (1-indexed). Pitch can be specified for both the source and destination; consult the CUDA documentation for more details. This call is executed asynchronously if async is set, otherwise stream is synchronized.

unsafe_copy3d!(dst, dstTyp, src, srcTyp, width, height=1, depth=1;
               dstPos=(1,1,1), dstPitch=0, dstHeight=0,
               srcPos=(1,1,1), srcPitch=0, srcHeight=0,
               async=false, stream=nothing)

Perform a 3D memory copy between pointers src and dst, at respectively position srcPos and dstPos (1-indexed). Both pitch and height can be specified for both the source and destination; consult the CUDA documentation for more details. This call is executed asynchronously if async is set, otherwise stream is synchronized.

memset(mem::CuPtr, value::Union{UInt8,UInt16,UInt32}, len::Integer; [stream::CuStream])

Initialize device memory by copying val for len times.

free_memory()

Returns the free amount of memory (in bytes), available for allocation by the CUDA context.

total_memory()

Returns the total amount of memory (in bytes), available for allocation by the CUDA context.

CuStream(; flags=STREAM_DEFAULT, priority=nothing)

Create a CUDA stream.

isdone(s::CuStream)

Return false if a stream is busy (has task running or queued) and true if that stream is free.

priority_range()

Return the valid range of stream priorities as a StepRange (with step size 1). The lower bound of the range denotes the least priority (typically 0), with the upper bound representing the greatest possible priority (typically -1).

priority(s::CuStream)

Return the priority of a stream s.

synchronize([stream::CuStream])

Wait until stream has finished executing, with stream defaulting to the stream associated with the current Julia task.

See also: device_synchronize

@sync [blocking=false] ex

Run expression ex and synchronize the GPU afterwards.

The blocking keyword argument determines how synchronization is performed. By default, non-blocking synchronization will be used, which gives other Julia tasks a chance to run while waiting for the GPU to finish. This may increase latency, so for short operations, or when benchmaring code that does not use multiple tasks, it may be beneficial to use blocking synchronization instead by setting blocking=true. Blocking synchronization can also be enabled globally by changing the nonblocking_synchronization preference.

See also: synchronize.

default_stream()

Return the default stream.

legacy_stream()

Return a special object to use use an implicit stream with legacy synchronization behavior.

You can use this stream to perform operations that should block on all streams (with the exception of streams created with STREAM_NON_BLOCKING). This matches the old pre-CUDA 7 global stream behavior.

per_thread_stream()

Return a special object to use an implicit stream with per-thread synchronization behavior. This stream object is normally meant to be used with APIs that do not have per-thread versions of their APIs (i.e. without a ptsz or ptds suffix).

CuEvent()

Create a new CUDA event.

record(e::CuEvent, [stream::CuStream])

Record an event on a stream.

synchronize(e::CuEvent)

Waits for an event to complete.

isdone(e::CuEvent)

Return false if there is outstanding work preceding the most recent call to record(e) and true if all captured work has been completed.

wait(e::CuEvent, [stream::CuStream])

Make a stream wait on a event. This only makes the stream wait, and not the host; use synchronize(::CuEvent) for that.

elapsed(start::CuEvent, stop::CuEvent)

Computes the elapsed time between two events (in seconds).

@elapsed [blocking=false] ex

A macro to evaluate an expression, discarding the resulting value, instead returning the number of seconds it took to execute on the GPU, as a floating-point number.

See also: @sync.

CuDim3(x)

CuDim3((x,))
CuDim3((x, y))
CuDim3((x, y, x))

A type used to specify dimensions, consisting of 3 integers for respectively the x, y and z dimension. Unspecified dimensions default to 1.

Often accepted as argument through the CuDim type alias, eg. in the case of cudacall or CUDACore.launch, allowing to pass dimensions as a plain integer or a tuple without having to construct an explicit CuDim3 object.

cudacall(f, types, values...; blocks::CuDim, threads::CuDim,
         cooperative=false, dependent=false, shmem=0, stream=stream())

ccall-like interface for launching a CUDA function f on a GPU.

For example:

vadd = CuFunction(md, "vadd")a = rand(Float32, 10)b = rand(Float32, 10)ad = alloc(CUDA.DeviceMemory, 10*sizeof(Float32))unsafe_copyto!(ad, convert(Ptr{Cvoid}, a), 10*sizeof(Float32)))bd = alloc(CUDA.DeviceMemory, 10*sizeof(Float32))unsafe_copyto!(bd, convert(Ptr{Cvoid}, b), 10*sizeof(Float32)))c = zeros(Float32, 10)cd = alloc(CUDA.DeviceMemory, 10*sizeof(Float32))cudacall(vadd, (CuPtr{Cfloat},CuPtr{Cfloat},CuPtr{Cfloat}), ad, bd, cd; threads=10)unsafe_copyto!(convert(Ptr{Cvoid}, c), cd, 10*sizeof(Float32)))

The blocks and threads arguments control the launch configuration, and should both consist of either an integer, or a tuple of 1 to 3 integers (omitted dimensions default to 1). The types argument can contain both a tuple of types, and a tuple type, the latter being slightly faster.

launch(f::CuFunction; args...; blocks::CuDim=1, threads::CuDim=1,
       clustersize::CuDim=1, cooperative=false, dependent=false,
       shmem=0, stream=stream())

Low-level call to launch a CUDA function f on the GPU, using blocks and threads as respectively the grid and block configuration. Dynamic shared memory is allocated according to shmem, and the kernel is launched on stream stream. If clustersize > 1 and compute capability is >= 9.0, thread block clusters are launched. If clustersize > 1 and compute capability is < 9.0, an error is thrown, as thread block clusters are not supported.

Setting dependent=true marks this as a programmatically dependent launch, allowing it to overlap with the preceding kernel in stream. The preceding kernel should call trigger_programmatic_launch_completion, and this kernel must call grid_dependency_synchronize before accessing its results. This feature requires compute capability 9.0 or higher.

Arguments to a kernel should either be bitstype, in which case they will be copied to the internal kernel parameter buffer, or a pointer to device memory.

This is a low-level call, prefer to use cudacall instead.

launch(exec::CuGraphExec, [stream::CuStream])
exec([stream::CuStream])

Launch an executable graph, by default on the current task's stream.

Just like other operations, launching a graph takes ownership of the memory it uses, so that it is safe to use that memory from other tasks, and so that it is only released after the graph has finished executing.

@profile [trace=false] [raw=false] code...
@profile external=true code...

Profile the GPU execution of code.

There are two modes of operation, depending on whether external is true or false. The default value depends on whether Julia is being run under an external profiler.

Integrated profiler (external=false, the default)

In this mode, CUDA.jl will profile the execution of code and display the result. By default, a summary of host and device-side execution will be show, including any NVTX events. To display a chronological trace of the captured activity instead, trace can be set to true. Trace output will include an ID column that can be used to match host-side and device-side activity. If raw is true, all data will always be included, even if it may not be relevant.

Slow operations will be highlighted in the output: Entries colored in yellow are among the slowest 25%, while entries colored in red are among the slowest 5% of all operations.

The results can also be written as JSON, e.g., for processing by scripts or coding agents, with show(stdout, "application/json", results). See Profile.ProfileResults for details.

To collect hardware metrics for every kernel instead, use CUDA.@metrics.

External profilers (external=true, when an external profiler is detected)

For more advanced profiling, it is possible to use an external profiling tool, such as NSight Systems or NSight Compute. When doing so, it is often advisable to only enable the profiler for the specific code region of interest. This can be done by wrapping the code with CUDA.@profile external=true, which used to be the only way to use this macro.

@metrics [metrics=:overview] [trace=false] code...

Collect hardware metrics for every kernel launched by code, similar to NSight Compute, and return them as a Profile.KernelProfile.

By default (metrics=:overview), this reports how close each kernel comes to the peak throughput of the GPU's compute units and memory subsystems. metrics=:memory reports DRAM traffic, while a vector of strings collects the given CUPTI metrics (as listed by ncu --query-metrics). Displaying the result shows a summary per kernel, or every launch when trace is true.

To collect metrics, kernels are executed one at a time, and replayed as often as needed to read all counters (restoring memory in between). Durations and other metrics thus describe each kernel in isolation, and the code runs considerably slower. Only kernels launched in the current context are profiled, at most 512 of them.

The results can also be written as JSON with show(stdout, "application/json", results).

See also: CUDA.@profile, which traces host and device activity.

@sample [period=14] [format=:tree] [noisefloor=2.0] code...

Sample where the warps of every kernel launched by code spend time, and why they are stalled, and return the samples as a Profile.PCProfile.

The program counter (PC) of a warp on every SM is sampled once every 2^period cycles. Shorter periods collect more samples, which take longer to process afterwards. Kernels execute concurrently and at nearly full speed, and all kernels executing in the current context are sampled, including those launched by other tasks.

The result p can be inspected in several ways:

• displaying it shows, like Profile.print, a tree of the source locations in every kernel with their number of samples and of stalled samples, where the children of a location are the functions that were inlined into it. format=:flat lists the locations instead. Locations with n samples are hidden if n <= noisefloor * √N, where N is the number of samples of their parent (or of the kernel, in the flat format). Stalls are colored by kind if the output supports colors, and lines are only cropped to the width of the display if the output is limited (as in the REPL), so string(p) is plain text where every row carries its location and function;
• CUDATools.code_sass(p) prints the SASS code of every sampled function, with the samples of every instruction;
• CUDATools.code_ptx(p) prints the PTX code of every sampled function, with the samples of the instructions generated for every line;
• show(io, MIME"application/json"(), p) writes all of this as JSON;
• the fields of p and of the Profile.SampledFunctions it contains hold the samples of every instruction per stall reason, for programmatic use.

The JSON and the fields are described in the documentation of Profile.PCProfile.

This requires CUDA 13 or later, and a GPU with compute capability 7.5 or higher.

See also: CUDA.@profile, CUDA.@metrics.

ProfileResults(...)

The results of a profiling run, as returned by CUDA.@profile. The recommended way to interpret these results is to visualize them using the I/O stack (e.g. by calling display, print, string, ...)

For programmatic access, show(io, MIME"application/json"(), results) writes the report that would be displayed as JSON, with times in nanoseconds relative to the start of the trace, and null for values that are not available:

• view: "summary", or "trace" when trace=true was passed;
• duration_ns and events: the duration of the trace and the number of events in it;
• host_time_ns: the time the busiest thread spent calling CUDA APIs;
• device_time_ns: the total duration of device activity, which exceeds the duration of the trace when operations overlap;
• host, device and nvtx: arrays of API calls, device operations, and NVTX ranges. In summary view, each entry summarizes the events with the same name: the number of calls, their total_ns and the time_fraction of the trace that represents, and their mean_ns, std_ns, min_ns and max_ns. In trace view, each entry describes one event: its start_ns and duration_ns, an id that matches API calls with the operations they launched, and details like launch configurations or the size of memory operations.

It is also possible to access the fields of this struct. However, the exact format is not guaranteed to be stable, and may change between CUDA.jl releases. Currently, it contains three tables (NamedTuples of vectors):

• host, containing host-side activity;
• device, containing device-side activity;
• nvtx, with information on captured NVTX ranges and events.

See also: CUDA.@profile

KernelProfile

The hardware metrics collected by CUDA.@metrics. Displaying this object shows a summary per kernel, or every launch when trace=true was passed.

For programmatic access, show(io, MIME"application/json"(), p) writes the report that would be displayed as JSON:

• view: "summary", or "trace" when trace=true was passed;
• device, and the number of kernel launches and of replay passes;
• metrics: the collected metrics, with their CUPTI name, the label used when displaying them, and their unit (null if unknown);
• kernels: in summary view, every kernel with its number of calls and the time_fraction they took, sorted by duration; in trace view, every launch with its id, grid and block size, registers per thread, and stream. Each entry has a metrics object with the value of every metric, averaged over the launches of a kernel in summary view.

The following fields are also available:

• kernels: a table (NamedTuple of vectors) with one row per kernel launch, in the order they executed: the launch's correlation id, kernel name, grid and block dimensions, registers per thread, and stream;
• metrics: the names of the CUPTI metrics that were collected;
• values: a matrix with, for every launch (row), the value of every metric (column). Durations are in nanoseconds, percentages between 0 and 100;
• passes: how often every kernel was executed to collect these metrics.

See also: CUDA.@metrics

PCProfile

The samples collected by CUDA.@sample, which describes the ways to display them.

The text report shows, for every sampled function, a tree of source locations where the children of a location are the functions that were inlined into it, or a flat list of locations. Every location lists the number of samples of the instructions generated for it (Samples), how many of them were Stalled, the samples of its own code, excluding inlined functions (Self), and the most common reasons for the stalls. Stall reasons are grouped into kinds (memory, throttle, sync, execution and issue).

show(io, MIME"application/json"(), p) writes the samples as JSON:

• view: "tree" or "flat", depending on the format;
• device, the sampling period (once every 2^period cycles), the noisefloor, and the number of total_samples (including samples outside of the sampled functions) and dropped_samples;
• reasons: the reasons warps were stalled for;
• frames: source locations, each with a function (empty for the sampled function itself), file and line;
• ptx: the PTX code of the sampled modules, as embedded in their cubins (see CUDATools.code_ptx);
• functions: every sampled function with its demangled name and mangled symbol, its count of samples and how many of them were stalled, its ptx code (by index, from 0, or null without line information), the locations that are displayed (in order, each with its depth in the tree, its frame (by index, from 0, or null if unknown), count, stalled and overhead samples, and stalled samples per reason), and every sampled instruction with its offset, sass text, the frames it was generated from (outermost first), the line of PTX code it was generated from (ptx_line, from 1, or null if unknown), and its count and stalled samples per reason. Reasons without samples are omitted.

The following fields are also available:

• device, period, noisefloor, total and dropped, as above, and the format;
• reasons: the reasons that index the columns of the sample matrices;
• functions: a vector of Profile.SampledFunction, sorted by their number of stalled samples.

SampledFunction

The samples collected in a GPU function (a kernel, and any non-inlined functions it calls) by CUDA.@sample, with the following fields:

• name: the (demangled) name of the function, and symbol its mangled name;
• instructions: a table (NamedTuple of vectors) with the function's SASS instructions: their offset in the function, the instruction text (sass, or labels with offset -1), and the source locations they were generated from (outermost first, including inlined frames; empty if unknown). If the code couldn't be disassembled, this only lists the sampled instructions, without text;
• samples and stalled: matrices with, for every instruction (row), the number of samples per reason (column), counting all samples, or only the samples where the warp scheduler didn't issue an instruction (i.e., where stalls weren't hidden by other warps);
• ptx: the PTX code of the module, as ptxas embedded it in the cubin (text, a vector of lines, with directives like .loc blanked out), and the line of that code every instruction was generated from (lines, 0 if unknown), or nothing if the code was compiled without line information.

See CUDA.@sample for the ways to display these samples.

code_sass([io=stdout], p::Profile.PCProfile; debuginfo=:default)

Print the SASS code of the functions sampled by CUDA.@sample, annotating every instruction with its number of samples, of stalled samples, and its most common stall reason. Unless debuginfo=:none, the source locations the instructions were generated from are shown as comments, including the frames of inlined functions, like @device_code_sass does. Functions without samples are omitted.

See also: CUDATools.code_ptx

code_ptx([io=stdout], p::Profile.PCProfile; raw=false, debuginfo=:default)

Print the PTX code of the functions sampled by CUDA.@sample, annotating every line with the samples of the SASS instructions that ptxas generated for it, as recorded in the cubin. This includes the code of functions that ptxas inlined. Unless debuginfo=:none, source locations are shown as comments, taken from the first instruction generated for a line that has one. With raw=true, the code is printed as ptxas embedded it in the cubin, with an empty line for every directive (like .loc) and with the labels used by debug information.

This requires line information, i.e., running Julia with a debug level of at least 1 (the default).

See also: CUDATools.code_sass

start()

Enables profile collection by the active profiling tool for the current context. If profiling is already enabled, then this call has no effect.

stop()

Disables profile collection by the active profiling tool for the current context. If profiling is already disabled, then this call has no effect.

CuTexture{T,N,P}

N-dimensional texture object with elements of type T. These objects do not store data themselves, but are bounds to another source of device memory. Texture objects can be passed to CUDA kernels, where they will be accessible through the CuDeviceTexture type.

CuTexture{T,N,P}(parent::P; address_mode, filter_mode, normalized_coordinates)

Construct a N-dimensional texture object with elements of type T as stored in parent.

Several keyword arguments alter the behavior of texture objects:

• address_mode (wrap, clamp, mirror): how out-of-bounds values are accessed. Can be specified as a value for all dimensions, or as a tuple of N entries.
• interpolation (nearest neighbour, linear, bilinear): how non-integral indices are fetched. Nearest-neighbour fetches a single value, others interpolate between multiple.
• normalized_coordinates (true, false): whether indices are expected to fall in the normalized [0:1) range.

!!! warning Experimental API. Subject to change without deprecation.

CuTexture(x::CuTextureArray{T,N})

Create a N-dimensional texture object withelements of type T that will be read from x.

CuTexture(x::CuArray{T,N})

Create a N-dimensional texture object that reads from a CuArray.

Note that it is necessary the their memory is well aligned and strided (good pitch). Currently, that is not being enforced.

CuTextureArray{T,N}(undef, dims)

N-dimensional dense texture array with elements of type T. These arrays are optimized for texture fetching, and are only meant to be used as a source for CuTexture{T,N,P} objects.

CuTextureArray(A::AbstractArray)

Allocate and initialize a texture array from host memory in A.

CuTextureArray(A::CuArray)

Allocate and initialize a texture array from device memory in A.

launch_configuration(fun::CuFunction; shmem=0, max_threads=0)

Calculate a suggested launch configuration for kernel fun requiring shmem bytes of dynamic shared memory. Returns a tuple with a suggested amount of threads, and the minimal amount of blocks to reach maximal occupancy. Optionally, the maximum amount of threads can be constrained using max_threads.

In the case of a variable amount of shared memory, pass a callable object for shmem instead, taking a single integer representing the block size and returning the amount of dynamic shared memory for that configuration.

Without dynamic shared memory, the result is cached in fun, and reused for every max_threads that admits the unconstrained number of threads. Setting a function attribute with attributes or the cache configuration with cache_config! invalidates the cache, but changing the configuration through the driver API directly (e.g. cuFuncSetCacheConfig) does not.

active_blocks(fun::CuFunction, threads; shmem=0)

Calculate the maximum number of active blocks per multiprocessor when running threads threads of a kernel fun requiring shmem bytes of dynamic shared memory.

occupancy(fun::CuFunction, threads; shmem=0)

Calculate the theoretical occupancy of launching threads threads of a kernel fun requiring shmem bytes of dynamic shared memory.

capture(f; [stream], mode=STREAM_CAPTURE_MODE_RELAXED, throw_error=true)::CuGraph

Capture the GPU operations that f performs into a graph, without executing them. The operations are captured on a dedicated stream, which is the task's stream while capturing, so other tasks can keep using the regular stream of the task. Instantiate the graph to execute it, possibly many times:

graph = capture() do    y .= a .* x .+ yendexec = instantiate(graph)for i in 1:100    exec()end

Only the GPU operations are captured: launching the graph doesn't run f. The operations are captured with the arguments they were called with, so launching the graph always uses the same arrays and scalars. The contents of those arrays may change, though, so to execute the graph with different inputs, copy them into the arrays used during capture.

Memory that is used by captured operations is kept alive by the graph. Arrays that are allocated during capture are allocated once, outside of the graph, and launching the graph overwrites their contents, so allocating operations like y = a .* x can be captured too. Data copied from the CPU (e.g., when using CuRef scalars) is copied when capturing.

Not all operations can be captured. Anything that waits for the GPU, like copying memory back to the CPU, or that creates library handles, results in a CaptureError or a CuError. It's typically a good idea to execute f once before capturing it, so that kernels are compiled and libraries are initialized. When throw_error is false, failures due to unsupported operations are not reported, and nothing is returned instead.

Tasks that are spawned by f take part in the capture: their operations are captured on the same stream, so they need to have finished before f returns. Other tasks are not affected by the capture.

To capture on a specific stream instead, pass it as the stream keyword argument. That needs to be a stream that was created with flags=STREAM_NON_BLOCKING in the current context, and that isn't used by anything else while capturing. Other tasks can still use memory that was last used on that stream before the capture.

CUDA.jl checks the operations it performs itself, like waiting for the GPU, but by default doesn't ask the driver to prohibit other operations that are potentially unsafe during capture. The driver performs those checks per thread rather than per task, so they would also reject operations by unrelated tasks that happen to run on the capturing thread while the capturing task is waiting. To have the driver perform these checks anyway, e.g., to debug a library that doesn't support capture, use mode=STREAM_CAPTURE_MODE_THREAD_LOCAL (which checks the capturing thread, and keeps the task on that thread), or mode=STREAM_CAPTURE_MODE_GLOBAL (which also checks other threads).

See also: instantiate, capture!.

capture!(f, graph::CuGraph; after=CuGraphNode[], [stream], mode=STREAM_CAPTURE_MODE_RELAXED)

Capture the GPU operations that f performs, like capture, but add them to an existing graph. The captured operations depend on the nodes in after, and are independent of the other operations in the graph. This makes it possible to construct graphs with operations that can execute concurrently, without having to capture multiple streams:

graph = CuGraph()a = capture!(graph) do    x .= sin.(x)endb = capture!(graph) do    y .= cos.(y)endcapture!(graph; after=[a; b]) do    z .= x .+ yend

Returns the nodes that operations depending on the captured operations should depend on. If capturing fails, the graph may contain some of the captured operations, and should not be used anymore. The graph cannot be used by other tasks while capturing, including tasks spawned by f.

This functionality requires CUDA 12.3 or higher.

CuGraph()

A graph of GPU operations. Graphs are typically created by recording operations with capture, but you can also create an empty graph and add operations to it using capture!. To execute a graph, instantiate it.

A graph keeps the memory that its operations use alive for as long as it exists.

CuGraphNode

A node in a CuGraph, representing an operation. Nodes are owned by their graph, and are returned by capture! to express dependencies between operations.

instantiate(graph::CuGraph, [flags])::CuGraphExec

Create an executable graph from a graph, which can then be launched many times. This is an expensive operation, so reuse the executable graph, updating it if needed.

See also: launch, update.

CuGraphExec

An executable graph, created by instantiating a CuGraph, which can be launched many times. It keeps the memory used by the graph's operations alive for as long as it exists.

See also: instantiate, launch, update.

launch(f::CuFunction; args...; blocks::CuDim=1, threads::CuDim=1,
       clustersize::CuDim=1, cooperative=false, dependent=false,
       shmem=0, stream=stream())

Low-level call to launch a CUDA function f on the GPU, using blocks and threads as respectively the grid and block configuration. Dynamic shared memory is allocated according to shmem, and the kernel is launched on stream stream. If clustersize > 1 and compute capability is >= 9.0, thread block clusters are launched. If clustersize > 1 and compute capability is < 9.0, an error is thrown, as thread block clusters are not supported.

Setting dependent=true marks this as a programmatically dependent launch, allowing it to overlap with the preceding kernel in stream. The preceding kernel should call trigger_programmatic_launch_completion, and this kernel must call grid_dependency_synchronize before accessing its results. This feature requires compute capability 9.0 or higher.

Arguments to a kernel should either be bitstype, in which case they will be copied to the internal kernel parameter buffer, or a pointer to device memory.

This is a low-level call, prefer to use cudacall instead.

launch(exec::CuGraphExec, [stream::CuStream])
exec([stream::CuStream])

Launch an executable graph, by default on the current task's stream.

Just like other operations, launching a graph takes ownership of the memory it uses, so that it is safe to use that memory from other tasks, and so that it is only released after the graph has finished executing.

upload(exec::CuGraphExec, [stream::CuStream])

Upload an executable graph to the device, without executing it. This makes the first launch of the graph faster, which is otherwise slower than subsequent ones.

update(exec::CuGraphExec, graph::CuGraph; throw_error::Bool=true)::Bool

Update an executable graph to perform the operations of graph instead, which needs to have the same structure as the graph it was instantiated from, e.g., because it was captured from the same code. This is much cheaper than instantiating a new executable graph.

Returns whether the update succeeded. Unless throw_error is false, an error is thrown if the update failed.

update!(f, exec::CuGraphExec, node::CuGraphNode)

Update the parameters of a single operation in an executable graph, by capturing the operation that f performs, which needs to be of the same kind as the existing one (e.g., launching the same kernel). This is cheaper than updating the entire graph, and makes it possible to change the arguments of an operation without capturing the graph again:

graph = CuGraph()node = only(capture!(graph) do    @cuda kernel(a)end)exec = instantiate(graph)exec()update!(exec, node) do    @cuda kernel(b)endexec()

Only kernel launches, memory copies and memory sets can be updated this way.

for ...
    @captured begin
        # code that executes several kernels or CUDA operations
    end
end

A convenience macro that captures the operations performed by a block of code into a graph, and executes that graph, reusing the executable graph from the previous execution of the same block. If the operations have different parameters than during the previous execution (e.g., because they use different arrays, or scalar arguments changed), the executable graph is updated, which is much cheaper than instantiating a new one.

As the code is still executed (and captured) every time, this does not reduce the CPU overhead of executing it as much as launching a previously instantiated graph does. It can still improve performance of code that executes many short-running kernels, by avoiding gaps between the kernels on the GPU.

If capturing fails, e.g., because a library needs to be initialized, the code is executed normally, and capturing it is tried again.

See also: capture.

CaptureError(msg)

An operation was attempted that isn't supported while capturing a graph, e.g., waiting for the GPU, which is not possible as captured operations only execute when the graph is launched.

is_capturing([stream::CuStream])

Return whether a stream is being captured into a graph.

capture_status([stream::CuStream])

Return the capture status of a stream as a named tuple (status, id), where status is a CUstreamCaptureStatus and id is the unique identifier of the capture sequence the stream is part of (or nothing when the stream is not being captured).

unsafe_wrap(CuContext, handle::CUcontext; own=false)

Wrap a context that was created outside of CUDA.jl. The handle needs to remain valid for as long as the returned object, or anything created in the context, is used.

Contexts are never destroyed implicitly, so own only documents whether ownership moves to the caller of unsafe_wrap: with own=true, the context may be destroyed with CUDACore.unsafe_destroy!. It has no effect otherwise, and in particular doesn't keep a context that is not owned from being destroyed. That doesn't apply to primary contexts, which are released through CuPrimaryContext, nor to contexts derived from green contexts.

unsafe_wrap(CuStream, handle::CUstream; own=false)

Wrap a stream that was created outside of CUDA.jl. Default streams and streams in a green context can't be wrapped. The stream's context is not owned by the returned object.

With own=true, CUDA.jl takes ownership of the stream and uses it like a stream it created: the stream is destroyed when the returned object is garbage collected, and the handle shouldn't be used by anything else after that. A stream that is being captured can't be adopted.

With own=false, the stream's owner may destroy it as soon as it doesn't need it anymore. CUDA.jl memory remembers the stream it was last used on, and uses it after that, e.g., to be freed, so a stream that CUDA.jl doesn't own can only be used for work that doesn't involve CUDA.jl memory: recording and waiting for events, synchronizing, and launching kernels that don't take arrays as arguments. Other uses, like launching a kernel that takes an array, copying an array, capturing a graph, or making the stream the task's stream with stream!, throw an error. To use memory on behalf of the stream's owner, order work on the task's stream with the borrowed one using events:

xla_stream = unsafe_wrap(CuStream, handle)  # e.g., the stream XLA passes to a callback# make the task's stream wait for the work XLA submitted before the callbackevent = CuEvent(CUDA.EVENT_DISABLE_TIMING)record(event, xla_stream)CUDA.wait(event, stream())a .+= 1  # use memory on the task's stream, as usual# make XLA's stream wait for the work the callback submittedrecord(event, stream())CUDA.wait(event, xla_stream)

unsafe_wrap(CuEvent, handle::CUevent; ctx::CuContext, own=false)

Wrap an event that was created outside of CUDA.jl, in context ctx. With own=true, CUDA.jl takes ownership of the event and destroys it when the returned object is garbage collected. Otherwise, the handle needs to remain valid for as long as the returned object is used.

Timing information is only available if the event was created without EVENT_DISABLE_TIMING.

unsafe_wrap(CuModule, handle::CUmodule; ctx::CuContext, own=false)

Wrap a module that was loaded outside of CUDA.jl, in context ctx. With own=true, CUDA.jl takes ownership of the module and unloads it when the returned object is garbage collected; that is not possible for modules that belong to a library or to the CUDA runtime. Otherwise, the handle needs to remain valid for as long as the returned object, or a function obtained from it, is used.

unsafe_wrap(CuMemoryPool, handle::CUmemoryPool; ctx::CuContext, own=false)

Wrap a memory pool that was created outside of CUDA.jl. Pools are never destroyed implicitly: with own=true, ownership moves to the caller, and the pool may be destroyed with CUDACore.unsafe_destroy! (in context ctx). Otherwise, the handle needs to remain valid for as long as the returned object is used.

unsafe_wrap(CuGraph, handle::CUgraph; ctx::CuContext, own=false)

Wrap a graph that was created outside of CUDA.jl, to be instantiated in context ctx. With own=true, CUDA.jl takes ownership of the graph and destroys it when the returned object is garbage collected. Otherwise, the handle needs to remain valid for as long as the returned object is used, and operations cannot be captured into the graph with capture!, as the graph would need to keep the memory they use alive.

The caller remains responsible for the memory used by operations that the graph already contains, for as long as the graph or an executable graph instantiated from it is used.

resource_finalizer(f, obj; blocking=true, ctx=context()) -> obj

Register f to clean up obj once it is garbage collected, like finalizer(f, obj), but without calling f from the garbage collector. Many CUDA API calls that release resources wait for all running kernels to finish, which can stall or deadlock the thread the garbage collector happens to run on. Instead, the finalizer only queues obj, and a regular task calls f(obj) later on, in the context ctx that was active when registering. Pass ctx=nothing if f activates the right context itself. If ctx has been destroyed by then, f is not called, as destroying a context releases its resources.

With blocking=true (the default), f is only called when CUDA.jl reclaims memory, i.e., when an allocation runs out of memory, or when calling CUDA.reclaim(). Only use blocking=false when f is known not to wait for the GPU. It is then called the next time CUDA.jl allocates memory or synchronizes, or within a second.

The cleanup function should use its argument instead of capturing obj, which would keep it alive. If f throws an error, it is logged, and obj is kept alive instead of retrying the cleanup, as it may have partially succeeded. As with finalizer, calling finalize(obj) only queues the cleanup, and there is no guarantee that f is called before the process exits. The callers remain responsible for keeping obj alive while the GPU uses it, and for not cleaning it up twice when also releasing it explicitly.