Skip to content

vllm.model_executor.layers.quantization.utils.machete_utils

query_machete_supported_group_sizes 

query_machete_supported_group_sizes(
    act_type: dtype,
) -> list[int]

Queries the supported group sizes for Machete based on the activation type.

Parameters:

Name Type Description Default
act_type dtype

The activation data type (torch.float16, torch.bfloat16).

 required

Returns:

Type Description
list[int]

A list of supported group sizes. The group size must
list[int]

be divisible by TileShapeK = 128 * 8 // num_bits(act_type).
list[int]

-1 indicates per-channel quantization.
Source code in vllm/model_executor/layers/quantization/utils/machete_utils.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
def query_machete_supported_group_sizes(act_type: torch.dtype) -> list[int]:
    """
    Queries the supported group sizes for Machete based on the activation type.

    Args:
        act_type: The activation data type (torch.float16, torch.bfloat16).

    Returns:
        A list of supported group sizes. The group size must
        be divisible by `TileShapeK = 128 * 8 // num_bits(act_type)`.
        -1 indicates per-channel quantization.
    """
    if act_type in [torch.float16, torch.bfloat16]:
        return [-1, 64, 128]
    else:
        return [-1, 128]