banhxeo.model.llm.config module

class banhxeo.model.llm.config.LayerNormConfig(eps: float = 1e-05, elementwise_affine: bool = True, bias: bool = True)

Bases: object

eps: float = 1e-05

elementwise_affine: bool = True

bias: bool = True

__init__(eps: float = 1e-05, elementwise_affine: bool = True, bias: bool = True) → None

class banhxeo.model.llm.config.MLPConfig(dim: int = 3072)

Bases: object

dim: int = 3072

__init__(dim: int = 3072) → None

class banhxeo.model.llm.config.MHAConfig(dim: int = 64, n_heads: int = 12)

Bases: object

dim: int = 64

n_heads: int = 12

__init__(dim: int = 64, n_heads: int = 12) → None

class banhxeo.model.llm.config.GPT2Config(*, vocab_size: int | None = None, d_model: int = 768, debug: bool = True, init_range: float = 0.02, n_ctx: int = 1024, n_layers: int = 12, mha: MHAConfig, ln: LayerNormConfig, mlp: MLPConfig)

Bases: ModelConfig

d_model: int

debug: bool

init_range: float

n_ctx: int

n_layers: int

mha: MHAConfig

ln: LayerNormConfig

mlp: MLPConfig

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].