Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llama : support Mamba Selective State Space Models #5328

Merged
merged 43 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
8cd0a28
mamba : begin working on support for Mamba SSM
compilade Jan 26, 2024
5a69a26
mamba : begin figuring out how to (ab)use the kv cache for Mamba
compilade Jan 27, 2024
f680364
mamba : recurrent inference almost works, but incoherent
compilade Jan 28, 2024
54d3e48
mamba : recurrent inference WORKS!!!
compilade Jan 28, 2024
74eea85
convert : optionally use d_conv and d_state from config.json for Mamba
compilade Jan 29, 2024
9e77061
mamba : refactor recurrent conv, resulting in 20% perf increase
compilade Jan 29, 2024
3f7233b
ggml : parallelize ggml_exp
compilade Jan 29, 2024
e9cc45e
mamba : simplify the conv step with a self-overlapping view
compilade Jan 31, 2024
81b57bb
mamba : fix self-overlapping view depth stride
compilade Jan 31, 2024
ffc116f
mamba : handle batches of more than 1 token
compilade Feb 1, 2024
78a853b
ggml : in ggml_ssm_scan, merge multiple rows in the same vec operation
compilade Feb 2, 2024
5816ae6
mamba : very basic quantization support
compilade Feb 2, 2024
a3f4a1c
mamba : fuse more steps of the SSM scan in the ggml_ssm_scan operator
compilade Feb 3, 2024
9f55809
convert : for Mamba, also consider the "MambaLMHeadModel" arch name
compilade Feb 4, 2024
cd0f33f
mamba : fix vocab size problems with official models
compilade Feb 4, 2024
de92f15
ggml : remove ggml_exp and ggml_soft_plus
compilade Feb 4, 2024
766db75
mamba : remove some useless comments
compilade Feb 4, 2024
c52fb3c
convert : fix flake8 linter errors
compilade Feb 5, 2024
6ff34da
mamba : apply suggestions from code review
compilade Feb 5, 2024
8a43ffc
mamba : multiple sequences, but one at a time
compilade Feb 14, 2024
e73eaa7
mamba : in comments, properly refer to KV cells instead of slots
compilade Feb 14, 2024
de50c54
mamba : reduce memory usage of ggml_ssm_scan
compilade Feb 18, 2024
9473ec2
mamba : simultaneous sequence processing
compilade Feb 19, 2024
3dcf798
mamba : support llama_kv_cache_seq_cp copy chains
compilade Feb 25, 2024
34e2fca
mamba : make the server and parallel examples work with whole sequences
compilade Feb 25, 2024
79d636c
mamba : dedicate an input tensor for state copy indices
compilade Feb 25, 2024
8f605cf
mamba : adapt perplexity, batched, and batched-bench examples
compilade Feb 27, 2024
206e8ee
mamba : stop abusing attention metadata
compilade Feb 28, 2024
1af1000
mamba : more correctly update the "used" field of the KV cache
compilade Mar 2, 2024
d52dd50
ggml : in ggml_ssm_scan, use a threshold for soft_plus
compilade Mar 3, 2024
b83fbc9
convert : for Mamba, fallback to internal NeoX tokenizer
compilade Mar 3, 2024
eefb794
mamba : support state saving and restoring
compilade Mar 3, 2024
2a99d1b
ggml : implicitly pass src tensors through dst for Mamba-related ops
compilade Mar 4, 2024
93fd4b8
mamba : clarify some comments
compilade Mar 4, 2024
5544f52
Merge branch 'master' into support-mamba-ssm
compilade Mar 5, 2024
916b586
Merge branch 'master' into support-mamba-ssm
compilade Mar 7, 2024
7cd5a1f
server : fix cache_tokens not getting correctly resized
compilade Mar 7, 2024
d8024a4
convert-hf : support new metadata keys for Mamba
compilade Mar 8, 2024
17e4d6c
mamba : rename metadata to be more similar to transformers library
compilade Mar 8, 2024
1c8ea55
mamba : add missing spaces
compilade Mar 8, 2024
d0d32dc
convert-hf : omit output.weight when identical with token_embd.weight
compilade Mar 8, 2024
3e5685f
readme : add Mamba to supported models, and add recent API changes
compilade Mar 8, 2024
39579d3
mamba : move state_seq and state_mask views outside layer loop
compilade Mar 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions convert-hf-to-gguf.py
Original file line number Diff line number Diff line change
Expand Up @@ -1844,6 +1844,71 @@ class StarCoder2Model(Model):
model_arch = gguf.MODEL_ARCH.STARCODER2


@Model.register("MambaForCausalLM")
class MambaModel(Model):
model_arch = gguf.MODEL_ARCH.MAMBA

def set_gguf_parameters(self):
d_model = self.hparams["d_model"]
d_inner = self.hparams.get("d_inner", 2 * d_model)
# Fail early for models which don't have a block expansion factor of 2
assert d_inner == 2 * d_model

self.gguf_writer.add_name(self.dir_model.name)
self.gguf_writer.add_context_length(2**20) # arbitrary value; for those who use the default
self.gguf_writer.add_embedding_length(d_model)
self.gguf_writer.add_feed_forward_length(0) # unused, but seemingly required when loading
self.gguf_writer.add_head_count(d_inner) # the number of rows in conv_state and ssm_state
self.gguf_writer.add_block_count(self.hparams["n_layer"])
self.gguf_writer.add_layer_norm_rms_eps(self.hparams.get("rms_norm_eps", 1e-5))
# NOTE: (ab)using the KV cache metadata to store dimensions for conv_state and ssm_state
# Since the first column of the conv_state is shifted out each time, it's not actually needed
self.gguf_writer.add_key_length(self.hparams.get("d_conv", 4) - 1)
self.gguf_writer.add_value_length(self.hparams.get("d_state", 16))
self.gguf_writer.add_file_type(self.ftype)

def write_tensors(self):
block_count = self.hparams["n_layer"]
tensor_map = gguf.get_tensor_name_map(self.model_arch, block_count)
for name, data_torch in self.get_tensors():
old_dtype = data_torch.dtype

# convert any unsupported data types to float32
if data_torch.dtype not in (torch.float16, torch.float32):
data_torch = data_torch.to(torch.float32)

# map tensor names
new_name = tensor_map.get_name(name, try_suffixes=(".weight", ".bias"))
if new_name is None:
print(f"Can not map tensor {name!r}")
sys.exit()

if name.endswith(".A_log"):
print("A_log --> A ==> " + new_name)
data_torch = -torch.exp(data_torch)

data = data_torch.squeeze().numpy()

n_dims = len(data.shape)
data_dtype = data.dtype

# if f32 desired, convert any float16 to float32
if self.ftype == 0 and data_dtype == np.float16:
data = data.astype(np.float32)

# TODO: Why cant we use these float16 as-is? There should be not reason to store float16 as float32
if self.ftype == 1 and data_dtype == np.float16 and n_dims == 1:
data = data.astype(np.float32)

# if f16 desired, convert big float32 2-dim weight tensors to float16
if self.ftype == 1 and data_dtype == np.float32 and new_name.removesuffix(".weight").endswith((".ssm_in", ".ssm_out", "token_embd", "output")) and n_dims == 2:
data = data.astype(np.float16)

print(f"{new_name}, n_dims = {n_dims}, {old_dtype} --> {data.dtype}")

self.gguf_writer.add_tensor(new_name, data)


###### CONVERSION LOGIC ######


Expand Down
Loading