Skip to content

Commit 3a3e67d

Browse files
alay2shahykhrustalev
alay2shah
and
ykhrustalev
authored
Fix code examples: update API patterns and sampling parameters (#70)
## Summary Fixed inconsistencies between docs, notebooks, and code snippets: - Updated Transformers examples to use `return_dict=True` and `**inputs` pattern for proper attention mask handling - Corrected sampling parameters in notebooks to match model card specifications (LFM2.5-1.2B-Instruct: temperature=0.1, top_k=50) - Added critical `lm_head` workaround for Transformers v5 vision model bug - Regenerated all quickstart MDX snippets from updated notebook sources to ensure consistency All code examples now align across deployment guides, Colab notebooks, and quickstart snippets. <!-- mintlify-editor-comments:start --> Mintlify --- 0 threads from 0 users in Mintlify - No unresolved comments <!-- mintlify-editor-comments:end --> <!-- mintlify-comment--> <a href="https://dashboard.mintlify.com/liquidai/liquidai/editor/alay2shah%2Ffix-code-snippets?source=pr_comment" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://d3gk2c5xim1je2.cloudfront.net/assets/open-mintlify-editor-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://d3gk2c5xim1je2.cloudfront.net/assets/open-mintlify-editor-light.svg"><img src="https://d3gk2c5xim1je2.cloudfront.net/assets/open-mintlify-editor-light.svg" alt="Open in Mintlify Editor"></picture></a> <!-- /mintlify-comment --> --------- Co-authored-by: Yuri Khrustalev <ykhrustalev@users.noreply.github.com> Co-authored-by: Yuri Khrustalev <yuri@liquid.ai>
1 parent b8de7f3 commit 3a3e67d

14 files changed

Lines changed: 46 additions & 127 deletions

‎deployment/gpu-inference/transformers.mdx‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -59,13 +59,13 @@ inputs = tokenizer.apply_chat_template(
5959
return_tensors="pt",
6060
tokenize=True,
6161
return_dict=True,
62-
)
63-
input_ids = inputs["input_ids"].to(model.device)
62+
).to(model.device)
6463
65-
output = model.generate(input_ids, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512)
64+
output = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512)
6665
6766
# Decode only the newly generated tokens (excluding the input prompt)
68-
response = tokenizer.decode(output[0][len(input_ids[0]):], skip_special_tokens=True)
67+
input_length = inputs["input_ids"].shape[1]
68+
response = tokenizer.decode(output[0][input_length:], skip_special_tokens=True)
6969
print(response)
7070
# C. elegans, also known as Caenorhabditis elegans, is a small, free-living
7171
# nematode worm (roundworm) that belongs to the phylum Nematoda.
@@ -137,7 +137,7 @@ generation_config = GenerationConfig(
137137
)
138138

139139
# Use it in generate()
140-
output = model.generate(input_ids, generation_config=generation_config)
140+
output = model.generate(**inputs, generation_config=generation_config)
141141
```
142142

143143
For a complete list of parameters, see the [GenerationConfig documentation](https://huggingface.co/docs/transformers/v4.57.1/en/main_classes/text_generation#transformers.GenerationConfig).
@@ -157,11 +157,10 @@ inputs = tokenizer.apply_chat_template(
157157
return_tensors="pt",
158158
tokenize=True,
159159
return_dict=True,
160-
)
161-
input_ids = inputs["input_ids"].to(model.device)
160+
).to(model.device)
162161

163162
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
164-
output = model.generate(input_ids, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, streamer=streamer, max_new_tokens=512)
163+
output = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, streamer=streamer, max_new_tokens=512)
165164
```
166165

167166
## Batch Generation
@@ -191,8 +190,7 @@ inputs = tokenizer.apply_chat_template(
191190
tokenize=True,
192191
padding=True,
193192
return_dict=True,
194-
)
195-
inputs = {k: v.to(model.device) for k, v in inputs.items()}
193+
).to(model.device)
196194

197195
# Generate for all prompts in batch
198196
outputs = model.generate(**inputs, do_sample=True, temperature=0.1, top_k=50, repetition_penalty=1.05, max_new_tokens=512)
@@ -217,6 +215,9 @@ model = AutoModelForImageTextToText.from_pretrained(
217215
device_map="auto",
218216
dtype="bfloat16"
219217
)
218+
# IMPORTANT: tie lm_head to input embeddings (transformers v5 bug)
219+
model.lm_head.weight = model.get_input_embeddings().weight
220+
220221
processor = AutoProcessor.from_pretrained(model_id)
221222

222223
# Load image and create conversation

‎lfm/models/lfm2-1.2b-extract.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,8 @@ If no system prompt is provided, defaults to JSON. Specify format (JSON, XML, or
9393
{"role": "user", "content": user_input}
9494
]
9595

96-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
97-
outputs = model.generate(inputs, max_new_tokens=256, temperature=0, do_sample=False)
96+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
97+
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0, do_sample=False)
9898
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
9999
print(response)
100100
```

‎lfm/models/lfm2-1.2b-rag.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,8 +94,8 @@ The following documents may provide you additional information to answer questio
9494
{"role": "user", "content": user_input}
9595
]
9696

97-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
98-
outputs = model.generate(inputs, max_new_tokens=256, temperature=0, do_sample=False)
97+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
98+
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0, do_sample=False)
9999
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
100100
print(response)
101101
# Output: The library serves 48 scientists and 85 technicians, along with many visiting staff and students.

‎lfm/models/lfm2-2.6b-transcript.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,8 +121,8 @@ Participants: Names (Roles)
121121
{"role": "user", "content": user_input}
122122
]
123123

124-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
125-
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.3, do_sample=True)
124+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
125+
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.3, do_sample=True)
126126
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
127127
print(response)
128128
```

‎lfm/models/lfm2-350m-enjp-mt.mdx‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,8 +74,8 @@ This model requires a specific system prompt to specify translation direction. S
7474
{"role": "user", "content": "What is C. elegans?"}
7575
]
7676

77-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
78-
outputs = model.generate(inputs, max_new_tokens=256)
77+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
78+
outputs = model.generate(**inputs, max_new_tokens=256)
7979
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
8080
print(response)
8181
# Output: C. elegansとは何ですか？
@@ -88,8 +88,8 @@ This model requires a specific system prompt to specify translation direction. S
8888
{"role": "user", "content": "今日は天気がいいですね。"}
8989
]
9090

91-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
92-
outputs = model.generate(inputs, max_new_tokens=256)
91+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
92+
outputs = model.generate(**inputs, max_new_tokens=256)
9393
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
9494
print(response)
9595
# Output: The weather is nice today.

‎lfm/models/lfm2-350m-extract.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,8 @@ Schema:
9191
{"role": "user", "content": user_input}
9292
]
9393

94-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
95-
outputs = model.generate(inputs, max_new_tokens=256, temperature=0, do_sample=False)
94+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
95+
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0, do_sample=False)
9696
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
9797
print(response)
9898
```

‎lfm/models/lfm2-350m-math.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,8 @@ LFM2-350M-Math is a tiny reasoning model optimized for mathematical problem solv
6060
{"role": "user", "content": "If a train travels at 60 mph for 2.5 hours, how far does it travel?"}
6161
]
6262

63-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
64-
outputs = model.generate(inputs, max_new_tokens=256)
63+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
64+
outputs = model.generate(**inputs, max_new_tokens=256)
6565
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
6666
print(response)
6767
```

‎lfm/models/lfm2-350m-pii-extract-jp.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,8 +83,8 @@ Extract specific entities by listing only what you need (e.g., `Extract <human_n
8383
{"role": "user", "content": user_input}
8484
]
8585

86-
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
87-
outputs = model.generate(inputs, max_new_tokens=256, temperature=0, do_sample=False)
86+
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
87+
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0, do_sample=False)
8888
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
8989
print(response)
9090
# Output: {"address": [], "company_name": [], "email_address": ["celegans@liquid.ai"],

‎notebooks/LFM2_Inference_with_Transformers.ipynb‎

Lines changed: 3 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -86,30 +86,7 @@
8686
"execution_count": null,
8787
"metadata": {},
8888
"outputs": [],
89-
"source": [
90-
"from transformers import GenerationConfig\n",
91-
"\n",
92-
"generation_config = GenerationConfig(\n",
93-
" do_sample=True,\n",
94-
" temperature=0.3,\n",
95-
" min_p=0.15,\n",
96-
" repetition_penalty=1.05,\n",
97-
" max_new_tokens=512,\n",
98-
")\n",
99-
"\n",
100-
"prompt = \"Explain quantum computing in simple terms.\"\n",
101-
"inputs = tokenizer.apply_chat_template(\n",
102-
" [{\"role\": \"user\", \"content\": prompt}],\n",
103-
" add_generation_prompt=True,\n",
104-
" return_tensors=\"pt\",\n",
105-
" return_dict=True,\n",
106-
").to(model.device)\n",
107-
"\n",
108-
"output = model.generate(**inputs, generation_config=generation_config)\n",
109-
"input_length = inputs[\"input_ids\"].shape[1]\n",
110-
"response = tokenizer.decode(output[0][input_length:], skip_special_tokens=True)\n",
111-
"print(response)"
112-
]
89+
"source": "from transformers import GenerationConfig\n\ngeneration_config = GenerationConfig(\n do_sample=True,\n temperature=0.1,\n top_k=50,\n repetition_penalty=1.05,\n max_new_tokens=512,\n)\n\nprompt = \"Explain quantum computing in simple terms.\"\ninputs = tokenizer.apply_chat_template(\n [{\"role\": \"user\", \"content\": prompt}],\n add_generation_prompt=True,\n return_tensors=\"pt\",\n return_dict=True,\n).to(model.device)\n\noutput = model.generate(**inputs, generation_config=generation_config)\ninput_length = inputs[\"input_ids\"].shape[1]\nresponse = tokenizer.decode(output[0][input_length:], skip_special_tokens=True)\nprint(response)"
11390
},
11491
{
11592
"cell_type": "markdown",
@@ -154,51 +131,7 @@
154131
"execution_count": null,
155132
"metadata": {},
156133
"outputs": [],
157-
"source": [
158-
"from transformers import AutoProcessor, AutoModelForImageTextToText\n",
159-
"from transformers.image_utils import load_image\n",
160-
"\n",
161-
"# Load vision model and processor\n",
162-
"model_id = \"LiquidAI/LFM2.5-VL-1.6B\"\n",
163-
"vision_model = AutoModelForImageTextToText.from_pretrained(\n",
164-
" model_id,\n",
165-
" device_map=\"auto\",\n",
166-
" dtype=\"bfloat16\"\n",
167-
")\n",
168-
"\n",
169-
"# IMPORTANT: tie lm_head to input embeddings (transformers v5 bug)\n",
170-
"vision_model.lm_head.weight = vision_model.get_input_embeddings().weight\n",
171-
"\n",
172-
"processor = AutoProcessor.from_pretrained(model_id)\n",
173-
"\n",
174-
"# Load image\n",
175-
"url = \"https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg\"\n",
176-
"image = load_image(url)\n",
177-
"\n",
178-
"# Create conversation\n",
179-
"conversation = [\n",
180-
" {\n",
181-
" \"role\": \"user\",\n",
182-
" \"content\": [\n",
183-
" {\"type\": \"image\", \"image\": image},\n",
184-
" {\"type\": \"text\", \"text\": \"What is in this image?\"},\n",
185-
" ],\n",
186-
" },\n",
187-
"]\n",
188-
"\n",
189-
"# Generate response\n",
190-
"inputs = processor.apply_chat_template(\n",
191-
" conversation,\n",
192-
" add_generation_prompt=True,\n",
193-
" return_tensors=\"pt\",\n",
194-
" return_dict=True,\n",
195-
" tokenize=True,\n",
196-
").to(vision_model.device)\n",
197-
"\n",
198-
"outputs = vision_model.generate(**inputs, max_new_tokens=64)\n",
199-
"response = processor.batch_decode(outputs, skip_special_tokens=True)[0]\n",
200-
"print(response)"
201-
]
134+
"source": "from transformers import AutoProcessor, AutoModelForImageTextToText\nfrom transformers.image_utils import load_image\n\n# Load vision model and processor\nmodel_id = \"LiquidAI/LFM2.5-VL-1.6B\"\nvision_model = AutoModelForImageTextToText.from_pretrained(\n model_id,\n device_map=\"auto\",\n dtype=\"bfloat16\"\n)\n\n# IMPORTANT: tie lm_head to input embeddings (transformers v5 bug)\nvision_model.lm_head.weight = vision_model.get_input_embeddings().weight\n\nprocessor = AutoProcessor.from_pretrained(model_id)\n\n# Load image\nurl = \"https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg\"\nimage = load_image(url)\n\n# Create conversation\nconversation = [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": image},\n {\"type\": \"text\", \"text\": \"What is in this image?\"},\n ],\n },\n]\n\n# Generate response\ninputs = processor.apply_chat_template(\n conversation,\n add_generation_prompt=True,\n return_tensors=\"pt\",\n return_dict=True,\n tokenize=True,\n).to(vision_model.device)\n\noutputs = vision_model.generate(**inputs, do_sample=True, temperature=0.1, min_p=0.15, repetition_penalty=1.05, max_new_tokens=64)\nresponse = processor.batch_decode(outputs, skip_special_tokens=True)[0]\nprint(response)"
202135
},
203136
{
204137
"cell_type": "markdown",
@@ -228,4 +161,4 @@
228161
},
229162
"nbformat": 4,
230163
"nbformat_minor": 0
231-
}
164+
}

‎notebooks/LFM2_Inference_with_vLLM.ipynb‎

Lines changed: 2 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -58,25 +58,7 @@
5858
"execution_count": null,
5959
"metadata": {},
6060
"outputs": [],
61-
"source": [
62-
"from vllm import LLM, SamplingParams\n",
63-
"\n",
64-
"# Initialize the model\n",
65-
"llm = LLM(model=\"LiquidAI/LFM2.5-1.2B-Instruct\")\n",
66-
"\n",
67-
"# Define sampling parameters\n",
68-
"sampling_params = SamplingParams(\n",
69-
" temperature=0.3,\n",
70-
" min_p=0.15,\n",
71-
" repetition_penalty=1.05,\n",
72-
" max_tokens=512\n",
73-
")\n",
74-
"\n",
75-
"# Generate answer\n",
76-
"messages = [{\"role\": \"user\", \"content\": \"What is C. elegans?\"}]\n",
77-
"output = llm.chat(messages, sampling_params)\n",
78-
"print(output[0].outputs[0].text)"
79-
]
61+
"source": "from vllm import LLM, SamplingParams\n\n# Initialize the model\nllm = LLM(model=\"LiquidAI/LFM2.5-1.2B-Instruct\")\n\n# Define sampling parameters\nsampling_params = SamplingParams(\n temperature=0.1,\n top_k=50,\n repetition_penalty=1.05,\n max_tokens=512\n)\n\n# Generate answer\nmessages = [{\"role\": \"user\", \"content\": \"What is C. elegans?\"}]\noutput = llm.chat(messages, sampling_params)\nprint(output[0].outputs[0].text)"
8062
},
8163
{
8264
"cell_type": "markdown",
@@ -149,7 +131,7 @@
149131
"execution_count": null,
150132
"metadata": {},
151133
"outputs": [],
152-
"source": "from vllm import LLM, SamplingParams\nfrom typing import List, Dict, Any\n\ndef build_messages(parts):\n content = []\n for item in parts:\n if item[\"type\"] == \"text\":\n content.append({\"type\": \"text\", \"text\": item[\"value\"]})\n elif item[\"type\"] == \"image\":\n content.append({\"type\": \"image_url\", \"image_url\": {\"url\": item[\"value\"]}})\n return [{\"role\": \"user\", \"content\": content}]\n\nIMAGE_URL = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n\nllm = LLM(\n model=\"LiquidAI/LFM2.5-VL-1.6B\",\n max_model_len=1024,\n)\n\nsampling_params = SamplingParams(\n temperature=0.0,\n max_tokens=1024,\n)\n\n# Batch multiple prompts - text-only and multimodal\nprompts: List[List[Dict[str, Any]]] = [ # type: ignore[no-redef]\n [{\"type\": \"text\", \"value\": \"What is C. elegans?\"}],\n [{\"type\": \"text\", \"value\": \"Say hi in JSON format\"}],\n [\n {\"type\": \"image\", \"value\": IMAGE_URL},\n {\"type\": \"text\", \"value\": \"Describe what you see in this image.\"},\n ],\n]\n\nconversations = [build_messages(p) for p in prompts]\noutputs = llm.chat(conversations, sampling_params)\n\nfor output in outputs:\n print(output.outputs[0].text)\n print(\"---\")"
134+
"source": "from vllm import LLM, SamplingParams\nfrom typing import List, Dict, Any\n\ndef build_messages(parts):\n content = []\n for item in parts:\n if item[\"type\"] == \"text\":\n content.append({\"type\": \"text\", \"text\": item[\"value\"]})\n elif item[\"type\"] == \"image\":\n content.append({\"type\": \"image_url\", \"image_url\": {\"url\": item[\"value\"]}})\n return [{\"role\": \"user\", \"content\": content}]\n\nIMAGE_URL = \"http://images.cocodataset.org/val2017/000000039769.jpg\"\n\nllm = LLM(\n model=\"LiquidAI/LFM2.5-VL-1.6B\",\n max_model_len=1024,\n)\n\nsampling_params = SamplingParams(\n temperature=0.1,\n min_p=0.15,\n repetition_penalty=1.05,\n max_tokens=1024,\n)\n\n# Batch multiple prompts - text-only and multimodal\nprompts: List[List[Dict[str, Any]]] = [ # type: ignore[no-redef]\n [{\"type\": \"text\", \"value\": \"What is C. elegans?\"}],\n [{\"type\": \"text\", \"value\": \"Say hi in JSON format\"}],\n [\n {\"type\": \"image\", \"value\": IMAGE_URL},\n {\"type\": \"text\", \"value\": \"Describe what you see in this image.\"},\n ],\n]\n\nconversations = [build_messages(p) for p in prompts]\noutputs = llm.chat(conversations, sampling_params)\n\nfor output in outputs:\n print(output.outputs[0].text)\n print(\"---\")"
153135
},
154136
{
155137
"cell_type": "markdown",

0 commit comments

Comments
 (0)