[{"data":1,"prerenderedAt":412},["ShallowReactive",2],{"content-\u002Fposts\u002Fllm":3,"blog-posts":140},{"id":4,"title":5,"body":6,"categories":125,"date":126,"description":127,"draft":128,"extension":129,"meta":130,"navigation":131,"path":132,"seo":133,"stem":134,"tags":135,"__hash__":139},"posts\u002Fposts\u002F为什么LLM在上线不久后使用体感上变差了.md","为什么模型上线后，体感像是“降智”了？",{"type":7,"value":8,"toc":117},"minimark",[9,13,18,29,50,59,63,66,74,77,80,88,91,94,97,100,103,114],[10,11,12],"p",{},"如果你觉得某个模型刚上线时表现更好，但过一段时间后，同一个模型名下的回答却像变迟钝了，serving 侧的量化值得排查。量化能降低显存占用和推理成本，也可能改变模型输出。",[14,15,17],"h2",{"id":16},"checkpoint-与-serving-配置","Checkpoint 与 serving 配置",[10,19,20,21,25,26,28],{},"仅知道训练得到的 checkpoint，并不能判断线上服务是否直接按 checkpoint 保存时的精度运行。Serving 配置涉及模型计算所用的 ",[22,23,24],"code",{},"dtype","、权重量化方案和 KV cache 的存储 ",[22,27,24],{}," 等方面；这些配置能否组合使用，以及量化是否同时覆盖权重与激活，取决于量化方案、模型和硬件。",[10,30,31,32,35,36,43,44,49],{},"以 vLLM 为例，在线量化可以在加载 BF16\u002FFP16 模型时把 Linear、MoE 权重转换为 FP8；KV cache 也有独立的 ",[22,33,34],{},"--kv-cache-dtype"," 设置。",[37,38,42],"a",{"href":39,"rel":40},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Ffeatures\u002Fquantization\u002Fonline\u002F",[41],"nofollow","vLLM 在线量化文档","、",[37,45,48],{"href":46,"rel":47},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fserve\u002F",[41],"serving 参数文档"," 展示了这条技术路径，但没有说明任何特定厂商怎样部署。",[10,51,52,53,58],{},"因此，量化不一定以替换 checkpoint 文件的方式发生；服务可以从上线之初就使用量化配置，也可以之后调整配置。对外模型名本身无法告诉用户后端使用的精确 revision、路由或数值格式；“上线时直接跑训练 checkpoint，后来才换成量化 checkpoint”只是众多可能部署过程中的一种。",[37,54,57],{"href":55,"rel":56},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fapi\u002Fvllm\u002Fconfig\u002Fmodel\u002F",[41],"vLLM 模型配置文档"," 也分别配置了对外服务名与模型 revision。",[14,60,62],{"id":61},"权重量化与-kv-cache-量化","权重量化与 KV cache 量化",[10,64,65],{},"权重量化用较少位元表示参数，能压缩权重显存占用，并减少解码时读取权重的内存流量；端到端延迟是否下降，取决于量化内核和硬件能否高效执行。数值近似会改变前向计算得到的 logits。如果最高分 token 与其他候选 token 接近，细微误差就可能改变下一个 token，自回归解码随后会沿着新的文本继续。输出因此可能变化，任务质量却不必然明显下降。",[10,67,68,73],{},[37,69,72],{"href":70,"rel":71},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2210.17323v2",[41],"GPTQ（GPT 模型后训练量化方法）论文"," 在 OPT（Open Pre-trained Transformer）、BLOOM（BigScience Large Open-science Open-access Multilingual Language Model）等模型及 perplexity、zero-shot 基准上评测了离线权重量化。",[10,75,76],{},"例如，OPT-175B 在 WikiText2 上的 perplexity，FP16 基线为 8.34，4-bit GPTQ 为 8.37。该结果表明这一方案在这个指标上保留了质量，但不能保证其他量化方法、模型或线上对话也有相同表现。论文报告的速度收益依赖定制 GPU kernel 和减少内存搬运的实现。",[10,78,79],{},"权重之外，推理还会保存已处理 token 的 key\u002Fvalue，供后续 token 生成时计算 attention。活动序列越长、并发请求越多，KV cache 通常占用越多显存，因此压缩它能腾出并发空间。KV cache 精度与权重精度是不同的配置维度，调整 KV cache 精度也可能影响回答。",[10,81,82,87],{},[37,83,86],{"href":84,"rel":85},"https:\u002F\u002Farxiv.org\u002Fhtml\u002F2402.02750v2",[41],"KIVI（KV cache 非对称量化方法）论文"," 给出了一个 2-bit KV cache 方案：key 按 channel 量化，value 按 token 量化，并为近期 token 保留全精度残余区。它在 Llama、Mistral 的若干基准上报告较小的质量下降，也评测了 LongBench 长上下文任务；论文同时指出，Falcon 的多查询注意力让原始 KV cache 较小，2-bit 方案在部分任务上可能明显退化。",[10,89,90],{},"论文摘要报告，Llama-2-7B 使用 KIVI 时的峰值总显存约为 FP16 基线的 1\u002F2.6（统计口径包含模型权重）。",[10,92,93],{},"第 4.2.4 节另报告一组吞吐实验：研究者在单张 80GB A100 上使用 ShareGPT 衍生负载，逐步增大 batch 至显存耗尽。以 FP16 基线为比较对象、显存上限相近时，KIVI 可容纳最高 4 倍 batch size，吞吐为基线的 2.35 到 3.47 倍。结果受论文所用模型、硬件和负载限制，不能直接外推为生产服务的速度保证。量化效果取决于方案、模型、任务和运行条件；长上下文质量也不能只凭位宽判断。",[14,95,96],{"id":96},"怎样判断服务是否改过精度",[10,98,99],{},"运营者采用低精度配置有成本动机：较小的权重与 cache 能释放显存，有机会容纳更多并发请求。但研究论文和开源引擎文档证明的是技术可行性，不是某家服务实际做过哪次变更。同样的体感还可能来自模型 revision 或路由、系统提示词、采样参数和推理预算的变化。",[10,101,102],{},"用户侧要判断服务是否变过，需要保留上线初期的同题输出作基线，固定可见的模型版本、输入、采样参数和工具设置，再重复测试短、长上下文任务并盲评。没有历史基线，只能测量当前表现；如果系统提示词、路由或模型 revision 不公开，前后结果的变化也无法定位到量化。",[10,104,105,106,109,110,113],{},"能访问 serving 配置时，可以固定 checkpoint ",[22,107,108],{},"revision","、代码与量化内核、硬件、tokenizer、模板和采样配置，分别比较权重量化与 ",[22,111,112],{},"kv_cache_dtype","。两者可能相互作用，因此还可以补齐四种配置：都不量化、仅量化权重、仅量化 cache、两者都量化。若权重量化方案同时改变激活精度，测到的是整套方案的效果；要单独归因权重，还需固定激活精度或选用 weight-only 方案。",[10,115,116],{},"因此，量化是“体感变弱”的一种可检验解释。要断定某项服务上线后才切换了精度，还需要部署变更记录或能够隔离变量的服务端对照。",{"title":118,"searchDepth":119,"depth":120,"links":121},"",2,3,[122,123,124],{"id":16,"depth":119,"text":17},{"id":61,"depth":119,"text":62},{"id":96,"depth":119,"text":96},"技术杂谈","2026-09-29 21:44:08","权重与 KV cache 的量化都能降低 serving 成本，也可能改变输出；但体感变弱不足以证明服务后来切换了精度。",false,"md",{},true,"\u002Fposts\u002Fllm",{"title":5,"description":127},"posts\u002F为什么LLM在上线不久后使用体感上变差了",[136,137,138],"LLMs","Serving","量化","whcCazMY_im8KnfPwcKazSW41TSoPtuj9KJ8R7m5FRA",[141,150,156,163,168,178,187,193,197,199,208,214,220,228,232,236,246,253,259,263,267,275,282,290,295,301,307,312,317,324,331,338,344,351,359,365,371,377,383,388,397,405],{"path":142,"title":143,"description":144,"date":145,"categories":146,"tags":147,"draft":131},"\u002Fposts\u002Ftransposition-borrowed-chords","平移借用：大调里立刻能用的和弦","把大调整体平移，借用各调的顺阶和弦；有符号平移量 ±2、±3、±5、±7 给出七个近关系调，合计 25 个现成三和弦与 31 个现成七和弦，全部 12 个平移量给 36 与 48。","2026-11-08 16:20:00","音乐",[148,149],"乐理","和弦",{"path":151,"title":152,"description":153,"date":154,"categories":146,"tags":155,"draft":131},"\u002Fposts\u002Fdiatonic-chords-major-system","大调体系下的三和弦和七和弦","以同主音 C 为基准，逐个推出七套音阶的顺阶三和弦与七和弦，标注每个和弦的出处，去重后得到 21 个三和弦与 28 个七和弦。","2026-11-08 15:20:00",[148,149],{"path":157,"title":158,"description":118,"date":159,"categories":160,"tags":161,"draft":131},"\u002Fposts\u002F作息感慨","作息感慨","2026-11-04 07:15:08","生活杂谈",[162],"日常",{"path":164,"title":165,"description":118,"date":166,"categories":118,"tags":167,"draft":131},"\u002Fposts\u002Fgp-tab-测试","gp-tab-测试","2026-10-12 10:51:22",null,{"path":169,"title":170,"description":171,"date":172,"categories":173,"tags":174,"draft":128},"\u002Fposts\u002Fnetwork-clean","怎么让网络变得干净","一个干净的网络，让人省心的使用服务。","2026-10-11 19:40:02","技术",[175,176,177],"VPN","网络","AI",{"path":179,"title":180,"description":181,"date":182,"categories":160,"tags":183,"draft":128},"\u002Fposts\u002Fthe-age-of-hesitation","[搬运] 不敢投入的时代","为什么我们现在越来越不敢投入了。很多时候我们以为的选择，比如你有两个工作offer，现在选择好像是我到底应该考公还是考编。明明这两个选项都不是真正的选项，因为你还没有开始，也不知道最后能不能成。这个时候我们就已经开始犹豫了。","2026-10-10 11:44:12",[184,185,186],"翻译","搬运","心理学",{"path":188,"title":189,"description":190,"date":191,"categories":160,"tags":192,"draft":131},"\u002Fposts\u002Fwhy-happiness-look-like-same","[搬运] 为什么大家的幸福都长一个样？","为什么大家的幸福都长一个样？一种认知被广泛传播后似乎变得不断扁平，这种感觉很奇怪。","2026-10-09 13:57:15",[184,185,186],{"path":194,"title":195,"description":118,"date":196,"categories":160,"tags":167,"draft":131},"\u002Fposts\u002Feverything-is-paid-twice","任何事情都要被消费两次","2026-10-06 12:30:06",{"path":132,"title":5,"description":127,"date":126,"categories":125,"tags":198,"draft":128},[136,137,138],{"path":200,"title":201,"description":202,"date":203,"categories":173,"tags":204,"draft":128},"\u002Fposts\u002Fspinning-up-policy-optimization","[译] 深度强化学习入门（三）：策略优化入门","OpenAI Spinning Up 教程第三部分的全文翻译：推导最简单的策略梯度、log-derivative trick、EGLP 引理、reward-to-go 与基线，配 PyTorch 示例代码。","2026-09-28 17:49:25",[205,206,207,184],"强化学习","RL","Spinning Up",{"path":209,"title":210,"description":211,"date":212,"categories":173,"tags":213,"draft":128},"\u002Fposts\u002Fspinning-up-rl-intro-2","[译] 深度强化学习入门（二）：强化学习算法的分类","OpenAI Spinning Up 第二部分的全文翻译：无模型与基于模型的 RL、策略优化与 Q-learning 的取舍，以及规划、专家迭代和经验增强等方法。","2026-09-28 10:20:21",[205,206,207,184],{"path":215,"title":216,"description":217,"date":218,"categories":173,"tags":219,"draft":128},"\u002Fposts\u002Fspinning-up-rl-intro","[译] 深度强化学习入门（一）：RL 的关键概念","OpenAI Spinning Up 教程第一部分的全文翻译：智能体与环境、状态与观测、动作空间、策略、轨迹、奖励与回报、价值函数、贝尔曼方程、优势函数，附 MDP 形式化。","2026-09-28 09:29:40",[205,206,207,184],{"path":221,"title":222,"description":223,"date":224,"categories":160,"tags":225,"draft":131},"\u002Fposts\u002Fwhen-machines-come-alive","冰箱之后，生活还会怎样改变？","从厨房里的果蝇想到冰箱、移动互联网与 AI 的多轮调用：AGI 已经到来，只是昂贵的智能会怎样改变生活？","2026-09-25 06:36:22",[177,226,227],"随笔","科技",{"path":229,"title":230,"description":118,"date":231,"categories":118,"tags":167,"draft":131},"\u002Fposts\u002Fmathematical-foundations-of-llms-1","LLM的数学基础（1）：","2026-09-19 22:59:55",{"path":233,"title":234,"description":118,"date":235,"categories":118,"tags":167,"draft":131},"\u002Fposts\u002Ftowards-maximize-serving-capatity","如何在 offline bench 里最大化 LLM serving system 的吞吐","2026-09-19 05:45:05",{"path":237,"title":238,"description":239,"date":240,"categories":125,"tags":241,"draft":128},"\u002Fposts\u002Fllm-memory-compute-tradeoff","LLM 的 Memory-Compute Trade-off：从 MHA 到 MLA、Sparse 与 Linear Attention","ds4.1f 上线了。它的架构 novel 在哪，便宜和快又是从哪省出来的。","2026-09-11 12:00:00",[136,242,243,244,245],"Attention","MLA","DeepSeek","长上下文",{"path":247,"title":248,"description":249,"date":250,"categories":125,"tags":251,"draft":131},"\u002Fposts\u002Fmarkdown-extensions","Markdown 扩展语法速查","本站 Markdown 支持的扩展语法演示与速查：admonition 提示块、GitHub 仓库卡片、spoiler 遮罩、ABC 乐谱、GP 乐谱、音频播放器、浏览器可运行代码块。","2026-09-01 15:30:00",[252],"博客",{"path":254,"title":255,"description":256,"date":257,"categories":160,"tags":258,"draft":131},"\u002Fposts","为什么还需要手写博客？","AI这么强，还要古法写博客吗？","2026-09-01 13:17:53",[252],{"path":260,"title":261,"description":118,"date":262,"categories":118,"tags":167,"draft":131},"\u002Fposts\u002Fllm-serving-1","LLM Serving（一）:以在H200上部署GLM-5.2为例","2026-07-21 06:15:44",{"path":264,"title":265,"description":118,"date":266,"categories":146,"tags":167,"draft":128},"\u002Fposts\u002Finteresting-music-rhythm","有趣的音乐节奏","2026-07-16 12:47:24",{"path":268,"title":269,"description":270,"date":271,"categories":173,"tags":272,"draft":128},"\u002Fposts\u002Ftowards-monosemanticity","LLM 可解释性（一）：SAE与单语义","读 Anthropic《Towards Monosemanticity》。从难以解释的神经元出发, 讲清superposition为什么让单个神经元混成一团, SAE怎么把多语义的激活分解成单语义特征。","2026-07-12 22:00:00",[136,273,274],"可解释性","SAE",{"path":276,"title":277,"description":278,"date":279,"categories":160,"tags":280,"draft":131},"\u002Fposts\u002Fpiketty-ai-china-k-shaped","用皮凯蒂的镜头看 AI 与中国 K 型社会","一次凌晨的对话整理。从 r>g 出发，看 AI 时代的资本分配、中国的世袭化趋势、体制内的二元劳动力市场，以及一个普通 00 后在一线打拼的真实牌面。","2026-06-24 03:00:00",[226,281,177],"经济",{"path":283,"title":284,"description":118,"date":285,"categories":118,"tags":286,"draft":131},"\u002Fposts\u002Fclaude-code-compact-behavior","cluade code compact 行为","2026-06-23 20:17:42",[287,288,289],"Agent","AI Agent","Claude Code",{"path":291,"title":292,"description":118,"date":293,"categories":173,"tags":294,"draft":131},"\u002Fposts\u002Fclaude-code-settings","Claude Code 配置分享","2026-06-23 14:22:12",[136,288,287,289],{"path":296,"title":297,"description":298,"date":299,"categories":125,"tags":300,"draft":128},"\u002Fposts\u002Fmy-thoughts-on-the-ai-industry","简单聊聊对 AI Agent 行业的理解","写这篇文章的初衷，是我发觉逐渐忘了前两个月的 AI\u002FLLMs 是什么样的能力和情况了。意识中似乎某些在突然加速，但理性和直觉告诉我这不对，作为行内人，我知道这是一个渐进的、累积的过程。我想自己回忆一下，Agent的发展。","2026-06-20 13:14:25",[136,288,287],{"path":302,"title":303,"description":118,"date":304,"categories":173,"tags":305,"draft":128},"\u002Fposts\u002Frotational-positional-encoding","RoPE","2026-06-17 18:31:40",[136,306,303],"Transformer",{"path":308,"title":309,"description":118,"date":310,"categories":173,"tags":311,"draft":128},"\u002Fposts\u002Fintroduction-to-large-language-model","大语言模型入门基础","2026-06-16 22:36:11",[306,136],{"path":313,"title":314,"description":118,"date":315,"categories":125,"tags":316,"draft":131},"\u002Fposts\u002Fcoding-agent","关于 Coding Agent 使用的遐思","2026-06-03 17:54:20",[287],{"path":318,"title":319,"description":320,"date":321,"categories":173,"tags":322,"draft":131},"\u002Fposts\u002Fabout-ssh-usage","关于SSH的使用——客户端篇","介绍SSH客户端的用法。","2025-06-17 02:44:57",[323],"工具",{"path":325,"title":326,"description":327,"date":328,"categories":125,"tags":329,"draft":128},"\u002Fposts\u002Fopensourcesoftwarestarttutorial","如何开始一个开源项目？","浅谈我对开源项目的认知。","2025-04-17 18:00:00",[330],"开源",{"path":332,"title":333,"description":334,"date":335,"categories":336,"tags":337,"draft":131},"\u002Fposts\u002Fmotivation","渴求动力","有意识的生活并主动地行动会使得生活更美好。","2025-03-05 00:44:57","生活-情感",[226],{"path":339,"title":340,"description":341,"date":342,"categories":336,"tags":343,"draft":131},"\u002Fposts\u002F2024summary","2024年终总结","啊，痛彻的时光；啊，悲怆的人！","2025-01-01 01:00:00",[226],{"path":345,"title":346,"description":347,"date":348,"categories":173,"tags":349,"draft":131},"\u002Fposts\u002Fvps-crash-startup","快速搭建个人VPS","如何快速搭建个人VPS。","2024-12-31 19:00:00",[350],"VPS",{"path":352,"title":353,"description":354,"date":355,"categories":173,"tags":356,"draft":131},"\u002Fposts\u002Fdata-label-software","AI数据（自动）标注软件概述","对市面上目前的AI数据（自动）标注软件进行调研。","2024-12-20 14:00:00",[357,358],"数据标注","深度学习",{"path":360,"title":361,"description":362,"date":363,"categories":336,"tags":364,"draft":128},"\u002Fposts\u002Fself-destruction","生活的崩坏与自我毁灭","是什么在牵引着我？我的行为，我的思想？我的感受？我所处的地方？它们在何处？","2024-12-19 12:00:00",[226],{"path":366,"title":367,"description":368,"date":369,"categories":173,"tags":370,"draft":131},"\u002Fposts\u002Fcs194-196_llmagents","LLM Agents 课程笔记","UCB CS294\u002F194-196 Large Language Model Agents 的课程笔记。","2024-11-30 20:00:00",[287],{"path":372,"title":373,"description":374,"date":375,"categories":336,"tags":376,"draft":131},"\u002Fposts\u002Fwhatsyourworth","价值、位置","你们的生存究竟有何价值？如果毫无价值，你们究竟为何存在？","2024-11-29 02:00:00",[226],{"path":378,"title":379,"description":380,"date":381,"categories":336,"tags":382,"draft":128},"\u002Fposts\u002Fblogrelifever2","重启博客v2","忙里偷闲学习前端，还是决定结合想法手搓一个自己的博客。","2024-11-25 00:00:00",[252],{"path":384,"title":385,"description":386,"date":387,"categories":336,"tags":167,"draft":131},"\u002Fposts\u002Fblogrelife","重启博客","在我的表达欲与现实所需(问题解决记录)驱动下，我决定重写博客。","2024-10-21 20:44:57",{"path":389,"title":390,"description":391,"date":392,"categories":160,"tags":393,"draft":128},"\u002Fposts\u002Fsurvival-welcome-to-university","生存手册——欢迎来到大学","大学同时生产三样东西：可验证技能、认证信号和驯化资本。你付了四年的钱，真正拿到手的是哪一样，取决于你毕业时要进的那个市场在为什么付费。","2024-10-12 20:00:00",[394,395,396],"大学","求职","方法论",{"path":398,"title":399,"description":400,"date":401,"categories":336,"tags":402,"draft":128},"\u002Fposts\u002Frevival","240205 复活","好吧，断更了许久，我又终于回来了，我又终于回来了。","2024-02-05 20:00:00",[226,403,404],"生活","返乡",{"path":406,"title":407,"description":408,"date":409,"categories":336,"tags":410,"draft":131},"\u002Fposts\u002Fclouds","「230717」云","我已经很久忘记了不我已经很少把自己的想法在lofter上发布的。","2023-07-17 20:00:00",[226,411],"散文",1795115925452]