Most active commenters

    ←back to thread

    DeepSeek-v3.1

    (api-docs.deepseek.com)
    776 points wertyk | 14 comments | 21 Aug 25 19:06 UTC | HN request time: 0.001s | source | bottom
    1. seunosewa ◴[21 Aug 25 20:05 UTC] No.44977398[source]▶
    It's a hybrid reasoning model. It's good with tool calls and doesn't think too much about everything, but it regularly uses outdated tool formats randomly instead of the standard JSON format. I guess the V3 training set has a lot of those.
    replies(2): >>44977985 #>>44978742 #
    2. ivape ◴[21 Aug 25 20:56 UTC] No.44977985[source]▶
    What formats? I thought the very schema of json is what allows these LLMs to enforce structured outputs at the decoder level? I guess you can do it with any format, but why stray from json?
    replies(2): >>44978153 #>>44979158 #
    3. seunosewa ◴[21 Aug 25 21:11 UTC] No.44978153[source]▶
    Sometimes it will randomly generate something like this in the body of the text: ``` <tool_call>executeshell <arg_key>command</arg_key> <arg_value>echo "" >> novels/AI_Voodoo_Romance/chapter-1-a-new-dawn.txt</arg_value> </tool_call> ```

    or this: ``` <｜toolcallsbegin｜><｜toolcallbegin｜>executeshell<｜toolsep｜>{"command": "pwd && ls -la"}<｜toolcallend｜><｜toolcallsend｜> ```

    Prompting it to use the right format doesn't seem to work. Claude, Gemini, GPT5, and GLM 4.5, don't do that. To accomodate DeepSeek, the tiny agent that I'm building will have to support all the weird formats.

    replies(3): >>44978260 #>>44983225 #>>44983486 #
    4. ◴[21 Aug 25 21:23 UTC] No.44978260{3}[source]▶
    5. darrinm ◴[21 Aug 25 22:14 UTC] No.44978742[source]▶
    Did you try the strict (beta) function calling? https://api-docs.deepseek.com/guides/function_calling
    6. refulgentis ◴[21 Aug 25 22:57 UTC] No.44979158[source]▶
    In the modes in APIs, the sampling code essentially "rejects and reinference" any token sampled that wouldn't create valid JSON under a grammar created from the schema. Generally, the training is doing 99% of the work, of course, it's just "strict" means "we'll check it's work to the point a GBNF grammar created from the schema will validate."

    One of the funnier info scandals of 2025 has been that only Claude was even close to properly trained on JSON file edits until o3 was released, and even then it needed a bespoke format. Geminis have required using a non-formalized diff format by Aider. Wasn't until June Gemini could do diff-string-in-JSON better than 30% of the time and until GPT-5 that an OpenAI model could. (Though v4a, as OpenAI's bespoke edit format is called, is fine because it at least worked well in tool calls. Geminis was a clown show, you had to post process regular text completions to parse out any diffs)

    replies(1): >>44979191 #
    7. dragonwriter ◴[21 Aug 25 23:02 UTC] No.44979191{3}[source]▶
    > In the modes in APIs, the sampling code essentially "rejects and reinference" any token sampled that wouldn't create valid JSON under a grammar created from the schema.

    I thought the APIs in use generally interface with backend systems supporting logit manipulation, so there is no need to reject and reinference anything; its guaranteed right the first time because any token that would be invalid has a 0% chance of being produced.

    I guess for the closed commercial systems that's speculative, but all the discussion of the internals of the open source systems I’ve seen has indicated that and I don't know why the closed systems would be less sophisticated.

    replies(1): >>44979848 #
    8. refulgentis ◴[22 Aug 25 00:33 UTC] No.44979848{4}[source]▶
    I maintain a cross-platform llama.cpp client - you're right to point out that generally we expect nuking logits can take care of it.

    There is a substantial performance cost to nuking, the open source internals discussion may have glossed over that for clarity (see github.com/llama.cpp/... below). The cost is very high, default in API* is not artificially lower other logits, and only do that if the first inference attempt yields a token invalid in the compiled grammar.

    Similarly, I was hoping to be on target w/r/t to what strict mode is in an API, and am sort of describing the "outer loop" of sampling

    * blissfully, you do not have to implement it manually anymore - it is a parameter in the sampling params member of the inference params

    * "the grammar constraints applied on the full vocabulary can be very taxing. To improve performance, the grammar can be applied only to the sampled token..and nd only if the token doesn't fit the grammar, the grammar constraints are applied to the full vocabulary and the token is resampled." https://github.com/ggml-org/llama.cpp/blob/54a241f505d515d62...

    replies(2): >>44980832 #>>44983711 #
    9. dragonwriter ◴[22 Aug 25 03:45 UTC] No.44980832{5}[source]▶
    Thanks for the explanation!
    10. irthomasthomas ◴[22 Aug 25 11:32 UTC] No.44983225{3}[source]▶
    Can't you use logit bias to help with this? Might depend how they are tokenized.
    11. ilaksh ◴[22 Aug 25 12:04 UTC] No.44983486{3}[source]▶
    Maybe you have your temperature turned up too high.
    12. 7thpower ◴[22 Aug 25 12:27 UTC] No.44983711{5}[source]▶
    This is a basic question but maybe you can help: what is a good resource to use to understand how to take advantage of logits?
    replies(1): >>44986335 #
    13. ivape ◴[22 Aug 25 16:12 UTC] No.44986335{6}[source]▶
    https://dottxt-ai.github.io/outlines/latest/

    For OpenAI, you can just pass in the json_schema to activate it, no library needed. For direct LLM interfacing you will need to host your own LLM or use a cloud provider that allows you too hook in, but someone else may need to correct me on this.

    If anyone is using anything other than Outlines, please let us know.

    replies(1): >>44997321 #
    14. 7thpower ◴[23 Aug 25 17:03 UTC] No.44997321{7}[source]▶
    Thank you!