Not a good starter guide for beginners IMO. It's weirdly fixated on cloud-only, GPU-only tooling, and only mentions running the FP16 full-weighted version of the model - when the real selling...
Not a good starter guide for beginners IMO.
It's weirdly fixated on cloud-only, GPU-only tooling, and only mentions running the FP16 full-weighted version of the model - when the real selling point of Mistral has been how resource-light/easy to run on consumer hardware it is.
I haven't had a chance to run Mistral myself yet, but for anyone who does want to run it easily, the HN thread was reporting that it was smooth to set up through ollama, with simply:
Fair warning though that the above model isn't instruction-tuned version. The -instruct version of the model is much easier for general use, but may require a bit more finagling to get set up. I'm planning to get my feet wet with it this weekend and test it out as a local 3.5-GPT replacement, so if anyone has questions or wants to trade notes, feel free to comment or PM.
Just a little anecdote about how nice it is to have uncensored models — I’ve been working on this little search tool that can analyze search results for me. Most of the time I have been using...
Just a little anecdote about how nice it is to have uncensored models — I’ve been working on this little search tool that can analyze search results for me. Most of the time I have been using Anthropic’s Claude model since it’s got high parallel availability which is super useful if I want to summarize a hundred documents at once. I regularly get a refusal to answer because of reasons like hate speech, violence, etc — from asking it to summarize news content from Reuters. Today I noticed it is refusing to answer requests to summarize several documents about truth and reconciliation day. It’s so frustrating dealing with these paternalistic policies.
Not a good starter guide for beginners IMO.
It's weirdly fixated on cloud-only, GPU-only tooling, and only mentions running the FP16 full-weighted version of the model - when the real selling point of Mistral has been how resource-light/easy to run on consumer hardware it is.
I haven't had a chance to run Mistral myself yet, but for anyone who does want to run it easily, the HN thread was reporting that it was smooth to set up through ollama, with simply:
ollama run mistralhttps://github.com/jmorganca/ollama
Fair warning though that the above model isn't instruction-tuned version. The
-instructversion of the model is much easier for general use, but may require a bit more finagling to get set up. I'm planning to get my feet wet with it this weekend and test it out as a local 3.5-GPT replacement, so if anyone has questions or wants to trade notes, feel free to comment or PM.Just a little anecdote about how nice it is to have uncensored models — I’ve been working on this little search tool that can analyze search results for me. Most of the time I have been using Anthropic’s Claude model since it’s got high parallel availability which is super useful if I want to summarize a hundred documents at once. I regularly get a refusal to answer because of reasons like hate speech, violence, etc — from asking it to summarize news content from Reuters. Today I noticed it is refusing to answer requests to summarize several documents about truth and reconciliation day. It’s so frustrating dealing with these paternalistic policies.