Let's talk Local LLMs - So many questions

zielperson
2023-06-14T08:17:49Z

Hello there
(oh god, I am opening my first thread here - so exciting)

I'd love to ask the people here about local LLMs.
To be honest, I got interested in this topic, but am leaving reddit, where a sub r/locallama exists.
I don't want to interact with that site anymore, so I am taking this here.

My questions, to start us off:

Models are available on huggingface (among other places), but where do I get the underlying software? I read "oogabooga" somewhere, but honestly, I am lost.

If I only want to USE a local model, what are the requirements, and how do I judge if I can use something from the values of "4bit / 8 bit" and "30B, 7B"??

If I get crazy and want to TRAIN a LorA ... what then?

Good resources / wiki pages, tutorials, etc?

Comments URL: https://tildes.net/~tech/169c/lets_talk_local_llms_so_many_questions

Votes: 21

Comments: 25