pull down to refresh

I'm fascinated by the promise of LLMs. A handful of my friends are GPT power users, and I've been using generative models for ~5 years at this point, so I feel like I have a decent sense of what they're capable of and what they're not good at.
What I don't have a good sense of, is how to use them in my own workflow. The reason for this is that I fundamentally don't trust the centralized providers of LLM technology not to create a shadow profile of me, and then to use that for something nefarious.
So, last week I worked on an enterprise implementation of a document analysis tool at a hackathon with my day job. It was awesome, and I understand more about how feasible this is on my own machine now. And I intend to implement such a tool with my journals and notes, locally, with the intent of building myself a collection of agents and helpers.
Anybody else doing this?
Yes. If you have a desktop with dedicated gpu or a macbook with unified memory just do it it. If you have other hardware configurations maybe don't.
reply
Yeah, I've got a M2 Mac right now, but eventually I'll pick up a beastly GPU to handle larger models. For the past year, or a bit more, I've been running Mistral (and a couple others) locally with Ollama.
I've spent some time today using Python + Ollama (Mistral) + ChromaDB to vectorize my journal entries, and now I'm tinkering with RAG implementation... it's not my favorite thing yet, but I've only just gotten the RAG to start returning documents from the DB
Mostly I'm interested in using the OpenWeb-UI front end (plus its various tools) to augment myself & processes.
reply
15 sats \ 1 reply \ @kepford 5h
Have a colleague that was showing me a proof of concept using some models to create a symatic search over some data... Kinda wild how simple it is to do. That's what surprised me most. The complexity is really in the models. Been dipping my toe into running stuff locally as well. It's interesting.
reply
I'm using this as a frontend locally: https://docs.openwebui.com
reply