The barriers to AI engineering are crumbling fast

After just spending 15 minutes trying to get something useful accomplished, anything useful at all, with latest beta Apple Intelligence with a M1 iPad Pro (16G RAM), this article appealed to me!

I have been running the 32B parameters qwen2.5-coder model on my 32G M2 Mac and and it is a huge help with coding.

The llama3.3-vision model does a great job processing screen shots. Small models like smollm2:latest can process a lot of text locally, very fast.

Open source front ends like Open WebUI are improving rapidly.

All the tools are lining up for do it yourself local AI.

The only commercial vendor right now that I think is doing a fairly good job at an integrated AI workflow is Google. Last month I had all my email directed to my gmail account, and the Gemini Advanced web app did a really good job integrating email, calendar, and google docs. Job well done. That said, I am back to using ProtonMail and trying to build local AIs for my workflows.

I am writing a book on the topic of local, personal, and private AIs.

Another thought: OpenAI has done a good enough job productizing ChatGPT with advanced voice mode and now also integrated web search. I don’t know if I would trust OpenAI with access to my Apple iCloud data, Google data, my private GitHub repositories, etc., but given their history of effective productization, they could be a multi-OS/platform contender.

Still, I would really prefer everything running under my own control.

Who can trust a company whose name contradicts its presence.

Have you tried RAG on Open WebUI. How does it do in asking questions from source docs?

I don’t disagree with you!

Not yet. It has ‘Knowledge sources’ that you can set up, and I think that supplies data for built in RAG - but I am not sure until I try it.

I wrote a script to queue and manage running llama vision on all my images and writing the results to an sqlite db used by my Media Viewer, and now I can do text or vector search on it. It's cool to not have to rely on Apple or Google to index my images and obfuscate how they're doing it from me. Next I'm going to work on a pipeline for doing more complex things like multiple frames in a video, doing multiple passes with llama vision or other models to separate out the OCR, description, and object, people recognition. Eventually I want to feed all of this in here https://lowkeyviewer.com/ and have the ability to manually curate the automated classifications and text.

I'm curious why you find descriptions of images useful for searching. I developed a similar flow and ended up embedding keywords into the image metadata instead. It makes them easily searchable and not tied to any databases, and it is faster (dealing with tens of thousands of images personally).

* https://github.com/jabberjabberjabber/LLavaImageTagger

It's not as good as tags but it does pretty ok for now especially since searching for specific text in an image is something I want to do a lot. I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there.

This is the prompt I've been using.

"Create a structured list of all of the people and things in the image and their main properties. Include a section transcribing any text. Include a section describing if the image is a photo, comic, art, or screenshot. Do not try to interpret, infer, or give subjective opinions. Only give direct, literal, objective descriptions of what you see."

> I'm trying to work on getting llama to output according to a user defined tagging vocabulary/taxonomy and ideally learn from manual classifications. Kind of a work in progress there.

Good luck with that. The only thing that I found that works is using gbnf to force it, which slows inference down considerably.

Not me. I learned that lesson after I tried to take a bite out of my Apple Macintosh.

can llama 3.3 vision do things like "there's a textbox/form field at location 1000, 800 with label "address"" ?

I did a quick and dirty prototype with Claud for this, but it returned everything with an offset and/or scaled.

Would be a killer app to be able to auto-fill any form using OCR.

I can't speak to the OPs decision, but I also have a similar script set up that adds a combination of YOLO, bakllava, tesseract etc. and also puts it along with a URI reference to the image file into a database.

I actually store the data in the EXIF as well, but the nice thing about having a database is that it's significantly faster than attempting to search hundreds of thousands of images across a nested file structure, particularly since I store a great deal of media on a NAS.

Well, clearly the apple already has a bite taken out of it, so that's user error

You are getting access to some of the best AI tools for free, by that definition isnt that open?

Open source frontend??? Wtf. Show your code. This post is bs. AI coding has not advanced software in any meaningful way and will make coders dumber

nice!

Were you using claude’s computer mode? It can do this

You wouldn't happen to have this on github or have some other way to share it? I am interested in seeing how you implemented it.

No, I used the regular Claude which can also (somewhat) do this and uses the same image processing backend as "computer use" as far as I know (source: Antropic CEO interview with Lex Friedman recently).

Computer use is also not very good at it (often mis-clicking for example).

I'm guessing this will work flawlessly within 6 months to a year or so, but it doesn't seem ready yet.