{"id":8523,"date":"2026-10-11T15:07:28","date_gmt":"2026-10-11T15:07:28","guid":{"rendered":"https:\/\/silvybrand.com\/?p=8523"},"modified":"2026-10-11T15:07:28","modified_gmt":"2026-10-11T15:07:28","slug":"learning-local-ai-hermes-qwen-model-mac-studio-testing-diary-progress","status":"publish","type":"post","link":"https:\/\/silvybrand.com\/?p=8523","title":{"rendered":"Learning to use local AI is exciting, overwhelming, and frustrating"},"content":{"rendered":"


\n<\/p>\n

\n
\n

I\u2019m embarking on a journey I\u2019ve so far been hesitant to go on: actually using AI on the regular. Giving my most personal data to cloud services has been a major hold up for me, but it\u2019s increasingly possible to run powerful models locally to see what they\u2019re capable of. Is this tech so useful that it\u2019s worth throwing gobs of money at a computer or a laptop with lots of overpriced RAM? I\u2019ll try to help you find out. I\u2019m not an AI expert, it\u2019s going to be a journey for me, but maybe I can help you see if that journey is worth taking \u2014 and I\u2019ll be posting my way through it.<\/p>\n<\/div>\n

\n

A big part of Apple\u2019s pitch for its new Mac desktops is how good they are for local AI, and there\u2019s an entire line of new RTX Spark<\/a> Windows machines<\/a> landing<\/a> imminently<\/a> with up to 128GB of RAM<\/a> aimed at using agentic AI. Local AI and agents are just, like, so<\/em> hot right now. But, in all seriousness, the thought of having a little gofer living solely in the box on my desk that answers only to me and not overlords from OpenAI, Google, Microsoft, or Anthropic hits a bit different than trusting cloud services or subscribing to a chatbot. Don\u2019t get me wrong, there\u2019s still no way I\u2019m gonna have it write for me, edit for me, or generate ugly fliers<\/a>. But now, with some privacy on my side maybe I can finally think of useful busy work to \u201ctrust\u201d it with.<\/p>\n<\/div>\n

\n

So as part of my ongoing testing of the M5 Ultra Mac Studio<\/a>, as well as experimenting on other Mac and Windows systems, I\u2019ve installed some local LLMs and I\u2019m beginning to tinker. Here\u2019s how things are going.<\/p>\n<\/div>\n

\n
\n
\n
\n

1\/4<\/strong>

Hermes Agent can be used with cloud or local models.<\/em><\/figcaption><\/span><\/p>\n<\/section>\n<\/div>\n<\/div>\n<\/div>\n
\n

I started by installing the open-source Hermes Agent<\/a> on the Mac Studio. Hermes is a self-hosted AI agent desktop app that works on macOS, Windows, and Linux. It\u2019s completely free to use if you stick to local LLMs. Since this M5 Ultra Mac Studio has 256GB of unified memory, I could run just about any model I want. This is how things quickly get overwhelming. There are more models out there than I can count<\/a>, and some have specialized uses. Generally, models with many billions of parameters are able to do more than smaller ones with only a few billion. And since we don\u2019t have to worry about paying for tokens when running a model locally, I figured I\u2019d go big to start.<\/p>\n<\/div>\n

\n

Using Hermes\u2019 straightforward onboarding interface and model picker, I chose the Qwen 3.8 Flash Next<\/a> model \u2014 a 125-billion parameter model that\u2019s around 105GB in size. I also plan on spending some time with much smaller Qwen models on devices like an M6 Mac Mini, M5 MacBook Air, an Asus TUF Gaming A14 with AMD Strix Halo processor, and, eventually, RTX Spark. (Care to share what local models and hardware you use? You know where to leave your comments. 🫡)<\/p>\n<\/div>\n

\n

It wasn\u2019t long to get up and running with Hermes and Qwen, and even make them controllable via my phone by using a Telegram bot<\/a>. But then, I was presented with my usual nemesis when it comes to using AI: the text box.<\/p>\n<\/div>\n

\n

\u201cWhat do I do with this thing?\u201d<\/p>\n<\/div>\n

\n
\n
\n
\n
\n
\"My<\/a><\/div>\n<\/div>\n<\/div>\n

My Hermes daily briefing, sent to me daily via Telegram. Basic, I know.<\/em><\/figcaption><\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n

First up, to get things going, I followed some helpful YouTube advice<\/a> and had Hermes make me a daily morning briefing \u2014 to just start small with a cron job and ensure things are working. It scans my email and calendar, tells me anything worth my immediate attention, and gives me a short weather report. It\u2019s not all that useful, and you obviously don\u2019t need a $12,000 computer to build it, but it\u2019s something I can develop over time. The brief kept failing to generate at first, but then I eventually figured out that macOS couldn\u2019t be asleep when the job fired up at 7:30am. Duh! It works like clockwork now, but I have to think of more info for it to pull for me to make the results more helpful.<\/p>\n<\/div>\n

\n

A more useful task was reorganizing my Steam library. I have over 400 games on Steam, and I recently started categorizing them for when I want to play something new on a whim. Steam doesn\u2019t categorize your games for you, so you instead have to do it manually, one-by-one. But here\u2019s where an agentic lackey could fit right in. I asked, \u201cCan you reorganize my Steam library?\u201d Hermes could see most of my games by just viewing the already installed Steam client. It quickly came up with some organization options for me. I decided to have it organize my games by genre, but leave some of my own specialized categories(favorites, co-op titles, party games, games to play with my wife, etc.).<\/p>\n<\/div>\n

\n

You have to give a local AI some permissions over your machine for it to take actions for you, and in the case of my game library I had to register a Steam web API key<\/a> and give it to Hermes. Once it had that access, it only took a few minutes for it to churn through sorting all the games into categories. And the API key was easy to revoke right after, since I don\u2019t need it further. The idea is for it to just do this heavy lifting for me once, and then I\u2019ll file new games into the categories as I eventually buy them. It did a fine job overall, and now my Steam library is easily browsable.<\/p>\n<\/div>\n

\n
\n
\n
\n
\n
\"<em>Starting<\/a><\/div>\n
\"<em>The<\/a><\/div>\n