June 17 - The era of local personal AI is coming.

For the past 2 and a half years Personal AI has mostly meant online AI resources such as ChatGPT, Perplexity.ai, Anthropic Claude, Grok, Google Gemini and Notebooklm, etc. that you access with your browser on your PC.

It meant queries on the content they already had, web searches, web scraping, and other cloud based capabilities, but not on your data unless you are willing to send it up to the cloud and hope no one else can see it. Not worth the risk

Now the opportunity for Local Personal AI is coming.

Total privacy, full AI, and custom AI applications all on your own content, on your own system, never leaving your space.

Of course the existing hardware and software is not optimal for this, you’re going to need a new computer, dude. (bro?)

You may need a fast GPU with 24gb or more memory, depending on the size and performance requirements of your AI applications. Maybe more.

And if you want to train your own models or fine tune an open source one on your data without sending it up, you will probably need more resources than that even.

You’ll probably want to start by duplicating the online resources you are currently using in the cloud but locally instead, on your own private documents, spreadsheets and databases.

ChatBots, Ideation, Image Generation, Automated workflows incorporating Agents, MultiModal Agents, Orchestrated Agent swarms, etc, etc. but all running locally and on your private data.

The age of Local Personal AI

Watch for DGX systems from Nvidia and others. We’ll offer some of them here in the newsletter

But for software, it’s going to be interesting.

We’re focusing on Docker as a way to simply install sophisticated AI applications and utilities and tools such as n8n, ollama, local models, etc so almost everything you are using now in the cloud can be made available locally and privately on Windows, MacOS and Linux.

There are a number of kinks to work out and it is going to take the coming months to straighten them out.

Here are some youtubes about it:

I asked perplexity.ai about this and here is what it told me:

First i said:

I'm focusing on Local Personal AI , with n8n running from docker on windows with nvidia 3090 gpu

• using ollama and openrouter and various n8n workflows as a main personal ai system

• then migrating to agents in n8n local,

• then python agent versions, again in docker

• in preparation for the coming systems from nvidia, microsoft, and others.

• There are a lot of gotchas involving amd and docker gpu, macos and docker, etc. some of which you've identified in the 2.5hr long local ai youtube

• the more you can provide on this the better for me

Here’s what it replied with

How to Check NVIDIA GPU Access in Docker on Windows

To verify that your NVIDIA GPU is accessible from within a Docker container on Windows, follow these steps:

Prerequisites

• Windows 10 or 11 with WSL 2 enabled

• NVIDIA GPU with up-to-date drivers supporting WSL 2 GPU Paravirtualization

• Docker Desktop installed and configured to use the WSL 2 backend

• NVIDIA Container Toolkit (comes with Docker Desktop for GPU support)

Step-by-Step Verification

1. Start Docker Desktop

Ensure Docker Desktop is running and configured to use the WSL 2 backend12.

2. Run a GPU Benchmark Container

Open a terminal (PowerShell, Command Prompt, or WSL shell) and execute the following command:

bash

docker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

• The --gpus=all flag ensures the container can access all available GPUs.

• This command runs a CUDA sample that benchmarks GPU performance123.

3. Interpret the Output

If GPU access is working, you will see output mentioning your GPU model and benchmark results, such as:

text

GPU Device 0: "GeForce RTX XXXX" with compute capability X.X ... = 2724.379 single-precision GFLOP/s at 20 flops per interaction

If the GPU is not available, you will see an error or no GPU devices listed123.

4. Additional Check: nvidia-smi

You can also run a container with the NVIDIA CUDA image to check GPU status:

bash

docker run --rm --gpus=all nvidia/cuda:12.0-base nvidia-smi

This should display a table with your GPU details if everything is set up correctly4.

Troubleshooting Tips

• Make sure your NVIDIA drivers are up to date.

• Ensure Docker Desktop is the latest version.

• Always use the --gpus=all flag in your docker run command52.

• If you encounter issues, restart Docker Desktop and your computer after updating drivers or Docker.

Summary Table

StepCommand/ActionExpected OutputRun CUDA benchmarkdocker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmarkGPU model and benchmark results123Check with nvidia-smidocker run --rm --gpus=all nvidia/cuda:12.0-base nvidia-smiGPU details table4

By following these steps, you can confirm that your Docker containers on Windows have access to your NVIDIA GPU123.

1. https://docs.docker.com/desktop/features/gpu/

2. https://www.devzero.io/blog/docker-gpu

3. https://grand-challenge.org/documentation/setting-up-wsl-with-gpu-support-for-windows-11/

4. https://blog.roboflow.com/use-the-gpu-in-docker/

5. https://www.reddit.com/r/docker/comments/1atfveb/how_do_i_give_a_docker_on_docker_desktop_access/

6. https://stackoverflow.com/questions/25185405/using-gpu-from-a-docker-container

7. https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html

8. https://learn.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/gpu-acceleration

9. https://dev.to/thenjdevopsguy/using-nvidia-gpus-with-docker-in-5-minutes-386g

10. https://forums.docker.com/t/applications-not-using-gpu-inside-the-container/140376

It’s going to take some time to go though all of this. You can see the challenges

If you’re interested in Local Personal AI , just reply to this email and make some comments. Yeah, you can reply to these. Next the email goes to the website and sits there for future reference..

I can help you get going with your own Beehiiv newsletters as well

Later