eGPU Setup for AI Homelab

Yes, we all are looking for the personal AI that will sit in our basement and do all the work for us. I guess that will still take a while, but let’s get started with some testing.

I tried several setups on my laptop, but even though it has a 2060 Chip in it, let’s be real, it just dies on anything serious I do. So I started to look around for alternative hardware that I could afford (we are talking hobby and home lab here). I was not willing to spend more than a 1000 Euros.

I ended up going the eGPU route, meaning you need to have some machine, NUC, PC, Laptop whatever that has working thunderbolt 3 or 4.

Using my Intel NUC 10 with an Intel Core i5-10210U, which seemed to be the only working thunderbolt device in my possession.

Looking for an eGPU housing I wanted one of those where you plugin and decide what card you want despite there being attractive all in one boxes out there as well. I found a few which are similar and ended up getting a used Sonnet eGPU Breakaway Box 750ex https://www.sonnettech.com/product/egpu-breakaway-box/overview.html the 750 without the „ex“ would have been fine, but this one was offered, used in great condition for 300 Euros. Which has got a 750W 80 Plus power supply.

Now in order to get a card in there you have to watch the dimensions. There are many nice cards out there expensive as hell. New was out of the question. All 4060s I checked where ok, but doing AI you want to get as much as you can. So I started watching the used market.

Pretty soon I noticed that there where 4070s in my price range. I decided to go hunting for 4070 SUPER (12 GB VRAM). Paying attention to the cooling (we are stuffing it into this little box) I saw some RTX 4070 SUPER Founders Edition on the used market. I missed getting one for 500 Euros, but managed to get one in great shape with original packaging for 560 Euros.

Done.

Stayed under 1000 Euros for my eGPU Setup for my AI Homelab and am now ready to get started with some LLaMA 7B, Stable Diffusion XL, Codellama etc.