Codatory's Scribbles

Fix drm-kmod after upgrading FreeBSD to 14.2-RELEASE

2024-12-12T00:00:00-05:00

Caution: this is probably not the best way to do this. I took a break from FreeBSD and am just getting back into it.

Do this from your “broken” install of 14.2. You may need to use an alternate console or SSH.

Prep Ports and OS Sources

pkg install -y git portmaster

git clone --branch releng/14.2 https://github.com/freebsd/freebsd-src.git /usr/src

git clone --branch main https://github.com/freebsd/freebsd-ports.git /usr/ports

Rebuild drm-kmod and drm-kmod-61

pkg remove drm-kmod drm-61-kmod drm-515-kmod drm-510-kmod

portmaster -f --packages-build -r drm-kmod drm-61-kmod

Reboot

Hope you work now!

Acadia National Park

2023-09-27T01:00:00-04:00

Honestly, what can I say that hasn’t already been said about Acadia? It was absolutely gorgeous, and picking just a few hikes to fit in the time I had was quite the challenge. If you have the opportunity, definitely go visit!

Quechee Gorge

2023-09-19T01:00:00-04:00

If you google for things to do along I-91 in Vermont, one of the top things on the list will inevitably be Quechee Gorge Trail. The state park made for a convenient and affordable stop for the night, with direct access to the trail and easy access to the shops nearby. Heads up though, the trail from the campground down is pretty serious so if you aren’t a big hiker take the shortcut out to the road and join the trail at the bridge where things are much easier.

The Journey Begins

2023-09-18T01:00:00-04:00

It’s been an adventure all its own to kick off this season of travel. I gave up my apartment as part of the kickoff, which required a lengthy and complex move of everything into storage. On top of that, in the middle of the move I had to take the RV back to the factory to get some long delayed repairs made.

Thankfully, with the help of my family and friends, I was able to get everything going and into good shape and it’s time to begin my travels.

The first leg of travel in this season is to the New England area, because I haven’t had a chance to spend time in that part of the country before and September / October are great times to see the foliage. Normally, I’d try to have an itinerary map to start off a segment, but between some technical issues and some very off the cuff planning I don’t really have a great one to share.

The general goal right now is to make my camping reservation at Acadia National Park on the 24th of September, and after that I’ll be able to more or less bop around as I won’t be on such a tight deadline.

Today, I’m enjoying the sound of rain on the metal roof in Pownal, VT at the lovely Pine Hollow Campground after spending the weekend ripping across New York and Ohio. Tomorrow I’ll move on to Quechee State Park to hike the Quechee Gorge Trail and Wednesday I’ll be heading to Shelborne, NH to finish out the work week.

And now, for some photos from the trip so far…

Scioto Park, Dublin, Ohio

Gorman Nature Center, Mansfield, Ohio

Der Dutchman, Bellville, Ohio

Cuyahoga Valley National Park

2022 Thor Tellaro L Year in Review

2023-04-24T01:00:00-04:00

The astute readers among you might recognize this post is a bit late for a year-end review. Unfortunately, part of owning a new RV is dealing with problems and my rig has been out of commission for about 8 weeks now due to a failure in the electrical system. Unfortunately, my repair experience will be one that, when complete, will join the horror stories you’ve undoubtedly heard before.

But, since that’s currently a developing story, let’s talk about the year I had with the rig before it cut my trip short.

Let’s start with the things that are good, because I kind of need to cheer myself up to write the rest of this post.

A Class B rig is super easy to get around in. Generally speaking, if a full-size pickup fits, I fit. With no slides, I’m barely longer than a regular parking space and if parks in my driveway at home with no drama whatsoever.
The tanks are perfectly sized for me. I can plan on dumping and filling once a week, and it only takes a few minutes.
The included features are quite reasonable for an entry-level Class B. The furnace is quiet and efficient, the AC is effective, the kitchen and bathroom is functional and generally speaking it’s a nice place to hang out.
I’ve gone on some pretty epic adventures.
https://photos.app.goo.gl/M33xJX62gHaoRyMt7

Now, some of the changes or additions I’ve made over the year…

Garmin RV GPS - Just because I can fit anywhere most trucks can doesn’t mean it makes for a nice drive. A Garmin RV890 was added after my regular GPS decided to route me through a mountain pass to save 10 minutes. I spent more than 10 minutes cleaning up the mess that shaking and rattling made!
Cradlepoint Dual modem 5G Router - Part of what makes RV travel appealing to me is the ability to be gone for a long time, but I still need to keep connected. Cradlepoint’s products are the industry leaders and this thing has been an amazing addition to my rig.
Black & Decker FlexVAC - Pretty obvious, right?
Google Chromecast w/ Google TV - You need something to play stuff on the TV and these are really cheap…

What hasn’t been so good, then?

Well, the electrical system still sucks. The Under-hood generator failed while I was on my recent trip through Texas in a relatively spectacular manner. I’ll probably write up a whole post on what happened there whenever it finally gets resolved. But more importantly, the dealer support after sales has been truly horrific. My selling dealer, LazyDays of Elkhart (now a Camping World location), was perfectly happy to schedule a service appointment to drop off the rig (3 hours away) but once they had the rig in their possession refused to answer the phone, provide any sort of status update or anything else. It wasn’t until I sat down and repeatedly called their number that someone accidentally answered and then basically shrugged me off and said “yeah, it’s fixed, come pick it up.”

When I went to pick it up, I can just say for sure it wasn’t fixed. Now the coach is in worse condition than when I dropped it off and Thor warranty support has been completely unhelpful in finding a new option that might have some clue on how to work on the thing (or at the very least answer the phone).

Hello Keepalived

2022-12-03T08:00:00-05:00

My K8S cluster has been pretty flaky, with lots of errors on the KubeAPI and Metrics Server going unavailable regularly. I was also frequently unable to get logs from my pods. Turns out I should’ve read the K0S documentation better.

Control plane high availability requires a tcp load balancer, which acts as a single point of contact to access the controllers.

Well, the hardware I have is really the hardware I want to use in this cluster. Sure I could spin up some VMs or Raspberry Pis to do the load balancing but I’d much rather use the hardware that’s there. We aren’t pushing things super hard, so it shouldn’t be super demanding or anything.

So, I pulled out my old pal Keepalived. My original plan was to use LVS to load balance the services, but after repeated problems with getting the port bindings to work reliably I decided load balancing wasn’t really that necessary either. Instead, I’m going to allow keepalived to float around an IP and k0s will bind to it directly. This means if I do get more API traffic than a single node can handle to the floating IP I’ll have problems. I don’t expect problems.

To make it work, you need to enable a Linux kernel feature called nonlocal bind. This allows services to bind to addresses that aren’t physically present on the server. Then I have keepalived configured to monitor the K0S processes and float the IP address to the server with the most services working. Per usual, here’s an Ansible Playbook with my configuration. You’ll probably need to sub in your interface IDs and the IP you want to use as a floater.

Once it’s configured, you’ll need to update your k0sctl config to have it use the new floating IP. You do this by setting the externalAddress on the API Server and populating the SANs with all valid addresses (and hostnames) for your cluster.

  k0s:
    version: 1.25.4+k0s.0
    dynamicConfig: false
    config:
      spec:
        api:
          externalAddress: 10.8.0.10
          sans:
            - 10.8.0.10
            - 127.0.0.1
            - 10.8.0.11
            - 10.8.0.12
            - 10.8.0.13

Hello Host Patching

2022-11-20T08:00:00-05:00

Well, it’s been a few days and now we want to check for updates on our host systems and be able to patch them. After all, we can’t deploy workloads until we have reached some baseline operational maturity. While there are some cluster-native ways to orchestrate patching I’m still happy to have my legacy tooling (Ansible) do it as a slightly simpler workflow.

Patching a K8S cluster isn’t entirely difficult or entirely straightforward, there are a few things that need to happen in sequence to make sure everything goes off well. First off, you don’t want to induce too much load on the cluster at once. Second off, becore you take a node offline you need to drain the pods off of it, and lastly when the node is ready you need to uncordon it to allow work to be scheduled on it again.

My Ansible Playbook to do that is here. It uses needrestart to check for services that need restarted and if a reboot is required does all the dran/uncordon work using kubectl on the admin workstation.

Hello Monitoring

2022-11-19T09:00:00-05:00

Perfect, we’ve got storage sorted out and now we can run real applications. Let’s deploy monitoring to our cluster so we can see how it’s performing and check that everything is working correctly. We’ll be deploying the Prometheus Stack, as this is the most popular and best supported monitoring solution for Kubernetes. Any guesses how this starts?

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Yep; a new helm repo. This chart needs enough options we’ll go ahead and provide them with a YAML file. We’re turning alertmanager off for now but you’re welcome to learn how to configure it if you want. In Grafana we’re setting a default timezone, enabling persistent storage, and exposing the service as a LoadBalancer (so that MetalLB will assign it an IP address) and working around a bug. The Prometheus section also has a long block to have it request storage as part of the deployment.

helm upgrade --install --wait --create-namespace --namespace monitoring prometheus-stack prometheus-community/kube-prometheus-stack --values=prometheus.yml

Once the deployment is completed, you should be able to run kubectl get svc -n monitoring to find the IP address for Grafana. Punch that into your browser and log in with admin prom-operator, and browse through the Dashboards page.

Hello NFS

2022-11-19T08:00:00-05:00

Ok, now we have a problem. We haven’t told Kubernetes where it can find storage for our apps. In this example, we’re going to use NFS but there are loads of other options like OpenEBS and Gluster. I’m going to assume you’ve already configured your NFS server of choice, and you know the appropriate mount options.

Guess what? Yep, we’re adding another helm repo.

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

Now we’re going to be lazy and create one NFS provisioner that will pick up requests for volumes and map them to NFS, and make sure it cleans them up when they’re no longer needed. You can do a lot more, but for now just make sure the server and path are correct for your environment.

helm install --create-namespace --namespace nfs-system \
    nfs-subdir-external-provisioner \
    nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server=10.8.0.3 \
    --set nfs.path=/KuberNFS \
    --set storageClass.defaultClass=true \
    --set storageClass.onDelete=delete

Hello MetalLB

2022-11-18T21:00:00-05:00

Ok, we need an easy way to connect services running in our cluster to the external network. The method I chose is MetalLB, because it’s a very simple and flexible way to expose services without relying on an external load balancer. MetalLB plays well with the CNI (Kube-Proxy) used by k0s, so we don’t need to do anything special to prepare for the installation. I already have Helm installed on my administration system, so deploying MetalLB is really simple. First we need to add the repository to Helm.

helm repo add metallb https://metallb.github.io/metallb

Once the repository is added, we can issue a command to have helm install (or update) MetalLB for us.

helm upgrade --install --create-namespace --namespace metallb-system metallb metallb/metallb

Once the chart has been installed, all the necessary pods will be downloaded and started on the appropriate servers. Lastly, we need to give the system some IP addresses to assign to services. We do this with a YAML file that we apply with kubectl apply -f. Be sure to select addresses that are available on the network your servers are attached to!

Oh did it fail for you too? In both of my installs applying the YAML file failed with a webhook error until I re-started the controller pod. No idea why. kubectl -n metallb-system get pod | grep control | awk '{print $1}' | xargs kubectl delete -n metallb-system pod ought to do the job to get you going.

Now, let’s deploy something so we can test and make sure this thing is working. We’ll deploy something called in Ingress, which is basically a shared web server that allows us to route traffic into the cluster using HTTP. The details don’t matter right now, so let’s just get going.

One again we need to add a repository to Helm:

helm repo add traefik https://traefik.github.io/charts

Then we will deploy the chart.

helm upgrade --install --create-namespace --namespace traefik-system traefik traefik/traefik

This will go and deploy some stuff for us, and once it’s done we can run kubectl -n traefik-system get services and in the External-IP column you should have an IP that was in the range in the YAML file you provided. If you try and load that IP you should get a 404 error. Congrats!