Projects and links more related to home networking than professional ops
Home Automation Ultimate Home Server - Massive Helm chart for a homelab, designed for a single-node k3s deploy
With a recent update to 89.0.4389.82 of ChromeOS, there was also an update to the underlying container that hosts Linux. Normally if this happens it’s a relatively short process. However, a few days ago this took far longer, and ended up with a “Error starting penguin container: 58” message. Not helpful at all! Fortunately with some searching, I found the solution on Masaki Muranaka’s blog. The article is in Japanese, but fortunately Google translate was able to work well enough for me to get the steps necessary to fix the issue.
Happy 2021! And to get the year started, let’s look at some modern logging stacks!
I love modern application and stack observability. Anyone that remembers the old days of stringing together Nagios and Syslog into a cobbled-together stack can certainly sympathize. Now solutions exist that provide a more integrated experience across metrics, logs, and tracing. Today I’ll be concentrating on the logs component with Grafana’s Loki, and demonstrate that stack on a Civo k3s cluster.
DNS Blacklists Not quite a proxy server, but similar functions with a little less configuration required for end clients:
PiHole: Extremely popular drop-in to block Ads and Spyware. PiHole Blocklists: Github projects with various pre-compiled blocklists BlockWorkr: Unified blocklist service. DNSWarden Blocklist: DNS blocklist - no longer maintains, but could be good starting point. Proxy Server E2Guardian: Successor of Dansgaurdian, which I had good success filtering most “inappropriate” content for K12 Captive Portal In K12 environments, user accounts and authentications are generally useful to provide a method to control who has internet access.
Some interesting hosted CLI utilities (telnet/ssh):
rainmaker.wunderground.com (Telnet): Has existed for longer then I can remember (at least the late 90s) wttr.in (HTTP) A super-nify text-based weather forcast netris.rocketnine.space (SSH): A text-based multi-user tetris clone. Source Time! Not really hosted, but NIST maintains an old-fasion “daytime” service. Example nc time.nist.gov 13
Current Stack Home Assistant - Python based, great available integrations. Hardware Resources Phillip Hue - Supported Devices: Comprehesive list of all devices supported by the Hue bridge
Kubernetes - Really the next base for server operations. I kind of wrap is up as “Core” infrstrucutre RancherOS - Lightweight distro dedicated to running Kubernetes K3OS - Distro developed to run k3s. Based on LinuxKit Operation Systems CentOS: My go-to for server installations. Being a Redhat Enterprise clone it is pretty boring, but that’s a good thing for servers :-) http://www.centos.org/ Ubuntu: My go-to for desktop installations.
‘‘when you don’t create things, you become defined by your tastes rather than ability. your tastes only narrow & exclude people. so create.’’ - Why the Lucky Stiff Source ‘‘Fashion is a form of ugliness so intolerable that we have to alter it every six months.’’ - Oscar Wilde ‘‘Those who don’t understand UNIX are doomed to reinvent it, poorly’’ - Henry Spencer ‘‘Luck is what happens when preparation meets opportunity.
This blog is an exploration of simplicity. Most of this is expressed in Eric Raymond’s [Art Of Unix Programming]((http://www.faqs.org/docs/artu/). The fundamental principle is “Keep It Simple, Stupid”. Complexity begets large, inflexible systems that are difficult to change and debug. Simplistic systems are more flexible, more robust, and easily combined to perform larger tasks. “Controlling complexity is the essence of computer programming” [Kernighan-Plauger] (From the Art of Unix Programming).
The name AVR Geek was chosen due to my fascination to Atmel 8-bit processors.
The Blackberry is no longer my phone of choice. I put together this guide for a few friends. The information is compiled from other sites and forum postings. No guarantees of the effectiveness of these directions. These directions were successful on a Blackberry Pearl 8300.
Bluetooth is used to perform the actual tethering. USB would have been ideal, but the protocol is very proprietary. The Bluetooth RFCOMM method is a bit more open.
Every now and then we do have to use windows. So let’s try to make it as Linux like as possible
Enabling WinRM As Ansible leverages WinRM for managing Windows servers, it’s guide below provides a lot of helpful guidance:
https://docs.ansible.com/ansible/latest/user_guide/windows_setup.html
The TL;DR is to use the following snippet below to run a powershell script that will correctly enable WinRM for the system:
$url = "https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1" $file = "$env:temp\ConfigureRemotingForAnsible.ps1" (New-Object -TypeName System.
Captures ad-hoc scripting and process needs. Also includes resources making these ad-hoc scripts more robust and part of an automation framework.
Shell Minimally Safe Bash Shell Script Template Bash Style Guidelines Bash - Using Loops - Nice example of several looping and conditional structures. Debugging Bash Like a Sire: A stacktrace like mechanism for Shell! Gum - Pretty Shellscript Addon Testing Shell Scripts/CLI leveraging Go’s Testing Framework Higher Level Python Shell-like Scripting In Go - Might be a nice bridge when a shell script gets complex enough that you want to use a more robust langauge, but maintain a lot of “shell-like” features Cheatsheet Passing variables to curl bash (Example from coder)
Various resources on working remotely.
Guides Gitlab Remote Resources GitLab Remote Culture GitLab Guide to Starting a Remote Position GitLab Information Communication in an all-remote environment Increment: A guide to distributed teams Microsoft: Staying productive while working remotely: Although focused slightly around Microsoft Teams, some good overall strategies for remote work. Toptal Remote Work Playbook Articles 47 Insanely Practical Work From Home Tips from Our 100% Remote Team Algorithms You Should Know Before You Take a System Design Interview Driving Engagement In a Remote Work World How to Ace an Online Job Interview: Also great tips for remote meetings in general How to Mentor Remote Workers How to build remote teams properly My 90 day plan as a new manager of an existing team: Useful outside of management roles.
Protect your infrastructure - some overlap with secure, so this will focus more on protection from non-security related incidents (buggy code, infrastructure outages, etc)
Backups Restic Kubernetes k8up: Open source k8s backup, allows for custom commands via annotations or generic RWX volume backups Kasten K10: Commercial Kubernetes backup, but up to 10 nodes for free. Velero: Backup both k8s manifests and volumes. Audit Kube Bench - Scan K8S clsuter against CIS best practices.
On Sunday, Feb 12th, I received a suspcious message about a DHL delivery from hello@namecheap.com. Pretty odd, but I ignored. Later I received notification from NameCheap that there had been a compromise. As of 18:18UTC on 2/13 it’s still being investigated. Below are the emails headers of my message, which appears to be an authenticated message from Sendgrid. I would venture to guess that an API key was leaked. Certainly highlights the need to protect third-party access to such systems.
Now that my day job involves more specific DevOps practices, I’ve grown my Sysadmin phylosophy to be more all-encompassing.
Beyond just automation of system configuration, modern practices requires more encompassing practices such as tight integration with developers, version control, test driven development, and continuous integration and delivery.
From ScaledAgileFramework
Collaboration and organization: A primary focus of Devops is around culture. Instead of an “us versus them” mentality between production and development teams, high integration of those roles is necessary.
OpenSSL Quick Commands Details of a certificate
openssl x509 -noout -text -in <cert> Generate the Sha1 or sha256 fingerprint for a certificate
openssl x509 - -noout -text -fingerprint [-sha1|-sha256] in <certificate> SSL Private CA Build a CA See JamieLinux.com for a good walkthrough. Build a new certificate openssl req -out mydomain.csr -new -sha256 -newkey rsa:2048 -nodes -keyout mydoamin.key Parameters Country: US State: Michigan Locality Name: Full City Name Organization Name: Company Organizational Unit: Department Common name: fqdn.
All things monitoring related.
Cloud Native My preferred stack: Prometheus, Grafana, Loki
Node Exporter: Prometheus exporter for server/OS statistics Elk Stack for Log Monitoring: ELK tends to be a bit heavy, but keeping this around just in case Changd: Notify if WebUI changes. Performance related articles at https://www.brendangregg.com Internet Monitoring (globally) AWS CloudWatch Internet Weather Map Contrack talkes - one thousand and one flows - Interesting article on monitoring the maximum number of entries in the Linux Contrack table, used for statefile firewall setup Pingdom’s State of the Internet Down Detector Oracle Internet Intelligence The Outage Mailing List - Network admins chatting about global issues Internet Monitoring (locally) Open Speed Test: Browser based, no client login required.
Kubernetes - the API for the modern data-center! At least why I find it so exciting.
General Kubernetes: Main Kubernetes page. Docs are fairly good and in depth. Kubernetes Production Check List: Good list of best practices when running kubernetes in production Ambassador K8S Initializer: Opinionated monitoring/Gitops stack generator. 47 Things To Become a Kubernetes Expert - Really good detailed items for k8s foundational work 100 Days of Kubernetes - Source Why you should build on Kubernetes from day one Beginners Guide to Kubernetes Components CRI: Container Runtimes CNI: Network Plugins CNI Comparison services that are potentially needed with running kuberntes in production.
Force a Redeployment If you want to recycle pods in a deployment
kubectl rollout restart deploy/<mydeployment> Get ALL resources Only gets pods/deployments/services
kubectl get all <-n namespace> Great hint from this kubectl issue on leveraging kubectl api-resources to get all resources within a namespace
kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -l <label>=<value> -n <namespace> Job from Cronjob kubectl create job --from=cronjob/<cronjob> <newjobname> Custom columns Super useful for a LOT of quick information.
A complete desktop environment utilizing the command line
ASCIINema: An ingenious way to share a text-based walkthrough/presentation Source along with Javascript Player Source and Webcast Server Source AsciiFlow - Not really a CLI App, but a handle ASCII chart design tool. SSH Access Pure SSH Javascript SSH for public console, with SSO Command Environment bash tmux to allow easy multi-tasking, or perhaps byobu Modern Unix Commands ripgrep-all: Ripgrep, that also searches PDFs, eBooks, Office Docs, etc.
General Workstations (Cloud) Asana: The main task tracker we use at home Asana CLI: Asana CLI in Go. Asana Project Mover: Move projects between workspaces GifCAP: Completely client based animated GIF capture tool! Source code Linux Workstations Ultra-Geek Workstation Guide: Really good overview of hardware/software for Workstations Nix Configs: An example of a nicely modularized nix configuration Windows Workstations Source
Nitenite: Simple installation wizard for new installs Remote Support MeshCentral RustDesk
The configuration step in the DevOps toolchain will be covered. Also including any core infrastructure links in this page.
Kubernetes/Container Kubernetes(k8s) is the best modern approach for deploying and maintaining code.
Infrastructure As Code CrossPlane: Infrastructure provider as k8s resources Is Crossplane the Infrastructure LLVM? Terraform: A very popular IaC platform tf-free: Provision free-tier resources across a variety of cloud providers. Infracost: Terraform Infrastructure Cost estimation, which can be baked into a CD pipeline for better review of what infrastructure costs will be.
Generic career resources
Remote - Remote working resources Career Path Awesome CTO DevOps Patterns/Anti-Patterns - A structure useful to determine in the interview process. TechLead Development Ladder Staff Engineer Day in a Life Day in the life of a BoxBoat DevOps Engineer Management Cancel Annoying Workplace Habits Engineering Manager New Employees Onboarding Anti-patterns Tools Business Card Generator Source Time Zone Converter - Open source web timezone converter Source World Time Buddy - Great site to quickly track local time.