It's no surprise to anyone reading this, that git is the standard for source control today. It has completely replaced SVN, Mercurial, and many other systems we're glad to leave in the past, becoming without doubt the industry standard source control system. Maybe someday it will be superceded by jujutsu or something even fancier, but for now, git is here to stay. It's also not surprising to say that people are moreso interested in programming as opposed to source control management systems, so naturally people often learn just enough git to make it get out of the way for them, and little else. There's a number of features that are criminally underused for this reason. The main ones I'm thinking of are git bisect, staging hunks with git add -p, and the topic of today's post, worktrees.

 Worktrees are branches on steroids. However, unlike steroids (or some other git features), worktrees are not anger inducing. They are in fact almost a drop-in replacement for branches, which is one of the few git features you can be certain anyone using a source control system is familiar with. It's best to get this kind of information straight from the source, but as a brief explanation (i.e. just enough working knowledge to understand this post), they are simply a means of checking out a new branch in a separate directory. As a horrible pseudo-equivalence, you can imagine git worktree add ../<branch name> <branch name> to be roughly equivalent to git clone . ../<branch name> && cd ../<branch name> && git checkout -b <branch name>.

 The reason this is useful, it that it means you can work on multiple branches concurrently with a much lower amount of effort to switch contexts.

 I can already hear the cries of detraction: Context switching is the devil! Just keep your working directory clean! This is already solved by <some bloated GUI>!

 Well these complaints just aren't practical. You often need to leave your current state of a repo dirty. Sometimes the different technology doesn't fit your workflow, and while nobody wants to context switch frequently, it's an inevitability in the workplace. You can't simply sit around with one open pull request being worked on and do nothing else until you get feedback on it! Given that you are going to be context switching, and that worktrees make it much less painless. Let's get into some examples to show this.

Demo

Examples

 Demo Meeting

  It's time for you to show off and get feedback on the features you've been working on! You arrange a meeting with some colleagues and you're ready to go. Are you going to demo the cool new features from some branch you had to messily octopus merge together? Maybe instead you plan to stop the demo halfway through so you can checkout another branch, set up the environment correctly, and then begin? Well with a worktree based workflow, you can spend 30 seconds before the meeting starts, and run each feature you want to demo on a server on a different port! Now when you're moving from one feature to another, you simply open another tab! Easy!

 Bug Squashing

  The new feature has been released, and customers are loving it. Unfortunately, they're expressing their love in a funny way - by pointing out many small UI inconsistencies... "These buttons are out of alignment!", "When I have 5,000 options in this dropdown, the search is slow!", you know the drill.

 Your options are:

• Bundling all these fixes into one PR, slowing down the deployment of every change you're making and frustrating the review process
• Checking out a new branch for each bug fix, and when you (inevitably) read the reviews showing you made a typo, or that there was a linting error, you have to stop your current work in progress, stash the changes, check out a different branch, make the quick fix, push the change, switch back to your WIP branch, pop your stash....

 If you were using worktrees however, you can just open the file in another directory to make a quick fix and push the change without ever having to interfere with your current working state.

 Proof of Concepts

  You have a great idea for a new feature, but you want to experiment with it before sharing with a wider audience. It's low priority, maybe just for fun, and because of that you can only really work on it in downtime or when blocked. Instead of you forgetting that you ever checked out a branch to do this investigation, and never opening it again, or having 25 different unnamed stashes in the stack, you can see every time you look at your Code/ directory that it's right there. Waiting for you in the exact same state as when you left it.

 Comparisons & Benchmarks

  Suppose you're writing a new API endpoint and you want to run some benchmarks. You can quickly and easily run two instances of your backend simultaneously and just flip the port to compare the implementations. No need to git clone the repo to another location and disrupt your workflow. This is immediately possible with worktrees.

Limitations

 However, there is one minor limitation and it is related to dependencies. I'm hesitant to really call it a limitation, but I want to get ahead of the git branch diehards and address it. Very likely, most of your branches don't end up requiring new dependencies. That means you don't need to install anything new as you're still operating within the same directory. However, since worktrees are actually completely separate directories, you do have to install all your dependencies in each new worktree. Depending on your tooling and overall environment, this mightn't be too big of a deal. For example, with bun and uv, most of my installs are extremely fast. So fast that this limitation doesn't really affect me.

 On the positive side of this coin though, you have the benefit of being able to try out new runtimes, major version changes, etc without any risk to your primary worktree.

 I'm sure that there is a smart way to bypass this limitation if you rarely end up changing dependencies. For example, could you simply symlink the venv/ or node_modules/ in your new worktree? Let me know if you have any experience of doing this or something similar!

I'm convinced, let me use them!

 First of all, git worktree list or git worktree remove --force is far too much to type out each time. I have the following very basic set of aliases:

alias gw="git worktree"
alias gwl="git worktree list"
alias gwr="git worktree remove"

export PATH="$PATH:/home/username/scripts"

 However, the real trick is setting up your own script for git worktree create to suit your workflow.

 There are only two reasons I create a new worktree. Either I want to test and review a colleague's work, or I want to create my own worktree to mess around with. Either way, I would have to manually create the worktree, cd to it, run my personal justfile rules for instantiating the environment, and only then could I start on whatever I was doing. Around the fifth time I was auto-piloting my way through this sequence of typing, I decided to spend 10 minutes creating a script to do exactly this. It was a good idea. Now when I run gwc <name>, the script will pull the branch from the remote if it exists and set up a new worktree for it, or if it doesn't exist, it will create a new worktree for it. Finally, it will copy over my personal, not git tracked utilities, and initialize the project. I highly recommend customizing this to suit your needs!

#!/bin/bash

set -e

if [ $# -eq 0 ]; then
    echo "Usage: gwc <branch-name>"
    exit 1
fi

BRANCH_NAME="$1"
WORKTREE_PATH="../$BRANCH_NAME"

# If worktree already exists, just cd to it:
# NOTE: Can't cd from a script, so only option is to manually cd.
# Other option is to have this as an alias in zshrc to run
# `gwc ... && cd <new-dir>` or a wrapper function in general -_-
if [ -d "$WORKTREE_PATH" ]; then
    echo "Worktree already exists!"
    # cd "$WORKTREE_PATH"
    exit 0
fi

# Always fetch first to ensure we have latest remote refs
git fetch origin

# Check if branch exists on remote
if git show-ref --verify --quiet refs/remotes/origin/"$BRANCH_NAME"; then
    # Remote branch exists - ensure worktree points to remote head
    if git show-ref --verify --quiet refs/heads/"$BRANCH_NAME"; then
        # Local branch exists - delete it and recreate from remote to ensure sync
        git branch -D "$BRANCH_NAME" 2>/dev/null || true
    fi
    # Create tracking branch and worktree from remote
    git worktree add --track -b "$BRANCH_NAME" "$WORKTREE_PATH" origin/"$BRANCH_NAME"
elif git show-ref --verify --quiet refs/heads/"$BRANCH_NAME"; then
    # Only local branch exists - use it as is
    git worktree add "$WORKTREE_PATH" "$BRANCH_NAME"
else
    # Branch doesn't exist anywhere - create new branch
    git worktree add "$WORKTREE_PATH" -b "$BRANCH_NAME"
fi

# Copy files that live out of git tracking
cp .env "$WORKTREE_PATH/.env"
cp web/.env "$WORKTREE_PATH/web/.env"
cp justfile "$WORKTREE_PATH/justfile"

# Set strictPort to false so that worktrees can run concurrently
sed -i '' 's/strictPort: true,/strictPort: false,/g' "$WORKTREE_PATH/web/vite.config.ts"

# Install dependencies
cd "$WORKTREE_PATH/web"
bun install
# When working on backend stuff, do just init_project also

# Switch to the worktree directory
# NOTE: Can't cd from a script, so only option is to manually cd.
# Other option is to have this as an alias in zshrc to run
# `gwc ... && cd <new-dir>` or a wrapper function in general -_-
# cd "$WORKTREE_PATH"

Conclusion

• I hope you either already are using worktrees, or I have managed to convince you to try them out!
• Keep learning your tools! Maybe new features haven't been released since you first learned to use them, but maybe you can now understand different means of using the existing features.

I was initially stubborn when it came to personally using "AI" powered tools. I still am, but I have now accepted that these tools are here to stay. The cat is out of the bag, Pandora's Box has been opened, etc etc. For better or worse, we should all be operating under the assumption that these tools are currently in use, and will become more relied upon in future. This is why naming in software development has become even more important.

These tools rely, at least partially, on the semantic meaning of your variable and function names. That means if your names are inconsistent or poorly conceived, you're going to get worse output. Some examples in regular code block: (Because I also reject Jupyter Notebooks)

import openai
import os
import sys
import time

from pydantic import BaseModel, Field


openai.api_key = os.getenv("OPENAI_API_KEY")


class GermanTranslationResponse(BaseModel):
    message_in_german: str = Field(
        description="The user message translated into German"
    )

class FrenchTranslationResponse(BaseModel):
    message_in_french: str = Field(
        description="The user message translated into French"
    )

class FrenchTranslationResponse2(BaseModel):
    message_in_german: str = Field(
        description="The user message translated into French"
    )

class FrenchTranslationResponse3(BaseModel):
    message_in_french: str = Field(
        description="The user message translated into German"
    )


def get_response(message: str, language: str, model) -> str:
    context = [{
        "role": "user",
        "content": message
    }]

    chat_completion = openai.ChatCompletion.create(
        model="gpt-3.5-turbo-0613",
        messages=context,
        functions=[
            {
                "name": f"get_{language}_translation",
                "description": f"get the message translated to {language}",
                "parameters": model.schema(),
            },
        ],
        function_call={"name": f"get_{language}_translation"},
        temperature=0.0,
    )

    return chat_completion.choices[0].message.function_call.arguments

if __name__ == "__main__":
    print(f"Input message: {sys.argv[1]}")

    # Translates to German
    print("German")
    print(get_response(sys.argv[1], "German", GermanTranslationResponse))
    time.sleep(1)

    # Translates to French
    print("French")
    print(get_response(sys.argv[1], "French", FrenchTranslationResponse))
    time.sleep(1)

    # Translates to English
    print("French and German")
    print(get_response(sys.argv[1], "French", GermanTranslationResponse))
    time.sleep(1)

    # Translates to English
    print("German and French")
    print(get_response(sys.argv[1], "German", FrenchTranslationResponse))
    time.sleep(1)

    # Translates to German
    print("German and French2")
    print(get_response(sys.argv[1], "German", FrenchTranslationResponse2))
    time.sleep(1)

    # Translates to French
    print("German and French3")
    print(get_response(sys.argv[1], "French", FrenchTranslationResponse3))

$ python3 ex.py "what's up"
Input message: what's up
German
{
  "message_in_german": "Was gibt's Neues?"
}
French
{
  "message_in_french": "Quoi de neuf"
}
French and German
{
  "message_in_german": "what's up"
}
German and French
{
  "message_in_french": "what's up"
}
German and French2
{
  "message_in_german": "Was gibt's Neues?"
}
German and French3
{
  "message_in_french": "Quoi de neuf"
}

From having lights turn on a little dim before the morning alarm on my phone, to making sure all devices are off whenever there is nobody home, the depth of what you can do with automation is immense. However, there is one UX issue I have with these automations - I just don't quite trust them. Too many times I've come home and realized that my automation didn't realize I had left the house and never turned off my devices. Similar to alarms and logs in software development, automations are only worthwhile if they are reliable and you can trust them. I set out to find a way to increase my trust in the automations I have at home, and also have an easier path to debugging them if they don't run when I expected them to.

My solution was to have a simple notification on my phone whenever an automation runs on Home Assistant. The notification medium was obvious for my workflow/style - use ntfy.sh to trigger a notification on my phone so that whenever I was concerned if the automation ran or not, I would already have the information available in a zero click manner. I thought this might be a little difficult, but as it turns out, there is a built in Apprise notification service built into Home Assistant (which I think was a great idea!). My second issue was making sure that this notification would fire for all of my automations and that I wouldn't have to manually add it to each and every one. This turned out to be easier than expected, as there is an event you can trigger on called AUTOMATION_TRIGGERED which fires on every single automation run. The plan was set!

The first thing I had to do was add the Apprise integraion to my configuration.yaml for Home Assistant with the URLs I wanted to trigger notifications on: ^[1]

notify:
  - platform: apprise
    url: ntfy://ntfy.sh/home-assistant-notification-topic

After that, I created an automation that would trigger whenever an automation ran, and would then notify me with the name of the automation that ran:

That was the sum total of what I had to do! This has made me feel much more at ease and trusting of when my automations have run, and if they don't run when expected, I can easily debug what didn't happen when I expected it to. One interesting thing to note is that the automation that sends a notification doesn't end up triggering the automation to fire again. There is no recursive behavior from it. I hope this helps anyone looking to set up automations on Home Assistant using Apprise, ntfy.sh, or just trying to trigger an automation to run whenever a different automation runs! If you have any fun automations for Home Assistant, or any thoughts/comments, please let me know through whatever means you found this post, and thanks for reading!

[1] I self-host ntfy.sh so this URL is just a dummy! If you want another blog post on self-hosting ntfy.sh, let me know, but those docs are great and it's one of the most stable services I have ever ran.

Introduction

 Everybody is a little vain. Vanity frequently spurs people to action where other, more pure, motives fail.  There's an interesting relationship between vanity and perfectionism. Perfectionism is often driven by vanity, the wish to be perceived as being a master in whatever actions you are currently undertaking. This is clearly an impossible task, but striving towards an impossible ideal can often lead to successful results along the way.

 Recognizing that perfectionist tendencies could have a different root cause than you previously thought leads to a different perspective on them. For me, perfectionist tendencies have been counterproductive to my desire for acknowledgement and recognition in the things I (at least think!) am knowledgeable about. However, you need to be visible in order to be recognized, so perfectionism really gets in the way. If my perfectionist tendencies were allowed to run free, I would rarely, if ever, release anything. The concept of "Building in Public" has been a great perspective for me in this regard, even though almost all it ever leads to is me writing half-baked blog posts and creating things like my most recent project - git-vain.

Buildlog

 Gitvain is a simple Python project that sends you a notification whenever the follower/stargazers on a Github change. It supports notifications using Apprise which is a fantastic tool that allows you to specify some simple configuration and then send notifications using many, many services. PyGithub is used to fetch changes on the watched repositories. There are many outstanding issues, that will most likely never be completed. This is by design, however, as I set out a plan to get to a point where I could be satisfied enough to leave it alone, say I sufficiently completed it, and thereby prevent my perfectionism from getting in the way.

 A major part of my recent work in software development has focused around best practices and ops-adjacent work. Recognizing that setting up CI/CD at the start of a project is very easy compared to getting it set up midway through, and that actually defining the feature you want to work on and writing short functions with no side-effects to get it done makes you progress quickly. Being realistic with the scope of what you're working on allows you to keep motivation and leave your work in a state that's trivial to restart whenever you next get the chance. To achieve this, I simply started by writing out a minimum set of features I wanted to have before being happy to cut a v1.0.0. I then broke these features down into small components and added them to Vikunja my favorite todo list application.

 I started by getting a basic Python project set up with continuous deployment using Github Actions to build a Dockerfile that would deploy to my home server. This is so simple that it literally just prints a Hello World statement in the Dockerfile! Even though this seems insignificant, from here you have the perfect jumping off point; a stable point you can solidly build on.

 After this, I prioritised one type of event I wanted to be notified on - stargazers - and wrote a basic script to star/unstar a repo, and another to use ntfy to send a notification when this list changed. Already it was starting to take shape enough to keep my motivation going.

 Once this was complete, I used the simple Python stdlib shelve to add some persistence, and integrated with Apprise to support a wider variety of channels for notification.

 There's no point in going into any further detail, because A) it's not very interesting, and B) it didn't progress much further! But this is a feature, not a bug - I set out with the intention of cutting a v1.0.0 and this was almost there already!

 But what next?

Next Steps

 I realised I had a few things I wanted to add as I was going through the development process. This is typical, and it's the source of all feature creep. I curtailed this by putting anything I wanted to do into Vikunja, and deciding to convert them to issues once v1.0.0 had been completed. If I didn't, I would certainly still be adding little bits and pieces here and there now as opposed to enjoying the feeling of having "finished" a piece of work. Even if it only lasts as long as it takes to go back to working on v2.0.0!

 Some things I want to improve in v2.0.0 or in the next side project I work on are as follows:

• Use pre-commit to add more stability and standardisation to the work
  • This feels particularly powerful and would be a good way to finally upgrade my development process which still consists of a pretty vanilla vim setup. Who knows? Maybe I'll even switch to NeoVim with this new-fangled LSP!
• There are some bugs such as not accounting for pagination of the response from Github.
  • I deliberately didn't try to address these during this effort as I could easily have gotten bogged down in it and it's a small, contained piece of work that should be easy to add in the future if I leave the project and come back to it after a period of time.
• SemVer and Conventional Commits seem to be a fantastic way to completely automate the release cycle.
  • If there's no Github Action for it, it shouldn't be too difficult to make either as I didn't find a suitable service for my needs so far - next project maybe??
• Python has a good few issues when it comes to naming conventions and project setup.
  • This is potentially exacerbated by the fact that the tooling I'm familiar with is outdated by now.
    • Using setup.py works nicely (especially if you have experience with it) for a small project like this, but it's not a good idea for more serious work.
  • The naming conventions I was having issues with surrounded small things like - vs _ vs camelCase. Package names vs module names vs binary names. Small things, that actually ended up adding some annoying decision fatigue and cognitive load to keep consistent.

Conclusion

 The Python standards and conventions are a bit of a mess. Dev Ops/best practices actually do provide great help when used in conjunction with common sense, and the "perfect" project template likely cannot exist due to the subtle variations in what "perfect" looks like for different purposes.

 Nonetheless, that's what my vanity is currently driving me towards finding, so for now, I'll keep looking.

  Hopefully not in vain!

The Tools

 Grafana

   Lets start with the main one - what is Grafana? Grafana is at its core a web-based data visualization platform. It acts as a front end to many time-series databases, and uses plugins to consume data from different sources and support custom dashboard visualizations.  It also has a simple graphical tool to help you craft queries on the data. The best place to try the Grafana platform out is at play.grafana.org.

 Prometheus

   Prometheus is a time-series database which operates on a pull model. You configure exporters which will have metrics requested from them by Prometheus on a regular schedule. There is a suite of components it can make use of, but one core feature we will be using is PromQL - the Prometheus Query Language. We will use this through Grafana to aggregate metrics collected by Prometheus. One thing that is important to note is that Prometheus is designed to work with numeric information only. This means it cannot be used to search through textual logs like you might do in Splunk or Sumo Logic.

 Loki

   Being restricted to just working with metrics is quite a limitation, so we will also be using Loki. Loki encompasses a set of tools/services, but my working model of it doesn't extend much further than Prometheus for log lines. It accepts data in any format, and similar to Prometheus, it allows you to build metrics and alerts based on them.

 Promtail

   Promtail is responsible for delivering log lines from log files to Loki. It is the roughly equivalent component in the Loki stack as Node-Exporter is in the Prometheus stack. This is confusing as Promtail looks like it should be part of the Prometheus stack, but alas naming of open source tooling is never great!

   Promtail will be used to collect log lines from containers of my own services, or of services being debugged.

 Node-Exporter

   Node Exporter monitors and exports hardware and kernel level metrics to Prometheus. It is highly configurable with a long list of metrics it can collect if you desire. Despite the warnings, we will be running node-exporter from a Docker container for now. This is just for ease of encapsulation until I can move my personal home server to using NixOS or similar.

   This will provide the host-level metrics we need, such as CPU usage, RAM usage, free space, etc.

 cAdvisor

   From the cAdvisor Github page:

[cAdvisor] is a running daemon that collects, aggregates, processes, and exports information about running containers.

   These metrics can be exposed for Prometheus, and will provide the per-container resource usage metrics we need.

Stack

   So now we have all the components explained, it's worthwhile visualizing the stack we will have. One crucial thing to remember is that while this seems like a large number of services, each one is very small and modular, so it won't be consuming a huge amount of resources.

The Data

   Now we know the how of observability, we need to get to the what. Honestly, I spent a long time putting this off, probably because this was the largest gap in my knowledge! However, I think an iterative approach works best here anyways - both in iteratively building up to "complete" observability/insight, and iteratively building up my knowledge of the Grafana stack.
   I suppose it makes sense to think about why I'm making some monitoring on these services. Primarily it's to see what my server is capable of. i.e. do I need to add some RAM/storage/replace the entire CPU? How many additional containers can I run? Has there been a large spike in usage? If so by what containers/services? How much network input/output is each service going through? As a percentage of the whole input/output? How much storage is each container using? Secondly, I want to have insights into actual logs of my own services (or for others if I really want I guess? But primarily my homemade services). This should be all logs for debug purposes and usage metrics in general.
   Lets make a list:

• Host
  • Metrics
    • CPU Usage
    • RAM Usage
    • Storage Usage %
    • Load (1 min, 5 min, 15 min seems standard)
    • Network Throughput (Input/Output volume)
  • Logs
    • syslog
    • auth.log
• Per-Service
  • Metrics
    • CPU Usage %
    • RAM Usage
    • Storage Usage %
    • Network Throughput
  • Logs
    • For specific containers

The Implementation

   Now we know what we're observing, and how we're going to ingest it, we just need to do it!

   Since we painstakingly mapped out the different component services, we can tell instantly that we need cAdvisor for the per-service metrics, NodeExporter for the host metrics, and loki for all the log lines. Lets start with the metrics.
   The metrics all need to feed into prometheus in order to end up in Grafana, so we need to edit the prometheus config file in order to do that. For getting all our container metrics from cAdvisor, we just need a few lines. For NodeExporter, just a few more:

scrape_configs:
  - job_name: "cadvisor"
    scrape_interval: 15s
    static_configs:
      - targets: ["cadvisor:8080"]
  - job_name: "node_exporter"
    scrape_interval: 15s
    static_configs:
      - targets: ["node-exporter:9100"]

   Then Loki needs to be configured for syslog and auth.log. This was achieved by a simple promtail config and mapping /var/log:/var/log/host_logs in docker-compose:

scrape_configs:
- job_name: hostlogs_job
  static_configs:
  - targets:
      - localhost
    labels:
      job: hostlogs
      __path__: /var/hostlogs/*log
- job_name: docker_container_logs
  docker_sd_configs:
  - host: unix:///var/run/docker.sock
    refresh_interval: 5s
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)'
      target_label: 'container'

Alerting

 Finally we have the full stack set up. Last remaining thing to get a semi-professional (emphasis on the semi!) is to get some alerting going. For alerting, I'm going to use ntfy and a small Grafana integration I found called grafana-to-ntfy. This took a little more work than expected, but eventually I got it all working. Firstly, I set up a personal ntfy instance, then added the grafana-ntfy container to my docker-compose along with a simple env file as explained in the README. I then integrated it with Grafana alerting. One of the key things to note here is that I just used plain http for communication with the grafana-ntfy container as I couldn't get it set up with SSL! I kept getting invalid cert errors with reference to a cert only valid for Traefik. Also not fully documented, but the BAUTH variables need to be passed too although they should be made optional. May submit a PR for that... Follow the README to do a test notification

Set up a query to make sure notifications are coming through and then just get on with standard alarming!

Conclusion

 So, as you've probably realized, I really lost steam towards the end of this post. I have been working on this post/stack setup for about three months and it has been frustrating me to no end and stopping me from writing things I would prefer to write about, and follow my current tech interests. I try to balance doing things I feel I should do with things that I have a strong (but usually fleeting) motivation to do as these rarely overlap. This time however, even though I can see the huge benefit of having a well set up monitoring stack for my home server and how all aspects of this will improve my quality of life when debugging/doing basic admin, the balance has just tipped to being more stressful than beneficial to me.

 I will update my stack in the future, and hopefully write a more concise post on setting up a home server monitoring stack, but for now, this is all you get!

DNS is infuriating to me. It exists in the uncanny valley of technologies where on paper I know a decent amount about it, but in reality I just flounder aimlessly and ultimately end up in an IRC channel somewhere spilling my woes to any (seemingly mystically intelligent) kind soul who will help me debug my issues. This does not pair well with trying to host your own server in your living room.

I had all my DNS issues tidied up enough to forget about them (the ideal state of affairs for DNS in my opinion) while I was living in Ireland and my ISP by default gave out static IPv4 addresses, or if not, changed them so infrequently that I didn't experience a change in over a year. However, with a new ISP in Berlin, I had quite a different experience. After congratulating myself on the quality of the infrastructure-as-code I had developed, so that even after 5 months of downtime, I could just plug in my server again, connect it to the WiFi, and it was back online, I went to bed happy. The following morning, however, I woke up to find that my IP address had changed overnight. I gave the DNS provider a call, and they informed me that a static IP address can easily be provided, but it will cost an additional ten euro per month. They reassign IP addresses every 24 hours too, just to be annoying about it. This means you can't rely on a rarely updating dynamic IP address either.

I felt personally attacked at the thought of paying for a static IP address, and for some inexplicable reason, I didn't want to use existing Dynamic DNS solutions. Maybe some vague notion of not wanting another external resource that I'd have to pay for or could break due to a misconfiguration, but most likely it was just stubbornness. Enter DIYnDNS! A simple Python script that will check if my IP address has changed in the last N minutes, and if so, connect to Cloudflare, update my DNS records to point at the new IP address, then go back to sleep until it notices the IP address has changed again.

This is all pretty straightforward, but there are a few nice aspects and things I learned along the way.

• First of all, this is all packaged up in a nice and small Docker container, which allows me to also make a simple docker-compose file to allow it to be installed with ease on any reverse proxy setup like Caddy or Traefik.
• Secondly, it uses a plain .ini file for configuration, which I never understood why people used before. I think I'll be using .ini files for my configuration in future Python projects, primarily due to the inbuilt configparser library and how easy it was to use.
• Thirdly, there's a watchtower container defined in the docker-compose file too, which will automatically update the diyndns container whenever there's a new image pushed to docker hub.
• Finally, this container is updated on every push to the repository's main branch using some straightforward Github Actions CI.

The fact that all of this is possible with just some glorified configuration files and one easy to throw together python script is amazing to me. I really feel like anyone who isn't on the boat of self-hosting simple services, and understanding even the basics of how to put together a full CI/CD system like this, is missing out.

Finally, there are a good few areas that need to be improved. For starters, I should be able to configure the cron from the config file somehow, but I had horrific trials trying to get the crontab to respect env vars, follow my PATH etc. Best solution for this is likely to put it in a systemd service file, or else some baked in Python threading tool. I also think my usage of configparser could be better. I want to explore the library more so I can use it for other projects in the near future.

Thank you for reading, and please get in touch if you have any comments, advice, or suggestions. My next post will be about the hassle I had and the effort I put in to try getting this setup working with Namecheap, my original DNS provider. It was significantly more involved than this and ultimately was ridiculous enough that I moved my DNS to Cloudflare!

Photoprism is the self-hosted photo/video library that does it all for me. It has labeling of images so you can search by picture content, it has calendar and map features for more natural finding of the pictures you're trying to find and it lets you have public/private content as you choose. To be clear, I want our reliance on huge mega-corps to be gotten rid of. I don't think that necessarily implies some form of neo-luddism or that we should have to deal with worse products. We get what we create and support, and this project is definitely worthy of both. Pictures are some of the most personal things we routinely create so we should try protect these moments from prying eyes. Moreover, with Google Photos continuing to degrade their product by reneging on their promise of free unlimited storage and by silently degrading your picture quality an alternative should be welcomed!

From self-hosting Photoprism, I've found it to be quite a smooth process relative to the size and scale of its features. The one process I found difficult to get working was one of the most crucial however; syncing with my Android phone. I mostly take 35mm analog pictures (because I'm insufferable) but I appreciate being able to scroll back through my screenshots for conversations with friends and taking a quick picture on the go. I found Photoprism's guide however lacking to solve the use case of my phone automatically uploading new content to Photoprism and having it imported ready for viewing, so I tried a few different apps and settled on FolderSync.

FolderSync allows you to sync files between different locations on your phone or use WebDAV to sync with an external location. Photoprism supports WebDAV so this should be a cinch! I decided to set up one folder on my device that would one-way upload to Photoprism and then locally sync any files in other media folders on my phone to that folder. This might be a needless step, but I think it's useful for supporting multiple different ingestion locations and also optionally not indiscriminately syncing all media from your phone with a potentially public library! From screenshots, my FolderSync setup looks something like this for SD Card Sync:

And for WebDAV we have the following:

Using these, you should be able to set up automatic photo syncing from your Android phone to Photoprism! If you have any issues, refer back to the Photoprism guide for this topic as it's definitely more up to date,  thorough, and useful than this short post. However, I strongly recommend FolderSync over their suggestions of PhotoSync and SMBSync2!

I love Python. It's easy to read, even easier to write, and best of all, the easiest language (I've used) to interact with other people's code. I spent too long being only on one end of that equation though, only consuming code other people have written as opposed to taking the things I was making and making them public so the cycle could continue. I felt it was too much hassle to get started with and for no reason because who would be using the libraries I write anyways? After finally learning how to create and publish packages to PyPi, I'm glad to report it's super easy and definitely worthwhile. Even if it's just so you can easily import code you wrote for a previous project without going into the horrors of relative imports, I feel it's worth the five minutes investment to get up and running with it.

First of all, you need to make a PyPi account and then add your credentials to ~/.pypirc. Your file should look something like this:

[distutils]
    index-servers = pypi
    
[pypi]
    repository: https://upload.pypi.org/legacy/
    username: <Your PyPi username>
    password: <Your PyPi password>

Realistically, you should set up API keys, but I am just trying to show the bare minimum to get up and running with PyPi. I am in no way an authority  on this and most often don't even follow best practices.

Next take a simple project or tool you've been working on and impose some structure. Personally, I am a fan of simple tools that do things I understand. Consequentially, I end up using tools that started falling out of fashion in the 80s. I find Makefile to be just the right level of build tool for me. I understand the file at a glance, I can hack it to get what I want done, and it is going to be the topic of another blog post so I'm not going to dwell on it too much here, suffice to say you should add a Makefile to the root of your repository/project and have these key rules:

PYTHON=python3.8

ENV_DIR=.env_$(PYTHON)
IN_ENV=. $(ENV_DIR)/bin/activate &&

upload_pip: build_dist
    twine upload --repository pypi dist/*

build:
    $(IN_ENV) $(PYTHON) -m pip install --editable .
    rm -fr dist/
    $(IN_ENV) $(PYTHON) setup.py sdist bdist_wheel

build_dist:
    rm -fr dist/
    $(IN_ENV) python setup.py sdist

setup:
    $(PYTHON) -m pip install --upgrade virtualenv
    $(PYTHON) -m virtualenv -p $(PYTHON) $(ENV_DIR)
    $(IN_ENV) $(PYTHON) -m pip install --upgrade -r requirements.txt
    $(IN_ENV) $(PYTHON) -m pip install --editable .

This will let you run make upload_pip and magically have your package arrive on PyPi once you install python3 -m pip install twine. But wait, what is it going to upload?? That's defined in your setup.py:

from setuptools import (
    find_packages,
    setup
)

INSTALL_REQUIRES = []

setup(
    name='my-first-pip-package-name',
    description='This is my first Pip package!',
    version=0.0.1,
    url='https://github.com/link/to/your/repo',
    python_requires='>=3.6',
    packages=find_packages('src'),
    package_dir={'': 'src'},
    install_requires=INSTALL_REQUIRES,
    entry_points={
        'console_scripts': []
    }
)

This should be pretty self evident for what should be where,  but examples are the best way to grok something and here are two personal repos where I use these structures for Makefile and setup.py to publish Pip packages.

I know this is quite a low quality introduction to making a Pip package, so please leave me feedback. I want to improve at my writing and hearing responses is the best way I can get there!

First of all, I would not suggest doing this for anything you want to remain stable or have any decent uptime on. This is to keep up to date with the latest bleeding edge of a project and can be useful for development and side projects when you merge your code you can show it to people live a few minutes later. If you do this with containers you're looking to rely on (such as Matrix, Vikunja or Home Assistant (ohh, foreshadowing?)), you can easily see how you could accidentally deploy to a breaking version and have to do some messy sysadmin work! However, there are valid reasons to automatically update to the latest tag, and to do this, we will be using Watchtower which is a Docker container that monitors other containers and if their image updates, then will SIGTERM it, pull down the newest version and restart it. Pretty meta! Onto the code:

version: '3.3'

services:
  via-web:
    container_name: "via-web"
    image: conorjf/via-web:latest
    restart: unless-stopped

    volumes:
      - /var/log/instance_logs/via:/var/log

  via-watchtower:
    container_name: "via-watchtower"
    image: containrrr/watchtower
    restart: unless-stopped

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

    command: via-web --interval 30

This relatively simple docker-compose.yaml shows how easy this is. We define two services, Via and Watchtower. Via's image is set to latest from Docker Hub and it uses a simple trick to easily view logs explained here. This is clearly something under development and so it's safe to set it up to deploy latest blindly. The via-watchtower service is also very simple, really only requiring 2 statements apart from the image. The volumes is so Watchtower can be informed about Docker updates/changes in state, etc and the command is of the format <container to watch for updates on> --interval <seconds to check for updates>. It's that easy! Using this and some continuous integration to build and deploy the Docker image, Via is kept constantly up to date for development and beta testing.

I hope this was instructive and please reach out if you have any questions, suggestions or comments in general :)