Jose Antunes

Docker and DBT starter template

TL;DR: Docker and DBT starter template

I am currently revising some data modeling concepts, and in the course I am doing they use Pentaho. However, the lecturer clearly states that Pentaho is just the tool, and the course is generic enough that you can use any tool you want.

At Cable, we used DBT (data build tool) to build our data models and run them on BigQuery. I became a big fan of DBT, and really wanted to use it during my course. So, to make my life easier, I created a starter template for Docker and DBT.

You can check out the GitHub repository here. All files should be self-explanatory, but open an issue if you have any questions.

Motivation

1. I did not want to setup a Python environment - I have been using Poetry and it feels clunky to me
2. I already had Docker installed on my machine, so I thought that I could have a similar workflow to VSCode’s devcontainers
3. I still wanted to use Zed as my editor - so I wanted to make sure that the changes in my host machine were reflected in the container
4. I also wanted to be able to generate and access the docs that DBT creates
5. And finally I knew I could just use docker-compose to run the dbt container and the Postgres container easily together

I hope this helps you get started with Docker and DBT.

I like virtual coffees

Virtual coffees are a divisive topic. The opinions I have seen around me range from “yuck, no” to “they’re okay”. Personally, they can be really great! I have had many fulfilling conversations, ranging from learning moments about other areas of the company to deep, meaningful life conversations. This post serves to advocate for virtual coffee, to both managers and ICs, and to share a few tips that hopefully will help make your one-on-one moments better.

Setting up a new Mac

I wanted to write a blogpost about how I setup my new Mac for work. However, I find myself being annoyed at my own process of setting up my Mac. I feel this because my process feels inefficient.

Let me walk you through how I setup things. And hopefully I will figure out the questions I should be asking myself to improve it.

Learning to be managed

I said to myself that I was going to write once a month during 2024. I want to make good on that promise, though I am already late for January… So let’s just say I want to write on average once a month. Just a small change of the goal posts. I think it is acceptable.

Goal posts relate to the topic I want to talk about today: management. “Learning how to be managed” is a topic I am interested in, because I think I have been doing it wrong. And it took failing at one of my own goals to dive into it.

Using Swift and CSVs to send data to CloudKit DB

Pre-requisite: Don’t forget to read the section “Accessing CloudKit Using a Server-to-Server Key” in this page here. You will need to create a key that you can work with.

TL;DR: This GitHub repository has the code I wrote to upload a CSV to CloudKit Database. Scroll to the bottom for a quick heads-up on a hard to find (un)documented behaviour.

Try the code by forking it and first adjusting the Location struct to your needs and then running:

$ swift build
$ swift run csvToCloudKit \
	--ck-key-id "your/key/id" \
	--private-key-file-path "path/to/file.pem" \
	--csv-file-path "path/to/file.csv"

Ideally, a blog post has something new to say. I am not sure my post fits that category to everyone, but it does to me.

A while ago, I had an idea for an app, and I decided to learn SwiftUI and Swift to make it happen. Why those technologies? Because I am a Mac and iPhone user, and I wanted to take full advantage of the libraries available. I wanted to be able to integrate with any possible Apple services.

I have not gotten far in my app, but I did solve one of my first hurdles. As such, I want to share some of the things that I have learned, and the code that I have built.

Let me set the scene:  I want to learn how to use CloudKit DB, and I want to seed it with data that will be viewable in my app.

In my case, the seed data consists of a set of locations that are relevant to the users. Though seeding databases is such a common problem, I did not find a trivial solution to seed my database, say: using a CSV and importing it into the table (or set of records) that match the CSV’s schema.

I know that spreadsheets and CSVs are the devil - I have talked about that at previous jobs many times - however, for this quick script, it should be fine. I am only using it for 28 locations, and by the time I need to manage more, I will have a better solution. This is just a proof of concept and a learning experience.

Back to our problem, some of the solutions I saw involved adding the CSV as a part of the app Bundle, and then creating a button that would push the data to the database from my iOS app. This way, it is possible to use the CloudKit library, and a lot of the functionality built into the environment that is used to build iOS apps.

I don’t know if this is the right way, but it doesn’t feel like it. So I decided I wanted to create a Swift CLI script that would take a file and parse each row to create it as an record in the database. Nevertheless, what sounds like an easy task is not. We can’t use CloudKit, we need to use CloudKit Web Services.

I divided my code in three parts:

• The reader that loads and parses my CSV - here
• The models that represent my record and the CloudKit requests - here
• The requests that I need to use to create my records - here

The biggest hurdle was that the type values in the Record Field Dictionary must be upper case! The documentation reads:

A record field dictionary represents the value of a field and is of the form { value: CKValue, type: string (optional) }. CKValue represents field values of type String, Number, Boolean, Reference, Asset, and Location.

But nowhere does it say that type needs to exactly match the text schema of the database, such as:

primitive-type:
    | ASSET
    | BYTES
    | DOUBLE
    | [ ENCRYPTED ] { BYTES | STRING | DOUBLE | INT64 | LOCATION | TIMESTAMP }
    | INT64
    | LOCATION
    | NUMBER [ PREFERRED AS { INT64 | DOUBLE } ]
    | REFERENCE
    | STRING
    | TIMESTAMP

Once I tried with upper case values and the code worked, I was super happy!

That’s it! Shoot me any questions if you have them! :)