Tom de Bruijn

Git: Pull changes from a local repository

2025-10-10T00:00:00+00:00

This may a bit of niche problem to have, to pull changes from one clone of a repository to another clone, on the same machine. Why would I need to do this?

The why

At work, I have multiple of the same repository locally on my machine. One 'main' clone and multiple clones of the repository as submodules in another repository. I work on these local clones in different contexts, and I don't always know or realize I need to use it in a different context later.

If I start work in the main repository, then want to test it as part of one of the test setups, what do I do? Push the changes to GitHub, then pull the branch in the other local repository? And risk everyone seeing my terrible Git commits? No~ooo.

The setup

What I do instead is connect the local repositories, so I can pull the changes from one local repository to another.

To set this up, I first find the path of the local clone of the repository I want to pull from, e.g. /home/tomdebruijn/work/main-repo. Then, in the repository I want to pull the changes into, set up a remote to that repository:

# Add a new remote with the name 'local' that points to the other local clone
git remote add local /home/tomdebruijn/work/main-repo

Then pull fetch changes from that repository and check out the branch I need.

# Fetch the contents of the other local clone
git fetch local
# Check out the other local clone's branch
git checkout <branch name>

Now I have the changes I committed in one local clone, available to me in another local clone as well, without having to sync it via an external SCM hosting provider.

Pull instead of push

Knowing this, we can also use this for other Git commands.

For example, you can push changes to another local repository clone, instead of pulling changes:

# Create a branch and add some changes to it
git checkout -b test-branch
echo "Test" > test.txt
git add .
git commit -m "Test commit"

# Push the branch to the other local clone of the repository
git push local test-branch

No need to fetch the changes in the other clone. It works like any remote repository.

Extra: Clone a local repository

You can also clone a local repository by pointing to its path.

# Go to another directory
cd /home/tomdebruijn/experiments
# Clone the local repository into this directory
git clone /home/tomdebruijn/work/main-repo 
# Open the newly cloned repository
cd main-repo
# List the repository contents
ls

Switching Zellij and Vim panes with ease

2024-11-28T00:00:00+00:00

I'm trying out Zellij after it announced non-colliding keybindings in version 0.41.0. Currently, I'm using iTerm and MacVim as two separate apps, but I'd like to try running everything in the terminal and switch to NeoVim.

I never got into tmux, but I know there are plugins to have you smoothly switch between tmux and Vim panes like they are from the same application. I want this behavior in Zellij too without having to switch Zellij modes.

As I got started with Zellij, I found a couple of different projects with different ideas on how to accomplish this.

For the default Zellij colliding keybindings, zellij-autolock looks like the way to go. It automatically switches between the "normal" and "locked" modes when entering and leaving Vim. But, as the author pointed out, it is not tested with non-colliding keybindings.

I couldn't find a solution that worked for me using the Zellij non-colliding keybindings in combination with Vim, while also allowing other applications that use the ctrl+hjkl keybindings to work, but please let me know if I missed one!

What I ended up doing to make it work for me is the following:

Install a Neovim Zellij plugin

Add the zellij-nav.nvim plugin * to your Vim config that will send commands to Zellij when you try to press one of the ctrl+hjkl keybindings and are on the edge of a Vim window.

*: There may be other plugins that work, too; this is the one that worked for me.

I'm using vim-plug for package management. There are other Vim package managers listed on the plugin's README.

" ~/.vimrc

call plug#begin('~/.vim/plugged')

" Other plugins here

Plug 'swaits/zellij-nav.nvim'

call plug#end()

lua require("zellij-nav").setup()

nnoremap <c-h> <cmd>ZellijNavigateLeft<cr>
nnoremap <c-j> <cmd>ZellijNavigateDown<cr>
nnoremap <c-k> <cmd>ZellijNavigateUp<cr>
nnoremap <c-l> <cmd>ZellijNavigateRight<cr>

Then quit Vim, restart it, and run the :PlugInstall command.

Install the Zellij plugin

Next up, we'll install a Zellij plugin. I started using the vim-zellij-navigator. It works great for switching between Vim and Zellij.

However, I ran into issues with other applications like fzf, triggered by running fzf or pressing ctrl-r/ctrl-t. When the fzf prompt is open, pressing ctrl-j or ctrl-k now does nothing. Instead, the Zellij plugin captures the key presses and switches panes or tabs, but the fzf prompt stays unchanged.

I forked the plugin and made some changes to support these scenarios. Don't worry I submitted a Pull Request to merge them upstream.

Until those are merged and released, you'll need to build the plugin manually.

Install Rust via rustup.

Clone the repository fork:

git clone https://github.com/tombruijn/vim-zellij-navigator.git

Compile the plugin.

# Open the project directory
cd vim-zellij-navigator
# Install the latest Rust version (at time of writing)
rustup override set 1.82
# Install Rust WASM target
rustup target add wasm32-wasi
# Compile the project
cargo build --release

Copy the plugin to the Zellij plugin directory.

# Create the plugins directory if it does not exist yet
mkdir -p ~/.config/zellij/plugins
# Copy the plugin to the plugins directory
cp ~/target/wasm32-wasi/release/vim-zellij-navigator.wasm ~/.config/zellij/plugins

Finally, configure Zellij to load the plugin and configure keybindings to call the plugin.

My configuration is down below, which is complete for my use case.

After configuring Zellij, restart it. The first time you press any of the keybindings you've configured to call the plugin (indicated with MessagePlugin), you'll get a prompt from the plugin asking permission to do a couple things. Press `y' to approve the permissions.

// ~/.config/zellij/config.kdl

plugins {
    vim-zellij-navigator location="file:~/.config/zellij/plugins/vim-zellij-navigator.wasm"
}

// I use the locked mode as the default mode.
// By default this is set to "normal".
default_mode "locked"

keybinds clear-defaults=true {
    locked { // Change this to "normal" if your default mode is "normal"
        // Switch panes with `Ctrl+hjkl`
        bind "Ctrl h" {
            MessagePlugin "vim-zellij-navigator" {
                name "move_focus_or_tab";
                payload "left";
            };
        }
        bind "Ctrl j" {
            MessagePlugin "vim-zellij-navigator" {
                name "move_focus";
                payload "down";
            };
        }
        bind "Ctrl k" {
            MessagePlugin "vim-zellij-navigator" {
                name "move_focus";
                payload "up";
            };
        }
        bind "Ctrl l" {
            MessagePlugin "vim-zellij-navigator" {
                name "move_focus_or_tab";
                payload "right";
            };
        }

        // Disable the plugin when any of these keybindings are used and still press that key too
        bind "Ctrl r" { // `Ctrl+r` is for forward shell history
            WriteChars "\u{0012}"; // Passthrough `Ctrl+r`
            MessagePlugin "vim-zellij-navigator" {
              payload "disable";
            };
        }
        bind "Ctrl s" { // `Ctrl+s` is for backward shell history
            WriteChars "\u{0013}"; // Passthrough `Ctrl+s`
            MessagePlugin "vim-zellij-navigator" {
              payload "disable";
            };
        }
        bind "Ctrl t" { // `Ctrl+s` is for backward shell history
            WriteChars "\u{0014}"; // Passthrough `Ctrl+t`
            MessagePlugin "vim-zellij-navigator" {
              payload "disable";
            };
        }
        // Reenable the plugin when any of these keybindings are used and still press that key too
        bind "ESC" {
            WriteChars "\u{001B}"; // Passthrough `ESC`
            MessagePlugin "vim-zellij-navigator" {
              payload "enable";
            };
        }
        bind "Enter" {
            WriteChars "\u{000D}"; // Passthrough `Enter`
            MessagePlugin "vim-zellij-navigator" {
              payload "enable";
            };
        }
        bind "Ctrl c" {
            WriteChars "\u{0003}"; // Passthrough `Ctrl+c`
            MessagePlugin "vim-zellij-navigator" {
              payload "enable";
            };
        }
        // Toggle the plugin on or off manually
        bind "Ctrl Alt a" {
            MessagePlugin "vim-zellij-navigator" {
                payload "toggle";
            };
        }
    }
}

Write your own Domain Specific Language in Ruby

2023-02-01T00:00:00+00:00

Let's do some metaprogramming in Ruby! This article assumes you have some familiarity with writing Ruby and are interested in learning how to use metaprogramming.

This article is also available as a video recording of a talk.

What's a DSL?

Let's start with some definitions on what we're going to be looking at. The abbreviation DSL stands for Domain Specific Language.

A DSL is a sub-language within a Ruby app (or any other programming language) that is specific to a certain domain: like a Ruby gem or part of an app. Without that gem or app, the DSL won't work. It's specific to that domain.

In Ruby there's a gem named Rake. This gem provides a DSL to create tasks to be run from the command line. A small example looks like this:

# Rakefile
task :hello do
  puts "Hello there"
end

You can call this task from the terminal with the following command. When called, it will perform the block we've given to the task method.

$ rake hello
Hello there

This is the power of a DSL. We don't need to know how to listen to arguments given to Ruby from the command line. Rake does this for us. We only need to define the commands.

Why use a DSL?

Configuration

One of the most common reasons to create your own DSL is for configuration, for an app or part of a gem. In Rails, the Application class contains configuration for the different gems that make up Rails. It's even possible to define your own custom config options specific to your app.

# config/application.rb
module MyApp
  class Application < Rails::Application
    config.load_defaults 7.0

    config.time_zone = "UTC"

    config.app.custom_option = "some value"
  end
end

Within the Application class, the config object can be used to set config options or load defaults, like shown above.

Automate writing code

A DSL can be used to abstract away a bunch of repetitive code. In a Rails app there can be many different HTTP endpoints the app listens to. To write a Ruby app that listens to these endpoints can require a lot of code. In Rails, we can declare several different routes with one method: resources

Rails.application.routes.draw do
  resources :posts do
    resources :comments
  end
  root "homepage#show"
end

The resources method will do several things:

create several endpoints like GET /posts, GET /posts/new, POST /posts, DELETE /posts, etc.;
nest endpoints, if they are nested within another resources method call's block, like in the example above for comments;
route the request data on the endpoint to the relevant controller.

This is all defined with one method call. We don't need to go through several files and methods to make sure these steps are all done for every route. This makes, what would be high churn code, more maintainable*.

*: At times, we may be swapping out a lot of code with less code, but that code is more complex because of the metaprogramming involved.

Writing your own Ruby DSL

The start of a DSL

The first thing that usually gets a DSL is configuration. A gem ships with a configuration class or a part of an app does. A new Config class gets created with some config options. (In this case a CLI tool that has options to print more information with verbose and a particular output format called documentation.)

class Config
  def initialize(options = {})
    @options = options
  end
end

config = Config.new(
  :verbose => true,
  :format => :documentation
)
# => #<Config:...
#        @options={:verbose=>true, :format=>:documentation}>

The above example is usually enough for a small component.

Let's look how we can make this DSL feel more like some of the DSLs we've seen. In the example below we don't configure the Config class with a Hash of options, but use method calls like the verbose= attribute writer.

Ruby has helpers for defining reader and writer methods on a class for instance variables. Calling attr_accessor :verbose in a class definition will define the verbose or verbose= methods.

class Config
  attr_accessor :verbose, :format
end

config = Config.new
config.verbose = true
config.verbose # => true
config.format = :documentation

pp config
# => #<Config:...
         @format=:documentation,
         @verbose=true>

While this is quick to set up, it has some downsides. Like before, we want to store all config options on a @options Hash instead of on their own instance variables. To export the config options as a Hash, we would need to export them all manually. Now there are more places that need to be updated the more config options are added. Easily forgotten, easily broken.

class Config
  # ...

  def options
    {
      :verbose => @verbose,
      :format => @format
    }
  end
end

pp config.options
# => {:verbose=>true, :format=>:documentation}

To make the Config class work with a Hash of options, we can define our own methods for each config option. In these methods we set the values on the @options Hash, using the key of the config option we want to set. Great! We now can export our configuration with ease.

class Config
  attr_reader :options

  def initialize
    @options = {}
  end

  def verbose=(value)
    @options[:verbose] = value
  end

  def format=(value)
    @options[:format] = value
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}

With this setup, we run into a familiar problem. Every time a new config option is added we need to add another method for that config option. As more config options are added, the Config class grows and grows, becoming more difficult to maintain.

Dynamically define methods in Ruby

Instead of defining a method per config option, we can dynamically define these methods. Dynamically defining methods allow us to quickly define several methods that share almost the same implementation.

In the example below I've created a new class method, as indicated with the def self.def_options syntax (1). This method, "define options", receives an array of option names to define methods for (2). In the def_options method it will loop through these option names (3). In each iteration of the loop it will call the define_method method (4). With this method we can define a new method, like with the def <method name> syntax, but with a dynamic method name.

class Config
  def self.def_options(*option_names) # 1
    option_names.each do |option_name| # 3
      define_method "#{option_name}=" do |value| # 4
        @options[option_name] = value # 5
      end
    end
  end

  def_options :verbose, :format # 2

  def initialize
    @options = {}
  end

  def options
    @options
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}

The define_method method receives the name of the method, and a block (4). This block will be the implementation of the method we are defining. In this case, it will call the @options instance variable to receive the Hash of options configured, and add a new value for the option name we've declared (5). The DSL syntax for the end-user remains the same.

Using Ruby blocks

In my opinion, no Ruby DSL is complete without blocks*. They look good for one, but they also provide a new scope of context to users of our DSL, in which our new DSL rules apply.

*: It's perfectly fine to have a DSL without blocks.

Config.configure do |config|
  config.verbose = true
  config.format = :documentation
end

In the next example, I've created a new class method called configure. To interact with blocks given to a method, we can use the yield keyword. If a method has been given a block, yield will call that block. If we give a value to yield, it will make it so that the block receives that value as an argument in the block, as is shown with |config| in the example above.

class Config
  def self.configure
    instance = new # Create a new instance of the Config class
    yield instance # Call the block and give the Config instance as an argument
    instance
  end

  # ...
end

Using instance_eval

We can make this shorter, and in my opinion, feel more like a DSL with its own block context. We can drop the |config| block argument and avoid having to call config. in front of every config option.

Config.configure do
  verbose true
  format :documentation
end

I've omitted the equals sign (=) from the example here. More on that later.

To make this work we go back to our configure class method. Instead of yield, we'll use instance_eval. (Eval may sound scary, and it has some things to look out for, but in this small example it's quite safe.)

To call instance_eval we need to pass in the block given to the method. We can't use yield for this. We need to explicitly specify the block argument of the configure method with &block. The key part being the ampersand (&) of the argument name &block, telling Ruby it's the block argument.

class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  # ...
end

(In Ruby 3.1 and newer the &block argument can also be written as only the ampersand &, the name is optional.)

When called on another object, like our Config class instance, instance_eval will execute that block within the context of the Config instance. The context of that block is changed from where it was defined to the Config instance. When a method is called within that block, it will call it directly on the Config instance.

Local variable assignment gotcha

Remember how in the example above I omitted the equals sign from the method names? When calling a method name ending with an equals sign in instance_eval, Ruby will interpret it as a local variable assignment instead. It will not call the method on the Config instance. That's how Ruby works, and we can't change that. I omitted the equals sign to make the DSL still work.
Config.configure do
  # This won't work
  verbose = true
  format = :documentation
end
It's possible to work around this by explicitly calling the methods on self., but this syntax becomes basically the same as using yield. Instead, I removed the equals sign from the method name.
Config.configure do
  # This works, but requires `self.` in front of every method call
  self.verbose = true
  self.format = :documentation
end

When to use yield or instance_eval?

I've shown two ways of calling blocks in Ruby. Which one should be used is up to the authors of the DSL. I have my personal preference for using instance_eval, but there are definitely scenarios where using yield works better.

When something needs to be configured, using a lot of methods with an equals sign (writer methods), using yield with a block argument is quite common.

Config.configure do |config|
  config.verbose = true # Set a value
end

When defining something or calling actions, I see the instance_eval approach used a lot.

Config.configure do
  load_defaults! # Perform an action
end

That doesn't mean one excludes the other. Gems like Puma use the instance_eval method for their configuration file.

# config/puma.rb
workers 3
preload_app!

Use the approach you feel works best for your DSL.

Share DSL behavior with modules

With the above methods you can start building your DSL. This gives us a good toolbox. As the DSL grows, or more DSLs get added to the domain, it's good to share behavior between the DSL classes.

For example, this domain has two separate configuration classes: a Config and an Output class. One configures how the gem works, the other how it outputs results to the terminal.

# Multiple configuration classes
Config.configure do
  enabled true
end

Output.configure do
  verbose true
  format :documentation
end

We don't want to repeat ourselves, so we move the DSL definition behavior to a module called Configurable. When that module is included the class method def_options is made available with the class definition.

class Config
  include Configurable

  def_options :enabled
end

class Output
  include Configurable

  def_options :verbose, :format
end

When the Configurable module is included, it adds the options method to the class. The module also extends the class it's included in, using the included callback in Ruby modules. In that callback, the class is extended with the ClassMethods module, defined inside the Configurable module. This module will add the class methods (def_options and configure) to the class. This is also how things like ActiveSupport::Concern work.

module Configurable
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def def_options(*option_names)
      # ...
    end

    def configure(&block)
      # ...
    end
  end

  def options
    @options ||= {}
  end
end

With all the logic moved to the Configurable module, we now have a reusable module to create as many config classes as the domain needs.

Nesting DSLs

The Output class is really a sub domain of the Config class though. It's part of the gem config: the configurations should be nested within one another.

Config.configure do
  enabled true

  output do
    verbose true
    format :documentation
  end
end

Rather than Output being a top-level config class, we store it on the Config class. In the output method in the example below, the method creates a new Output instance. When a block is given, it will instance_eval the block on the @output object. The method will always return the created @output object so the sub-config can be accessed.

class Config
  include Configurable

  def_options :enabled

  class Output
    include Configurable

    def_options :verbose, :format
  end

  def output(&block)
    @output ||= Output.new
    @output.instance_eval(&block) if block_given?
    @output
  end
end

Module builder pattern

The Module builder pattern is a really neat design pattern in Ruby that allows us to do away with the two step process of including a module and then calling methods included by it. This pattern is described in more detail in The Ruby Module Builder Pattern by Chris Salzberg.

In the example below, a new ConfigOption module is used to include a dynamically defined module. For the end-user, the resulting DSL remains the same.

class Config
  include ConfigOption.new(:verbose, :format)
end

When ConfigOption.new is called, the desired config option names are given to the initialize method. Like before, we iterate over this list. Using define_method the necessary methods are defined on the module. It's possible to do so in this initialize method, because it's creating a new implementation of the ConfigOption module.

class ConfigOption < Module
  def initialize(*option_names)
    define_method :options do
      @options ||= {}
    end

    option_names.each do |option_name|
      define_method option_name do |value|
        options[option_name] = value
      end
    end
  end
end

For a much more in-depth look, I can highly recommend reading The Ruby Module Builder Pattern by Chris Salzberg.

Create DSL objects

Another step I recommend is to separate the behavior of the DSL itself and the actual app code that uses it, by creating DSL classes. These classes are only used when the end-user interacts with the DSL, like configuring a gem. When the configuration is done, the options are read from the DSL class and set on whatever config object the gem uses.

Like before, we have a Config class. When configured using Config.configure, it creates a ConfigDSL class (1). This ConfigDSL class will become the context of the block given to the configure class method.

class Config
  def self.configure(&block)
   dsl = ConfigDSL.dsl(&block) # 1
   new(dsl.options) # 2
  end

  attr_reader :options

  def initialize(options)
    @options = options
  end
end

class ConfigDSL
  include ConfigOption.new(:verbose, :format)

  def self.dsl(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end
end

After the configure block has been called, the Config class reads the options from the ConfigDSL and initializes a new instance of itself with these options (2).

With this approach the DSL that configures the gem is only used when the app starts. The app doesn't carry around all that extra weight of how the DSL works all the time. The separation of this behavior makes it easier to not accidentally call the configure DSL when it should not be available.

This approach also solves a downside of using instance_eval. When calling blocks this way, private methods are accessible inside the block. That's something we usually don't want to allow.

class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  private

  def secret_method
    "super secret method"
  end
end

Config.configure do
  # This should raise an error about calling
  # a private method, but it doesn't
  secret_method
  # => "super secret method"
end

Private methods can be accessed in blocks called using instance_eval, because the block is evaluated in the context of the instance. It's as if it's being run from a method within that object. With separate DSL classes you have to worry less about private methods that you don't want anyone to call.

Conclusion

Now that our toolbox is complete we can create our own Domain Specific Language using Ruby. We have the following tools at our disposal:

Use define_method to dynamically define new methods on classes.
Use Ruby blocks to give your DSL that real DSL feeling and create a context wherein your DSL shines.
- Use yield to return a DSL object as a block argument, or;
- Use instance_eval to change how to block works, and allow end-users to directly call methods on the new context of new block.
Use Modules to:
- share behavior between many classes, and;
- use the Module builder pattern to remove any logic of how the DSL is constructed from the class that uses the DSL.
DSL objects separate the logic of how the DSL works and how the rest of the gem works. Creating a better separation of responsibilities in your code.

Now go forth, and build your own DSL!

Calculating String length and width – Fun with Unicode

2022-06-14T00:00:00+00:00

Let's calculate string length in Rust¹! How many characters there really are in a string and how much space these strings take up when displayed.

¹: This article may also apply to other languages. This article will only focus on strings with the Rust default UTF-8 encoding. There's an appendix for how it works in Ruby at the end of the article. I'm simplifying this content to keep the article short.

String.len()

The first function you'll probably come across is String.len()/str/len(), or length of string. Given the string "abc" it will returns the length of three. All looks good so far.

"abc".len() // => 3

That is, until we take a closer look at the docs for this function. It says the following:

Returns the length of this string, in bytes, not chars or graphemes. In other words, it might not be what a human considers the length of the string.

Source: String.len()

The Rust docs are giving us a warning here that it may not always return the number we'd expect. It will return the string length in bytes, and it sounds like not all characters are counted as one byte.

Let's try something that's not just plain "a" through "z", but something like a character with an accent.

"é".len() // => 2 bytes

We can see here that the result is a larger number than what we consider the string length to be. A lot of characters are comprised of multiple bytes. There are only so many characters we can make from an eight number byte, this is what ASCII is. To support all characters of all languages in the world in 256 possible different bytes wouldn't fit. Let's try another approach.

Chars.count()

Rust has a built-in "Chars" module we can use to split up the string into a list of characters, this gives a more accurate result.

"abc".chars().count() // => 3 characters
"é".chars().count()   // => 1 characters

When we ask the str.chars() method to give us a breakdown of what it considers characters we get a pretty good result. The character with the accent is seen as one character.

"é".chars() // => Chars(['é'])

We've learned that characters can be composed of multiple bytes. Relying on the string byte length for the actual length is not accurate enough.

Will Chars always return an accurate string length? This depends on your use-case. If you're looking for the number of characters in a string, probably yes, but also no. If you're looking for the actual string display width as rendered, the size it takes up on screen, then very much no.

If we look at the documentation again it will give us another warning:

It's important to remember that char represents a Unicode Scalar Value, and might not match your idea of what a 'character' is. Iteration over grapheme clusters may be what you actually want.

Source: str.chars()

Graphemes

What if the string being checked doesn't only contain numbers and letters, accents or not? Emojis are very popular nowadays and present in every kind of string. Emojis can have a byte size of much more than two bytes, but also consist of multiple characters even though it looks like one object.

// Person emoji
"🧑".len()           // => 4 bytes
"🧑".chars().count() // => 1 characters

// Woman scientist emoji
"👩‍🔬".len()           // => 11 bytes
"👩‍🔬".chars().count() // => 3 characters

// Family: Man, Man, Girl, Boy emoji
"👨‍👨‍👧‍👦".len()           // => 25 bytes
"👨‍👨‍👧‍👦".chars().count() // => 7 characters

The "woman scientist" emoji shown above takes up eleven bytes and three characters. The family takes up 27 bytes and seven characters, even though we only see one item in the string.

If we print the list of characters we'll get a better idea of what is happening.

"👩‍🔬".chars()
// => Chars(['👩', '\u{200d}', '🔬'])

"👨‍👨‍👧‍👦".chars()
// => Chars([
//   '👨',
//   '\u{200d}',
//   '👨',
//   '\u{200d}',
//   '👧',
//   '\u{200d}',
//   '👦'
// ])

Emoji can consists of multiple characters, that together construct another emoji and tell computers what to render. In the example above, we first see the "woman" emoji, then what is called a "Zero Width Joiner" character, and finally the microscope emoji. For the family we see the whole list of different genders and ages joined together in the same way.

The Zero Width Joiner character used by these combined emoji is, as the name describes, a Unicode character with zero width (which makes it invisible). This character is used to join together multiple emoji to make a new one.

These additional emoji characters do mess up our string length calculation. Luckily we can use the unicode-segmentation Rust crate to do more accurate character counting. We can ask the crate to split the string based on something called Graphemes and return the list of characters. A grapheme cluster in this context is a group of Unicode codepoints that consist of "user-perceived character", what we consider a character when we type it.

use unicode_segmentation::UnicodeSegmentation;

"abc".graphemes(true).count // => 3
"é".graphemes(true).count   // => 1
"🧑".graphemes(true).count() // => 1
"👩‍🔬".graphemes(true).count() // => 1
"👨‍👨‍👧‍👦".graphemes(true).count() // => 1

(The true argument given to graphemes function means we want to count it using the Unicode extended grapheme clusters and not the legacy clusters. Let's use the non-legacy cluster.)

Finally! We have an accurate string length. Right?

Yes. For the scenarios in which you want to know the number of characters in a string, I'd say yes.

But...

Graphemes widths

As you can see in the code example below, emoji aren't shown one character column wide. Most emoji are rendered with a display width of two columns, two other Latin characters, like A through Z. For illustration purposes I'll be using a monospaced font so that the latin characters have a predicable width.

// These two lines should render with the same width
"ab"
"🧑"

While the graphemes-count-method (shown in the previous section) will give you a correct string length in terms of how many characters you see, it's not always same as the horizontal space in columns a string takes up.

In my Lintje project (a Git linter) I ran into this string length vs display width problem for the program's rules and output formatting. The terminal output didn't align properly when a string in a Git commit message included a emoji. In the example output below, the ^ marker should be aligned with the last character of the string, but with an emoji in the string it wouldn't be properly aligned.

# Badly aligned output
String with a ❌ emoji
                    ^ End of sentence
# Well aligned output
String with a ✅ emoji
                     ^ End of sentence

For rules that checked string length, I decided to count the display width of how every character is rendered as the string length. If a string has a max length of 50 characters, because of readability and width constraints, it should not be allowed to jam that string full with 50 emoji. It would take up double the horizontal space of a string with the same length without emoji.

Using the unicode-width Rust crate we can calculate a more accurate string display width.

UnicodeWidthStr::width("🧑") // => 2
UnicodeWidthStr::width("👩‍🔬") // => 4
// Consists of 👩, a Zero Width Joiner and 🔬
UnicodeWidthStr::width("👨‍👨‍👧‍👦") // => 8
// Consists of multiple face emoji and Zero Width Joiner characters

That is, the display width that every character is defined as being part of each grapheme cluster in Unicode. As you can see, the emoji with a Zero Width Joiner characters are as wide as the number of emoji they join times two.

There are also "emoji" that only return a display width of one, not two, columns. These one column display width characters are considered to have a "narrow" display width. The two column wide characters are "wide". This difference is something you can see if your terminal and other applications, as it sometimes overlaps with the next character. This will not be visible in the browser you're reading this in.

"❤️".len()           // => 6 bytes
"❤️".chars().count() // => 2 characters
"❤️".chars()         // => Chars(['❤', '\u{fe0f}'])
UnicodeWidthStr::width("❤️") // => 1 display width

The heart emoji ❤️ is a "Heavy black heart" character ❤ from the dingbat collection combined with what's called a "Variation Selector-16" character. This is another invisible character in graphemes clusters to indicate certain characters should be displayed as their emoji counterparts.

And the actual width is?

Why is this happening? From what I understand the unicode-width library is following the Unicode reference to the letter, counting only the base character's display width and not the emoji variation. That results in output that I would not have expected. It's not something that can be fixed, unless you change something in Unicode first.

Unfortunately I don't have another function or library to call on to fix this and return what we see as the actual display width.

What I did in my Lintje project was first split up the string into graphemes clusters, then scan every sub string for Zero Width Joiner characters, and then only return "two" as the width. This is somewhat more accurate to what we see displayed, but it's not 100% accurate. It's good enough for my purpose right now. There probably are better ways out there to do this, but I fear that it will mean maintaining a list of the display width of all emojis. At least the exceptions with a display width of one. Let me know if you know of a better solution!

// Very simplified example
let string = "👨‍👨‍👧‍👦";
let mut width = 0;
let unicode_chars = string.graphemes(true);
for character in unicode_chars.into_iter() {
    if character.contains("\u{200d}") {
        width += 2;
    } else {
        width += UnicodeWidthStr::width(character);
    }
}
width // 2

The full implementation with additional checks for other modifier characters can be found in the Lintje GitHub project's utils module.

In conclusion

We've learned that what we humans consider string length is not the byte length. Characters can be multiple characters joined together, like emoji. Not all characters have the same display width. Some characters take up more horizontal space than others and some don't even take up any horizontal space.

So what should you use to calculate string length or display width?

Do you want the length of a string in bytes? Use String.len().
Do you want the number of characters in a string? The Chars.count() method is an option, but I suggest the next solution.
Do you want the visible number of characters in a string? Use str.graphemes(true).count().
Do you want the somewhat accurate width of the string? Use UnicodeWidthStr.width(str).
If you're like me and want the string display width calculated more accurately? You'll have to write something yourself I'm afraid.

Appendix: Ruby

I write a lot of Ruby code, so here's how the above applies to Ruby, gor as much as I looked into it.

When calling Ruby's String#length, it returns the length of characters like Rust's Chars.count. If you want the length in bytes you need to call String#bytesize.

"abc".length   # => 3 characters
"abc".bytesize # => 3 bytes
"é".length     # => 1 characters
"é".bytesize   # => 2 bytes

Calling the length on emoji will return the individual characters as the length. The 👩‍🔬 emoji is three characters and eleven bytes in Ruby as well.

"👩‍🔬".length   # => 3 characters
"👩‍🔬".bytesize # => 11 bytes

Do you want grapheme clusters instead? You're in luck: it's built-in to Ruby with String#grapheme_clusters.

"👩‍🔬".grapheme_clusters.length # => 1 cluster

To calculate the display with, we can use the unicode-display_width gem. The same multiple counting of emoji in the grapheme cluster still applies here.

require "unicode/display_width"

Unicode::DisplayWidth.of("👩‍🔬") # => 4
Unicode::DisplayWidth.of("❤️") # => 1

When not to use instance variables in RSpec

2021-02-02T00:00:00+00:00

Using RSpec there is some confusion about the differences between let, let!, and instance variables in specs. I'd like to focus on how instance variables work in RSpec in combination with before :context blocks, and in what kind of scenarios you should and should not use them.

Why use instance variables?

The advantage of declaring instance variables in before :context (or before :all) blocks is that whatever value is assigned is only queried or calculated once for many specs. The before :context block is only executed once for all specs in that context. Those specs can use the instance variable without repeating the same setup for every spec, which should speed up the test suite.

Reading the above it might be tempting to put a lot of spec setup in before :context blocks. A fast test suite creates happy developers, right? But there's a downside to using instance variables in RSpec, which could make for very unhappy developers.

From the RSpec docs:

It is very tempting to use before(:context) to speed things up, but we recommend that you avoid this as there are a number of gotchas, as well as things that simply don't work.

[...]

Instance variables declared in before(:context) are shared across all the examples in the group. This means that each example can change the state of a shared object, resulting in an ordering dependency that can make it difficult to reason about failures.

The RSpec docs give us a warning about changing values of the instance variable. State can leak between specs using instance variables defined in a before :context block this way.

Let's look at some examples of specs using instance variables and in what scenarios in will break.

Specs sharing instance variables

In the example below specs only assert if the value of the instance variable matches the expected value. Since the instance variable is not changed, all the specs will pass. If the instance variable value took a long time to query or calculate we have saved that time for two specs in this file.

# spec/lib/example_1_spec.rb
describe "Example 1" do
  before :context do
    # Imagine this being a complex value to prepare
    # This block is only run once in the `describe "Example 1"` block
    @my_instance_variable = :my_value
  end

  it "spec 1" do
    expect(@my_instance_variable).to eql(:my_value)
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql(:my_value)
  end
end

$ bundle exec rspec spec/lib/example_1_spec.rb --order defined
Finished in 0.00223 seconds
2 examples, 0 failures

(I'm using --order defined in the examples in this post so that the spec execution order is predictable and reproducible.)

But specs can be more complicated than this. They may pass the instance variable to some other part of the app, which modifies the given value. This is where things go wrong, what the RSpec docs warn us about.

Reassigning the instance variable

If changing the instance variable is the problem, let's reassign it and see what happens in other specs.

In the example below the "spec 1" spec changes the instance variable to test a slightly different scenario.

# spec/lib/example_2_spec.rb
describe "Example 2" do
  before :context do
    # Imagine this being a complex value to prepare
    @my_instance_variable = :my_value
  end

  it "spec 1" do
    @my_instance_variable = :new_value
    expect(@my_instance_variable).to eql(:new_value)
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql(:my_value)
  end
end

$ bundle exec rspec spec/lib/example_2_spec.rb --order defined
Finished in 0.00223 seconds
2 examples, 0 failures

In this example "spec 2" does not fail, even though "spec 1"—which runs before "spec 2"—changes the instance variable. There was no need for us to reset the original value of the instance variable at the end of the spec even though we changed it.

The way that RSpec runs the specs ensures that every spec uses the original instance variables. Every spec in RSpec is its own Ruby class, in which the spec is performed. Before the instance of the spec class is performed, RSpec sets the instance variables from the before :context block on the spec class. When an instance variable is reassigned in a spec, it only reassigns it on that spec class instance. It doesn't not reassign the instance variable on the same scope as the before :context instance variables are stored, and does not interfere with any other specs. An example of how this looks can be found later on in this post.

This behavior doesn't always quite work though. Let's see what happens if we use a bit more complex value. That way we know the limitations of using instance variables in RSpec.

Modifying instance variable values

In the next example the @my_instance_variable is assigned a more complex value: an array with multiple values. We will intentionally break the spec in this scenario.

In "spec 1" we're testing a slightly different scenario again, modifying the value before running the assertion. Instead of reassigning the variable we're adding a value to the array on @my_instance_variable.

# spec/lib/example_3_spec.rb
describe "Example 3" do
  before :context do
    @my_instance_variable = [:one, :two]
  end

  it "spec 1" do
    @my_instance_variable << :three
    expect(@my_instance_variable).to eql([:one, :two, :three])
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql([:one, :two])
  end
end

$ bundle exec rspec spec/lib/example_3_spec.rb
Failures:

  1) Example 3 spec 2
     Failure/Error: expect(@my_instance_variable).to eql([:one, :two])

       expected: [:one, :two]
            got: [:one, :two, :three]

       (compared using eql?)

       Diff:
       @@ -1 +1 @@
       -[:one, :two]
       +[:one, :two, :three]

     # ./spec/lib/example_3_spec.rb:14:in `block (2 levels) in <top (required)>'

Finished in 0.01596 seconds (files took 0.10576 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/lib/example_3_spec.rb:12 # Example 3 spec 2

Unlike before, the "spec 2" spec has now failed. It fails because the instance variable still has the value from the first spec. State has leaked from "spec 1" into "spec 2". Let's look at how the values from these instance variables have moved between specs.

How instance variables work in RSpec

To recap what we learned from the examples earlier:

Assigning instance variables in before :context means they'll only be assigned once, as the before :context block is only once run before all specs in the spec context.
Every spec in RSpec is performed as its own class, with its own scope. Instance variable from the before :context block are set on the spec class before it is performed.
Instance variable values do not leak between specs when the values are basic Ruby objects such as Symbols, numbers, etc.
Instance variable values do leak between specs when more complex object types such as Arrays, Strings, Class instances, etc. are modified.

Let's take a closer look at how RSpec handles instance variables for spec classes to see how this could break in our test suite.

How RSpec handles instance variables

When RSpec runs specs in a context, it first runs the before :context blocks. After a before :context block is executed RSpec then stores the list of the instance variables on the class it creates for the context.

When RSpec then starts a spec in that context it creates a new class for that spec and sets instance variables of that context on the spec class.

Let's look at how this works using the same scenario from the second example, where we reassigned the instance variables.

class Context # RSpec context class
  def self.before_context_ivars
    # Where the `before :context` instance variables are stored
    @ivars ||= {}
  end

  def self.set_before_context_ivars_on(spec_instance)
    # Set instance variables from `before :context` on spec instance
    before_context_ivars.each do |name, value|
      spec_instance.instance_variable_set(name, value)
    end
  end
end

class Spec # RSpec spec class
  def run # During spec
    @var = 2 # Reassign instance variable
  end
end

# Mock a `before :context` block and
# store an instance variable on the Context class
Context.before_context_ivars[:@var] = 1 # Basic Ruby object value
# Initialize the spec class
spec_instance = Spec.new
# Set the `before :context` instance variables on the spec
Context.set_before_context_ivars_on(spec_instance)

puts "Context @var before spec:",
  Context.before_context_ivars[:@var].inspect

# Run the spec
spec_instance.run

puts "Spec @var:", spec_instance.instance_variable_get(:@var).inspect
puts "Context @var after spec:",
  Context.before_context_ivars[:@var].inspect

(Simplified example for demonstration purposes.)

Context @var before spec:
1
Spec1 @var:
2
Context @var after spec:
1

RSpec sets the original value of the instance variable on every spec class instance. Reassigning the instance variable in the spec class does not modify the value of the instance variable on the context. It only changes it on the spec class instance.

When the instance variable value was modified however, the value stored on the Context class is also modified, as it is still the same value.

# Using the same context class as from the previous example

# Mock a `before :context` block and
# store an instance variable on the Context class
Context.before_context_ivars[:@var] = [1, 2] # More complex Ruby object
# Initialize the spec class
spec_instance = Spec.new
# Set the `before :context` instance variables on the spec
Context.set_before_context_ivars_on(spec_instance)

puts "Context @var before spec:",
  Context.before_context_ivars[:@var].inspect

# Run the spec
spec_instance.run

puts "Spec @var:", spec_instance.instance_variable_get(:@var).inspect
puts "Context @var after spec:",
  Context.before_context_ivars[:@var].inspect

Context @var before spec:
[1, 2]
Spec @var:
[1, 2, 3]
Context @var after spec:
[1, 2, 3]

In this example we can see the instance variable's value has changed, because the value was modified, rather than the instance variable being reassigned. This value was modified in memory, also changing the value in the context instance variable storage, which causes the modified state to leak into other specs.

Pointers

What happens is, RSpec (or rather Ruby) is assigning pointers to the values in memory when it sets the instance variable values on the spec class instance. Variables in Ruby are pointers to a place in the application's memory. Assigning a value to another variable does not make a copy of it, but points to the same location in memory.

When RSpec sets the instance variables, it doesn't set a copy of the original Array value. Instead it sets the pointer to the Array value in memory. If the Array in memory has changed during the spec run, it will set not the original value for the next spec, but the modified Array instead. This is part of how Ruby works, this is not something RSpec can "fix". And which is why the RSpec docs warn us about using instance variables in before :context blocks.

Alternatives

To prevent state from leaking into other specs by modified values, it's possible to "freeze" objects in Ruby. If we freeze an Array, String or other object instance, Ruby will not allow any modifications.

var = [1, 2].freeze
var << 3
# Raises an error to prevent modification
# => FrozenError (can't modify frozen Array: [1, 2])

But this will be more difficult to do for larger objects with nested objects, as it only freezes the top object and not all nested objects.

var = [[1, 2], [4, 5]].freeze
var[0] << 3
puts var.inspect
# => [[1, 2, 3], [4, 5]] # The nested value was modified

Alternatively it's possible to deep clone or dup the object. The problem with this is that it will take up a lot more memory, as every object will be kept in memory multiple times, so I can't recommend it.

Conclusion

RSpec scoping instance variables between specs in classes is a great help from RSpec for basic Ruby objects, but this behavior shouldn't be relied upon for more complex Ruby objects such as Arrays, Strings, and other object instances.

If a spec is modifying an instance variable value, you can't be sure what the value of that instance variable will be in the next spec. State may leak to other specs, breaking them in unexpected ways. This will be especially difficult to track down when the specs are run in a random order each time.

Make sure that if you use instance variables, you absolutely do not modify any value set on the instance variable if you want a predictable and reproducible test suite. And that's something we should all want. I'm all for fast test suites, but what I like more is a stable test suite.

A big thanks to Benoit Tigeot for fact checking this article!

Git: Review changes before committing

2020-10-26T00:00:00+00:00

Making a git commit can be as quick as git add --all && git commit -m "WIP". The problem with this approach is that we didn't check what changes we committed. Who knows what got committed that shouldn't have. Usually we find out when it's too late. We want to see the Git changes before we commit.

While working on an issue, I personally make a lot of changes that don't belong in the same commit. For that reason the approach of "now commit everything I have changed" rarely works for me.

Let's look at the things we don't want to commit first and then improve the way we make commits by reviewing what we commit first and how.

Things we don't want to commit

Some things that we don't want stored in git commits are:

App secrets
- Tokens, passwords, personal information, etc. These are things we don't want to commit to the git history, especially public repositories. A bot will scrape our hosting provider credentials and spin up a lot of machines on our dime before we know what happened.
Debugging code
- Some code may still be present in the project that helped debug an issue, such as a print/puts, a console.log, or a binding.pry/debugger-statement. While not necessarily harmful it creates a lot of output or hangs the process while running the app and test suites. It creates unwanted noise.

Removing these changes from commits isn't always as easy. Force pushing branches can mess up the team's flow and commits in public repositories don't disappear. Someone with a direct link can still access the old commits.

To go through the process of creating new app secrets, rolling those out across the app is very time consuming and cumbersome. Removing debugging code isn't as much trouble but it does require another commit, which adds noise to the git history. This doesn't help communicating in git.

Review what to commit

To avoid including things that shouldn't be committed and create better commits, let's look at ways to create commits.

This is the process I recommend going through when creating a commit:

Start with reviewing changes:
- Are we about to stage files we should not be staging, such as credential files?
- Are there debug statements in the changes we should remove first?
- Are there changes that are not required to fix the bug or add the feature? In a commit where a bug is fixed, don't include code style changes in an unrelated part of the code.
- Finally, this is also a review for you, the committer, to double check the changes that were made are actually all necessary and correct. Are there tests for every logic change? Did we add or update the docs?
Stage the changes that you approve and are necessary.
Only after this review, commit the changes.

The main idea of this is that we look at what files we stage, but more importantly, we review the changes in each file before we stage them.

Commit tools

Let's look at some tools that help with creating git commits. These are alternatives to git add --all command from the beginning of this post, which stages all changes: modifications, additions and deletions.

There are some basic tools that git provides us and there are third party tools we can use to make staging specific changes easier.

Git add file

Using the git add command it's possible to add one file at a time with git add path/to/file.md. This makes sure we don't add files we shouldn't commit, such as secret files, which is good.

$ git status
On branch main
Changes not staged for commit:
  modified:   path/to/file.md

$ git add path/to/file.md

$ git status
On branch master
Changes to be committed:
  modified:   path/to/file.md

But what about the changes in those files? We still don't know exactly what changes are being staged and will end up getting committed. We need a more detailed staging process.

Interactively stage changes

To review and select changes within a file we want to stage we can use git add --patch. This opens a basic selection interface in the terminal with which we can select "hunks" to be staged. Hunks are groups of changed lines in a file that git can easily stage and unstage.

$ git add --patch
diff --git a/app/models/blog_post.rb b/app/models/blog_post.rb
index 8b13789..257cc56 100644
--- a/app/models/blog_post.rb
+++ b/app/models/blog_post.rb
@@ -154,6 +154,12 @@ class BlogPost < ActiveRecord::Base
   end
+
+  def published?
+    binding.pry
+    Time.now >= published_at
+  end

   def active?
(1/10) Stage this hunk [y,n,q,a,d,e,?]?

In the example above we run git add --patch and it prompts us with the question if we want to "Stage this hunk". What's useful here is that we can see the "hunk" it's referring to in the preview above it, and review if it's any code we actually want to commit.

Choosing "y" for "yes", or "n" for "no", we can step by step stage changes or skip them. Git will take us through all the changes one "hunk" at a time until we've reviewed them all.

But maybe we spot an issue with the hunk. We can "q" for "quit" and open the file in our editor to fix the issue and then restart the process by running git add --patch again.

This interface is quite limiting. If we don't agree with git's hunk boundaries we can't easily change them. It's not very easy to only stage one line or omit a line. If we stage the hunk in the example above we also commit some debugging code that will break things for us later.

Git GUI

A git GUI can help give us more control over what we commit. Apps that give us humans a more easy to digest interface for git and may even help with more advanced tools such as merge conflict resolution and rebasing.

Git GUIs are great! Don't let anyone tell you otherwise.

When making commits it's important to have a view of the current changes, stage them interactively, and commit knowing we only commit those changes that were selected. A GUI can display very detailed changes in a concise way, and the selection process for staging changes is more precise.

Personally I use tig most of the time in the terminal, and Fork any time I run into tig's limitations. To get started I recommend Fork, or a similar GUI app, as it's a very complete Git tool. Both tools give great overviews of git history and staged and unstaged files. It's possible to stage entire files in one go, or deep dive and only add the one specific line we want to stage and nothing else.

There are many other git GUIs, and no one tool is the best or only one you're allowed to use. Experiment, try them out, and use the tool that works best for you.

An example commit flow

When I am making a commit I often use tig in the terminal, but for convenience I'll use Fork in this example as it will look the same for everyone.

First I open the app, then I'm presented with the git history. After opening the "Local Changes" view I get an overview of all uncommitted changes.

In this example I want to stage the env_helpers.rb file, but not all changes. There are changes in the same file for another feature I'm working on. If I stage the entire hunk I also stage the code I don't want to include. I could open the file in my editor again and remove the line temporarily, but then I should remember to restore it again later. (I'm going to forget that, aren't I?)

This is the process I go through in the video:

In the left panel's "Unstaged Changes" box I look for the file I've just made changes to and want to commit.
I open the file by clicking it and see the file changes in the right panel.
There's a change on a line I do not want to commit, as indicated by the "TODO" comment.
By selecting the line I do want to commit the Fork app gives me context specific actions to "Stage" the changes.
I press the "Stage" button and only the selected change gets staged. The remaining unstaged file change is the line with the "TODO" comment I do not want to commit.
I click on the file in the "Staged Changes" view in the left panel. Here I see the changes the one line change I just staged and nothing else. If these are all the changes I need, I can now commit them without also committing the unstaged changes.

Conclusion

I hope this peek into how I review my changes before committing helps you make better commits. It's a process to keep being aware of. When I haven't kept a close eye on what I commit, I have included and pushed things I shouldn't have. So keep an eye on it and let's make better commits together!

Review the changes you stage so you don't accidentally include any app secrets, debugging code, personal information, or anything else that shouldn't be committed.

Try out a GUI or any of the other tools I've listed in this post and let me know which is your favorite!