To make structured, analytical decisions, we can leverage a very simple yet powerful tool known as Multi-Criteria Decision Matrix, as described in the book, “Psychology of Intelligence Analysis”, by a former CIA veteran Richards J. Heuer, Jr.. Before I introduce it, let’s understand the problem context to which it is most relevant.

Problem: Buying a house is complicated

Let us imagine the following scenario; let’s say we would like to buy a house. There are many factors involved in buying the house: obviously its price, square footage, where it is located (e.g., close to or far from work?), the crime levels in the surrounding neighborhood, quality of the building, future maintenance fees, and so on.

We often buy the houses with our partners, and that can lead to conflicting preferences. For example, what is more important: having a backyard, but paying more, or perhaps vice versa? There is no “right” or “wrong” decision, just different set of consequences (higher vs lower price; there is a backyard vs there is none).

Buying a house, in general, is a high-stakes situation due to the amount of money that we need to spend. We also often consider how long we are going to live at that place, along with “investment” opportunities, as in how much it is going to appreciate or depreciate over the next decade.

In short, it is a complicated and stressful process that takes weeks, if not months. However, there is a way to simplify it by approaching the process from the analytical mindset.

Solution: Multi-criteria Decision Matrix

The essence of making a good decision is to gather and structure available information at hand (the known knowns). Humans are not good at holding a lot of information at the same time in their head. Thus, we need to externalize the information onto a visible form, such as an A4 paper. In addition to the externalization, we break down complex information into its simpler constituents.

One way to externalize and break down the information in a structured way is known as Multi-Criteria Decision Matrix (MCDM). It is a hands-on, paper-based method for making a decision that considers multiple constraints (criteria). Let’s see how it applies to the housing purchase situation above.

First, we come up with a list of attributes, such as:

• Location
• Price
• Square footage
• Estimated maintenance costs
• Whether there is a garden or not
• etc.

Next, we both assign relative importance to each of the attributes:

The ranked list of attributes helps focus on what matters most and leave out things that do not matter. For example, is garden really important to us or can we live without it? Are we okay with paying more but get a better location, or vice versa, get a house further away but for a cheaper price? Should we include Maintenance in the Price attribute rather than keeping it separate? Hence, we can now make a more structured decision on what matters to us before searching for a house. On top of that, we will immediately notice the difference in our preferences and that of our partners, be able to quantify that difference, and discuss it in detail.

Now that we have visited a few houses, we are ready to apply the structured analytical method to make a decision. You now need to quantify each house’s “score”: distribute (imaginary) 100 points among the options we have. For example, in the table below, House A gets 50, House B gets 10 and House C gets 40 points for the same Location attribute. Simply put, we liked House A the most, and House C is close enough to it, but we did not like House B as much; all in terms of the “goodness” of location.

Having split 100 points across our options for each attribute, we can now calculate the total score by multiplying the %-ge importance by the score of the house and sum it up:

Voila! In the table above, House A is the winner with total of 36 points. House B and C have the same total scores despite scoring wildly differently in some attributes, such as Location and Price. Note that the points themselves only matter in relative terms, i.e., what house is better than another and by how much.

We can also compare each of our results with our partners (if done separately). The separate result will show the difference in both of our preferences and help re-evaluate houses in case there is a vast difference between our results.

You can extend the matrix and do sensitivity analysis to determine how much change in one attribute’s score can swing our decision from one house to another. For example, we can calculate the amount that the House B’s price has to go down for us to make it our primary choice rather than House A. This kind of add-on can help us understand how sensitive our choices are (firm on one option vs being open to consideration of alternatives).

Conclusion

In summary, we broke down a problem (“buying a house”) into smaller sub-problems (attributes & their relative importances for each option) and externalized it onto a paper. These steps helped us evaluate and rank the options, and ultimately make a decision.

Multi-Criteria Decision Matrix is one among many approaches to structured, analytical decision making. It can be applied anywhere where the stakes are high and a good enough decision has to be made, whether at home, in our career, in relationships, or elsewhere. I highly recommend to read Richard Heuer’s book, as it is packed with valuable and insightful information on how to effectively make sense of a vast amount of information in today’s world.

References

• Heuer, Richards J. Psychology of intelligence analysis. Center for the Study of Intelligence, 1999.

Writing to PostgreSQL

The SQL in PostgreSQL is already a spoiler for what language is used to retrieve or store the data. One of the common ways, which is also part of SQL standard, is to use INSERT command:

INSERT INTO customer (id, name, age)
VALUES (123, "john doe", 29);

In the example below, we insert (store) a single row containing 3 columns. This classic usage of INSERT is quite common when you need to store a few rows (e.g. when you need to insert new information in real time).

But does INSERT still work with larger amount of data? What if we write 100s of columns? One way is to loop over your data records and insert them one by one:

import psycopg

customers = [( 123, "john doe", 29 ), ( 125, "jane doe", 31 )]

# Connect to an existing database
with psycopg.connect("dbname=marketplace user=mytestusername") as conn:

    # Open a cursor to perform database operations
    with conn.cursor() as cur:

        # Pass data to fill a query placeholders and let Psycopg perform
        # the correct conversion (no SQL injections!)
	for id, name, age in customers:
		cur.execute(
		    "INSERT INTO customer (id, name, age) VALUES (%s, %s, %s)", (id, name, age)
		)

        # Make the changes to the database persistent
        conn.commit()

This works fine for a handful of queries. Once you get into 1000s or even more records, the code above will progressively start getting worse. It is not scalable.

Another option is to use cur.executemany() where the same query will be executed for a list of tuples instead of a single tuple. It is better than serially executing INSERTS because the transaction is committed only once instead of multiple times. However, executemany() will still be very slow because it is not optimized for very large amounts of data (>=1 million rows).

Best option for “batch” loading (when we load large amount of data at once) is to use COPY. COPY is done only within a single transaction and treats the entire file as an input stream. It is specifically optimized for batch loading.

COPY customer FROM 'customer_data.csv' DELIMITER ',';

Benchmarking

I created a repository with benchmarking code that shows the performance difference between the three methods (i.e. execute, executemany, and copy). Feel free to fork it and reproduce locally. The results might differ, but only slightly. The speed-up ratios between the approach should stay about the same.

Here are the results from a single run:

Function 'very_slow_insert' [execute one by one] executed in 4309.07 secs / 71.82 mins
Function 'slow_insert' [executemany] executed in 168.56 secs / 2.81 mins
Function 'fast_copy' [copy] executed in 61.26 secs / 1.02 mins

COPY command was about ~71x faster than running INSERT in a loop and ~2.8x faster than running a single-batch INSERT command.

Conclusion

It is clear that COPY is the best option for large data loading in Postgres. The single-batch INSERT (with many values) works too, but is still behind the copy. The individual INSERT commands are the slowest and I do not recommend running them unless you are only inserting a handful of rows and do not generally have a large load.

With the exponentially increasing volume of data, it would be nice to have the ability to read data larger than the available memory. Inspired by Daniel Beach’s post on DuckDB vs Polars, I would like to do a similar analysis of data processing libraries that focus on high performance. The only difference is that I would not be reading data from a cloud storage like S3. Instead, I would have the data downloaded locally on my computer.

Setup

Data. I will use the same source of raw HD test dataset that Daniel used (Backblaze hard drive data stats such as models, capacities, and failures). I could not determine what files exactly he downloaded, but I will use 2022 Q1 and Q2, and all of 2023 data for our setup. So we have a total of ~45 GB of CSV files.

$ du -sh data_* --total

6.2G	data_Q1_2022
7.1G	data_Q1_2023
6.4G	data_Q2_2022
7.6G	data_Q2_2023
8.6G	data_Q3_2023
9.0G	data_Q4_2023
45G	total

Compute. For the compute power, I have 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz with 4 cores (2 threads per core; so total 8 threads). I have 16 GB RAM total, but I will limit it to 4 GB only using ulimit -v 4000000 (4 million bytes) for each test.

Task. Our task will be to compute the number of failures grouped by dates in the set of ~45 GB of CSV files. In SQL, it looks like:

SELECT date, SUM(failure) as failures
FROM table_with_all_failures 
GROUP BY date

Setting up the tools

Polars. Polars is a high-performance data processing library. It can be used to manipulate structured data in a very fast way. While the core of the library is written in Rust, the library has APIs in Python, R, and NodeJS. Basically, think of it as a very fast alternative to Pandas (but remember, it’s not quite a drop-in replacement due to some major differences between the two).

Our test code using Polars looks like this:

import polars as pl
import time

# Uncomment line below for more logs.
# pl.Config.set_verbose(True)

def polars_test():
    lazy_df = pl.scan_csv("*/*.csv")

    sql = pl.SQLContext()
    sql.register("harddrives", lazy_df)   
    results = sql.execute("""
        SELECT date, SUM(failure) as failures
        FROM harddrives 
        GROUP BY date
    """)

    results_filename = "results_polars.csv"
    results.collect(streaming=True).write_csv(results_filename)

start_time = time.time()
polars_test()
end_time = time.time()
print(f"It took {end_time - start_time} seconds to run Polars test.")

In the code snippet above, we are lazily scanning the set of CSV files into a polars LazyFrame, register the dataframe as a (quasi) SQL table so that we can run the aforementioned SQL query on it. Note the .collect(streaming=True) part with the streaming parameter: it will process the data in chunks because our dataset is larger than available memory. Once we get the results of the grouping operation, we write them to a CSV file results_polars.csv.

DuckDB. DuckDB is a “fast in-process analytical database”. Think of it as in-memory database that allows you to perform very fast computations on columns (a.k.a. OLAP). Similar to Polars, it supports a SQL dialect that can be used to query and manipulate data.

Our test code using DuckDB looks like this:

import duckdb
import time


def duckdb_test():
    duckdb.sql("""
        SET preserve_insertion_order = false;
        SET temp_directory = './temp';

        CREATE VIEW metrics AS 
        SELECT date, SUM(failure) as failures
        FROM read_csv('*/*.csv', union_by_name = true)
        GROUP BY date;
    """)

    duckdb.sql("""
        COPY metrics TO 'results_duckdb.csv';
    """)

start_time = time.time()
duckdb_test()
end_time = time.time()
print(f"It took {end_time - start_time} seconds to run DuckDB test.")

In the snippet above, we read the CSV files within the SQL statement using read_csv() function. I had to set union_by_name parameter to circumvent the duckdb.duckdb.InvalidInputException: Invalid Input Error: Mismatch between the schema of different files. The parameter combines schemas of different files by column name.

Also note couple configuration parameters, namely preserve_insertion_order = false and temp_directory = './temp'. The former is set to flag DuckDB that it does not have to preserve the order of the files it reads. Disabling the insertion order preserves memory. For the latter, setting temp_directory should have enabled us processing data larger than memory. According to DuckDB,

If DuckDB is running in in-memory mode, it cannot use disk to offload data if it does not fit into main memory. To enable offloading in the absence of a persistent database file, use the SET temp_directory statement”.

Despite many tries with different parameters, I could not make it work. Some folks needed to set number of threads to 1 to make it work. Others recommended using some nightly build that fixes the issue, but it looks like the issue is still there.

That’s quite unfortunate given that DuckDB claims that the larger-than-memory workloads are its “key strength”:

A key strength of DuckDB is support for larger-than-memory workloads, i.e., it is able to process data sets that are larger than the available system memory (also known as out-of-core processing). It can also run queries where the intermediate results cannot fit into memory.

Welp, did not work for me :/

Dask. Dask is a library for parallel computing in Python. It is a feature-rich library that lets you scale Python code from a single computer to large distributed clusters.

Here is our setup with Dask:

import dask.dataframe as dd 
import time


def dask_test():
    dfs = dd.read_csv("*/*.csv", dtype={'failure': 'float64'})
    result_df = dfs.groupby("date").failure.sum()
    result_df.to_csv("results_dask.csv", single_file=True)

start_time = time.time()
dask_test()
end_time = time.time()
print(f"It took {end_time - start_time} seconds to run DuckDB test.")

The code above reads the CSV files into Dask DataFrame, groups data by date and computes the sum of failures, saving the results in another CSV file. Note that I had to cast failure column to float64 because it would throw ValueError recommending that I change the type from int64 to float64, even though I never specified int64. It is most likely that most of the entries in the column are indeed of type int64, however it recommends float64 due to the presence of NaNs.

Results

We are finally in the results section! So here are they for each of tool set up above.

Polars. Polars is a winner with its setup script taking between 8-15 seconds to get the CSV file with the results. The setup, as you have seen, looks simple enough, it is readable, no surprises.

DuckDB. Unfortunately, DuckDB kept throwing “Out Of Memory” exceptions:

duckdb.duckdb.OutOfMemoryException: Out of Memory Error: Failed to allocate block of 32002048 bytes

As mentioned in the Setup section, I tried playing with several configuration parameters like temp_directory and preserve_insertion_order. Neither of them fixed the issue.

Dask. Our Dask test took between 175 seconds (~3 minutes), much slower than Polars. There were no surprises with Dask, but I also think it was not built for single-computer data processing. Even though I have not tested it yet, it would most likely shine in distributed computing environments where one can run data-intensive programs across many computers.

Conclusion

So here we have the results for the three tools. Polars is a clear winner here, but bear in mind that I provided results for one specific test which is about performing data analysis on data that is larger than memory. I am sure DuckDB is as performant as Polars, although I could not get the out-of-memory issue resolved. Dask is in between, likely a choice better for distributed computing environments rather than a single computer case.

The source code is open source and available here on GitHub.

Today’s world is dynamic and chaotic making it nearly impossible to predict what is expected in future. Similar behavior can be observed in software development, where requirements may drastically change within days even with careful, long-term planning. That is perhaps why Agile methodology is a popular and the “standard” approach to software development in today’s world.

Imagine a scenario in which you talk to a client about some cool mobile app idea, and they ask you to store, process, and display some data in the app. However, next week, they ask you us for something else, so you need to address the change in client’s requirements on data. As a consequence, you need to make changes to the application and the way it stores, processes, and renders the data.

If we had something hard-coded in code that needed to be changed, it would be simply changing the value of the associated structures (e.g., variables or lists of strings). However, software developers detach code from data, and typically use storage systems for storing the data.

There is a variety of database systems, such as relational and non-relational (e.g., NoSQL). The relational systems are great at establishing “relations” between different data entities, which provide safeguards against data anomalies and often make it efficient to retrieve related data together (e.g., through table joins and column indexes).

We know how developers commonly “version” source code changes to keep track of where the source code was and where it is at the moment. Additionally, it is important for effective collaboration with other developers. However, changing database schema is not as clear cut as changing code. Accidentally modifying (or even worse, deleting data) can result in large negative impact, including loss of monetary value and/or reputation.

Luckily, we are not the first ones to encounter such a problem. In this post, I would like to focus on changing the schema in the relational SQL databases, such as PostgreSQL, in effective ways that keeps the database state consistent. By the end of this blog post, we will learn how to effectively manage changes in SQL databases.

Schema changes as migrations

Software requirements change frequently, and so do database schema associated with them. Changing the schema is a tricky task that can easily result in data inconsistency and even unexpected downtimes. Not surprisingly, making changes to the schema can be a nerve-wracking experience.

If schema changes are important, then we should monitor them. We should track who makes the change, when it was made, and why it was made. We should have something similar to git workflow, where incremental changes to code are noted in history.

Akin to git commits, that keep track of changes in the application code, schema migrations are changes to the file(s) that represent the schema. Those changes can be applied incrementally (on top of each other) and can be easily reverted, if something goes wrong.

Take a look at the example code snippet below where we want to split one column into two. The initial version contained a column TEXT full_name that stored a person’s full name. A change to the schema was made to split full_name into first_name and last_name columns.

CREATE TABLE contact (
	id INT GENERATED ALWAYS AS IDENTITY,
-	TEXT full_name
+	TEXT first_name
+	TEXT last_name
	TEXT email
);

The changes made to the above SQL file can be tracked using git. That is an incomplete way of applying changes to the schema because they are not immediately reflected in the database. Yes, the file changed in the git repository, but that file cannot just be loaded in the database to apply the changes. So, how do we apply such a change in the database to actually split the column into two?

Migration tools

To apply the changes in the database, we can use a SQL schema migration tool, also known as database-as-code migration tool. There are plenty of amazing tools, both open source and free, as well as proprietary ones.

I am going to use Alembic, a lightweight schema migration tool that uses SQLAlchemy as its engine.

If you have never worked with SQLAlchemy, it is a cool library that provides both object-relational mapping (ORM) and database toolkit for Python applications. It helps map SQL schema onto Python data structures for easier, in-code data management.

Alembic keeps track of changes to the database schema through revision scripts. The revision scripts contain the “delta”, or the change that is applied to the schema. Let’s create an example script to split a single column full_name into two columns first_name and last_name. To create a revision script, we can run the following command:

alembic revision -m "split full_name into first_name and last_name"

The command will generate a file named something like a1829f4e7900_split_full_name.py. Note the prefix of the file name - that’s a revision hash used to mark a schema change, similar to a git commit hash. The contents of the file may look like this:

"""Split full_name into first_name and last_name

Revision ID: a1829f4e7900
Revises:
Create Date: 2023-02-02 11:40:27.089406
"""

# revision identifiers, used by Alembic.
revision = 'a1829f4e7900'
down_revision = None
branch_labels = None

from alembic import op
import sqlalchemy as sa

def upgrade():
    pass

def downgrade():
    pass

The file comes with a docstring with the short description of the change, revision ID, and creation date. Note that the comment also mentions "Revises: ", which indicates the previous revision ID. It is obviously empty in our file because we just created our first revision. If we created another revision in addition to the one we just generated, the new revision script would have "Revises: a1829f4e7900". Further, the variable down_revision indicates the same thing.

Note that we also have two empty functions generated for us, namely upgrade() and downgrade(). The former allows us to add logic for the new schema change, while the latter lets us add the logic to revert that new change in case of potential problems down the line (e.g., during deployment to QA).

Let’s fill those functions with some concrete logic:

def upgrade():
	# Add new columns 'first_name' and 'last_name' to the table 'contact'
	op.add_column('contact', sa.Column('first_name', sa.Text))
	op.add_column('contact', sa.Column('last_name', sa.Text))

	# Split 'full_name' and move into 'first_name' and 'last_name'
	results = op.execute("SELECT id, full_name FROM contact");
	for id, full_name in results:
		# Logic to split the name and insert into table
		first_name, last_name = full_name.split(' ')
		op.execute(f"UPDATE contact SET first_name = {first_name} WHERE id = {id}")
		op.execute(f"UPDATE contact SET last_name = {last_name} WHERE id = {id}")
	
	# Finally, drop the column
	op.drop_column('contact', 'full_name')

def downgrade():
	# Add 'full_name' column back.
	op.add_column('contact', sa.Column('full_name', sa.Text))

	# Join 'first_name' and 'last_name' into 'full_name'
	results = op.execute("SELECT id, first_name, last_name FROM contact");
	for id, first_name, last_name in results:
		# Logic to split the name and insert into table
		full_name = ' '.join(first_name, last_name)
		op.execute(f"UPDATE contact SET full_name = {full_name} WHERE id = {id}")
	
	# Finally, drop 'first_name' and 'last_name' columns
	op.drop_column('contact', 'first_name')
	op.drop_column('contact', 'last_name')

The snippet above only serves as a simplified example to showcase a schema revision script. It is not an optimal logic. Be careful when actually splitting full name into first and last names. In some cultures, there are no last names, or the last names may consist of multiple space-separate words. Lastly, make sure to RESTART your identity columns. We don’t want to accidentally cause an integer overflow in transaction ids.

The upgrade() function above performs three important steps: (1) creates two columns, (2) populates the two columns from an existing, older column, and (3) removes the older column that is no longer needed. Not surprisingly, the downgrade() function is the inverse operation of upgrade(): we add one column back, re-populate it from the two columns, and remove those two columns.

It’s best to keep migration scripts as small as possible. Even the snippet above could be split into multiple migration scripts. For example, in one revision, we can just add two columns. In the next one, we split the name into two parts and populate these two new columns. In the third and final revision, we get rid of the older column. Having smaller migration scripts allows users of the database to adjust their code without immediately causing breaking changes. More on evolutionary database design later.

After creating the script, we can now run alembic upgrade head to apply the change in the database.

$ alembic upgrade head
INFO  [alembic.context] Context class PostgresqlContext.
INFO  [alembic.context] Will assume transactional DDL.
INFO  [alembic.context] Running upgrade None -> a1829f4e7900

In case something goes terribly wrong, we can also revert that revision:

$ alembic downgrade -1
INFO  [alembic.context] Context class PostgresqlContext.
INFO  [alembic.context] Will assume transactional DDL.
INFO  [alembic.context] Running downgrade a1829f4e7900 -> None

Interestingly (and actually quite importantly), Alembic runs the above revision script within a transaction. Transactions are atomic in SQL databases (e.g., PostgreSQL), so if something fails within the transaction, it will be automatically rollbacked to the previous state to keep the database consistent. That’s amazing!

Benefits of migration tools

With the help of tools such as Alembic, teams can seamlessly work together on the schema changes and not be worried of corrupting the database state. Such tools provide developers with a nice view of schema changes as scripts, which makes it easier for them to keep track of, as well as, perform code reviews together. Migration tools are also a great addition to continuous integration and delivery (CI/CD) pipelines, which can run automated migrations in different environments such as Development and Production.

If you would like to learn more about applying Agile methodologies to databases, check out the following articles:

• Evolutionary Database Design by Martin Fowler describes a novel (at the published time) approach for database change management.
• If you are PostgreSQL user, their documentation is great along with the “Don’t Do This” best practices wiki.

Disadvantages

There is no silver bullet in software engineering. The same applies to using migration tools. Some of the biggest disadvantages are that:

• Tools can be expensive. Some of the tools come with great benefits … at a $-value cost. I have personally not used such tools, but perhaps they can provide greater benefits such as a separate UI for managing migrations.
• It is unclear what schema looks like at a given point. If schema evolves rapidly, there could be a rapid flow of new revision scripts being created and applied to the database. These add more complexity and make it difficult for developers to determine what the schema looks like at a given point of time.
• If something does not work, we need a new revision script. If some migration script is found to have some logical issues (a.k.a. bugs) after it was already applied, another migration script is likely required to fix the issue, since modifying existing scripts could lead to corrupt database state.

Final thoughts

Tools such as Alembic are great for managing change in database schema. As everything else in life, they come with pros and cons that software development teams need to consider before adapting such tools in their development lifecycle. In general, changing database schema can be a tricky thing and cause a nerve-wracking experience, but it becomes much simpler and smoother with the right migration tools and processes (CI/CD) at hand.

P.S. Wanna give Alembic a try?

I created a simple Dockerfile that will let you play with Alembic. This assumes that you have Docker installed locally.

# Use an official PostgreSQL image as the base image
FROM postgres:latest

ENV PYENV="/app/alembic_env"

# Install necessary packages for Python and Alembic
RUN apt-get update && \
    apt-get install -y python3 python3-pip python3-venv && \
    mkdir -p "$PYENV" && \
    python3 -m venv "$PYENV" && \
    . $PYENV/bin/activate && \
    pip3 install alembic

# Set environment variables for PostgreSQL
ENV POSTGRES_USER myuser
ENV POSTGRES_PASSWORD mypassword
ENV POSTGRES_DB mydb

# Expose the PostgreSQL port
EXPOSE 5432

Copy and paste the contents above to your local computer. Then, build and run the image.

$ docker build -t postgres_alembic:latest . && \
	docker run -it postgres_alembic:latest /bin/bash

To answer the question above, we first need to get as many 5-letter words as possible. In other words, we need to build a population of 5-letter words to be able to calculate precise probabilities of word occurrences, whether complete (guessing entire target word) or partial (guessing one or more character of a target word). While it may be difficult to get a complete data set of all 5-letter words in English, we can use some approximate datasets, such as the one with 5757 5-letter words from the Stanford GraphBase.

Probability of guessing a target word on the 1st try is 1 / (number of all 5-letter words). Evidently, it is a very low probability of success. What we can look for instead is good starting word(s). Those are the words that will maximise our chance of guessing a target word in 6 (or hopefully fewer) attempts.

Good starter words

We can use the following logic to derive a good starter word:

1. Calculate frequency of all letters (i.e., number of unique occurrences in words; do not double count). Sum all frequency counts.

# Given a list of words,
"aaa"
"abb"
"abc"
"cba"

# we get the corresponding dictionary of frequencies:
{
    "a": 4,  # 'a' occurs in 4 words, and so on.
    "b": 3,
    "c": 2,
}
# and sum of all frequencies is:
total = 4 + 3 + 2 = 9

1. Calculate weighted letter “coverage”, which shows the weighted percentage of letters (from alphabet of all words) covered by a word. The idea here is that we want to cover as many unique letters as possible for maximum diversity. The diversity helps us maximize likelihood of hitting at least one letter. We want a good starting word, not the “one-and-done” kind.

"aaa"
# ==> take unique letter 'a'
# ==> freq['a'] / n
# ==> 4/9 = 44%
# Coverage: 44%

"abb"
# ==> take unique letters 'a', 'b'
# ==> (freq['a'] / n) + (freq['b'] / n)
# ==> (4/9) + (3/9) = 77%
# Coverage: 77%

"abc"
# ==> take unique letters 'a', 'b', 'c'
# ==> (freq['a'] / n) + ... + (freq['c'] / n)
# ==> (4/9) + (3/9) + (2/9) = 100%
# Coverage: 100%

"cba"
# ==> take unique letters 'a', 'b', 'c'
# ==> (freq['c'] / n) + ... + (freq['a'] / n)
# ==> (4/9) + (3/9) + (2/9) = 100%
# Coverage: 100%

1. Sort by coverage in descending order.
2. Drink some coffee and enjoy a chocolatine.

Hold on for a sec. "abc" and "cba" have the same coverages (of 100%). Which one is better? Can one be better than another?

Taking positions into consideration

In case of having two equally “good” words, we should consider proportions of each letter in each position (0th to 4th indices) of each of the two words.

1. We look at each position “vertically” (over all k-letter words in a very large vocabulary) and calculate proportion (or “coverage”) of each letter in a given position. For example, with the four 3-letter words above, we will have the following positioned coverages:

# Position 1 ('aaac'):
{
    "a": "75%",
    # "b": "0%"  (not shown explicitly)
    "c": "25%"
}

# Position 2 ('abbb'):
{
    "a": "25%",
    "b": "75%",
}

# Position 3 ('abca'):
{
    "a": "50%",
    "b": "25%",
    "c": "25%"
}

We can also make an observation that the sum of relative frequencies at each position sums up to 100%.

1. Given that k is a fixed number of letters/positions (it is 3 in this case), then for each word, for each letter, get the positional coverage, e.g.:

Voila. We found the best word of the two: "abc". Congratulations, "abc"! :tada:

Actually, not quite. If you are diligent enough, you would see that "abb" has total (relative) coverage of 58.33%, i.e., the same as "abc". The problem stems from the repeating characters.

Why? Because, as letters start to repeat, the probability of hitting an existing letter from the target word becomes lower. For example, "abb" has only two distinct letters, namely "a" and "b", whereas "abc" has three. So, statistically speaking, "abc" has a higher probability of hitting some or all letters correct.

You may wonder why then the two words "abb" and "abc" have the same relative coverage. The answer is that we should not be counting positional coverage of a letter if we already saw it before. The following example shows the correct calculations:

Notice how we skip adding the proportion of the letter "b" from position 3 because we already processed it in position 2. This begs the question of what position should we choose if a letter occurs multiple times (i.e., in different positions). We just have to be greedy: for a given letter, we have to choose a position (and one only) that has the maximal proportion. By choosing the maximal proportion, we maximize the end value of the total (relative) coverage.

Alternatively, we can just ignore the words with repeating characters before returning a set of good starter words (ordered by total relative coverage). Note that we still use those words to calculate (relative) positional proportion (coverage) of each letter.

Here is a list of top-10 starter words, along with their total (relative) coverage (based on the dataset of ~5K 5-letter words):

{
    'cares': 0.16803890915407332,
    'bares': 0.1677609866249783,
    'cores': 0.16737884314747262,
    'bores': 0.1671009206183776,
    'pares': 0.16616293208268193,
    'tares': 0.16581552892131318,
    'canes': 0.1657807886051763,
    'pores': 0.16550286607608128,
    'banes': 0.16550286607608128,
    'cones': 0.16512072259857566,
}

Aaand, we are done. Or so I hope.

General Analysis

Here I was just playing with letter frequencies. Check it out.

It’s interesting that "s" is the most frequent letter in positions 1 and 5 (0 and 4, resp., if you are a techie).

General letter frequency

Frequency of letters in all 5-letter words (counting duplicate letters within each word):

{
    'a': 2348, 'b': 715, 'c': 964, 'd': 1181,
    'e': 3009, 'f': 561, 'g': 679, 'h': 814,
    'i': 1592, 'j': 89,  'k': 596, 'l': 1586,
    'm': 843,  'n': 1285,'o': 1915,'p': 955,
    'q': 53,   'r': 1910,'s': 3033,'t': 1585,
    'u': 1089, 'v': 318, 'w': 505, 'x': 139,
    'y': 886,  'z': 135
}

Letter frequency at each position

{
    0: {
        'a': 296, 'b': 432, 'c': 440, 'd': 311,
        'e': 129, 'f': 318, 'g': 279, 'h': 239,
        'i': 74,  'j': 73,  'k': 91,  'l': 271,
        'm': 298, 'n': 118, 'o': 108, 'p': 386,
        'q': 39,  'r': 268, 's': 724, 't': 376,
        'u': 75,  'v': 109, 'w': 228, 'x': 4,
        'y': 47,  'z': 24
    },
    1: {
        'a': 930, 'b': 32,  'c': 82,  'd': 43,
        'e': 660, 'f': 12,  'g': 24,  'h': 271,
        'i': 673, 'j': 4,   'k': 29,  'l': 360,
        'm': 71,  'n': 168, 'o': 911, 'p': 113,
        'q': 10,  'r': 456, 's': 40,  't': 122,
        'u': 534, 'v': 27,  'w': 81,  'x': 33,
        'y': 65,  'z': 6
    },
    2: {
        'a': 605, 'b': 128, 'c': 184, 'd': 178,
        'e': 397, 'f': 87,  'g': 139, 'h': 39,
        'i': 516, 'j': 8,   'k': 90,  'l': 388,
        'm': 209, 'n': 410, 'o': 484, 'p': 169,
        'q': 4,   'r': 475, 's': 248, 't': 280,
        'u': 313, 'v': 121, 'w': 98,  'x': 67,
        'y': 68,  'z': 52
    },
    3: {
        'a': 339, 'b': 99,  'c': 210, 'd': 218,
        'e': 1228,'f': 100, 'g': 176, 'h': 73,
        'i': 284, 'j': 4,   'k': 243, 'l': 365,
        'm': 188, 'n': 386, 'o': 262, 'p': 196,
        'q': 0,   'r': 310, 's': 257, 't': 447,
        'u': 154, 'v': 61,  'w': 70,  'x': 5,
        'y': 41,  'z': 41
    },
    4: {
        'a': 178, 'b': 24,  'c': 48,  'd': 431,
        'e': 595, 'f': 44,  'g': 61,  'h': 192,
        'i': 45,  'j': 0,   'k': 143, 'l': 202,
        'm': 77,  'n': 203, 'o': 150, 'p': 91,
        'q': 0,   'r': 401, 's': 1764,'t': 360,
        'u': 13,  'v': 0,   'w': 28,  'x': 30,
        'y': 665, 'z': 12
    }
}

Paid Financial Times … for free

I once stumbled upon an article from Financial Times, an online newspaper with emphasis on business and economic news.

The article, named What happens when you visit FT.com?, describes what happens in the background when one makes a request to ft.com. Besides many interesting technical details, I noted that FT used Heroku for hosting its services.

If you have ever used Heroku, you know that apps get public <NAME>.herokuapp.com domain by default. For example, say I register a Heroku app and name it apple. When I deploy my apple app (e.g., a RESTful web server), I will be able to access it on https://apple.herokuapp.com.

The moment I saw a word Heroku in their article, I wondered if I could randomly guess any of their services’ URL. Out of curiosity, I tried a few. Surprisingly, one of the URLs (namely, financialtimes.herokuapp.com) worked; I managed to access the same contents as in ft.com. What was more exciting is that I could access paid articles for free. In other words, one could read all articles, paid or not, without any subscription whatsoever.

Being not sure how serious this issue is, I nevertheless hoped to get some bounty for finding out a “backdoor” that let anyone read paid articles for free (in other words, authorization could be bypassed). After reporting the issue, even though I got nothing back (*sigh*), the URL link now does not render any FT content and the issue seem to be resolved.

It may have happened that one or more engineering staff members forgot to remove that app. Who knows, maybe it was used for testing (e.g., black-box system testing) or something else.

P.S. You can access archived versions of the website from 2017 at archive.org.

There are some things that should be installed before we get started:

• Docker
• Node JS
• Visual Studio Code (a.k.a VS Code)

Keywords

• Host - this is your computer where you are working. In computer networking terms (roughly saying), it is a computer that communicates with other computers.
• Docker image - the set of layers/instructions you describe to run (it is more like a sequence, where the order of commands matter).
• Docker container - instance of your image. Roughly saying, it is like an instance of some “class” (OOP).

Getting Started

Clone this project and let’s get started!

It all starts with npm init

Imagine that you are working on a computer where Node.js is not installed. One way to proceed with initialization of your project is to install Node.js locally using your package manager (e.g. apt) and then proceed with npm init. However, there is a cooler, more portable, and more cross-platform way of doing this; where, no version conflicts occur, no manual explicit configurations are needed to be set up with changing OS environment, etc. In other words, heaven Docker!

Docker is a container management service. The keywords of Docker are develop, ship and run anywhere. The whole idea of Docker is for developers to easily develop applications, ship them into containers which can then be deployed anywhere. What a brilliant and lovely idea for the DevOps workflow.

I assume you have latest Docker installed. Docker uses images (check the definition above) to run containers which are, roughly saying, isolated processes that share the same OS kernel. Note that that Docker containers are NOT magical, lightweight VMs! If you are interested how Docker Containers work behind the scenes, take a look at the following talk given by Jérôme Petazzoni at DockerCon EU.

Let’s initialize our project by using the latest Node.js image. The following command runs an interactive bash terminal, which lets us access the container with Node.js installed in it, and binds a current directory of the host to the /app directory in the container, which lets us persist our files (e.g. package.json, etc.).

docker run -it -v $(pwd):/app node /bin/bash

You will immediately notice that your terminal’s hostname has changed to something like root@b95028b5a79c:/#. Congrats! Now, you are in a container with Node.js present inside! How cool is that, huh? :)

Now, in order to access our files and initialize the project, open the /app folder and run npm init:

cd /app     # opens app folder
npm init    # initialize the Node.js project (creates a package.json)

You can also run ls command to see what’s in the current directory. You will see that you have everything there from the current directory of the host.

Now run exit to exit from the container. Since you have attached a volume using a -v option when running a container, your package.json could be seen in your current directory in the host.

Congratulations! So far, you have managed to initialize a Node.js project using Docker (images and running containers) without actually installing Node.js in your computer. It is quite lovely, isn’t it?

Dockerfile and docker-compose

If your app uses MongoDB as a database service, you already have 2 services interacting (i.e. API <-> DB). If you want to dockerize your multi-service application, then you have to define 2 images for those services using Dockerfiles (one per service).

Docker-compose is another Docker tool which lets us manage multi-container applications. With Docker-compose, it is simpler to manage and scale your services. Docker-compose works almost the same way as the docker command; instead of providing a Dockerfile, you can configure your services by creating the docker-compose.yml file. Though in a bit different way, you can configure the same way as in a Dockerfile: mounting volumes, running commands, pulling images, etc.

For instance, take a look at the image above. We can define a single Dockerfile for the API, and define the configs for Mongo just inside the Compose file. Why don’t we create Dockerfiles for it? Indeed, you could do so and configure your DB (e.g. create users, roles, priviliges, etc. upon initialization). However, in this case, we don’t need to configure anything, we just need the MongoDB service running in the respective container.

NOTE: We would use Dockerfile for the Node.js API service because we have to build the app first, i.e. install its dependencies, transpile (if we use Typescript, etc.). This is how Dockerfile for the Node.js would look like:

# use Node.js version latest
FROM node

# create app folder in the container (not the host)
RUN mkdir -p /app

# sets the working directory inside the container (where RUN/CMD commands will be executed)
WORKDIR /app

# copies package.json from the current directory into the /app folder inside the container
COPY package.json /app

# runs "npm install" command inside the container
RUN ["npm", "install"]

# copy the node_modules and the rest of the files into /app
COPY . /app

So, what have we done? We wrote a sequence of instructions which defines your image. We pulled Node (version 9) from the Docker Hub, created a directory /app inside the container, we “told” Docker to work with the /app directory, and copied everything (i.e. package.json, src folders, readme, etc.) from the current folder of the host machine into the /app folder in the container. And, ultimately, we ran the npm install command inside the container, so we get the dependencies from the package.json installed.

Now, let’s create the docker-compose.yml file where we will define set the configurations (env variables, volumes, networks, etc.) for the services in our architecture. Here is an example of how to define these:

version: "3"          # use version Compose version 3
services:             # our services
  api:
    build: .          # use Dockerfile from current directory at build time
    volumes:          # volumes are there to let us persist data when containers are exited
      - .:/app        # bind a current directory of the host to the /app directory in the container
    depends_on:
      - database      # not started until "example-mongo" service is started
    networks:         # let's us be discoverable and reachable by other services in the same network
      - api-net       # join "api-net" network
    ports:
      - 3000:3000     # bind port 3000 on host to port 3000 on container
    command: npm run start  # execute the following command when the image is running, e.g. run the Node server

  database:
    image: mongo      # if tag is not specified, gets latest image (e.g. MongoDB image)
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin      # set the root username to "admin"
      - MONGO_INITDB_ROOT_PASSWORD=admin123   # set the root password to "admin123"
    volumes:
      - ./data:/data/db   # persist data from mongodb
    networks:
      - api-net       # makes "database" reachable (via hostname) by other services in the same network
    ports:
      - "27017:27017" # bind port

Note that you could leverage links as well, though I personally love networks because of their simplicity: you can reach other services in the same network just by using their service name (in our case, they are api or database). One could also define working directory inside the compose file (by using working_dir field) instead of specifying it in the Dockerfile.

For more details on Docker Compose, I would recommend the following Compose file Reference

Typescript and Node.js

Typescript is a programming language that brings us an optional static type-checking and latest ECMAScript features. By using Typescript, you can leverage the power of OOP, i.e. interfaces, classes, inheritance, polymorphism etc. I would personally recommend to everyone, especially to those who come from the Java/C# side and are just starting out with Javascript. .ts files are compiled to .js files, meaning that Typescript is compiled to Javascript. So, in the end, you end up with Javascript anyway :)

In order to get started with Typescript, we have to install typescript module via npm. We can do this by running our Node container again. Do not forget to attach a volume so the change in package.json is actually saved on our host:

docker run -it -v $(pwd):/app node /bin/bash    # access the Node container
cd /app                                         # get into /app folder
npm install typescript --save-dev               # install and save as development dependency

At the same time, let’s install express and the type for it to run the Express server! Note that I will not be using MongoDB in this case (though you should experiment and try yourself!).

npm install express --save
npm install @types/express --save-dev
exit        # exit the container

As there is already some boilerplate code defined in src/server.ts file, let’s change the package.json so that we first compile the code from TS to JS, and then run it! Make sure you are out of the container.

"build": "tsc",   /* Transpile to JS  */
"start": "npm run build && node ./dist/server.js"  /* Build and run the server */

We also need to create a tsconfig.json file that configures the Typescript compiler. More details on TS compilation configurations, check this link out. Here is our example:

{
    "compileOnSave": true,
    "compilerOptions": {
        "outDir": "dist",
        "module": "commonjs",
        "target": "es6",
        "noImplicitAny": true,
        "moduleResolution": "node",
        "sourceMap": true,
        "baseUrl": ".",
        "skipLibCheck": true,
        "paths": {
            "*": [
                "node_modules/*",
                "/src/types/*"
            ]
        }
    },
    // includes all typescript files
    "include": [
        "./src/**/*.ts",
    ],
    // excludes the folder containing the compiled JS files 
    "exclude": [ "./dist"]
}

Now, let’s create a docker-compose.yml file with which we will run a Node.js server. Here, we will not use a Dockerfile; we will instead configure the API service straight in the Compose yaml file.

version: "3"          # use version Compose version 3
services:             # our services 
  api:
    image: node       # use latest node image
    working_dir: /app # set the working directory to /app
    volumes:          # volumes are there to let us persist data when containers are exited
      - .:/app        # bind a current directory of the host to the /app directory in the container
    ports:
      - 3000:3000     # bind port 3000 on host to port 3000 on container
    command: "npm run start"

Before running the container, we have to install our dependencies. You may not have Node.js in your computer, or you may have a different version, so let’s run a Node container (of latest version) and install our dependencies (do not forget to attach a volume):

docker run -it -v $(pwd):/app node /bin/bash
# we are inside the container
npm install
exit 

To run the actual service, type the following command (the -f option specifies the file to run) to run the server:

docker-compose -f docker-compose.yml up

Note if you are getting ERROR: If you are getting the ERROR: Error processing tar file(exit status 1): unexpected EOF, run the following commands:

cd ..       # get out to the parent directory 
sudo chown -R $(whoami) nodejs-debugging/  # this gives you and Docker the rights to the nodejs-debugging folder

If you see Listening on port 3000! message, yay! Open a browser and type localhost:3000, and you will see the server is up. You can also try opening the routes, i.e. /greet and /time.

Debugging with Visual Studio Code

Visual Studio Code (VS Code) has a built-in debugging support for Node.js runtime and can debug any languages that are transpiled to JavaScript.

Since the VS Code Node.js debugger communicates to the Node.js runtimes through wire protocols, the set of supported runtimes is determined by all runtimes supporting the wire protocols:

• legacy: the original V8 Debugger Protocol which is currently supported by older runtimes.
• inspector: the new V8 Inspector Protocol is exposed via the --inspect flag in Node.js versions >= 6.3. It addresses most of the limitations and scalability issues of the legacy protocol.

As we are running a server from a Docker container, we have to attach a remote debugger. We need to add a launch configuration to the .vscode folder, i.e. launch.json. Here is an example of the launch configuration file:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "attach",
            "name": "Remote Debugging",
            "address": "0.0.0.0",
            "port": 9229,
            "localRoot": "${workspaceFolder}/dist",
            "remoteRoot": "/app/dist",
            "protocol":"inspector"
        }
    ]
}

We set a remote root to be path in the container, where our program lives.

Now, in order to add remote debugging, we have to add another script to package.json:

"debug": "npm run build && node --inspect-brk=0.0.0.0:9229 ./dist/server.js"

The debug script will build (i.e. transpile the TS code) project and will start Node runtime in debugging mode accessible remotely on port 9229 (remember the port we specified above?).

Let’s create another Compose file, which we will use for running the server in the debug mode:

version: "3"          # use version Compose version 3
services:             # our services 
  api:
    image: node       # use latest node image
    working_dir: /app # set the working directory to /app
    volumes:          # volumes are there to let us persist data when containers are exited
      - .:/app        # bind a current directory of the host to the /app directory in the container
    ports:
      - 3000:3000     # bind port 3000 on host to port 3000 on container
      - 9229:9229     # bind port 9229 for debugging
    command: "npm run debug"

The only differences are that we are now running in debug mode and we attached extra port for debugging (9229).

Type the following command in the terminal so you can run the debug server:

docker-compose -f docker-compose.debug.yml up

And you will see the following message

api_1 | Debugger listening on ws://0.0.0.0:9229/f3cdf4f2-4685-21e6-8c31

Yay! Now, as the debugger is listening on 0.0.0.0:9229, we have to start debugging via VS Code. If you click CTRL + SHIFT + D keys, the “debug” mode will open. You will see the “Remote Debugging” slider upper-left corner and the button (looks like green triangle). E.g.:

Be courageous and click on the green triangle button. Congratulations! You have just started debugging your app using VS Code and Docker containers!

Bonus round! Simplifying debugging further …

Launching container manually, and proceeding to Debug view on VS Code may be a bit of a daunting and annoying task. However, we can automate that and make our lives a bit easier: let’s make it so that we can start debugging by just clicking F5 key! :squirrel:

Let’s first create the tasks file inside the .vscode directory, where our configurations for the VS Code reside. Copy-paste the following into the tasks.json:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "launch-debug-container",
            "command": "docker-compose -f docker-compose.debug.yml up",
            "type": "shell",
            "group": "build",
            "isBackground": true
        }
    ]
}

Tasks help you automate building and testing your app. Whenever you start debugging, they may help you build your app first. There may be many tasks per launch configuration, so you can automate anything that annoys you and the steps you constantly repeat whenever you start debugging.

In the task above, we label it as “launch-debug-container” and make it execute a command to start the containers specified in the docker-compose.debug.yml file.

Now, how do we perform a task when we actually launch our debugging? We have to adjust launch.json by adding another field in our “Remote Debugging” configuration:

{
  /* ... */
  "preLaunchTask": "launch-debug-container"
  /* ... */
}

By giving a label from tasks.json to the preLaunchTask property, our task will be executed first before launching our debugger. Note that I also change timeout to 60 seconds (default is 10 sec), as the containers docker-compose.debug.yml take some time to start.

In addition, we want our containers to be stopped and removed after the debugging. If you don’t want to do so, then you are done! Otherwise, if you don’t like the containers still running after you have finished debugging, let’s add another task that will execute a command to stop and remove containers after the debugging session.

Add the following to the list of tasks in your tasks.json:

{
    "label": "end-debug-container",
    "command": "docker-compose -f docker-compose.debug.yml down",
    "type": "shell",
    "group": "build",
    "isBackground": true
}

This will turn the containers down whenever you finish the debugging process. Let’s also adjust the launch configuration in the launch.json by adding a postDebugTask property:

{
  /* ... */
  "postDebugTask": "end-debug-container"
  /* ... */
}

Now is the moment … Just click F5 and your debugging starts automagically. If you exit debugging, you will see the containers terminating. Good job!

Note: if you are getting The specified task cannot be tracked error, click the Debug anyway button and your debugger will start.

Thank you + Feedback

Thank you and good job for reading the entire tutorial! It’s been a long markdown to read; anyway, you have just learned something new which will help you a lot debugging your Node.js apps and become a better developer!

Constructive feedback is always welcome via issues on this repo.

PS. If something does not work, or if you have any problems, please open an issue in this repo and I will do my best to help you asap.

References

• Node.js Debugging in VS Code
• Debugging TypeScript in a Docker Container
• Docker - Compose