Merging Git Repositories with Bash

I’ve been participating in TidyTuesday since 2018. TidyTuesday is a weekly data project in R from the R4DS community. A dataset is provided and allows you to practice data manipulation, visualization, and story telling. It’s traditional to share code so that others may learn from your contribution and it’s also traditional to share that code via Git.

Since the beginning, I’ve been creating a new repository for each week that I participated. This has added up to quite a few repos and has become cumbersome to look through. Inspired by some other TidyTuesday repos, I decided to attempt merging the Git repos into one master repo so I could easily display the final visualizations in one README.

While researching this, I found numerous ways to merge repos on blogs and stack overflow. Many of which were fairly complex as maintaining history is important for most developers. It wasn’t for me so I settled on the subtree method as it was the simplest and helped me accomplish what I had in mind.

I used the method outlined here: How to use the subtree merge strategy. This page outlines 5 git commands to merge a repo into a subdirectory of a master repo. This got repetitive fast so I wrote a simple bash script to accomplish this much faster. I can confirm this works well on Linux Mint 19.3.

#!/bin/bash
# git-merge.sh -- git merge script
# 1 - local name (alias)
# 2 - git project name
# 3 - year
# 4 - month-day


echo "
* * * git add remote * * *
"
# NOTE: git@gitlab.com:tidy_tuesday/*.git is the path to my repositories. You should 
# change this if you cconsider using it. Also note that the only variation is project name ($2)
git remote add -f $1 git@gitlab.com:tidy_tuesday/$2.git

echo "
* * * git merge * * *
"
git merge -s ours --no-commit --allow-unrelated-histories $1/master

echo "
* * * git read-tree * * *
"
git read-tree --prefix=$3/$4 -u $1/master

echo "
* * * git commit * * *
"
git commit -m "merge project into subfolder"

echo "
* * * git pull * * *
"
git pull -s subtree $1 master

echo "
* * * git push * * *
"
git push

echo "
* * * merge complete * * *
"

It’s very simple with no catches so mistakes would be propogated throughout, but I was pretty careful (overall). If I were more consistent with naming the repos, I would have preferred to wrap this in a loop. Still, this script helped remove a bit of human error thus making the process smoother and faster.

Let’s take a quick look at the arguments.

  1. The first argument calls for the local alias for the remote git to add – this can be a local folder as well.
  2. The second argument refers to the name of the git project.

I planned to organize the new repo by year and date.

  1. So years make up the first subfolders and the third argument.
  2. Month and day (eg, “12-25”) make up the project subfolder and the fourth argument.

For example:
      git-merge.sh localName remoteProject 2020 02-22

Assuming that you run this script from the base directory of the new repository, this will result in the remote repo merging into folder 2020/02-22/.

It was fairly straight forward in the end, but took a lot of trial and error to get there. And the final result is my new TidyTuesday repo. Check it out here: TidyTuesday.