Convert a Git folder to a submodule retrospectively?

Quite often it is the case that you're writing a project of some kind, and after a while it becomes clear that some component of the project is actually useful as a standalone component (a library, perhaps). If you've had that idea from early on, then there's a fair chance that most of that code is in its own folder.

Is there a way to convert one of the sub directories in a Git project to a submodule?

Ideally this would happen such that all of the code in that directory is removed from the parent project, and the submodule project is added in its place, with all the appropriate history, and such that all the parent project commits point to the correct submodule commits.

Solution 1:

To isolate a subdirectory into its own repository, use filter-branch on a clone of the original repository:

git clone <your_project> <your_submodule>
cd <your_submodule>
git filter-branch --subdirectory-filter 'path/to/your/submodule' --prune-empty -- --all

It's then nothing more than deleting your original directory and adding the submodule to your parent project.

Solution 2:

First change dir to folder which will be a submodule. Then:

git init
git remote add origin <repourl>
git add .
git commit -am 'first commit in submodule'
git push -u origin master
cd ..
rm -rf <folder> # the folder which will be a submodule
git commit -am 'deleting folder'
git submodule add <repourl> <folder> # add the submodule
git commit -am 'adding submodule'

Solution 3:

I know this is an old thread, but the answers here squash any related commits in other branches.

A simple way to clone and keep all those extra branches and commits:

1 - Make sure you have this git alias

git config --global alias.clone-branches '! git branch -a | sed -n "/\/HEAD /d; /\/master$/d; /remotes/p;" | xargs -L1 git checkout -t'

2 - Clone the remote, pull all branches, change the remote, filter your directory, push

git clone [email protected]:user/existing-repo.git new-repo
cd new-repo
git clone-branches
git remote rm origin
git remote add origin [email protected]:user/new-repo.git
git remote -v
git filter-branch --subdirectory-filter my_directory/ -- --all
git push --all
git push --tags

Solution 4:

Status quo

Let's assume we have a repository called repo-old which contains a subdirectory sub that we would like to convert into a submodule with its own repo repo-sub.

It is further intended that the original repo repo-old should be converted into a modified repo repo-new where all commits touching the previously existing subdirectory sub shall now point to the corresponding commits of our extracted submodule repo repo-sub.

Let's change

It is possible to achieve this with the help of git filter-branch in a two step process:

Subdirectory extraction from repo-old to repo-sub (already mentioned in the accepted answer)
Subdirectory replacement from repo-old to repo-new (with proper commit mapping)

Remark: I know that this question is old and it has already been mentioned that git filter-branch is kind of deprecated and might be dangerous. But on the other hand it might help others with personal repositories that are easy to validate after conversion. So be warned! And please let me know if there is any other tool that does the same thing without being deprecated and is safe to use!

I'll explain how I realized both steps on linux with git version 2.26.2 below. Older versions might work to some extend but that needs to be tested.

For the sake of simplicity I will restrict myself to the case where there is just a master branch and a origin remote in the original repo repo-old. Also be warned that I rely on temporary git tags with the prefix temp_ which are going to be removed in the process. So if there are already tags named similarily you might want to adjust the prefix below. And finally please be aware that I have not extensively tested this and there might be corner cases where the recipe fails. So please backup everything before proceeding!

The following bash snippets can be concatenated into one big script which should then be executed in the same folder where the repo repo-org lives. It is not recommended to copy and paste everything directly into a command window (even though I have tested this successfully)!

0. Preparation

Variables

# Root directory where repo-org lives
# and a temporary location for git filter-branch
root="$PWD"
temp='/dev/shm/tmp'

# The old repository and the subdirectory we'd like to extract
repo_old="$root/repo-old"
repo_old_directory='sub'

# The new submodule repository, its url
# and a hash map folder which will be populated
# and later used in the filter script below
repo_sub="$root/repo-sub"
repo_sub_url='https://github.com/somewhere/repo-sub.git'
repo_sub_hashmap="$root/repo-sub.map"

# The new modified repository, its url
# and a filter script which is created as heredoc below
repo_new="$root/repo-new"
repo_new_url='https://github.com/somewhere/repo-new.git'
repo_new_filter="$root/repo-new.sh"

Filter script

# The index filter script which converts our subdirectory into a submodule
cat << EOF > "$repo_new_filter"
#!/bin/bash

# Submodule hash map function
sub ()
{
    local old_commit=\$(git rev-list -1 \$1 -- '$repo_old_directory')

    if [ ! -z "\$old_commit" ]
    then
        echo \$(cat "$repo_sub_hashmap/\$old_commit")
    fi
}

# Submodule config
SUB_COMMIT=\$(sub \$GIT_COMMIT)
SUB_DIR='$repo_old_directory'
SUB_URL='$repo_sub_url'

# Submodule replacement
if [ ! -z "\$SUB_COMMIT" ]
then
    touch '.gitmodules'
    git config --file='.gitmodules' "submodule.\$SUB_DIR.path" "\$SUB_DIR"
    git config --file='.gitmodules' "submodule.\$SUB_DIR.url" "\$SUB_URL"
    git config --file='.gitmodules' "submodule.\$SUB_DIR.branch" 'master'
    git add '.gitmodules'

    git rm --cached -qrf "\$SUB_DIR"
    git update-index --add --cacheinfo 160000 \$SUB_COMMIT "\$SUB_DIR"
fi
EOF
chmod +x "$repo_new_filter"

1. Subdirectory extraction

cd "$root"

# Create a new clone for our new submodule repo
git clone "$repo_old" "$repo_sub"

# Enter the new submodule repo
cd "$repo_sub"

# Remove the old origin remote
git remote remove origin

# Loop over all commits and create temporary tags
for commit in $(git rev-list --all)
do
    git tag "temp_$commit" $commit
done

# Extract the subdirectory and slice commits
mkdir -p "$temp"
git filter-branch --subdirectory-filter "$repo_old_directory" \
                  --tag-name-filter 'cat' \
                  --prune-empty --force -d "$temp" -- --all

# Populate hash map folder from our previously created tag names
mkdir -p "$repo_sub_hashmap"
for tag in $(git tag | grep "^temp_")
do
    old_commit=${tag#'temp_'}
    sub_commit=$(git rev-list -1 $tag)

    echo $sub_commit > "$repo_sub_hashmap/$old_commit"
done
git tag | grep "^temp_" | xargs -d '\n' git tag -d 2>&1 > /dev/null

# Add the new url for this repository (and e.g. push)
git remote add origin "$repo_sub_url"
# git push -u origin master

2. Subdirectory replacement

cd "$root"

# Create a clone for our modified repo
git clone "$repo_old" "$repo_new"

# Enter the new modified repo
cd "$repo_new"

# Remove the old origin remote
git remote remove origin

# Replace the subdirectory and map all sliced submodule commits using
# the filter script from above
mkdir -p "$temp"
git filter-branch --index-filter "$repo_new_filter" \
                  --tag-name-filter 'cat' --force -d "$temp" -- --all

# Add the new url for this repository (and e.g. push)
git remote add origin "$repo_new_url"
# git push -u origin master

# Cleanup (commented for safety reasons)
# rm -rf "$repo_sub_hashmap"
# rm -f "$repo_new_filter"

Remark: If the newly created repo repo-new hangs during git submodule update --init then try to re-clone the repository recursively once instead:

cd "$root"

# Clone the new modified repo recursively
git clone --recursive "$repo_new" "$repo_new-tmp"

# Now use the newly cloned one
mv "$repo_new" "$repo_new-bak"
mv "$repo_new-tmp" "$repo_new"

# Cleanup (commented for safety reasons)
# rm -rf "$repo_new-bak"

Solution 5:

It can be done, but it's not simple. If you search for git filter-branch, subdirectory and submodule, there are some decent write-ups on the process. It essentially entails creating two clones of your project, using git filter-branch to remove everything except the one subdirectory in one, and removing only that subdirectory in the other. Then you can establish the second repository as a submodule of the first.