Github remote push pack size exceeded

I am new to Git and have a fairly large project that I want to push to a remote repo (Repo B) on Github. The original project was on Github as well but from a different repo (Repo A). I have to make some changes to files from Repo A before I can setup the project up on Repo B. I have setup the remotes, ssh keys etc. and I run into an issue when pushing the codebase to Repo B.

I get the following error all the time:

$ git push <remote_repo_name> master
Enter passphrase for key '/c/ssh/.ssh/id_rsa':
Counting objects: 146106, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (35519/35519), done.
fatal: pack exceeds maximum allowed size00 GiB | 154 KiB/s
fatal: sha1 file '<stdout>' write error: Invalid arguments
error: failed to push some refs to '[email protected]:<repo>.git

I changed the following settings in my local gitconfig

git config pack.packSizeLimit 1g
git config pack.windowMemory 1g

... and ran git gc (which I see reorganized the packs so that each pack stayed within the packsize of 1GB). This did not work and I get the error seen above.

I tried to lower the size of each pack as well ....

git config pack.packSizeLimit 500m
git config pack.windowMemory 500m

... and ran git gc (which I see reorganized the packs so that each pack stayed within the packsize of 500MB). This did not work either and I ran into the same error.

I am not sure of what Github's default packsize limits are (if any). The account is a micro account if that matters.

Solution 1:

The packsize limit does not affect git protocol commands (your push).

From git-config under pack.packSizeLimit:

The maximum size of a pack. This setting only affects packing to a file when repacking, i.e. the git:// protocol is unaffected.

When executing a push git will always create exactly one pack no matter the size!

To fix this use two (or more) pushes:

git push remoteB <some previous commit on master>:master
...
git push remoteB <some previous commit after the last one>:master
git push remoteB master

These pushes will all have smaller packs and will succeed.

Solution 2:

As onionjake noted in his answer, the pack.packSizeLimit setting does not affect pushes. As he suggested, this can often be addressed by using multiple pushes with fewer commits each. rurban posted a comment on how to automatically push chunks of 500 commits. Following is a modified version of his comment, generalized to work correctly regardless of whether the branch on the remote does not exist or exists and contains some of the commits already. I also added the --first-parent argument to the git log calls to prevent errors when the repository contains multiple root commits. I also made some tweaks to improve effiency, and added an additional call to git push to push the final (partial) batch of commits:

# Adjust the following variables as necessary
REMOTE=origin
BRANCH=$(git rev-parse --abbrev-ref HEAD)
BATCH_SIZE=500

# check if the branch exists on the remote
if git show-ref --quiet --verify refs/remotes/$REMOTE/$BRANCH; then
    # if so, only push the commits that are not on the remote already
    range=$REMOTE/$BRANCH..HEAD
else
    # else push all the commits
    range=HEAD
fi
# count the number of commits to push
n=$(git log --first-parent --format=format:x $range | wc -l)

# push each batch
for i in $(seq $n -$BATCH_SIZE 1); do
    # get the hash of the commit to push
    h=$(git log --first-parent --reverse --format=format:%H --skip $i -n1)
    echo "Pushing $h..."
    git push $REMOTE ${h}:refs/heads/$BRANCH
done
# push the final partial batch
git push $REMOTE HEAD:refs/heads/$BRANCH

Solution 3:

Well, limit commit count (for example 500) in each push is helpful in most time. But it can not resolve the error caused by a single large commit.

If a single large commit exceeds the limit size of git server, it won't help by limiting commit count(even to 1).

To fix a single large commit:

1. if this commit contains multiple files, it can be resolved by create sub-commits and merge commits.
2. if it's a single large file, then there's no good solution.

To fix a single large commit with multiple files ( say file1, file2, ..., file10)

git checkout -b tmp SINGLE_LARGE_COMMIT^
git add file1 file2 file3 file4  # add a sub-class of files inside SINGLE_LARGE_COMMIT
git commit -m 'sub-commit'
git push origin tmp
git merge master  # or any other branch which contains SINGLE_LARGE_COMMIT
git push origin tmp
git checkout master
git push origin master # success