The maximum size of each push should be smaller than 10MB. If you try to push one or more commits which size is bigger then you are receiving a remote: fatal: pack exceeds maximum allowed size. Usually it is one big file that you have committed and after that removed from your filesystem. The problem is that the commit containing the file will be pushed anyhow, causing the problem.
Prevent the problem
The most important recommendation is to check the size of your files BEFORE committing. You MUST check Neil's recommendations and use the ducommand to check the size before commiting. Note that the du can be executed only from a Linux terminal. You can use for instance your GIT Bash.
Once you have added files to your stage area BUT BEFORE you commit them you can check the size of all the files that are ready to be committed:
git diff --staged --name-only | xargs du -sh
If you want to see all the files in your stage area ordered by size (remember this command must be executed after the add but before the commit):
git diff --staged --name-only | xargs du -s | sort -rn
The numbers on the left are KB.
IMPORTANT NOTE: Previous commands only work if your directory and file names do not have spaces. Otherwise, it wont work. In this case go to:
git diff --staged --name-only | xargs -d '\n' du -sh
or
git diff --staged --name-only | xargs -d '\n' du -s | sort -rn
In this case you need to remove the file from the stage area:
git restore --staged <offending file>
and then remove the file
git rm <offending file>
Other option is that you run the command:
du -sh * | sort -rn
And check the size of ALL the files in your repo (not only the ones that you have just uploaded)
Steps to solve the problem
If you managed to have the problem there are many few ways to solve this issue. One student from Waag (I cannot find it now) showed a good tutorial on how he solved the problem. As soon as I find it I will link it.
I recommend the following 4 step process:
- Finding the commit with the offending file
- Finding the offending file
- Redo history, removing the file using
rebase -
pushthe commits.
Before doing this, sometimes the command
git gc
solves all issues. This command calls the garbage collector in git to remove unnecessary files.
Finding the commit with the offending file/files
Due to GIT architecture it is not straight forward to find the size of a GIT commit. This stack-overflow thread explains a bit more on how to do this. This other provides a much complex perl script.
What our students have done is trying to push each one of the commits separately one after another. In order to know which was the last commit that was pushed and the hash of the different commits you need to run the command:
git log --oneline -10 This command something like this (thanks @tatiana.avsievich for the picture).
In this case the last commit that has been pushed is the 5542576. You can see that the (origin/master is pointing there).
Now we can push each subsequent commit one after another:
git push origin <Commit_hash>:master
where <Commit_hash> is the commit id that you want to push.
For instance in the picture above we should do in order:
git push origin ec24cc8:mastergit push origin 367e267:mastergit push origin 5679a8:mastergit push origin adfe98f:master
The first commit that fails is the one that is having the offending file/s. Now we know which is the commit that we have to change.
Finding the offending file/s
Now we need to find which are the file/s that give us the headache. I have been playing with 3 different options in order to find which is the offending file/s
Option 1
This is the best option, if the files are still unmodified in your working space. We are going to check the files that has been modified in the commit that we found in the previuos step:
Go to the root of your repo and type:
git diff --name-only <Commit SHA> <Commit SHA>~1 | xargs du -sh
or to show them ordered:
git diff --name-only <Commit SHA> <Commit SHA>~1 | xargs du -s | sort -rn
Note the numbers on the left are KB.
**IMPORTANT NOTE: ** If you have spaces in your file names or directory names you must escape the xargs using '\n´ as delimiter. The command woudl look like:
git diff --name-only <Commit SHA> <Commit SHA>~1 | xargs -d '\n' du -sh
and
git diff --name-only <Commit SHA> <Commit SHA>~1 | xargs -d '\n' du -s | sort -rn
Option 2
You can use this one if the file has been modified in later commits.
You first move to detached HEAD to the commit you think is the offending file. Then the workspace will change all files to the state the files were in that commit.
git checkout <Commit SHA>
After that you can use the git diff to find the modified files from the previous commit as well as its size (using the du command). Go to the root of your repo and type:
git diff --name-only HEAD~1 | xargs du -sh
Or if you want them sorted:
git diff --name-only HEAD~1 | xargs du -s | sort -rn
Note that the numbers on the left are KB.
Important NOTE: This will only work if you do not have spaces in your folder/file names. Otherwise you must use the new line character as xargs delimiter. The commands would then look like:
git diff --name-only HEAD~1 | xargs -d '\n' du -sh
and
git diff --name-only HEAD~1 | xargs -d '\n' du -s | sort -rn
Finally, do not forget to move out from detached HEAD
git checkout master
Option 3
If the two previuos option do not work, we can use git ls-tree that will provide the sizes of the different tree objects.
Go to the root of your repo and type:
git ls-tree -lr <Commit SHA>
where is the commit which contains the offending file.
This command will return a screen similar to this one:
As you see the 4th column contains the size of the blob (in this case files) and the 5th column the path to the file. You must find in the list the bigger files, so you can remove them. If you have a lot of files, this might be crazy, but then sort comes to the rescue. Type the following command:
git ls-tree -r --long <Commith SHA> | sort --key 4 --numeric-sort
Now you are going to see the bigger files at the bottom of your screen. Very handy indeed.
The problem is that the ls-tree command return the size of all the files in your repo in certain commit, but you do not know if the file was included in this commit or in a previous one. If you want to list all the files that has been modified in a commit use:
git show --name-only {commit}
or equivalent
`git diff --name-only {commit} {commit}~1
Now, you can have an idea on which is the offending file/s. It is time to remove them while keeping the rest of the files. To that end, we need to modify the history.
Redo history, removing the file using interactive rebase
The rebase interactive command allows to modify our history, by changing files in the different commits. This command is very powerful, but you need to be careful when you use it. You can use this command if you have NOT push the commits to the remote server. Otherwise, different people will see different histories, and this will cause lots of problems.
What we are going to do with the rebase is:
- Mark the commit as editable
- Edit the commit by removing the files
- Save the new version of the commit (without the offending files)
- Continue rebuilding the history.
Let's see the commands:
-
git rebase -i <Commit SHA>~1Here we are just saying that we want to edit commits starting from the parent of <Commit SHA)
This will open your configured text editor with something similar to this:
You just need to change the pick word by the edit word in all those commits that you want to change. In our case only one:
After that save and close your text editor.
NOTE: If you have not configured a default text editor quite likely you are using vim. There you need to modify the file (change pick to delete), then press ESC, then type :wq and finally press enter. This is the equivalent of saving and closing in a text editor.
In your bash or command line you will see something like this:
Now we can edit the commit. In our case we are just going to remove the offending file/s
git rm <offending file>
We can check that the file/s has been removed using git status. Now we need to save the modified version of the commit:
git commit --amend
And continue the rebasing.
git rebase --continue
The rebase will stop for each commit marked as editable. When rebasing is finished, you will see a success message: Successfully rebased and updated refs/heads/master_
Our local repository is fixed. Now we need to synchronize with our remote repository.
push the commits.
No secrets here:
git push origin master
If you still have some problems, it might be that you have another commit with issues. Then, my recommendation is that you start again pushing commit by commit until you find the problematic commit.