BitBucket Server Unreachable When Merge Pull Request

We recently had an BitBucket incident – one project team found that they are unable to merge the pull request after approval, and the server always returns an error message:

So we checked their project repository, and here is something that astonish us:

• The repository have a few hundreds of branches.
• The size is nearly 7G!

No matter if it is related to the above issue or not, as a common practise you should not keep so many branches. As branches should be short-lived regardless it is for feature or bugs, they should be deleted once merged back to the main stream. Also, the binary files should not be saved to repository directly. Use something like Git LFS to only save pointer to repository, and save binary blob to a separate place.

To troubleshoot we enabled the debug logging and profiler in BitBucket, here is what we found:

Caused by: com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
    at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:113)
    at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:201)
    at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:31)
    at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:153)
    at com.atlassian.bitbucket.scm.BaseCommand$CommandFuture.internalGet(BaseCommand.java:287)
    at com.atlassian.bitbucket.scm.BaseCommand$CommandFuture.get(BaseCommand.java:251)
    at com.atlassian.bitbucket.scm.BaseCommand.call(BaseCommand.java:87)
    at com.atlassian.stash.internal.scm.git.porcelain.AbstractTimedRequest.configureAndCall(AbstractTimedRequest.java:42)
    at com.atlassian.stash.internal.scm.git.merge.DefaultMergeStrategy.merge(DefaultMergeStrategy.java:57)
    at com.atlassian.stash.internal.scm.git.merge.TimedMergeStrategy.merge(TimedMergeStrategy.java:37)
    at com.atlassian.stash.internal.scm.git.merge.MergeCommand.doMerge(MergeCommand.java:139)
    at com.atlassian.stash.internal.scm.git.merge.MergeCommand.doWithWorkTree(MergeCommand.java:144)

Basically the log tells us that ‘An error occurred while executing an external process: process timed out’ and the external process is ‘git merge’. Most likely it is caused by the large repository size. The best solution is to reduce the repository size, but as a workaround/ quick fix, we increased the value of pullrequest.merge.timeout to a larger value (it is 300 seconds by default). This has temporarily fixed it, but we all know the issue will come back if repository size keeps growing.

As a side note, you should always check the ‘Supported Platforms‘ against the BitBucket version to ensure it can perform well. One quick example is that Git 2.2.x – 2.4.0 have some performance issues when interacting with NFS. These versions cannot be used for Bitbucket Data Center or for Bitbucket Server installations that use NFS mounts for the home directory. The documentation also lists the versions of Git that have been specifically tested against the particular releases.

References:
https://confluence.atlassian.com/bitbucketserverkb/server-unreachable-timeout-when-merging-or-viewing-diff-of-pull-request-779171259.html
https://confluence.atlassian.com/bitbucketserver/bitbucket-server-config-properties-776640155.html
https://confluence.atlassian.com/bitbucketserver/supported-platforms-776640981.html
https://www.perforce.com/blog/storing-large-binary-files-in-git-repositories
https://jira.atlassian.com/browse/BSERV-7360