Troubleshooting Server Installations

To download a support bundle, select Support from the Management Console menu bar, and then select Download Support Bundle. CircleCI support engineers will often request support bundles to help diagnose/fix the problem you are experiencing.

If your Services component is fine, but builds are not running, or all builds are queueing, follow the steps below.

Run sudo docker logs dispatcher, if you see log output that is free of errors you may continue on the next step.

If the logs dispatcher container does not exist or is down, start it by running the sudo docker start <container_name> command and monitor the progress. The following output indicates that the logs dispatcher is up and running correctly:

If you see errors or do not see the above output, investigate the stack traces because they indicate that there is an issue with routing builds from 1.0 to 2.0. If there are errors in the output, then you may have a problem with routing builds to 1.0 or 2.0 builds.

If you can run 1.0 builds, but not 2.0 builds, or if you can only run 2.0 builds and the log dispatcher is up and running, continue on to the next steps.

Run the sudo docker logs picard-dispatcher command. A healthy picard-dispatcher should output the following:

The output should be filled with the above messages. If it is a slow day and builds are not happening very often, the output will appear as follows:

As soon as you run a build, you should see the above message to indicate that it has been dispatched to the scheduler. If you do not see the above output or you have a stack trace in the picard-dispatcher container, contact support@circleci.com.

If you run a 2.0 build and do not see a message in the picard-dispatcher log output, it often indicates that a job is getting lost between the dispatcher and the picard dispatcher.

Stop and restart the CircleCI app in the Management Console at port 8800 to re-establish the connection between the two containers.

Run sudo docker logs picard-scheduler . The picard-scheduler schedules jobs and sends them to nomad through a direct connection. It does not actually handle queuing of the jobs in CircleCI.

Check to see if there are any nomad nodes by running the nomad node-status -allocs command and viewing the following output:

If you do not see any nomad clients listed, please consult our <nomad#,Introduction to Nomad Cluster Operation> for more detailed information on managing and troubleshooting the nomad server.

If you see output like the above, but your builds are still queued, then continue to the next step.

Run the sudo docker exec -it nomad nomad status command to view the jobs that are currently being processed. It should list the status of each job as well as the ID of the job, as follows:

After a job has completed, the Status shows dead. This is a regular state for jobs. If the status shows running, the job is currently running. This should appear in the CircleCI app builds dashboard. If it is not appearing in the app, there may be a problem with the output-processor. Run the docker logs picard-output-processor command and check the logs for any obvious stack traces.

If the nomad client logs contain the following error message typw, check port 8585:

If a restore_cache step is failing for one of your jobs, it is worth checking the size of the cache - you can view the cache size from the CircleCI Jobs page within the restore_cache step. We recommend keeping cache sizes under 500MB – this is our upper limit for corruption checks because above this limit check times would be excessively long. Larger cache sizes are allowed but may cause problems due to a higher chance of decompression issues and corruption during download. To keep cache sizes down, consider splitting into multiple distinct caches.

Disable cache warming by completing the following steps: