TeamCity Build Agent Issues: Complete Troubleshooting Guide

# TeamCity Build Agent Issues: Complete Troubleshooting Guide

TeamCity build agents are the workers that execute your builds. When agents have problems, builds queue indefinitely or fail mysteriously. Agent issues range from registration failures to connection problems and authorization errors.

Let me walk through every common TeamCity agent problem and how to fix each one.

Understanding Agent States

TeamCity agents show different states:

Fix 1: Agent Registration Failure

New agents don't appear in TeamCity or show "Unauthorized".

Symptoms: - Agent appears in "Unauthorized" agents list - Agent shows "Unregistered" status - Builds don't run on new agent

Diagnosis:

Check agent logs on the agent machine:

```bash # Find TeamCity agent logs cd /opt/TeamCityBuildAgent/logs cat teamcity-agent.log

# Look for lines like: # "Authorization token is not provided" # "Server URL is invalid" # "Failed to connect to TeamCity server" ```

Solution A: Authorize the agent:

1. 1.Go to TeamCity web UI
2. 2.Navigate to Agents → Unauthorized agents
3. 3.Click on your agent
4. 4.Click "Authorize"
5. 5.Optionally add a description and enable

Solution B: Fix agent configuration:

```bash # Edit agent configuration cd /opt/TeamCityBuildAgent/conf nano buildAgent.properties

# Check these settings: serverUrl=http://teamcity-server:8111 # Must be correct URL name=MyBuildAgent # Agent name

# Save and restart agent ./bin/agent.sh restart ```

Solution C: Use authorization token:

For security, TeamCity may require authorization tokens:

1. 1.Go to TeamCity server
2. 2.Administration → Agents → Agent Push
3. 3.Find authorization token
4. 4.Add to agent config:

authorizationToken=YOUR_TOKEN_HERE

Fix 2: Agent Disconnection Issues

Agent repeatedly disconnects from server.

Symptoms: - Agent shows "Disconnected" intermittently - Builds fail with "Agent disconnected" - Agent reappears after restart

Diagnosis:

```bash # Check agent logs tail -100 /opt/TeamCityBuildAgent/logs/teamcity-agent.log

# Check server connectivity curl -I http://teamcity-server:8111

# Check network ping teamcity-server netstat -an | grep 8111 ```

Solution A: Fix network connectivity:

```bash # Test from agent machine telnet teamcity-server 8111

# If connection refused, check firewall sudo firewall-cmd --list-all # RHEL/CentOS sudo ufw status # Ubuntu

# Open TeamCity port sudo firewall-cmd --add-port=8111/tcp --permanent sudo firewall-cmd --reload ```

Solution B: Increase heartbeat timeout:

# In buildAgent.properties
heartbeatTimeout=300  # Increase from default (120 seconds)

Solution C: Fix DNS issues:

```bash # Check DNS resolution nslookup teamcity-server

# If failing, add to hosts file echo "192.168.1.100 teamcity-server" >> /etc/hosts ```

Solution D: Use IP address instead:

# In buildAgent.properties
serverUrl=http://192.168.1.100:8111

Fix 3: Agent Not Picking Up Builds

Agent shows "Connected" and "Authorized" but builds queue.

Symptoms: - Builds stay in queue - Agent shows "Idle" when builds are queued - No builds assigned to agent

Diagnosis:

Check agent requirements vs capabilities:

1. 1.Go to Agents → [Your Agent]
2. 2.Click "Agent Details"
3. 3.Check "Compatible builds" section
4. 4.Look for "Incompatible builds" to see why

Solution A: Check build requirements:

Build configurations have requirements:

1. 1.Go to your build configuration
2. 2.Edit Build Configuration → Agent Requirements
3. 3.Check what requirements are defined

Common requirements: - teamcity.agent.name - env.JAVA_HOME - env.PATH contains specific tools - Custom properties

Solution B: Add agent capabilities:

On agent machine:

```bash # Edit buildAgent.properties nano /opt/TeamCityBuildAgent/conf/buildAgent.properties

# Add custom properties (system. or env.) system.Docker=true env.Python=/usr/bin/python3 env.NodeJS=/usr/bin/node

# Restart agent ./bin/agent.sh restart ```

Solution C: Enable the agent:

1. 1.Go to Agents → Authorized agents
2. 2.Find your agent
3. 3.Check if "Enabled" toggle is on
4. 4.Click "Enable" if needed

Solution D: Check agent pool:

1. 1.Go to Agents → Agent Pools
2. 2.Check which pool your agent is in
3. 3.Verify build configuration uses that pool

# Or in buildAgent.properties
agentPoolId=Default  # Pool ID

Fix 4: Agent Process Crashes

Agent process dies unexpectedly.

Symptoms: - Agent shows "Disconnected" suddenly - No agent process running - System logs show crashes

Diagnosis:

```bash # Check if agent process is running ps aux | grep teamcity-agent

# Check system logs for crashes dmesg | grep -i "killed process" journalctl -xe | grep -i teamcity

# Check for out of memory free -m cat /proc/meminfo ```

Solution A: Increase agent memory:

```bash # Edit agent startup script nano /opt/TeamCityBuildAgent/bin/agent.sh

# Add JVM options TEAMCITY_AGENT_OPTS="-Xmx1024m -Xms256m"

# Or create wrapper configuration nano /opt/TeamCityBuildAgent/bin/wrapper.conf wrapper.java.maxmemory=1024 ```

Solution B: Fix agent as service:

```bash # If running as systemd service systemctl status teamcity-agent

# Check service logs journalctl -u teamcity-agent -n 100

# Restart service systemctl restart teamcity-agent ```

Solution C: Monitor agent process:

# Create a cron job to restart crashed agent
# /etc/cron.d/teamcity-agent-monitor
*/5 * * * * root pgrep -f "teamcity-agent" || /opt/TeamCityBuildAgent/bin/agent.sh start

Fix 5: Build Fails on Specific Agent

Builds succeed on some agents but fail on one.

Symptoms: - "Works on Agent A, fails on Agent B" - Environment differences between agents - Tool version mismatches

Diagnosis:

Compare agent environments:

1. 1.Run diagnostic build on both agents:

# Create a diagnostic build step
echo "=== Environment ==="
env | sort
echo "=== Tools ==="
java -version
python --version
node --version
echo "=== Paths ==="
echo $PATH
ls -la /usr/bin/

Solution A: Standardize agent configuration:

Create consistent agent setup:

```bash # On all agents, ensure same tools sudo apt-get install -y java-17-openjdk python3 nodejs

# Set same environment variables export JAVA_HOME=/usr/lib/jvm/java-17-openjdk export PATH=/usr/bin:/usr/local/bin:$JAVA_HOME/bin:$PATH ```

Solution B: Use Docker for builds:

To avoid environment differences:

<!-- Build step in TeamCity -->
<runner type="docker">
  <parameters>
    <param name="docker.image" value="node:20" />
    <param name="script" value="npm install && npm test" />
  </parameters>
</runner>

Solution C: Add agent requirements:

Ensure builds only run on compatible agents:

1. 1.Edit build configuration
2. 2.Agent Requirements → Add Requirement
3. 3.Add: env.NodeJS exists
4. 4.Add: system.NodeVersion equals 20

Fix 6: Agent Authorization Revoked

Agent that was working becomes unauthorized.

Symptoms: - Agent shows "Unauthorized" suddenly - Builds stop running on that agent - Manual authorization needed again

Cause: This can happen due to: - Server restart with new authorization policy - Agent push re-installation - Security settings change

Solution:

1. 1.Go to Agents → Unauthorized agents
2. 2.Click your agent
3. 3.Click "Authorize"
4. 4.Enable the agent

Prevent future issues:

# Use stable authorization token
# In buildAgent.properties
authorizationToken=permanent-token-here

Or in TeamCity:

1. 1.Administration → Agents → Authorization Policy
2. 2.Set to "Automatically authorize new agents" (less secure)
3. 3.Or "Manually authorize" and use tokens (more secure)

Fix 7: Agent Upgrade Issues

Agent fails to upgrade when server updates.

Symptoms: - "Agent version is incompatible with server" - Upgrade task fails - Agent won't connect after server upgrade

Solution A: Manual upgrade:

```bash # Stop agent /opt/TeamCityBuildAgent/bin/agent.sh stop

# Download new agent from server curl -O http://teamcity-server:8111/update/buildAgent.zip

# Extract unzip buildAgent.zip -d /opt/TeamCityBuildAgent-new

# Copy configuration cp /opt/TeamCityBuildAgent/conf/buildAgent.properties /opt/TeamCityBuildAgent-new/conf/

# Replace old agent mv /opt/TeamCityBuildAgent /opt/TeamCityBuildAgent-old mv /opt/TeamCityBuildAgent-new /opt/TeamCityBuildAgent

# Start agent /opt/TeamCityBuildAgent/bin/agent.sh start ```

Solution B: Force upgrade:

# In buildAgent.properties
forceUpgrade=true

Then restart agent.

Solution C: Use Agent Push:

From TeamCity server:

1. 1.Administration → Agents → Agent Push
2. 2.Configure push settings
3. 3.Select agents to upgrade
4. 4.Push new version

Fix 8: Cloud Agent Issues (AWS, Azure)

Cloud agents fail to provision.

Symptoms: - "Failed to start cloud instance" - Cloud agents stuck in "Starting" - Instance launch timeout

Solution A: Check cloud profile:

1. 1.Administration → Agents → Cloud Profiles
2. 2.Verify AWS/Azure credentials
3. 3.Check image ID/AMI
4. 4.Verify instance type
5. 5.Check security group/firewall

Solution B: Fix IAM permissions (AWS):

// IAM policy for TeamCity
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "ec2:DescribeInstances"
      ],
      "Resource": "*"
    }
  ]
}

Solution C: Increase instance timeout:

1. 1.Cloud Profile → Edit
2. 2.Increase "Instance startup timeout" (default 5 minutes)

Fix 9: Agent Working Directory Issues

Builds fail due to workspace problems.

Symptoms: - "Cannot create directory" - "Permission denied" in work directory - Disk full errors

Solution A: Fix permissions:

```bash # Check work directory ls -la /opt/TeamCityBuildAgent/work

# Fix ownership sudo chown -R teamcity-user:teamcity-user /opt/TeamCityBuildAgent/work

# Fix permissions chmod -R 755 /opt/TeamCityBuildAgent/work ```

Solution B: Change work directory:

# In buildAgent.properties
workDir=/mnt/build-work  # Larger disk

Solution C: Clean up disk:

```bash # Check disk usage df -h /opt/TeamCityBuildAgent

# Clean old builds find /opt/TeamCityBuildAgent/work -type d -mtime +30 -exec rm -rf {} +

# Or in TeamCity: # Administration → Clean-up → Configure ```

Fix 10: Multiple Agents on Same Machine

Running multiple agents causes conflicts.

Symptoms: - Agents interfere with each other - Port conflicts - Shared resource issues

Solution:

```bash # Install second agent cp -r /opt/TeamCityBuildAgent /opt/TeamCityBuildAgent2

# Edit configuration nano /opt/TeamCityBuildAgent2/conf/buildAgent.properties

# Change name and ports name=Agent2 ownPort=9090 # Different port

# Set different work directory workDir=/opt/TeamCityBuildAgent2/work

# Start second agent /opt/TeamCityBuildAgent2/bin/agent.sh start ```

Quick Reference: Agent Troubleshooting

Debugging Commands

```bash # Agent status /opt/TeamCityBuildAgent/bin/agent.sh status

# Agent logs tail -100 /opt/TeamCityBuildAgent/logs/teamcity-agent.log

# Check configuration cat /opt/TeamCityBuildAgent/conf/buildAgent.properties

# Test server connection curl -I http://teamcity-server:8111/app/rest/server

# Check agent process ps aux | grep teamcity

# Check disk df -h /opt/TeamCityBuildAgent

# Check memory free -m ```