Zabbix server stopped processing data or is showing database errors in the logs. Without the database, your entire monitoring infrastructure is compromised. Database issues in Zabbix can range from simple connection failures to complex performance problems.

Understanding Database Errors

Zabbix relies heavily on its database for storing configuration, history data, and trends. Database errors typically manifest as:

  • Zabbix server process stops or crashes
  • "Database error" messages in logs
  • Slow data processing or timeouts
  • Missing or incomplete graphs
  • Alerting failures

Common error patterns:

bash
database connection error: Cannot connect to MySQL server
bash
DB error: Lost connection to MySQL server during query
bash
database is down: unable to start Zabbix server
bash
query failed: Table 'zabbix.history' doesn't exist

Initial Diagnosis

Check Zabbix server status and database connectivity:

```bash # Check Zabbix server status systemctl status zabbix-server

# Check Zabbix server logs tail -n 100 /var/log/zabbix/zabbix_server.log | grep -i "database|db|error|mysql|postgres"

# Check database connection mysql -u zabbix -p -h localhost zabbix -e "SELECT 1" # Or for PostgreSQL psql -U zabbix -d zabbix -c "SELECT 1"

# Check database process systemctl status mysql # Or systemctl status postgresql

# Check database logs tail -n 50 /var/log/mysql/error.log | grep -i "error" tail -n 50 /var/log/postgresql/*.log | grep -i "error" ```

Common Cause 1: Database Connection Failure

Zabbix cannot connect to the database server.

Error pattern: `` Cannot connect to MySQL server on 'localhost' (111)

bash
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused

Diagnosis:

```bash # Check database service status systemctl status mysql systemctl status postgresql

# Check database is listening netstat -tlnp | grep 3306 # MySQL netstat -tlnp | grep 5432 # PostgreSQL

# Test direct database connection mysql -u zabbix -p -h localhost psql -U zabbix -h localhost -d zabbix

# Check database credentials in Zabbix config grep DB /etc/zabbix/zabbix_server.conf

# Test with credentials from config DBPASSWORD=$(grep '^DBPassword=' /etc/zabbix/zabbix_server.conf | cut -d= -f2) mysql -u zabbix -p"$DBPASSWORD" -h localhost zabbix ```

Solution:

Fix database connection:

```bash # Start database if stopped systemctl start mysql systemctl start postgresql

# Check Zabbix database configuration # /etc/zabbix/zabbix_server.conf DBHost=localhost DBName=zabbix DBUser=zabbix DBPassword=yourpassword DBPort=3306

# For remote database, update DBHost DBHost=db-server-ip

# Verify database user has correct permissions mysql -u root -p GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'localhost' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON zabbix.* TO 'zabbix'@'zabbix-server-ip' IDENTIFIED BY 'password'; FLUSH PRIVILEGES; ```

For PostgreSQL:

```bash # Update pg_hba.conf for connection echo "host zabbix zabbix zabbix-server-ip/32 md5" >> /var/lib/postgresql/data/pg_hba.conf

# Restart PostgreSQL systemctl restart postgresql ```

Common Cause 2: Database User Authentication Failure

Invalid credentials prevent database access.

Error pattern: `` Access denied for user 'zabbix'@'localhost' (using password: YES)

bash
FATAL: password authentication failed for user "zabbix"

Diagnosis:

```bash # Check credentials in Zabbix config grep -E '^DBUser|^DBPassword' /etc/zabbix/zabbix_server.conf

# Test credentials directly mysql -u zabbix -p'password_from_config' -h localhost zabbix -e "SELECT 1"

# Check if user exists in database mysql -u root -p -e "SELECT User, Host FROM mysql.user WHERE User='zabbix'" ```

Solution:

Reset database credentials:

```bash # MySQL - reset password mysql -u root -p ALTER USER 'zabbix'@'localhost' IDENTIFIED BY 'newpassword'; FLUSH PRIVILEGES;

# Update Zabbix configuration sed -i "s/^DBPassword=.*/DBPassword=newpassword/" /etc/zabbix/zabbix_server.conf

# Restart Zabbix server systemctl restart zabbix-server ```

For PostgreSQL:

```bash # Reset password psql -U postgres -c "ALTER USER zabbix WITH PASSWORD 'newpassword';"

# Update Zabbix config sed -i "s/^DBPassword=.*/DBPassword=newpassword/" /etc/zabbix/zabbix_server.conf systemctl restart zabbix-server ```

Common Cause 3: Database Performance Issues

Slow queries or insufficient resources cause timeouts.

Error pattern: `` DB error: Lock wait timeout exceeded

bash
query timeout exceeded: history_uint sync

Diagnosis:

```bash # Check database performance metrics mysql -u root -p -e "SHOW GLOBAL STATUS LIKE 'Threads_running';" mysql -u root -p -e "SHOW PROCESSLIST;"

# Check long running queries mysql -u root -p -e "SELECT * FROM information_schema.processlist WHERE TIME > 10;"

# Check InnoDB status mysql -u root -p -e "SHOW ENGINE INNODB STATUS;"

# For PostgreSQL psql -U postgres -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"

# Check Zabbix housekeeper process grep housekeeper /var/log/zabbix/zabbix_server.log

# Check table sizes mysql -u zabbix -p zabbix -e " SELECT table_name, ROUND(((data_length + index_length) / 1024 / 1024), 2) AS size_mb FROM information_schema.TABLES WHERE table_schema = 'zabbix' ORDER BY size_mb DESC LIMIT 20; " ```

Solution:

Optimize database configuration:

```bash # MySQL optimization for Zabbix # /etc/mysql/mysql.conf.d/zabbix.cnf [mysqld] # InnoDB settings innodb_buffer_pool_size = 2G innodb_log_file_size = 512M innodb_flush_log_at_trx_commit = 2 innodb_flush_method = O_DIRECT

# Connection settings max_connections = 500 thread_cache_size = 50

# Query cache (note: deprecated in MySQL 8) query_cache_size = 64M query_cache_type = 1

# Buffer settings read_rnd_buffer_size = 256K sort_buffer_size = 256K

# Restart MySQL systemctl restart mysql ```

For PostgreSQL:

```bash # /var/lib/postgresql/data/postgresql.conf shared_buffers = 512MB effective_cache_size = 2GB maintenance_work_mem = 256MB work_mem = 64MB max_connections = 200

# Restart PostgreSQL systemctl restart postgresql ```

Optimize Zabbix database:

```bash # Run database partitioning for large history tables # This significantly improves performance for large Zabbix installations

# Check current history and trends settings grep -E 'HistoryStoragePeriod|TrendStoragePeriod' /etc/zabbix/zabbix_server.conf

# Reduce if necessary HistoryStoragePeriod=7d TrendStoragePeriod=30d

# Run housekeeping manually mysql -u zabbix -p zabbix -e "CALL zabbix_housekeeping();" ```

Common Cause 4: Database Storage Full

Disk space exhaustion prevents database operations.

Error pattern: `` DB error: The table 'history' is full

bash
ERROR: could not extend file "base/16384/16388": No space left on device

Diagnosis:

```bash # Check disk space df -h /var/lib/mysql df -h /var/lib/postgresql

# Check database directory size du -sh /var/lib/mysql/ du -sh /var/lib/postgresql/

# Check largest tables mysql -u zabbix -p zabbix -e " SELECT table_name, ROUND((data_length + index_length)/1024/1024) as MB FROM information_schema.tables WHERE table_schema='zabbix' ORDER BY MB DESC LIMIT 10; " ```

Solution:

Free database storage:

```bash # Option 1: Run aggressive housekeeping # Reduce history retention # /etc/zabbix/zabbix_server.conf HistoryStoragePeriod=3d TrendStoragePeriod=7d

# Restart Zabbix to apply systemctl restart zabbix-server

# Option 2: Manually delete old history mysql -u zabbix -p zabbix DELETE FROM history WHERE clock < UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 3 DAY)); DELETE FROM history_uint WHERE clock < UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 3 DAY)); DELETE FROM history_str WHERE clock < UNIX_TIMESTAMP(DATE_SUB(NOW(), INTERVAL 3 DAY));

# Option 3: Drop and recreate problematic table # Only for non-critical data mysql -u zabbix -p zabbix TRUNCATE TABLE history_log; TRUNCATE TABLE history_text;

# Option 4: Move database to larger storage systemctl stop mysql mv /var/lib/mysql /mnt/larger-storage/mysql ln -s /mnt/larger-storage/mysql /var/lib/mysql systemctl start mysql ```

Common Cause 5: Database Corruption

Tables or indexes are corrupted.

Error pattern: `` Table 'zabbix.history' is marked as crashed and should be repaired

bash
ERROR: index "history_clock_idx" is corrupted

Diagnosis:

```bash # Check for crashed tables in MySQL mysqlcheck -u zabbix -p --check zabbix

# Check specific table mysql -u zabbix -p zabbix -e "CHECK TABLE history;"

# For PostgreSQL, check for corruption psql -U zabbix -d zabbix -c "REINDEX DATABASE zabbix;"

# Check for inconsistent data mysql -u zabbix -p zabbix -e "SELECT COUNT(*) FROM history WHERE itemid NOT IN (SELECT itemid FROM items);" ```

Solution:

Repair corrupted tables:

```bash # MySQL repair mysqlcheck -u zabbix -p --repair zabbix

# Repair specific table mysql -u zabbix -p zabbix -e "REPAIR TABLE history;"

# For InnoDB tables, try different approach mysql -u root -p SET GLOBAL innodb_force_recovery = 1; # Check tables, dump data if needed

# For PostgreSQL psql -U postgres -d zabbix -c "REINDEX TABLE history;" psql -U postgres -d zabbix -c "VACUUM FULL ANALYZE history;" ```

Common Cause 6: Zabbix Schema Issues

Schema version mismatch or missing tables.

Error pattern: `` Cannot find table 'zabbix.history_uint' in database

bash
database is not initialized

Diagnosis:

```bash # Check database schema version mysql -u zabbix -p zabbix -e "SELECT * FROM dbversion;"

# Check Zabbix expected version grep DB /var/log/zabbix/zabbix_server.log | head -10

# List all tables mysql -u zabbix -p zabbix -e "SHOW TABLES;" ```

Solution:

Update or create schema:

```bash # Check current Zabbix version zabbix_server --version

# Run schema upgrade if needed # Find schema files ls /usr/share/zabbix-server-mysql/ # or ls /usr/share/doc/zabbix-server-mysql/

# Apply schema updates mysql -u zabbix -p zabbix < /usr/share/zabbix-server-mysql/update.sql

# For fresh installation, create schema mysql -u zabbix -p zabbix < /usr/share/zabbix-server-mysql/create.sql ```

Common Cause 7: Database Lock Contention

High concurrency causing lock issues.

Error pattern: `` DB error: Deadlock found when trying to get lock

bash
Lock wait timeout exceeded; try restarting transaction

Diagnosis:

```bash # Check current locks mysql -u root -p -e "SHOW ENGINE INNODB STATUS;" | grep -A 20 "TRANSACTIONS"

# Check lock waits mysql -u root -p -e " SELECT r.trx_id waiting_trx_id, r.trx_mysql_thread_id waiting_thread, b.trx_id blocking_trx_id, b.trx_mysql_thread_id blocking_thread FROM information_schema.innodb_lock_waits w INNER JOIN information_schema.innodb_transactions b ON b.trx_id = w.blocking_trx_id INNER JOIN information_schema.innodb_transactions r ON r.trx_id = w.requesting_trx_id; " ```

Solution:

Reduce contention:

```bash # Reduce Zabbix syncers # /etc/zabbix/zabbix_server.conf StartDBSyncers=4 # Reduce from default if causing locks

# Increase timeout DBTimeout=30

# MySQL tuning for better concurrency innodb_lock_wait_timeout = 60 innodb_thread_concurrency = 8

# Restart services systemctl restart mysql systemctl restart zabbix-server ```

Verification

After fixing database issues:

```bash # Check Zabbix server is running systemctl status zabbix-server

# Check database connectivity zabbix_server -R config_cache_reload

# Check database is processing queries mysql -u zabbix -p zabbix -e "SELECT COUNT(*) FROM history WHERE clock > UNIX_TIMESTAMP(NOW() - INTERVAL 5 MINUTE);"

# Check Zabbix server stats zabbix_server -R stats

# Monitor processing tail -f /var/log/zabbix/zabbix_server.log | grep -i "processed"

# Check Zabbix UI # Verify graphs are showing recent data # Check Configuration > Hosts are all visible ```

Prevention

Set up database monitoring:

```yaml groups: - name: zabbix_database rules: - alert: ZabbixDatabaseDown expr: zabbix_server_db_available == 0 for: 2m labels: severity: critical annotations: summary: "Zabbix database is unavailable"

  • alert: ZabbixDatabaseSlow
  • expr: zabbix_server_db_latency_seconds > 5
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Zabbix database queries are slow"
  • alert: ZabbixDatabaseStorageLow
  • expr: mysql_database_size_remaining_percent < 20
  • for: 5m
  • labels:
  • severity: warning
  • annotations:
  • summary: "Zabbix database storage space low"
  • `

Regular maintenance:

```bash #!/bin/bash # zabbix-db-maintenance.sh # Run weekly via cron

# Check and repair tables mysqlcheck -u zabbix -p --check --repair zabbix

# Optimize large tables mysql -u zabbix -p zabbix -e "OPTIMIZE TABLE history, history_uint, trends;"

# Analyze table statistics mysql -u zabbix -p zabbix -e "ANALYZE TABLE history, history_uint, trends;"

# Check disk usage df -h /var/lib/mysql ```

Database errors in Zabbix typically stem from connectivity, performance, or storage issues. Start with basic connectivity checks, then investigate performance metrics and storage capacity. Regular maintenance prevents most issues.