Weaviate Migration Guide upgrading to Client v4 and Server 1.27+

This guide explains how to migrate from Weaviate client v3 to v4.17.0 and upgrade your Weaviate server from version 1.19.0 to 1.27.0 or higher. This migration is required for Dify versions that include the weaviate-client v4 upgrade.

Overview

Starting with Dify v1.9.2, the weaviate-client has been upgraded from v3 to v4.17.0. This upgrade brings significant performance improvements and better stability, but requires Weaviate server version 1.27.0 or higher.

BREAKING CHANGE: The new weaviate-client v4 is NOT backward compatible with Weaviate server versions below 1.27.0. If you are running a self-hosted Weaviate instance on version 1.19.0 or older, you must upgrade your Weaviate server before upgrading Dify.

Who Is Affected?

This migration affects:

Self-hosted Dify users running their own Weaviate instances on versions below 1.27.0
Users currently on Weaviate server version 1.19.0-1.26.x
Users upgrading to Dify versions with weaviate-client v4

Not affected:

Cloud-hosted Weaviate users (Weaviate Cloud manages the server version)
Users already on Weaviate 1.27.0+ can upgrade Dify without additional steps
Users running Dify’s default Docker Compose setup (Weaviate version is updated automatically)

Breaking Changes

Client v4 Requirements

The weaviate-client v4 introduces several breaking changes:

Minimum Server Version: Requires Weaviate server 1.27.0 or higher
API Changes: New import structure (weaviate.classes instead of weaviate.client)
gRPC Support: Uses gRPC by default on port 50051 for improved performance
Authentication Changes: Updated authentication methods and configuration

Why Upgrade?

Performance: Significantly faster query and import operations via gRPC (50051)
Stability: Better connection handling and error recovery
Future Compatibility: Access to latest Weaviate features and ongoing support
Security: Weaviate 1.19.0 is over a year old and no longer receives security updates

Version Compatibility Matrix

Dify Version	Weaviate-client Version	Compatible Weaviate Server Versions
≤ 1.9.1	v3.x	1.19.0 - 1.26.x
≥ 1.9.2	v4.17.0	1.27.0+ (tested up to 1.33.1)

This migration applies to any Dify version using weaviate-client v4.17.0 or higher.

Weaviate server version 1.19.0 was released over a year ago and is now outdated. Upgrading to 1.27.0+ provides access to numerous improvements in performance, stability, and features.

Prerequisites

Before starting the migration, complete these steps:

Check Your Current Weaviate Version
```
curl http://localhost:8080/v1/meta
```
Look for the version field in the response.
Backup Your Data
- Create a complete backup of your Weaviate data
- Backup your Docker volumes if using Docker Compose
- Document your current configuration settings
Review System Requirements
- Ensure sufficient disk space for database migration
- Verify network connectivity between Dify and Weaviate
- Confirm gRPC port (50051) is accessible if using external Weaviate
Plan Downtime
- The migration will require service downtime
- Notify users if running in production
- Schedule migration during low-traffic periods

Migration Paths

Choose the migration path that matches your deployment setup and current Weaviate version.

Choose Your Path

Path A – Migration with Backup (from 1.19): Recommended if you are still on Weaviate 1.19. You will create a backup, upgrade to 1.27+, repair any orphaned data, and then migrate the schema.
Path B – Direct Recovery (already on 1.27+): Use this if you already upgraded to 1.27+ and your knowledge bases stopped working. This path focuses on repairing the data layout and running the schema migration.

Do not attempt to downgrade back to 1.19. The schema format is incompatible and will lead to data loss.

Path A: Migration with Backup (From 1.19)

Safest path. Creates a backup before upgrading so you can restore if anything goes wrong.

Prerequisites

Currently running Weaviate 1.19
Docker + Docker Compose installed
Python 3.11+ available for the schema migration script

Step A1: Enable the Backup Module on Weaviate 1.19

Edit docker/docker-compose.yaml so the weaviate service includes backup configuration:

  weaviate:
    image: semitechnologies/weaviate:1.19.0
    volumes:
      - ./volumes/weaviate:/var/lib/weaviate
      - ./volumes/weaviate_backups:/var/lib/weaviate/backups
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      ENABLE_MODULES: backup-filesystem
      BACKUP_FILESYSTEM_PATH: /var/lib/weaviate/backups
      # ... rest of your environment variables

Restart Weaviate to apply the change:

cd docker
docker compose down
docker compose --profile up -d
sleep 10

Step A2: Create a Backup

List your collections:

curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
  "http://localhost:8080/v1/schema" | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
print("Collections:")
for cls in data.get('classes', []):
    print(f"  - {cls['class']}")
"

Trigger the backup: include specific collection names if you prefer.

curl -X POST \
  -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
  -H "Content-Type: application/json" \
  "http://localhost:8080/v1/backups/filesystem" \
  -d '{
    "id": "kb-backup",
    "include": ["Vector_index_COLLECTION1_Node", "Vector_index_COLLECTION2_Node"]
  }'

Check backup status:

sleep 5
curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
  "http://localhost:8080/v1/backups/filesystem/kb-backup" | \
  python3 -m json.tool | grep status

Verify backup files exist:

ls -lh docker/volumes/weaviate_backups/kb-backup/

Step A3: Upgrade to Weaviate 1.27+

Upgrade Dify to a version that ships Weaviate 1.27+:

cd /path/to/dify
git fetch origin
git checkout main  # or a tagged release that includes the upgrade

Confirm the new Weaviate image:

grep "image: semitechnologies/weaviate" docker/docker-compose.yaml

Restart with the new version:

cd docker
docker compose down
docker compose up -d
sleep 20

Step A4: Fix Orphaned LSM Data (if present)

cd docker/volumes/weaviate

for dir in vector_index_*_node_*_lsm; do
  [ -d "$dir" ] || continue

  index_id=$(echo "$dir" | sed -n 's/vector_index_\([^_]*_[^_]*_[^_]*_[^_]*_[^_]*\)_node_.*/\1/p')
  shard_id=$(echo "$dir" | sed -n 's/.*_node_\([^_]*\)_lsm/\1/p')

  mkdir -p "vector_index_${index_id}_node/$shard_id/lsm"
  cp -a "$dir/"* "vector_index_${index_id}_node/$shard_id/lsm/"

  echo "✓ Copied $dir"
done

cd ../../
docker compose restart weaviate
sleep 15

Step A5: Migrate the Schema

Install dependencies (in a temporary virtualenv is fine):

cd /path/to/dify
python3 -m venv weaviate_migration_env
source weaviate_migration_env/bin/activate
pip install weaviate-client requests

Run the migration script:

python3 migrate_weaviate_collections.py

Restart Dify services:

cd docker
docker compose restart api worker worker_beat
sleep 15

Verify in the UI: open Dify, test retrieval against your migrated knowledge bases.

After confirming a healthy migration, you can delete weaviate_migration_env and the backup files to reclaim disk space.

Path B: Direct Recovery (Already on 1.27+)

Only use this path if you already upgraded to 1.27+ and your knowledge bases stopped working. You cannot create a 1.19 backup anymore, so you must repair the data in place.

Prerequisites

Currently running Weaviate 1.27+ (including 1.33)
Docker + Docker Compose installed
Python 3.11+ for the migration script

Step B1: Repair Orphaned LSM Data

cd docker
docker compose stop weaviate

cd volumes/weaviate

for dir in vector_index_*_node_*_lsm; do
  [ -d "$dir" ] || continue

  index_id=$(echo "$dir" | sed -n 's/vector_index_\([^_]*_[^_]*_[^_]*_[^_]*_[^_]*\)_node_.*/\1/p')
  shard_id=$(echo "$dir" | sed -n 's/.*_node_\([^_]*\)_lsm/\1/p')

  mkdir -p "vector_index_${index_id}_node/$shard_id/lsm"
  cp -a "$dir/"* "vector_index_${index_id}_node/$shard_id/lsm/"

  echo "✓ Copied $dir"
done

Restart Weaviate:

cd ../..
docker compose start weaviate
sleep 15

List collections and confirm object counts are non-zero:

curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
  "http://localhost:8080/v1/schema" | python3 -c "
import sys, json
for cls in json.load(sys.stdin).get('classes', []):
    if cls['class'].startswith('Vector_index_'):
        print(cls['class'])
"

curl -s -H "Authorization: Bearer <WEAVIATE_API_KEY>" \
  "http://localhost:8080/v1/objects?class=YOUR_COLLECTION_NAME&limit=0" | \
  python3 -c "import sys, json; print(json.load(sys.stdin).get('totalResults', 0))"

Step B2: Run the Schema Migration

Follow the same commands as Step A5. Create the virtualenv if needed, install weaviate-client 4.x, run migrate_weaviate_collections.py, then restart api, worker, and worker_beat.

Step B3: Verify in Dify

Open Dify’s Knowledge Base UI.
Use Retrieval Testing to confirm queries return results.
If errors persist, inspect docker compose logs weaviate for additional repair steps (see Troubleshooting).

Data Migration for Legacy Versions

CRITICAL: Data Migration Required

Your existing knowledge bases will NOT work after upgrade without migration!

Why Migration is Needed:

Old data: Created with Weaviate v3 client (simple schema)
New code: Requires Weaviate v4 format (extended schema)
Incompatible: Old data missing required properties

Migration Options:

Option A: Use Weaviate Backup/Restore

Option B: Re-index from Original Documents

Option C: Keep Old Weaviate (Don’t Upgrade Yet) If you can’t afford downtime or data loss.

Automatic Migration

In most cases, Weaviate 1.27.0 will automatically migrate data from 1.19.0:

Stop Weaviate 1.19.0
Start Weaviate 1.27.0 with the same data directory
Weaviate will detect the old format and migrate automatically
Monitor logs for migration progress and any errors

Manual Migration (If Automatic Fails)

If automatic migration fails, use Weaviate’s export/import tools:

1. Export Data from Old Version

Use the Cursor API or backup feature to export all data. For large datasets, use Weaviate’s backup API:

# Using backup API (recommended)
curl -X POST "http://localhost:8080/v1/backups/filesystem" \
  -H "Content-Type: application/json" \
  -d '{"id": "pre-migration-backup"}'

2. Import Data to New Version

After upgrading to Weaviate 1.27.0, restore the backup:

curl -X POST "http://localhost:8080/v1/backups/filesystem/pre-migration-backup/restore" \
  -H "Content-Type: application/json"

For comprehensive migration guidance, especially for complex schemas or large datasets, refer to the official Weaviate Migration Guide.

Configuration Changes

New Environment Variables

The following new environment variable is available in Dify versions with weaviate-client v4:

WEAVIATE_GRPC_ENDPOINT

Description: Specifies the gRPC endpoint for Weaviate connections. Using gRPC significantly improves performance for batch operations and queries. Format: hostname:port (NO protocol prefix) Default Ports:

Insecure: 50051
Secure (TLS): 443

Examples:

# Docker Compose (internal network)
WEAVIATE_GRPC_ENDPOINT=weaviate:50051

# External server (insecure)
WEAVIATE_GRPC_ENDPOINT=192.168.1.100:50051

# External server with custom port
WEAVIATE_GRPC_ENDPOINT=weaviate.example.com:9090

# Weaviate Cloud (secure/TLS on port 443)
WEAVIATE_GRPC_ENDPOINT=your-instance.weaviate.cloud:443

Do NOT include protocol prefixes like grpc:// or http:// in the WEAVIATE_GRPC_ENDPOINT value. Use only hostname:port.

Updated Environment Variables

All existing Weaviate environment variables remain the same:

WEAVIATE_ENDPOINT: HTTP endpoint for Weaviate (e.g., http://weaviate:8080)
WEAVIATE_API_KEY: API key for authentication (if enabled)
WEAVIATE_BATCH_SIZE: Batch size for imports (default: 100)
WEAVIATE_GRPC_ENABLED: Enable/disable gRPC (default: true in v4)

Complete Configuration Example

# docker/.env or environment configuration
VECTOR_STORE=weaviate

# HTTP Endpoint (required)
WEAVIATE_ENDPOINT=http://weaviate:8080

# Authentication (if enabled on your Weaviate instance)
WEAVIATE_API_KEY=your-secret-api-key

# gRPC Configuration (recommended for performance)
WEAVIATE_GRPC_ENABLED=true
WEAVIATE_GRPC_ENDPOINT=weaviate:50051

# Batch Import Settings
WEAVIATE_BATCH_SIZE=100

Verification Steps

After completing the migration, verify everything is working correctly:

1. Check Weaviate Connection

Verify Weaviate is accessible and running the correct version:

# Check HTTP endpoint and version
curl http://your-weaviate-host:8080/v1/meta | jq '.version'

# Should return 1.27.0 or higher

2. Verify Dify Connection

Check the Dify logs for successful Weaviate connection:

docker compose logs api | grep -i weaviate

Look for messages indicating successful connection without “No module named ‘weaviate.classes’” errors.

3. Test Knowledge Base Creation

Log into your Dify instance
Navigate to Knowledge Base section
Create a new knowledge base
Upload a test document (PDF, TXT, or MD)
Wait for indexing to complete
Check that status changes from “QUEUING” → “INDEXING” → “AVAILABLE”

If documents get stuck in “QUEUING” status, check that the Celery worker is running: docker compose logs worker

4. Test Vector Search

Create or open a chat application with knowledge base integration
Ask a question that should retrieve information from your knowledge base
Verify that relevant results are returned with correct scores
Check the citation/source links work correctly

5. Verify gRPC Performance

If gRPC is enabled, you should see improved performance:

# Check if gRPC port is accessible
docker exec -it dify-api-1 nc -zv weaviate 50051

# Monitor query times in logs
docker compose logs -f api | grep -i "query_time\|duration"

With gRPC properly configured, vector search queries should be 2-5x faster compared to HTTP-only connections.

Troubleshooting

Issue: “No module named ‘weaviate.classes’”

Cause: The weaviate-client v4 is not installed, or v3 is still being used. Solution:

# For Docker installations, ensure you're running the correct Dify version
docker compose pull
docker compose down
docker compose up -d

# For source installations
pip uninstall weaviate-client
pip install weaviate-client==4.17.0

Issue: Connection Refused on gRPC Port (50051)

Cause: Port 50051 is not exposed, not accessible, or Weaviate is not listening on it. Solution:

For Docker Compose users with bundled Weaviate: The port is available internally between containers. No action needed unless you’re connecting from outside Docker.

For external Weaviate:

# Check if Weaviate is listening on 50051
docker ps | grep weaviate
# Look for "0.0.0.0:50051->50051/tcp"

# If not exposed, restart with port mapping
docker run -p 8080:8080 -p 50051:50051 ...

Check firewall rules:

# Linux
sudo ufw allow 50051/tcp

# Check if port is listening
netstat -tlnp | grep 50051

Issue: Authentication Errors (401 Unauthorized)

Cause: API key mismatch or authentication configuration issue. Solution:

Verify API key matches in both Weaviate and Dify:

# Check Weaviate authentication
curl http://localhost:8080/v1/meta | jq '.authentication'

# Check Dify configuration
docker compose exec api env | grep WEAVIATE_API_KEY

If using anonymous access:

# Weaviate docker-compose.yaml
weaviate:
  environment:
    AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
    AUTHENTICATION_APIKEY_ENABLED: 'false'

Then remove WEAVIATE_API_KEY from Dify configuration.

Issue: Documents Stuck in “QUEUING” Status

Cause: Celery worker not running or not connected to Redis. Solution:

# Check if worker is running
docker compose ps worker

# Check worker logs
docker compose logs worker | tail -50

# Check Redis connection
docker compose exec api redis-cli -h redis -p 6379 -a difyai123456 ping
# Should return "PONG"

# Restart worker
docker compose restart worker

Issue: Slow Performance After Migration

Cause: gRPC not enabled or configured incorrectly. Solution:

Verify gRPC configuration:

docker compose exec api env | grep WEAVIATE_GRPC

Should show:

WEAVIATE_GRPC_ENABLED=true
WEAVIATE_GRPC_ENDPOINT=weaviate:50051

Test gRPC connectivity:

docker exec -it dify-api-1 nc -zv weaviate 50051
# Should return "succeeded"

If still slow, check network latency between Dify and Weaviate

Issue: Schema Migration Errors

Cause: Incompatible schema changes between Weaviate versions or corrupted data. Solution:

Check Weaviate logs for specific error messages:
```
docker compose logs weaviate | tail -100
```
List current schema:
```
curl http://localhost:8080/v1/schema
```

If necessary, delete corrupted collections (⚠️ this deletes all data):

# Backup first!
curl -X DELETE http://localhost:8080/v1/schema/YourCollectionName

Restart Dify to recreate schema:
```
docker compose restart api worker
```

Deleting collections removes all data. Only do this if you have a backup and are prepared to re-index all content.

Issue: Docker Volume Permission Errors

Cause: User ID mismatch in Docker containers. Solution:

# Check ownership of Weaviate data directory
ls -la docker/volumes/weaviate/

# Fix permissions (use the UID shown in error messages)
sudo chown -R 1000:1000 docker/volumes/weaviate/

# Restart services
docker compose restart weaviate

Rollback Plan

If the migration fails and you need to rollback:

Step 1: Stop Services

cd /path/to/dify/docker
docker compose down

Step 2: Restore Backup

# Remove current volumes
rm -rf volumes/weaviate

# Restore from backup
tar -xvf ../weaviate-backup-TIMESTAMP.tgz

Step 3: Revert Dify Version

cd /path/to/dify
git checkout <previous-version-tag>
cd docker
docker compose pull

Step 4: Restart Services

docker compose up -d

Step 5: Verify Rollback

Check that services are running with old versions:

# Check versions
docker compose exec api pip show weaviate-client
curl http://localhost:8080/v1/meta | jq '.version'

# Check for errors
docker compose logs | grep -i error

Always test the rollback procedure in a staging environment first if possible. Maintain multiple backup copies before attempting major migrations.

Additional Resources

Official Documentation

Community Resources

Migration Tools

Summary

This migration brings important improvements to Dify’s vector storage capabilities: Better Performance: gRPC support dramatically improves query and import speeds (2-5x faster) Improved Stability: Enhanced connection handling and error recovery Security: Access to security updates and patches not available in Weaviate 1.19.0 Future-Proof: Access to latest Weaviate features and ongoing support While this is a breaking change requiring server upgrade for users on old versions, the benefits significantly outweigh the migration effort. Most Docker Compose users can complete the migration in under 15 minutes with the automatic update.

If you encounter any issues not covered in this guide, please report them on the Dify GitHub Issues page with the label “weaviate” and “migration”.

Quick Start

Advanced Deployments

Configuration

Platform Guides

Troubleshooting

​Overview

​Who Is Affected?

​Breaking Changes

​Client v4 Requirements

​Why Upgrade?

​Version Compatibility Matrix

​Prerequisites

​Migration Paths

​Choose Your Path

​Path A: Migration with Backup (From 1.19)

​Prerequisites

​Step A1: Enable the Backup Module on Weaviate 1.19

​Step A2: Create a Backup

​Step A3: Upgrade to Weaviate 1.27+

​Step A4: Fix Orphaned LSM Data (if present)

​Step A5: Migrate the Schema

​Path B: Direct Recovery (Already on 1.27+)

​Prerequisites

​Step B1: Repair Orphaned LSM Data

​Step B2: Run the Schema Migration

​Step B3: Verify in Dify

​Data Migration for Legacy Versions

​CRITICAL: Data Migration Required

​Why Migration is Needed:

​Migration Options:

Option A: Use Weaviate Backup/Restore

Option B: Re-index from Original Documents

Option C: Keep Old Weaviate (Don’t Upgrade Yet) If you can’t afford downtime or data loss.

​Automatic Migration

​Manual Migration (If Automatic Fails)

​1. Export Data from Old Version

​2. Import Data to New Version

​Configuration Changes

​New Environment Variables

​WEAVIATE_GRPC_ENDPOINT

​Updated Environment Variables

​Complete Configuration Example

​Verification Steps

​1. Check Weaviate Connection

​2. Verify Dify Connection

​3. Test Knowledge Base Creation

​4. Test Vector Search

​5. Verify gRPC Performance

​Troubleshooting

​Issue: “No module named ‘weaviate.classes’”

​Issue: Connection Refused on gRPC Port (50051)

​Issue: Authentication Errors (401 Unauthorized)

​Issue: Documents Stuck in “QUEUING” Status

​Issue: Slow Performance After Migration

​Issue: Schema Migration Errors

​Issue: Docker Volume Permission Errors

​Rollback Plan

​Step 1: Stop Services

​Step 2: Restore Backup

​Step 3: Revert Dify Version

​Step 4: Restart Services

​Step 5: Verify Rollback

​Additional Resources

​Official Documentation

​Community Resources

​Migration Tools

​Summary

Overview

Who Is Affected?

Breaking Changes

Client v4 Requirements

Why Upgrade?

Version Compatibility Matrix

Prerequisites

Migration Paths

Choose Your Path

Path A: Migration with Backup (From 1.19)

Prerequisites

Step A1: Enable the Backup Module on Weaviate 1.19

Step A2: Create a Backup

Step A3: Upgrade to Weaviate 1.27+

Step A4: Fix Orphaned LSM Data (if present)

Step A5: Migrate the Schema

Path B: Direct Recovery (Already on 1.27+)

Prerequisites

Step B1: Repair Orphaned LSM Data

Step B2: Run the Schema Migration

Step B3: Verify in Dify

Data Migration for Legacy Versions

CRITICAL: Data Migration Required

Why Migration is Needed:

Migration Options:

Automatic Migration

Manual Migration (If Automatic Fails)

1. Export Data from Old Version

2. Import Data to New Version

Configuration Changes

New Environment Variables

WEAVIATE_GRPC_ENDPOINT

Updated Environment Variables

Complete Configuration Example

Verification Steps

1. Check Weaviate Connection

2. Verify Dify Connection

3. Test Knowledge Base Creation

4. Test Vector Search

5. Verify gRPC Performance

Troubleshooting

Issue: “No module named ‘weaviate.classes’”

Issue: Connection Refused on gRPC Port (50051)

Issue: Authentication Errors (401 Unauthorized)

Issue: Documents Stuck in “QUEUING” Status

Issue: Slow Performance After Migration

Issue: Schema Migration Errors

Issue: Docker Volume Permission Errors

Rollback Plan

Step 1: Stop Services

Step 2: Restore Backup

Step 3: Revert Dify Version

Step 4: Restart Services

Step 5: Verify Rollback

Additional Resources

Official Documentation

Community Resources

Migration Tools

Summary