Troubleshoot Zero Downtime Migration

Set the ZDM Proxy log level to print the messages that you need.

The default log level is INFO, which is adequate for most logging.

If you need more detail for temporary troubleshooting, you can set the log level to DEBUG. However, this can slightly degrade performance, and DataStax recommends that you revert to INFO logging as soon as possible.

How you set the log level depends on how you deployed ZDM Proxy:

If you used ZDM Proxy Automation to deploy ZDM Proxy, then you can get logs for a single proxy instance, and you can use a playbook to retrieve logs for all instances.

Some log messages contain text that seems like an error but they aren’t errors. Instead, the message’s level indicates severity:

If the meaning of a warn or error message isn’t clear, you can report an issue.

Here are some of the most common messages in the ZDM Proxy logs.

If some configuration changes aren’t applied to your ZDM Proxy instances after a rolling restart, this typically means that you modified an immutable configuration variable.

Not all ZDM Proxy configuration variables can be changed after deployment, with or without a rolling restart. For a list of variables that you can change on a live deployment, see Change mutable configuration variables.

Any configuration variables excluded from the mutable variables list are considered immutable, and you must fully redeploy your instances to change them. This is by design because immutable configuration variables store values that must not change between the time that you finalize the deployment and start the migration. Allowing these values to change from a rolling restart could propagate a misconfiguration and compromise the deployment’s integrity.

If you change the value of an immutable configuration variable, you must run the deploy_zdm_proxy.yml playbook again. You can run this playbook as many times as needed. Each time, ZDM Proxy Automation recreates your entire ZDM Proxy deployment with the new configuration. However, this doesn’t happen in a rolling fashion: The existing ZDM Proxy instances are torn down simultaneously, and then they are recreated. This results in a brief period of downtime where the entire ZDM Proxy deployment is unavailable.

If you are running version 4.0 to 4.9 of the Cassandra Java driver, the following errors can occur during or after session initialization:

These errors are caused by a Java driver bug that was resolved in version 4.10.0.

To resolve this issue, do one of the following:

PROTOCOL ERROR messages in ZDM Proxy logs are a normal part of the handshake process while the protocol version is being negotiated.

These messages indicate that a protocol version downgrades happened because ZDM Proxy or one of the clusters doesn’t support the version requested by the client.

V5 downgrades are enforced by ZDM Proxy. Any other downgrade results from a request by a cluster that doesn’t support the version that the client requested. ZDM Proxy supports V3, V4, DSE_V1 and DSE_V2.

In the following example, notice that the PROTOCOL ERROR message is introduced by level=debug, indicating that it isn’t a true error:

PROTOCOL ERROR messages recorded at a higher log level, especially level=error, might indicate a bug because this means that the error occurred outside of the handshake process. This is a fatal unexpected error that terminates the connection. If you observe this behavior in your logs, report an issue so the issue can be investigated by the ZDM team.

If the ZDM Proxy logs contain debug messages with Invalid or unsupported protocol version: 3, this means that one of the origin clusters doesn’t support protocol version V3 or later.

Specifically, this happens with Cassandra 2.0 and DSE 4.6. ZDM cannot be used for these migrations because the ZDM Proxy control connections don’t perform protocol version negotiation, they only attempt to use V3.

Authentication errors indicate that credentials are incorrect or have insufficient permissions.

There are three sets of credentials used with ZDM Proxy:

Authentication errors mean that at least one of these three sets of credentials is incorrect or has insufficient permissions.

If the authentication error prevents ZDM Proxy from starting, then the issue is in the origin or target cluster credentials. The log message shows whether the origin or target handshake failed.

If ZDM Proxy starts but the logs contain a message like Proxy started. Waiting for SIGINT/SIGTERM to shutdown, then the authentication error occurs when a client application tries to open a connection to the proxy. This means that the client application itself has invalid credentials, such as an incorrect username/password, expired token, or insufficient permissions.

If ZDM Proxy is listening on a custom port (not 9042), you might see either of the following issues:

This happens because the application specifies the custom port as part of the contact points using the format PROXY_IP_ADDRESS:CUSTOM_PORT.

For example, if the ZDM Proxy instances were listening on port 14035, the contact points for the Cassandra Java driver might be specified as .addContactPoints("172.18.10.36:14035", "172.18.11.48:14035", "172.18.12.61:14035").

The contact point is the first point of contact to the cluster, but the driver discovers the rest of the nodes through CQL queries. However, this discovery process finds the addresses only, not the ports. The driver uses the addresses it discovers with the port that is configured at startup. As a result, the custom port is used for the initial contact point only, and the default port is used with all other nodes.

To resolve this issue, ensure that the custom port is explicitly set in your application. The way that you do this depends on your driver language and version. For example, for the Java driver, use .withPort(CUSTOM_PORT):

ZDM Proxy log messages such as the following indicate that the server doesn’t recognize the word "CALL" in the query string, which typically means that it is a remote procedure call (RPC):

From the proxy logs alone, you cannot determine which method was called by the query, but it’s typically the RPC that Cassandra drivers use to send DSE Insights data to the server.

Most DataStax-compatible drivers have DSE Insights reporting enabled by default when they detect a server version that supports it, even if the feature is disabled on the server side. The driver might also have it enabled for Astra DB depending on what server version Astra DB is returning for queries involving the system.local and system.peers tables.

These log messages are harmless, but if you need to remove them, you can disable DSE Insights in the driver configuration. For example, in the Java driver, you can set advanced.monitor-reporting to false.

When you deploy the ZDM Proxy Automation metrics component, a Grafana instance is deployed that doesn’t use Grafana’s default admin/admin credentials.

Instead, ZDM Proxy Automation specifies a custom set of credentials.

You can find the credentials for your ZDM Proxy Automation Grafana instance in the vars/zdm_monitoring_config.yml file in the ZDM Proxy Automation directory. You can also modify these credentials before deploying the metrics stack.

If the ZDM Proxy logs contain messages like Couldn’t connect to, context timed out or cancelled while opening connection, and context deadline exceeded, it can indicate that the ZDM Proxy couldn’t establish a connection with a particular node.

This can happen because ZDM Proxy has connectivity to a specific subset of the nodes. The control connection, which is established during ZDM Proxy startup, cycles through the nodes until it finds one that it can connect to successfully.

For client connections, each proxy instance cycles through its assigned nodes only. Each proxy instance has a different group of assigned nodes, which are a subset of the cluster nodes. Generally, these are unique for each proxy instances to avoid interference with load balancing that is already in place at client-side driver level. The assigned nodes aren’t necessarily contact points: Even discovered nodes undergo assignment to proxy instances.

In the following example, ZDM Proxy doesn’t have connectivity to 10.0.63.20, which was chosen as the origin node for the incoming client connection, but it connected to 10.0.63.163 during startup:

To avoid this issue, ensure that a stable network connection exists between the ZDM Proxy instances and all nodes of your origin and target clusters in the client application’s local datacenter.

When a ZDM Proxy instance comes back online after being unavailable for some time, then your client application might take too long to reconnect.

If ZDM Proxy doesn’t send topology events to the client application, the driver’s reconnection policy determines the time required for the driver to reconnect to the ZDM Proxy instance.

You can restart the client application to force an immediate reconnection attempt.

If you expect ZDM Proxy instances to go down frequently, change the driver’s reconnection policy to shorten the interval between reconnection attempts.

ZDM Proxy Automation logs might report errors that contain your Astra DevOps API endpoint:

This can indicate that the Astra DevOps API is temporarily unavailable. You can either wait to retry the operation, or you can download your database’s Secure Connect Bundle (SCB) from the Astra Portal, and then provide the path the SCB zip file in the ZDM Proxy Automation configuration.

If ZDM Proxy doesn’t start, the logs might contain messages with not successful status code:

There are two possible causes for this:

To resolve this issue, sign in to the Astra Portal, and then check the database status.

If the database isn’t in Active status, reactivate or wait for the database to return to Active status, and then retry the connection.

If the database is in Active status, then the credentials likely have insufficient permissions. Try using an application token scoped to a database, specifically a token with the Database Administrator role for your target database.

When dual reads are enabled, you might find the following messages in the ZDM Proxy logs:

The last message is logged when the async connection runs out of stream IDs. The async connection is a connection dedicated to the asynchronous dual reads feature. This can be caused by timeouts, as indicated in the first log message, or the connection being unable to keep up with the load.

Errors in the async request path (dual reads) don’t affect the client application. These log messages can be useful to predict what could happen when reads are switched over to the target cluster permanently, but async read errors and warnings don’t, by themselves, have any impact on the client.

If you find many of these messages in the logs, it is likely that an outage occurred. This causes all responses to arrive after requests timed out, as reported in the second log message. In this case the async connection might not recover.

Starting in version 2.1.0, you can tune the maximum number of stream IDs available per connection. The default is 2048, and you can increase it to match your driver configuration with the zdm_proxy_max_stream_ids property. If you are running a version prior to 2.1.0, upgrade ZDM Proxy.

If these errors are constantly written to the log files over a period of minutes or hours, then you likely need to restart the client application or ZDM Proxy to fix the issue. If you find an error like this, report an issue so the ZDM team can investigate it.

This issue is fixed in ZDM Proxy 2.1.0.

In ZDM Proxy versions earlier than 2.1.0, the logs can report that the Astra DB TARGET-CONNECTOR is disconnected every 10 minutes. This happens because Astra DB terminates idle connections after 10 minutes of inactivity. In the absence of asynchronous dual reads, the target cluster won’t get any traffic when the client application produces only read requests because ZDM forwards all reads to the origin cluster only.

To resolve this issue, DataStax recommends that you upgrade your ZDM Proxy instances to 2.1.0 or later to take advantage of the heartbeats feature, which keeps the connection alive during periods of inactivity. You can tune the heartbeat interval with the heartbeat_interval_ms variable, or by directly setting the ZDM_HEARTBEAT_INTERVAL_MS environment variable if you aren’t using ZDM Proxy Automation.

If upgrading is impossible, you can try the following alternatives:

If you run separate benchmarks against Astra DB directly, the origin cluster directly, and ZDM Proxy with both Astra DB and the origin cluster, then the results of these tests might show that latency or throughput is worse with ZDM Proxy than when connecting to Astra DB or origin cluster directly.

This is observed because ZDM always increases latency and, depending on the nature of the test, reduces throughput. Whether this performance hit is expected or not depends on the difference between the ZDM test results and the test results with the cluster that performed the worst.

Writes through ZDM require successful acknowledgement from both clusters, whereas reads require only the result from the primary cluster, which is typically the origin cluster. This means that if the origin cluster has better performance than the target cluster, then ZDM will inevitably have worse write performance than the target cluster alone.

Although it is typical for latency to increase with ZDM Proxy, you can minimize performance degradation with ZDM Proxy:

If the ZDM Proxy logs contain messages such as the following, it’s likely that you have an origin DSE cluster where Metrics Collector is enabled, and the user named in the logs doesn’t have sufficient permissions to report Insights data:

These are reported as level=debug, so ZDM Proxy isn’t affected by them.

There are two ways to resolve this issue: