Summary
During S3 operations, you may encounter the message "Worker i-... did not establish connection" in the log of an S3 copy or S3 restore operation:
Error - Worker i-1234567890abcdef did not establish connection - terminating operation
During File Level Restore operations (which starting with v2.6 requires a worker too), you may encounter a "worker i-... did not complete initializing" in the CPM Server log:
ERROR: execute(mount_volumes.py:180) worker i-1234567890abcdef did not complete initializing
Both errors are caused by lack of connectivity between the Worker and CPM Server.
Description
Please check the following:
1. The worker appliance security group settings, the following must be allowed for both backup and restore:
OUTBOUND HTTPS (port 443):
- To the AWS S3 public gateway IPs, same region as the S3 bucket.
- To the CPM server private or public IP.
OUTBOUND SSH (port 22) - for File Level Restore workers only:
- To the CPM server private or public IP.
Note 1: Amazon regularly rotates public gateway IPs, check current IP information here: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html
Note 2: If restoring from S3 to a different region, select a security group at the target region which allows outbound connections to the source region S3 bucket. Please see 4. below as well.
2. The CPM server security group settings:
INBOUND HTTPS (port 443):
- To the subnet the worker is configured to use.
INBOUND SSH (port 22) - for File Level Restore workers only:
- To the subnet the worker is configured to use.
3. If using an ACL, be sure the proper ports are open for bi-directional communication between the worker and S3: https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/
Also be sure access to the CPM server is allowed via port 443 where necessary.
4. The worker will connect to S3 via internet, even if the CPM server and S3 bucket are in the same region.
- The worker must have a public IP assigned by the subnet; -or- the subnet must be configured with a NAT gateway, for the worker to communicate with S3.
- Only if the EBS backups and S3 bucket are in the same region, this can be worked around by configuring an S3 endpoint as shown here: https://docs.n2ws.com/user-guide/appendix-a-recommended-configuration-for-copy-to-s3
5. The worker must be able to resolve the DNS name of the region with your S3 bucket. (e.g. https://s3.us-west-2.amazonaws.com/). Please ensure the worker will be able to resolve DNS names of S3 endpoints.
6. If an HTTP proxy is necessary to access public IPs, ensure this is properly configured in the worker settings and not blocking the transfer to s3. It may be advisable to test by opening the internet to the worker without a proxy in some cases to be sure the proxy is not causing an issue.
Testing connectivity from the worker to CPM
To test the proper connectivity from the worker to CPM, you can run the following commands from the worker (put CPM's IP or hostname instead of "CPMIP"):
wget --no-check-certificate https://cpmip/
This command should result in status 302 (redirecting to "/signin/") followed by 200
ssh cpmuser@CPMIP
This command should result in "Permission denied (publickey)". This is only needed for File Level Restore workers, not S3.
Important: In order to be able to login to the worker instance over SSH (using "ubuntu" username) you must ensure that you have configured the workers to use a key pair:
You also have this KB which shows how to connect to a worker:
How To Test Connectivity from a CPM Worker to AWS endpoints
Diagram:
Below diagram is aimed at helping illustrate the setup of the worker, this is just one example and might change based on your configuration/settings. For exact technical details, please see the above KB instruction and our User Guide.
Confirmation Steps
Once you have confirmed that the required permissions and connectivity are set, then the S3 copy and FLR operations should be successful.
Comments
0 comments
Please sign in to leave a comment.