Streamlining Data Flow: Integrating SFTP User Access with S3, SQS, and VM Directory Mounting
Enhance Data Efficiency: Seamlessly Integrate SFTP User Access with S3, SQS, and VM Directory Mounting for Streamlined File Management
Table of contents
Introduction :
In the blog post, we can explore how to set up a seamless data flow by integrating various components such as SFTP user access, S3 storage, SQS queues, and VM directory mounting. We'll discuss the steps involved in configuring SFTP users with limited access to specific directories, mounting S3 buckets to a virtual machine (VM) directory, and utilizing SQS for event notification.
Pre-requisite
Create s3 bucket.
Create sqs queue.
create ec2 instance (ubuntu)
create IAM User for accessing s3 bucket and add s3 permission to it.
Creating user for sftp
Install s3fs
sudo apt update
sudo apt install s3fs
Setup creds for s3fs
Create file
sudo vim /etc/passwd-s3fs
# write below content with its actual value for iam user created
ACCESS_KEY_ID:SECRET_KEY_ID
create user for sftp
#!/bin/bash
# Usage: sudo ./create-sftp-user.sh username
USERNAME=$1
S3_BUCKET_NAME="s3-bucket-name"
S3_BUCKET_PATH="$S3_BUCKET_NAME:/incoming/$USERNAME"
MOUNT_DIR="/mnt/sftp-s3/$USERNAME"
sudo useradd -m -d "$MOUNT_DIR" -s /bin/bash -G sftponly "$USERNAME"
# Set password for the user
sudo passwd "$USERNAME"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
sudo mkdir -p "$MOUNT_DIR/data"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR/data" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
echo "User $USERNAME added and configured."
sudo vim /etc/ssh/sshd_config
at last write
Match User userName
ChrootDirectory /mnt/sftp-s3/$USERNAME
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no
PasswordAuthentication yes
now try to access over sftp .
Update sqs queue access policy :
{
"Version": "2012-10-17",
"Id": "Policy1679925546977",
"Statement": [
{
"Sid": "Stmt1679925532180",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:*",
"Resource": "arn:aws:sqs:region:1111111111:sqs_name",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:s3:::s3-buket-name"
}
}
}
]
}
Create s3 event notification.
Thanks
For more such content follow me on :
Twitter : https://x.com/akash202k_