amazon web services – Fail to upload file to SFTP host with golang-ThrowExceptions

Exception or error:

I have the following golang function to upload a file to SFTP:

func uploadObjectToDestination(sshConfig SSHConnectionConfig, destinationPath string, srcFile io.Reader) {
    // Connect to destination host via SSH
    conn, err := ssh.Dial("tcp", sshConfig.sftpHost+sshConfig.sftpPort, sshConfig.authConfig)
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    // create new SFTP client
    client, err := sftp.NewClient(conn)
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    log.Printf("Opening file on destination server under path %s", destinationPath)
    // create destination file
    dstFile, err := client.OpenFile(destinationPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
    if err != nil {
        log.Fatal(err)
    }
    defer dstFile.Close()

    log.Printf("Copying file to %s", destinationPath)
    // copy source file to destination file
    bytes, err := io.Copy(dstFile, srcFile)
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("%s - Total %d bytes copied\n", dstFile.Name(), bytes)
}

The code above works 95% of the cases but fails for some files. The only relation between this files which are failing is the size (3-4kb). The other files which succeed are smaller (0.5-3kb). In some cases files with size 2-3kb are failing as well.

I was able to reproduce the same issue with different SFTP servers.

When changing the failing code (io.Copy) with sftp.Write I can see the same behavior, except that the process does not return an error, instead I see that 0 bytes were copied, which seems to be the same like failing with io.Copy.

Btw, when using io.Copy, the error I receive is Context cancelled, unexpected EOF.

The code is running from AWS lambda and there is no memory or time limit issue.

How to solve:

After few hours of digging, it turns out, my code was the source of the issue.
Here is the answer for future reference:
There was another function not in the original question which downloads the object(s) from S3:

    func getObjectFromS3(svc *s3.S3, bucket, key string) io.Reader {
    var timeout = time.Second * 30
    ctx := context.Background()
    var cancelFn func()
    ctx, cancelFn = context.WithTimeout(ctx, timeout)
    defer cancelFn()
    var input = &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }
    o, err := svc.GetObjectWithContext(ctx, input)
    if err != nil {
        if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
            log.Fatal("Download canceled due to timeout", err)
        } else {
            log.Fatal("Failed to download object", err)
        }
    }
    // Load S3 file into memory, assuming small files
    return o.Body
  }

The code above is using context and for some reason, the object returned object size was wrong.
Since I don’t use contexts here I simply converted my code to use GetObject(input) which fixed the issue.

func getObjectFromS3(svc *s3.S3, bucket, key string) io.Reader {
    var input = &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }

    o, err := svc.GetObject(input)
    if err != nil {
        if aerr, ok := err.(awserr.Error); ok {
            switch aerr.Code() {
            case s3.ErrCodeNoSuchKey:
                log.Fatal(s3.ErrCodeNoSuchKey, aerr.Error())
            default:
                log.Fatal(aerr.Error())
            }
        } else {
            // Print the error, cast err to awserr.Error to get the Code and
            // Message from an error.
            log.Fatal(err.Error())
        }
    }

    // Load S3 file into memory, assuming small files
    return o.Body
}

Leave a Reply

Your email address will not be published. Required fields are marked *