Attempt to trace a process in AWS lambda

By Brian Fitzgerald


On the surface, AWS Lambda appears to be a serverless resource that runs your code. However, Lambda users will quickly notice that the code runs on an EC2-like Linux container. There are times when a system-related error appears, and you want to trace the code to find out the cause, or the point of failure.


We setup the lambda. File

from os import getlogin
from json import dumps

def lam(ev, cx):
    ret = {
        'login': dumps(getlogin())
    return ret
if __name__ == '__main__':
    lam(None, None)

the output is:

START RequestId: bc81ac8d-5c3b-421a-9432-82a6ab279767 Version: $LATEST
[Errno 6] No such device or address: OSError
Traceback (most recent call last):
File "/var/task/", line 6, in lam
'login': dumps(getlogin())
OSError: [Errno 6] No such device or address

Suppose we want to know more about the error message. In Linux, strace will tell you what system call led to the error message. You can try strace. File

from os import system

def lam(ev, cx):
    system('strace python')
    return {}

However, strace is not in lambda:

sh: strace: command not found

You can search:

     system('find / -name strace -ls')

No file is found. You can copy in strace. Start from EC2. Copy in the strace binary and your python files:

[ec2-user@ip-172-31-80-17 trc]$ mkdir -p bin
[ec2-user@ip-172-31-80-17 trc]$ cp -p /usr/bin/strace bin/strace
[ec2-user@ip-172-31-80-17 trc]$ find * -type f
[ec2-user@ip-172-31-80-17 trc]$ zip -rq ../ * 
[ec2-user@ip-172-31-80-17 trc]$ aws s3 cp ../ s3://test.bucket/lambda/
upload: ../ to s3://test.bucket/lambda/

Upload the zip file:


Now your lambda function has 3 files: the two python files and the strace binary. Change the call to system:

    system('/var/task/bin/strace python')

Run it and you get this message:

/var/task/bin/strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted

That did not work. The next thing you can try is:

  1. Start a second process, and get the pid
  2. The second process will sleep for 1s and then run the failing statement.
  3. In the first process, trace the second process.

from subprocess import Popen
from os import system

def lam(ev, cx):
    proc = Popen('python', shell=True)
    fmt = 'bin/strace -p %s'
    cmd = format(fmt %
    return {}

if __name__ == '__main__':
    lam(None, None)

from time import sleep
from os import getlogin

lg = getlogin()

In that case, the error message is:

bin/strace: attach: ptrace(PTRACE_ATTACH, 5): Operation not permitted


We tried two different invocations of strace inside lambda. In the first attempt, we ran “strace command”. Internally, strace should trace itself by calling ptrace(PTRACE_TRACEME…) and then exec the command. In the second case, we want strace to attach to a process with strace(PTRACE_ATTACH…). AWS Lambda permits neither call.


The traditional approach to tracing Linux processes is not permitted in AWS Lambda.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s