Memory leak when using PyExec

[Grovkillen]

Run this script AS COMPILED and observe the process memory usage. It will grow rapidly until it crashes.

Code: Select all

Label>START_OF_SCRIPT
//format=1
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")),ISO_8601_format_1
//format=2
PYExec>import datetime%CRLF%print(datetime.datetime.now().astimezone().replace(microsecond=0).isoformat()),ISO_8601_format_2
//format=3
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-W%V")),ISO_8601_format_3
//format=4
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-W%V-%u")),ISO_8601_format_4
//format=5
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-%j")),ISO_8601_format_5
//format=6
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-%m-%d")),ISO_8601_format_6
//format=7
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("T%H:%M:%SZ")),ISO_8601_format_7
//format=8
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("T%H%M%SZ")),ISO_8601_format_8
//format=9
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")),ISO_8601_format_9
//format=10
PYExec>import datetime%CRLF%print(datetime.datetime.now().astimezone().isoformat()),ISO_8601_format_10
//bonus formats
//epoch time
PYExec>import time%CRLF%print(int(time.time())),epoch
//epoch time as float
PYExec>import time%CRLF%print(time.time()),epoch_float
//epoch in nanoseconds
PYExec>import time%CRLF%print(time.time_ns()),epoch_ns
Wait>1
Let>OUTPUT_DATA=OUTPUT:
ConCat>OUTPUT_DATA,%CRLF%%ISO_8601_format_1%
ConCat>OUTPUT_DATA,ISO_8601_format_2
ConCat>OUTPUT_DATA,ISO_8601_format_3
ConCat>OUTPUT_DATA,ISO_8601_format_4
ConCat>OUTPUT_DATA,ISO_8601_format_5
ConCat>OUTPUT_DATA,ISO_8601_format_6
ConCat>OUTPUT_DATA,ISO_8601_format_7
ConCat>OUTPUT_DATA,ISO_8601_format_8
ConCat>OUTPUT_DATA,ISO_8601_format_9
ConCat>OUTPUT_DATA,ISO_8601_format_10
ConCat>OUTPUT_DATA,epoch
ConCat>OUTPUT_DATA,epoch_float
ConCat>OUTPUT_DATA,epoch_ns
Message>OUTPUT_DATA
Goto>START_OF_SCRIPT

[Grovkillen]

I have a feeling that MS is starting a new instance of the Python code each PyExec. The memory allocation is increasing at a fairly linear pace and rate. Change the Wait>1 to Wait>0.01 and it will crash within a minute.

[Dorian (MJT support)]

We have run some analysis using our memory leak tool and we have not found any leaks on the Macro Scheduler side. However we are seeing leaks inside the Python37.dll process. Unfortunately we have no control over this. Python is third party. It looks like there may be leaks in Python itself. We tried the latest python3.7 dll but get the same thing.

Python scripts usually run on their own and cease to exist at the end. Therefore memory leaks are never noticed. That isn't the case when run inside macro Scheduler. But if each one of those pyexecs were inside separate compiled macros and they were run one after the other no memory leaks would ever be noticed. The memory leaks are inside python and there's nothing at all we can do to change that.

[Grovkillen]

All right, then my roadmap is to convert these executions of python code as sub-processes that will exit them selves and thus not crash. Thanks for the feedback!

[JRL]

I wouldn't call this a memory leak. This is programmed memory loading. Each iteration of "Import datetime" loads the datetime module into memory and there's no Python function available to remove it. As long as the program runs the loop, its memory usage will grow. Python wasn't designed to be used this way. You could fix it by rewriting the python code to run the entire loop in Python with only one "import" line.

Run the following in Macro Scheduler (No need to create an executable). Watch msched.exe memory in Task Manager and you will see memory usage increase about every 5 seconds while the program is running. Each "import" loads the datetime module and memory usage increases.

Code: Select all

Message>Collecting Data...

Label>START_OF_SCRIPT
IfWindowOpen>Macro Scheduler Message*
Else
  Exit>0
EndIf
PYExec>import datetime%CRLF%print(datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")),ISO_8601_format_1
SetControlText>Macro Scheduler Message,TMemo,1,ISO_8601_format_1
Wait>5
Goto>START_OF_SCRIPT

The same thing can happen with library functions. The difference being that library functions were designed to be used by programming languages and Microsoft designed methods to uninstall them from memory.

Use the following DrawLine subroutine in an infinite loop and it will run for a good long time. Then, remark or remove the last libfunc line "LibFunc>user32,ReleaseDC,RDCres,HDC_1,HDC" and run in an infinite loop and your computer will shortly run out of memory and crash.

Code: Select all

//DrawLine Usage:
//GoSub>DrawLine,WindowHandle,PenSize,PenColor,XStart,YStart,XEnd,YEnd

SRT>DrawLine
  LibFunc>user32,GetDC,HDC,%DrawLine_var_1%
  LibFunc>gdi32,CreatePen,Penres,0,%DrawLine_var_2%,%DrawLine_var_3%
  LibFunc>gdi32,SelectObject,SOPres,hdc,Penres
  Libfunc>gdi32,MoveToEx,mtres,HDC,%DrawLine_var_4%,%DrawLine_var_5%,0
  LibFunc>gdi32,LineTo,ltres,hdc,%DrawLine_var_6%,%DrawLine_var_7%
  LibFunc>gdi32,DeleteObject,DOres,Penres
  LibFunc>user32,ReleaseDC,RDCres,HDC_1,HDC
END>DrawLine

The problem with Python's "import" is that they had no need to design an "unimport" function. That said, Macro Scheduler does have a "LibFree>" function for removing functions loaded with "LibLoad>". Is it possible to have a "PyFree>" function?

[Grovkillen]

JRL, thanks for the feedback. I understand how you see it.

In my current project I have now made the python code run as sub processes.

[Phil Pendlebury]

In my current project I have now made the python code run as sub processes.

Nice one. If you get a min, can you show how you did this?

[Grovkillen]

Yes sure! The thing is though... You need to use a database to pass the data back to the main process. Or find another way, if you do please tell me how you did it.

The idea is to compile the script to an executable and from the main process you then start "yourself" but pass a variable from the command line (/run_as_sub_process=1 is my way of doing this). Then you just look for this variable before the start of the main loop and add an exit at the end of the specific code you want to run. I can maybe add an example later if you want.