Ahh, who could've thought? I spent my entire summer coding away as part of Google Summer of Code! Feels like the summer just began yesterday.
Prologue
Konnichiwa, I'm Syed Naimul Hasan Asif (alias SD Asif Hossein), an Electrical Engineering undergrad student from Bangladesh who is passionate about low-level programming, programming language research, operating systems development, deep learning and high-performance computing. This short blog post covers my work at Google Summer of Code 2024.
Over the time period, I have contributed to Pallene, an ahead-of-time compiled companion language to Lua used to write Lua's most performance critical code. Pallene is arguably more predictable and efficient than LuaJIT and Lua C API (benchmarks in the paper attached below). Pallene shares identical syntax as Lua enabling users to write high-performance code swiftly without worrying about API. Although, Pallene and Lua have a lot in common including runtime, Pallene lacks significantly in case of user debugging experience.
As first line of work to tackle this challenge, our goal in this project was to introduce proper stack-trace support upon error by tracking function calls during runtime enhancing overall debugging experience in Pallene. But the Lua call-stack lacks the feature of storing line number information for Lua-to-C calls and is completely agnostic to both inter- and intra-module C-to-C calls. It also affects Pallene which eventually transpiles to C.
To address the problem not only for Pallene but also for broader Lua C modules, a separate project was introduced, Pallene Tracer. Pallene Tracer provides tools, protocols and mechanisms to enable function tracebacks, generating a proper stack-trace with all the C-to-C call-frames and respective line numbers, without touching the Lua source code. Hence, over the course of summer we worked on Pallene Tracer, adoption of Pallene Tracer in Pallene and other Lua C libraries.
You can learn more about Pallene in this paper, written by my Mentor Hugo Musso Gualandi.
Get the Code
A good amount of code was written as part of this project for both Pallene and Pallene Tracer regarding function traceback and generating nice stack-trace during summer period.
My,
Merged PRs in Pallene can be found here.
Merged PRs in Pallene Tracer can be found here.
An amazing fact I enjoyed about working in this project is the collaboration between student and mentors. Coming up with a separate project like Pallene Tracer to deliver the tools, mechanisms and protocols used in Pallene to other Lua C modules was out of our idea list. But we did it regardless, just thinking about thousands of Lua C modules getting benefited from it!
Hence, my mentor Hugo and I both worked in Pallene Tracer together 🔥! You can find Hugo's merged PRs here.
My Mentors
In this project, I was lucky enough to have Hugo Musso Gualandi as my mentor, Srijan Paul and Gabriel de Quadros Ligneul as co-mentors. They are amazing, period.
The Pallene Tracer
For comprehensive technical overview and learn how Pallene Tracer works, please refer to the documentation.
As aforementioned, Lua call-stack does not store line number information for Lua-to-C calls and is completely oblivious to C-to-C calls. Changing Lua source code to implement those features was not an option, so we decided to create a call-stack ourselves storing the relevant information. We call it the Pallene Tracer call-stack.
At start, we decided to take the Linked List stack implementation route for Pallene Tracer call-stack, where we stored each frame structure in C stack memory and link between the call-frames. This approach was neat and didn't require any separate buffer allocation. But, there was a fundamental flaw in our idea.
While using Pallene Tracer you are expected to utilize macros. A macro to push a call-frame into Pallene Tracer call-stack used at the beginning of the function, a macro to set line number information to the topmost call-frame and finally another macro to pop frame out of the call-stack when we exit the function execution. we call it FRAMEENTER, SETLINE and FRAMEEXIT macros respectively. In order for a function to be traced, it must create a frame in call-stack by calling FRAMEENTER macro and should remove the frame by calling FRAMEEXIT whenever function is done executing. A function will only appear in stack-trace when respective call-frame is in the call-stack. In most cases, calling FRAMEENTER is generally safe because no runtime error is invoked as it's called at the very beginning of the function. But same cannot be said for respective FRAMEEXIT because when runtime error occurs, function execution terminates prior reaching FRAMEEXIT. This causes stack corruption and memory bugs specifically for the Linked List stack approach. This is generally not a problem since Lua interpreter exits right after the error. But it is a problem, when the error is caught by pcall() and its variants and interpreter continues. This bug has been mentioned in this issue.
To solve the problem, the first approach was to switch to more buffer-like stack, e.g. heap allocated stack. That did fix the memory errors but the stack corruption remained because of insufficient FRAMEEXITs. To fix that problem we exploited the property of To-be-Closed Objects with __close metamethod. To learn more about how it is implemented, please refer to the finalizer object section of documentation.
Accumulating frames into a separate stack is useless if not printed. The builtin Lua traceback function is incapable of taking advantage of the separate call-stack we have. That's why we had to come up with our separate debug traceback function which use both of the call-stacks (our call-stack and Lua call-stack), generating a reasonable stack-trace. Our debug traceback function is the default traceback for custom Pallene Tracer Lua frontend, pt-lua. An error handler function is also provided as a Lua global, pallene_tracer_errhandler to use the custom traceback function against xpcall() and its variants.
To know more about how Pallene Tracer is implemented, please refer to implementation section of the documentation.
Enabling Tracebacks in Pallene
To enable tracebacks in Pallene, simply pass the --use-traceback flag while compiling a Pallene translation.
pallenec some.pln --use-tracebackThis will enable tracing across all the Pallene functions. To generate stack-trace in case of runtime errors, run the Lua script using Pallene module with our custom Lua frontend:
pt-lua main.lua args ...To learn more about how Pallene Tracer was adopted into Pallene, please refer to the traceback documentation of Pallene.
Using Pallene Tracer in Lua C Libraries
As I have mentioned earlier, we have to utilize macros in functions to enable tracebacks. First, install Pallene Tracer in your machine. First clone the repository and then follow the instructions mentioned in the landing page to install it.
Suppose, we have a module named depth-recursion and a Lua script main.lua using that module. The module consists of a single C source named recursion.c. The idea here is C and Lua function calling each other until a specific depth has been reached. We are deliberately invoking a runtime error for demonstration.
recursion.c
#include <lua.h>
#include <lauxlib.h>
void depth_fn(lua_State *L, int depth) {
lua_pushvalue(L, 1);
if(depth == 0)
lua_pushinteger(L, depth);
else lua_pushinteger(L, depth - 1);
lua_call(L, 1, 0);
}
int depth_fn_lua(lua_State *L) {
/* Look at the macro definitions. */
if(luai_unlikely(lua_gettop(L) < 2))
luaL_error(L, "Expected atleast 2 parameters");
/* ---- `lua_fn` ---- */
if(luai_unlikely(lua_isfunction(L, 1) == 0))
luaL_error(L, "Expected the first parameter to be a function");
if(luai_unlikely(lua_isinteger(L, 2) == 0))
luaL_error(L, "Expected the second parameter to be an integer");
int depth = lua_tointeger(L, 2);
/* Dispatch. */
depth_fn(L, depth);
return 0;
}
int luaopen_recursion(lua_State *L) {
lua_newtable(L);
/* ---- depth_fn ---- */
lua_pushcfunction(L, depth_fn_lua);
lua_setfield(L, -2, "depth_fn");
return 1;
}main.lua
local recursion = require "recursion"
function lua_fn(depth)
if depth == 0 then
error "Depth reached 0!"
end
recursion.depth_fn(lua_fn, depth - 1)
end
lua_fn(10)Here, we have a C function depth_fn and a Lua function lua_fn, both calling each other until the provided depth number reaches 0. Here in recursion.c, I have used the dispatch mechanism to dispatch from Lua C function to a normal C function, doing parameter check beforehand. Following this type of mechanism is oftentimes performant while writing modules with large function call depths. Now, after compiling the module with
gcc -std=c99 -pedantic -Wall -Wextra -fPIC -shared recursion.c -o recursion.soif we run the main.lua script with traditional Lua interpreter, we get the following error:
lua: main.lua:5: Depth reached 0!
stack traceback:
[C]: in function 'error'
main.lua:5: in function 'lua_fn'
[C]: in function 'recursion.depth_fn'
main.lua:8: in function 'lua_fn'
[C]: in function 'recursion.depth_fn'
main.lua:8: in function 'lua_fn'
[C]: in function 'recursion.depth_fn'
main.lua:8: in function 'lua_fn'
[C]: in function 'recursion.depth_fn'
main.lua:8: in function 'lua_fn'
[C]: in function 'recursion.depth_fn'
main.lua:8: in function 'lua_fn'
main.lua:11: in main chunk
[C]: in ?Which is not super useful, because we don't have line numbers in C call-frame. To improve the traceback, we can integrate Pallene Tracer into the module. It would take to use copy-pasting some boilerplate macros, using the macros in the functions and finally binding the functions with Pallene Tracer call-stack Upvalues.
recursion.c
#include <lua.h>
#include <lauxlib.h>
#define PT_IMPLEMENTATION
#include <ptracer.h>
/** =============== BOILERPLATES ================ **/
/* Here goes user specific macros when Pallene Tracer debug mode is active. */
#ifdef PT_DEBUG
#define MODULE_GET_FNSTACK \
pt_fnstack_t *fnstack = lua_touserdata(L, \
lua_upvalueindex(1))
#else
#define MODULE_GET_FNSTACK
#endif // PT_DEBUG
/* ---------------- LUA INTERFACE FUNCTIONS ---------------- */
#define MODULE_LUA_FRAMEENTER(fnptr) \
MODULE_GET_FNSTACK; \
PALLENE_TRACER_LUA_FRAMEENTER(L, fnstack, fnptr, \
lua_upvalueindex(2), _frame)
/* ---------------- LUA INTERFACE FUNCTIONS END ---------------- */
/* ---------------- FOR C INTERFACE FUNCTIONS ---------------- */
#define MODULE_C_FRAMEENTER() \
MODULE_GET_FNSTACK; \
PALLENE_TRACER_GENERIC_C_FRAMEENTER(fnstack, _frame)
#define MODULE_C_SETLINE() \
PALLENE_TRACER_GENERIC_C_SETLINE(fnstack)
#define MODULE_C_FRAMEEXIT() \
PALLENE_TRACER_FRAMEEXIT(fnstack)
/* ---------------- FOR C INTERFACE FUNCTIONS END ---------------- */
/** =============== BOILERPLATES END ================ **/
void depth_fn(lua_State *L, int depth) {
MODULE_C_FRAMEENTER();
lua_pushvalue(L, 1);
if(depth == 0)
lua_pushinteger(L, depth);
else lua_pushinteger(L, depth - 1);
/* Set line number to current active frame in the Pallene callstack and
call the function which is already in the Lua stack. */
MODULE_C_SETLINE();
lua_call(L, 1, 0);
MODULE_C_FRAMEEXIT();
}
int depth_fn_lua(lua_State *L) {
int top = lua_gettop(L);
MODULE_LUA_FRAMEENTER(depth_fn_lua);
/* Look at the macro definitions. */
if(luai_unlikely(top < 2))
luaL_error(L, "Expected atleast 2 parameters");
/* ---- `lua_fn` ---- */
if(luai_unlikely(lua_isfunction(L, 1) == 0))
luaL_error(L, "Expected the first parameter to be a function");
if(luai_unlikely(lua_isinteger(L, 2) == 0))
luaL_error(L, "Expected the second parameter to be an integer");
int depth = lua_tointeger(L, 2);
/* Dispatch. */
depth_fn(L, depth);
return 0;
}
int luaopen_recursion(lua_State *L) {
/* Our stack. */
pt_fnstack_t *fnstack = pallene_tracer_init(L);
lua_newtable(L);
/* One very good way to integrate our stack userdatum and finalizer
object is by using Lua upvalues. */
/* ---- depth_fn ---- */
lua_pushlightuserdata(L, fnstack);
/* `pallene_tracer_init` function pushes the frameexit finalizer to the stack. */
lua_pushvalue(L, -3);
lua_pushcclosure(L, depth_fn_lua, 2);
lua_setfield(L, -2, "depth_fn");
return 1;
}After compiling with,
gcc -DPT_DEBUG -std=c99 -pedantic -Wall -Wextra -fPIC -shared recursion.c -o recursion.soif we run main.lua against pt-lua, our Lua frontend, we get:
pt-lua: main.lua:5: Depth reached 0!
stack traceback:
C: in function 'error'
main.lua:5: in function 'lua_fn'
recursion.c:55: in function 'depth_fn'
main.lua:8: in function 'lua_fn'
recursion.c:55: in function 'depth_fn'
main.lua:8: in function 'lua_fn'
recursion.c:55: in function 'depth_fn'
main.lua:8: in function 'lua_fn'
recursion.c:55: in function 'depth_fn'
main.lua:8: in function 'lua_fn'
recursion.c:55: in function 'depth_fn'
main.lua:8: in function 'lua_fn'
main.lua:11: in <main>
C: in function '<?>'which includes line number information. It would also include all the missing traces if there is any.
It is highly encouraged to read through the mechanism section of the documentation.
Future Expectations
Even though we have put a lot of work in Pallene Tracer and it's adoption to Pallene, there are still a lot of work to be done here. As we aim to make Pallene Tracer reachable to broad Lua C library hackers, we have to come up with more generalized macros, making it more developer friendly.
Pallene Tracer is in very early stage of development, a lot of bugs may still be lingering around that we are unaware of. We aim to fix these bugs if appears and improve Pallene Tracer as time goes by.
Epilogue
I am really glad and proud that I got to work with a real-world open-source project learning a ton of things from my mentor. I also got the chance to collaborate with my mentors, continuously discussing and implementing code and working side-by-side to make a difference in the community.
LabLua is an amazing organization. I would recommend my organization to anyone who is starting out GSoC. I would also like to do mentorship from behalf of LabLua someday.
Finally, at the end of the day, I absolutely adored my GSoC journey ❤️. Peace 😇.