Parallel Perl – Autoparallelizing interpreter with JIT

Prologue Since last time...

GPW 2016 · Nuremberg · "KI: Wie testet man ein Hirn?"

That was 10 years ago. A lot happened.

"Patience you must have, my young Padawan"

This prologue is not about Perl — but it is necessary context.

The classified stuff

Some things are better left unspoken.

What I can show you today is roughly half of what happened.

→ moving on

PV & Energy Systems

Designed, built, automated

Victron ecosystem  ·  Perl monitoring & control

PV & Energy Systems

Villa-A

A technocrat's house — 2050s standard

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Construction

Villa-A — Systems

• Erdwärmekörbe, WP — heizen, kühlen (passiv)
• "Klimamatten" — Deckenheizung, Kühlung (BEKA → Berlin)
• Luftbrunnen, KWL
• Privatkraftwerk (Offgrid)
• Wasserwerk / Wasseraufbereitung
• Digitale Lichtsteuerung, DALI "Full-Stack DC" (DALI-SELV)
• ... und haste nicht gesehen

...and it all needs to be controlled.

A house this complex needs an automation system

→

02 WHIP

Witty House Infrastructure Processor

PV — Perl Integration

First tools: Victron Modbus + ECS BMS — all in Perl

$ ecs_bms_tool -range 1-16          # query all battery modules
$ ecs_bms_tool -get cell_voltage -get cell_temperature
$ ecs_bms_tool -otype json           # JSON for pipeline integration

$ Wmodbus discover 192.168.2.0/24    # find Modbus devices on network
$ Wmodbus --host 192.168.2.201 --unit 2 read holding 0-10
$ Wmodbus --host 192.168.2.201 --profile vents-dbe900l monitor

ecs_bms_tool — ECS LiPro BMS management (SoC, cell voltage, balancing)

Scripts work. But a house is more than solar panels.

Looking for a Smarthome

Preferably in Perl, obviously.

The FHEM Experience

svn.fhem.de/trac/browser/trunk/fhem

(FHEM people: no offense. Well, maybe a little.)

Nice idea. Wrong execution.

I felt there had to be something better.

Birth of a Node

Birth of a Node

Birth of a Node

Why CAN?

Birth of a Hub

So you have 20 nodes — now what?

RasPi 4B/5 · 2-ch CAN HAT · Relay Board · DIN Rail Mount · CAN/IP & CAN/CAN Gateway

WHIP Architecture

Nodes — STM32 MCUs · FreeRTOS · 1MBit CAN bus · Autonomous C / Embedded

Hubs — RasPi · CAN/IP gateway · Hub Aggregation · Protocol bridges · Mojolicious Perl

Server — Orchestration · External connectivity Perl

Higher layers are always a supplement, never a requirement.

Nodes — STM32 + FreeRTOS

Ganglion

Ganglion = GANG of Lightweight I/O Nodes — insect-brain model.

Ganglion — In Action

DEF LightTimeout = 300       # 5 minutes

# Motion detected: light on, start timer
IF motion:detected THEN lights:on; SET $T_0 = LightTimeout

# Timer expired: light off
IF !$T_0 THEN lights:off

# Cross-node: kitchen smoke → alarm everywhere
DEF Kitchen = 42
IF Kitchen:smoke:detected THEN buzzer:alarm(1)

Hubs — a Pantheon

Specialized RasPi hubs. Named by function, not by accident.

No hub is a single point of failure. Each domain runs independently.

SELV-DALI — Lighting without mains

Inverters fail? Lights stay on — they bypass AC entirely.

Switch next to the bathtub? No problem. No electrician needed.

WHIP — Protocols & Integrations

All protocol handlers in Perl · Mojolicious async I/O

WHIP — In production

Invisible when it works. Competent when it matters. Built for decades, not warranties.

03 AI does Perl

Using AI to write Perl — the practical reality

AI contributed...

"Small" side projects

The CPAN Situation

SNMP & MIBs — quick primer

SNMP::MIB::Compiler

2 days with AI · CPAN module was 93% working · targeted fixes, no rewrite

4740 MIBs · 301 fixes · 52 fewer failures than Go

github.com/petajoulecorp/SNMP-MIB-Compiler

Grpc::FFI

Wanted gRPC in Perl. Everything on CPAN: dormant, dead, or broken.

So we built it. From scratch. FFI::Platypus bindings to the gRPC C API.

And yet — this was a prelude

What if FFI wasn't just a tailored library connector...

Navigator / Orchestrator

AI doesn't do this on its own.

gRPC example: I decided "start with UUID, then SQLite, then gRPC"

Without the navigator — the AI builds impressive things that go nowhere.

So... AI is quite good at Coding.

But how far can this actually go?

04 AI does Perl

Turning the predicate around.

pperl

PetaPerl  /  ParallelPerl

A Perl 5 interpreter — designed by humans.

Written in Rust — by many AI agents.

Serious — no toy or academic exercise.

pperl

PetaPerl  /  ParallelPerl

A Perl 5 interpreter Platform — designed by humans.

Written in Rust — by many AI agents.

Serious — no toy or academic exercise.

pperl — Not the first attempt

Common failure mode: underestimating Perl 5's complexity

pperl — Scope

pperl — Status

22,000+

tests total

~61–400 failures — give or take

Performance: good, bad and ugly

Quotes from the AI

13095 pass (+25 from previous 13070), 31 fail (down from 46!). The File::Path native implementation not only works, it unblocked 15 previously-failing tests that depended on File::Path. Zero regressions.

pperl — Benchmarks

Native Rust implementations — not XS, not C

pperl — Beyond Perl5

Maximum compatibility. But more.

Autoparallelization

Powered by Rayon — Rust's data-parallelism library

# This just works. In parallel.
my @results = map { expensive_computation($_) } @large_list;

# No threads. No MCE. No forks.
# pperl detects safe loops → Rayon handles the rest.

--parallel flag · list ≥ 1000 items · no shared mutation

JIT Compilation

Just-In-Time — compile to machine code while running

Cranelift — the compiler backend behind Wasmtime and Rust's alternative codegen.

# pperl detects this as a hot loop pattern
my $sum = 0;
for my $i (1 .. 1_000_000) {
    $sum += $i;
}
# → Cranelift compiles to native machine code

JIT — First Win

Inner loop JIT — single hot loop compiled to native code

Good. But only the innermost loop is compiled. What about nested loops?

JIT — Full Nested

All 3 loop levels compiled as one native function

200 million escape iterations of float arithmetic.

Perl. With JIT. That's a sentence nobody expected.

Autoparallel JIT — Full Win

JIT + Rayon: compile to native, then split across cores

JIT alone: 76×. Adding 8 threads: another ~7× on top.

Demo Time!

✓ Auto-FFI

No XS. No Inline::C. No compilation. Just call C.

# Layer 0 — Raw: any library, you provide type signatures
use Peta::FFI qw(dlopen call);
my $lib = dlopen("libz.so.1");
my $ver = call($lib, "zlibVersion", "()p");
say "zlib: $ver";    # 1.3.1

# Layer 1 — Pre-baked: curated signatures, zero ceremony
use Peta::FFI::Libc qw(getpid strlen strerror uname);
say strlen("hello");           # 5
my @info = uname();
say "$info[0] $info[2]";      # Linux 6.18.6-arch1-1

Pack-style type codes: (p)L = strlen(const char*) → size_t

Auto-FFI — Details

Powered by libffi — any signature works, no pre-generated stubs

# Layer 2 — What's on this system?
use Peta::FFI qw(scan dlopen call);
my $libs = scan();
say scalar(keys %$libs), " libraries found";
if (exists $libs->{"libz.so.1"}) {
    my $z = dlopen("libz.so.1");
    say "zlib: ", call($z, "zlibVersion", "()p");
}

Libc: ~30 functions (process, strings, env, math, file, time)

✓ Bytecode Cache (.plc)

Like Python's .pyc — but for Perl. Opt-in.

# Default: no caching (safe for development)
$ pperl script.pl

# Enable: compile once, load from cache on subsequent runs
$ pperl --cache script.pl

# Invalidate all caches
$ pperl --flush

Bytecode Cache — Details

Storable-model: bincode deserializes directly to final runtime types. Zero intermediate conversion.

Net module-loading cost: 33–37% faster with cache. Biggest win on fallback modules. Native Rust modules already near-zero cost.

SHA-256 keyed · mtime + version validation · aggressive format versioning

✓ Daemonize

Emacs-style daemon/client model

$ pperl --daemon script.pl   # compile, warm up, listen
$ pperl --client script.pl   # connect → fork → run → respond
$ pperl --stop   script.pl   # clean shutdown

First run: parse → codegen → execute (warm-up) → listen

fork() gives each client a fresh address space

Daemonize — Details

Eliminates both startup costs: process creation (~3-4ms) + module compilation (0-15ms)

Unix domain socket · JSON wire protocol · copy-on-write pages via fork()

Daemonize — Prior Art

All prior solutions: same interpreter across requests — state leakage by design

Future pperl

When to use pperl

Rule of thumb: the longer and more complex the script,

Correctness Case Study

How serious is "maximum compatibility"?

if (SvGMAGICAL(ofs) || SvOK(ofs))

if (PL_ors_sv && SvOK(PL_ors_sv))

// Same mask for both — SVS_GMG included for ORS. Wrong.
if flags & (SVF_IOK | SVF_NOK | SVF_POK | SVF_ROK | SVS_GMG) != 0

I have been doing

Get it here:

perl.petamem.com

Danke.

Richard Jelinek  ·  rj@petamem.com

One more thing.

psh

An interactive Perl shell

ls "-la";                    # it's just a sub call
cd "/tmp";                   # chdir wrapper
ps "aux";                    # system command

# But you're already in a scripting language:
for my $f (glob("*.log")) {
    if (-M $f > 7) {
        rm $f;
        say "cleaned $f";
    }
}

ps() | grep { $_->{mem} > 100_000 }
     | sort { $b->{cpu} <=> $a->{cpu} };

PowerShell's philosophy · Perl's text power · pperl's JIT speed