OpenArchFlow
← Back to Blog

Run AWS Glue jobs natively with OpenArchFlow and MiniStack

Published on 2026-11-02by Rafael Sales

Designing a data pipeline on paper is easy. Knowing that the PySpark job actually runs is another story — and until now that usually meant an AWS account, IAM roles and a bill. With OpenArchFlow v0.13.0 and MiniStack you can design the pipeline, deploy it and run the Glue job on your own machine.

How it works

MiniStack is a free, open-source AWS emulator that exposes 60+ services on a single port. Most of them are emulated in-process, but some run real software: RDS starts real Postgres, ECS starts real containers, and Glue executes jobs for real (catalog, crawlers, jobs, triggers and workflows). Athena runs real SQL over your S3 files through DuckDB when the :full image is used.

For Glue, MiniStack launches your job inside a Docker container. OpenArchFlow ships a container image built on top of the official amazon/aws-glue-libs:glue_libs_4.0.0_image_01 image (AWS Glue 4.0), with a small log forwarder that sends the JVM and Spark output to MiniStack's local CloudWatch Logs endpoint.

What OpenArchFlow adds: Glue Studio

Once a Glue Catalog node is deployed, open Glue Studio from the node. It has five tabs:

  • Catalog and Tables — browse the databases and tables of the deployed catalog.
  • Jobs — manage your Glue jobs, with a PySpark editor.
  • Runs — start a run and turn on Live to stream the Spark and JVM logs into the dashboard.
  • Athena Query — run Athena SQL over the files in your local S3.

Try it

You need Docker, with the Docker socket available to MiniStack so it can start the Glue container. From the OpenArchFlow repository:

docker compose -f docker-compose.ministack-aws-glue-full.yml up -d

The compose file pulls the Glue image from GHCR before MiniStack starts, enables persistent S3 and adds a small nginx sidecar on port 4567 that sets CORS headers, so the hosted app can talk to your local MiniStack. Prefer plain Docker?

docker run -d -p 4566:4566 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e GLUE_DOCKER_IMAGE=ghcr.io/dmux/openarchflow/ministack_glue_libs_4.0.0_image_01:latest \
  -e S3_PERSIST=1 \
  ministackorg/ministack:full

Then, in app.openarchflow.cloud:

  1. Describe a pipeline — for example "A data lake: an S3 raw bucket, an AWS Glue catalog and PySpark ETL job, an S3 curated bucket, queried with Athena".
  2. Click the Rocket icon, set the endpoint to http://localhost:4567 (or http://localhost:4566 if you run OpenArchFlow locally) and click Deploy All.
  3. Open Glue Studio, start a run and watch the logs.

Good to know

  • The :full MiniStack image is required for the Athena Query tab. The default image has no DuckDB engine and returns static mocked data.
  • Terraform export covers S3, the Glue catalog database and the Athena workgroup. Glue jobs themselves are authored in Glue Studio.
  • One-click local deploy currently covers 14 service types: S3, Lambda, DynamoDB, SQS, SNS, EventBridge, Kinesis, API Gateway, IAM, KMS, Secrets Manager, SSM, CloudWatch Logs and Glue.

Learn more

See the full list of use cases and reference architectures, the Getting Started guide and the OpenArchFlow repository. If you hit a problem with the Glue image, the README has a troubleshooting section — and issues are welcome.