noreasontopanic

Querying Billions of GitHub Events Using Modal and DuckDB (Part 1: Ingesting Data)

Patrick DeVivo — Fri, 29 Aug 2025 12:33:36 GMT

There have been two technologies I’ve been really excited about recently - and will jump at any excuse to use them. One is DuckDB, my new go-to for most local data processing, especially with the recent addition of a UI mode. DuckDB makes it very convenient to query fairly large (multiple GB) datasets on disk - CSVs, JSON files, etc. without having to write much boilerplate code. It’s also great for querying small JSON, like the output from an API request (or many requests).

The second is Modal, an infrastructure platform that truly feels like magic, which I’ll focus on more throughout this series.

This post and the following will demonstrate how to use Modal.com and DuckDB to ingest, process, and query huge amounts of public GitHub data (several terabytes of compressed JSON). It’s meant to serve as an example and introduction to these tools, and to show how well they work together!

This first post (Part 1) will cover how to use Modal’s highly concurrent, serverless infrastructure to download lots (50K+ files, 3.5 TB) of public GitHub data and store it on a Modal Volume - in about 15 minutes.

The GH Archive

Anyone who knows me, knows that I have a soft spot for querying git data with SQL - so when I came across the GH Archive again, I couldn’t help wanting to poke around to see what kind of queries I could run.

You can read more on the website, but the quick background is that it’s an open-source project that continuously archives public GitHub events (15+ types, such as git pushes, new issues, new PRs, stars, etc.), for all public GitHub repos, going back to 2011. That’s a lot of events!

The events are stored in gzip compressed JSON files and available for download at data.gharchive.org, partitioned by the hour - so each data file stores 1 hour of public GitHub event records. The URL format to access a data file looks like this:

GH Archive data is also available to query in BigQuery - but that’s no fun, let’s roll our own setup!

Downloading Lots of JSON

It’s great that we can access these files, but how can we actually query and analyze their content?

DuckDB can query remote data sources over HTTP(s) using the httpfs extension, and this is actually very cool. In fact, combined with the ability to decompress and handle line-delimited JSON on the fly, you could just run a simple query like so:

SELECT * FROM 'https://data.gharchive.org/2015-01-01-15.json.gz';

To treat that single hour of event data as a table. The file is “downloaded” as the query executes (it’s also decompressed and each line of JSON is mapped into a schema, inferred by sampling the JSON attributes). The output will look like this:

However, given that we want to execute queries across a lot of these files (let’s say over 1 year: 365 days * 24 hours = 8760 files) we’re probably going to want to download these files ahead of time. Storing them on disk and pre-processing them will certainly lead to faster querying - we don’t want to be waiting for thousands of network requests as our queries execute! Not only would this be really slow, it wouldn’t be very reliable either.

I’ll skip ahead and show you where we end up, which is every json.gz file from 2020 to now (August, 2025) stored on disk (a Modal Volume), in a folder structure organized by year, month, then day, where each file is named from {0-23}.json.gz, indicating the hour of the day. You can see that it takes up quite a bit of space on a Modal Volume: ~3.5 TB total. You can also see the slow growth in data size by year (corresponding to the overall user/activity growth of GitHub).

Using Modal’s debug shell with an attached Volume to inspect the disk usage of all the GitHub events we’ve stored.

All of this data was downloaded in ~15m using Modal’s highly concurrent infrastructure. The files are stored in a Modal Volume, a filesystem primitive which can be conveniently attached to other functions (and even Modal Notebooks!).

Here’s what the execution of the “parent” function (more on this in a moment) looks like, showing the timeline of the full download:

And here’s a screenshot of the Modal monitoring UI, showing resource utilization during the download run. Note the 300+ containers that were spun up and executed concurrently.

Parallelizing Downloads

Because GH Archive breaks out data into 1 hour chunks (1 file per 1 hour), we’re able to massively parallelize the overall download process - and this is where Modal can really shine.

Instead of spinning up a single big VM (or bare-metal server), or a Kubernetes cluster running a bunch of workers (and writing lots of YAML along the way…), we just decorate some Python code and instruct Modal to do a large fan-out for us, even handling retries with exponential back-off (important for dealing with network requests)!

This is where Modal really starts to feel like magic - and it’s probably easier to show rather then tell, but here’s a basic overview before I show you the code:

We define a function called download_file decorated with @app.function, indicating to Modal that it can be executed remotely. This function will handle the download of a single data file (1 hour of events). It will download the file into /tmp (attached disk), and then copy it into the Modal Volume, according to the recommendations here. This is the function that will execute many times in parallel, across hundreds of containers (for different inputs of year/month/day/hour combinations).
A “parent” function called download_range is responsible for generating the inputs for the calls to download_file that represent the time range we want to download data for. This does some calendar math to produce the set of year/month/day/hour combinations. Then, it uses .starmap(…) to have Modal execute download_file for all of those inputs in parallel!
Finally, a main function decorated with @app.local_entrypoint() calls download_range.remote(date(2020, 1, 1)), this tells Modal to start the download process for all files from Jan 1, 2020 until today.

Here’s some code!

\n \n

\n\n \n

\n\n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n

	import modal
	from datetime import date
	from .app import app, gharchive, GHARCHIVE_DATA_PATH
	\n
	\n
	@app.function(
	volumes={GHARCHIVE_DATA_PATH: gharchive},
	timeout=36000,
	retries=modal.Retries(
	max_retries=8,
	backoff_coefficient=2,
	initial_delay=1,
	max_delay=30,
	),
	ephemeral_disk=800 * 1024,
	)
	@modal.concurrent(max_inputs=12)
	def download_file(year: int, month: int, day: int, hour: int) -> tuple[str, float, int]:
	import os, time, tempfile, shutil, pycurl, random, io
	\n
	# Tiny jitter to avoid synchronized bursts (helps with WAF/rate shaping)
	time.sleep(random.uniform(0.01, 0.05))
	\n
	# URLs & paths
	url = f"https://data.gharchive.org/{year}-{month:02d}-{day:02d}-{hour}.json.gz"
	vol_path = f"{GHARCHIVE_DATA_PATH}/{year}/{month:02d}/{day:02d}/{hour}.json.gz"
	\n
	# Stage to a temporary file in /tmp which is on the attached SSD, as recommended in Modal docs: https://modal.com/docs/guide/dataset-ingestion
	tmp_dir = f"/tmp/{year}/{month:02d}/{day:02d}"
	os.makedirs(tmp_dir, exist_ok=True)
	fd, tmp_path = tempfile.mkstemp(
	dir=tmp_dir, prefix=f"{year}-{month:02d}-{day:02d}-{hour}.json.gz."
	)
	os.close(fd)
	\n
	# Configure curl
	c = pycurl.Curl()
	c.setopt(c.URL, url)
	c.setopt(c.FOLLOWLOCATION, 1)
	c.setopt(c.HTTP_VERSION, pycurl.CURL_HTTP_VERSION_2TLS) # allow HTTP/2
	c.setopt(
	c.USERAGENT, "gharchive-downloader/1.0 (contact: patrick.devivo@gmail.com)"
	) # may help with rate limiting
	c.setopt(c.NOSIGNAL, 1)
	c.setopt(c.CONNECTTIMEOUT, 10)
	c.setopt(
	c.TIMEOUT, 60 * 5
	) # total timeout - most downloads should occur within this
	c.setopt(c.LOW_SPEED_LIMIT, 20_000) # bytes/sec
	c.setopt(c.LOW_SPEED_TIME, 20) # if below limit for 20s -> timeout
	\n
	hdr_buf = io.BytesIO()
	c.setopt(c.HEADERFUNCTION, hdr_buf.write)
	\n
	# Execute the download → /tmp, then fsync for durability
	with open(tmp_path, "wb", buffering=1024 * 1024) as f:
	c.setopt(c.WRITEDATA, f)
	c.perform()
	f.flush()
	os.fsync(f.fileno())
	\n
	# Telemetry from curl
	status = int(c.getinfo(pycurl.RESPONSE_CODE))
	size_b = int(c.getinfo(pycurl.SIZE_DOWNLOAD))
	total_s = float(c.getinfo(pycurl.TOTAL_TIME))
	mbps = float(c.getinfo(pycurl.SPEED_DOWNLOAD)) / (1024 * 1024)
	ttfb_s = float(c.getinfo(pycurl.STARTTRANSFER_TIME))
	headers_s = hdr_buf.getvalue().decode("latin1", errors="replace")
	c.close()
	\n
	# Handle non-200s
	if status != 200:
	# Clean partial temp file
	try:
	os.remove(tmp_path)
	except FileNotFoundError:
	pass
	\n
	# Sometimes there's missing hours, but raise on 404 anyways to retry and report
	if status == 404:
	time.sleep(random.uniform(0.5, 2.0))
	raise RuntimeError(f"HTTP 404 for {url} (retryable)")
	\n
	# Transient / retry-worthy codes → raise so Modal retries
	retry_statuses = {403, 429, 500, 502, 503, 504}
	if status in retry_statuses:
	first_hdr = headers_s.splitlines()[0] if headers_s else f"HTTP {status}"
	raise RuntimeError(f"HTTP {status} for {url} ({first_hdr})")
	\n
	# Other client errors: surface as failures (Modal will retry)
	raise RuntimeError(f"HTTP {status} for {url}")
	\n
	# Publish into the Modal volume atomically
	vol_dir = os.path.dirname(vol_path)
	os.makedirs(vol_dir, exist_ok=True)
	tmp_dest = vol_path + ".part"
	with (
	open(tmp_path, "rb", buffering=1024 * 1024) as rf,
	open(tmp_dest, "wb", buffering=1024 * 1024) as wf,
	):
	shutil.copyfileobj(rf, wf, length=8 * 1024 * 1024)
	wf.flush()
	os.fsync(wf.fileno())
	os.replace(tmp_dest, vol_path) # atomic within the same FS
	dfd = os.open(vol_dir, os.O_DIRECTORY)
	try:
	os.fsync(dfd)
	finally:
	os.close(dfd)
	\n
	# Clean local temp
	try:
	os.remove(tmp_path)
	except FileNotFoundError:
	pass
	\n
	print(
	f"{os.path.basename(vol_path)} — {size_b / 1_048_576:.1f} MB "
	f"in {total_s:.2f}s @ {mbps:.2f} MB/s (TTFB {ttfb_s:.2f}s) → {vol_path}"
	)
	\n
	return vol_path, total_s, size_b
	\n
	\n
	@app.function(timeout=36000, volumes={GHARCHIVE_DATA_PATH: gharchive})
	def download_range(start: date, end: date = date.today()):
	from datetime import timedelta
	import time
	\n
	# Build hour-level inputs
	inputs = []
	delta = end - start
	for d in range(delta.days + 1):
	day = start + timedelta(days=d)
	for hour in range(24):
	inputs.append((day.year, day.month, day.day, hour))
	\n
	print(
	f"Downloading events from {start} to {end} — {len(inputs)} files over {delta.days + 1} days"
	)
	\n
	t0 = time.time()
	total_size = 0
	ok = 0
	failures = []
	\n
	for result in download_file.starmap(
	inputs,
	return_exceptions=True,
	wrap_returned_exceptions=False,
	order_outputs=False,
	):
	if isinstance(result, Exception):
	failures.append(result)
	else:
	_, _, sz = result
	total_size += sz
	ok += 1
	\n
	elapsed = time.time() - t0
	total_gb = total_size / (1024**3)
	agg_gbps = (total_gb / elapsed) if elapsed > 0 else 0.0
	print(
	f"Done: {ok}/{len(inputs)} files in {elapsed:.1f}s — {total_gb:.2f} GB total, avg {agg_gbps:.2f} GB/s"
	)
	\n
	if failures:
	print(f"Encountered {len(failures)} failures (Modal handled retries): ")
	for f in failures:
	print(f"[FAIL] {f!s}")
	\n
	gharchive.commit()
	\n
	\n
	@app.local_entrypoint()
	def main():
	download_range.remote(date(2020, 1, 1))

\n\n\n

\n\n

\n view raw\n \n download.py\n \n hosted with ❤ by GitHub\n

\n","stylesheet":"https://github.githubassets.com/assets/gist-embed-59543e005c9c.css"}" data-component-name="GitgistToDOM">

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters

	import modal
	from datetime import date
	from .app import app, gharchive, GHARCHIVE_DATA_PATH


	@app.function(
	volumes={GHARCHIVE_DATA_PATH: gharchive},
	timeout=36000,
	retries=modal.Retries(
	max_retries=8,
	backoff_coefficient=2,
	initial_delay=1,
	max_delay=30,
	),
	ephemeral_disk=800 * 1024,
	)
	@modal.concurrent(max_inputs=12)
	def download_file(year: int, month: int, day: int, hour: int) -> tuple[str, float, int]:
	import os, time, tempfile, shutil, pycurl, random, io

	# Tiny jitter to avoid synchronized bursts (helps with WAF/rate shaping)
	time.sleep(random.uniform(0.01, 0.05))

	# URLs & paths
	url = f"https://data.gharchive.org/{year}-{month:02d}-{day:02d}-{hour}.json.gz"
	vol_path = f"{GHARCHIVE_DATA_PATH}/{year}/{month:02d}/{day:02d}/{hour}.json.gz"

	# Stage to a temporary file in /tmp which is on the attached SSD, as recommended in Modal docs: https://modal.com/docs/guide/dataset-ingestion
	tmp_dir = f"/tmp/{year}/{month:02d}/{day:02d}"
	os.makedirs(tmp_dir, exist_ok=True)
	fd, tmp_path = tempfile.mkstemp(
	dir=tmp_dir, prefix=f"{year}-{month:02d}-{day:02d}-{hour}.json.gz."
	)
	os.close(fd)

	# Configure curl
	c = pycurl.Curl()
	c.setopt(c.URL, url)
	c.setopt(c.FOLLOWLOCATION, 1)
	c.setopt(c.HTTP_VERSION, pycurl.CURL_HTTP_VERSION_2TLS) # allow HTTP/2
	c.setopt(
	c.USERAGENT, "gharchive-downloader/1.0 (contact: patrick.devivo@gmail.com)"
	) # may help with rate limiting
	c.setopt(c.NOSIGNAL, 1)
	c.setopt(c.CONNECTTIMEOUT, 10)
	c.setopt(
	c.TIMEOUT, 60 * 5
	) # total timeout - most downloads should occur within this
	c.setopt(c.LOW_SPEED_LIMIT, 20_000) # bytes/sec
	c.setopt(c.LOW_SPEED_TIME, 20) # if below limit for 20s -> timeout

	hdr_buf = io.BytesIO()
	c.setopt(c.HEADERFUNCTION, hdr_buf.write)

	# Execute the download → /tmp, then fsync for durability
	with open(tmp_path, "wb", buffering=1024 * 1024) as f:
	c.setopt(c.WRITEDATA, f)
	c.perform()
	f.flush()
	os.fsync(f.fileno())

	# Telemetry from curl
	status = int(c.getinfo(pycurl.RESPONSE_CODE))
	size_b = int(c.getinfo(pycurl.SIZE_DOWNLOAD))
	total_s = float(c.getinfo(pycurl.TOTAL_TIME))
	mbps = float(c.getinfo(pycurl.SPEED_DOWNLOAD)) / (1024 * 1024)
	ttfb_s = float(c.getinfo(pycurl.STARTTRANSFER_TIME))
	headers_s = hdr_buf.getvalue().decode("latin1", errors="replace")
	c.close()

	# Handle non-200s
	if status != 200:
	# Clean partial temp file
	try:
	os.remove(tmp_path)
	except FileNotFoundError:
	pass

	# Sometimes there's missing hours, but raise on 404 anyways to retry and report
	if status == 404:
	time.sleep(random.uniform(0.5, 2.0))
	raise RuntimeError(f"HTTP 404 for {url} (retryable)")

	# Transient / retry-worthy codes → raise so Modal retries
	retry_statuses = {403, 429, 500, 502, 503, 504}
	if status in retry_statuses:
	first_hdr = headers_s.splitlines()[0] if headers_s else f"HTTP {status}"
	raise RuntimeError(f"HTTP {status} for {url} ({first_hdr})")

	# Other client errors: surface as failures (Modal will retry)
	raise RuntimeError(f"HTTP {status} for {url}")

	# Publish into the Modal volume atomically
	vol_dir = os.path.dirname(vol_path)
	os.makedirs(vol_dir, exist_ok=True)
	tmp_dest = vol_path + ".part"
	with (
	open(tmp_path, "rb", buffering=1024 * 1024) as rf,
	open(tmp_dest, "wb", buffering=1024 * 1024) as wf,
	):
	shutil.copyfileobj(rf, wf, length=8 * 1024 * 1024)
	wf.flush()
	os.fsync(wf.fileno())
	os.replace(tmp_dest, vol_path) # atomic within the same FS
	dfd = os.open(vol_dir, os.O_DIRECTORY)
	try:
	os.fsync(dfd)
	finally:
	os.close(dfd)

	# Clean local temp
	try:
	os.remove(tmp_path)
	except FileNotFoundError:
	pass

	print(
	f"{os.path.basename(vol_path)} — {size_b / 1_048_576:.1f} MB "
	f"in {total_s:.2f}s @ {mbps:.2f} MB/s (TTFB {ttfb_s:.2f}s) → {vol_path}"
	)

	return vol_path, total_s, size_b


	@app.function(timeout=36000, volumes={GHARCHIVE_DATA_PATH: gharchive})
	def download_range(start: date, end: date = date.today()):
	from datetime import timedelta
	import time

	# Build hour-level inputs
	inputs = []
	delta = end - start
	for d in range(delta.days + 1):
	day = start + timedelta(days=d)
	for hour in range(24):
	inputs.append((day.year, day.month, day.day, hour))

	print(
	f"Downloading events from {start} to {end} — {len(inputs)} files over {delta.days + 1} days"
	)

	t0 = time.time()
	total_size = 0
	ok = 0
	failures = []

	for result in download_file.starmap(
	inputs,
	return_exceptions=True,
	wrap_returned_exceptions=False,
	order_outputs=False,
	):
	if isinstance(result, Exception):
	failures.append(result)
	else:
	_, _, sz = result
	total_size += sz
	ok += 1

	elapsed = time.time() - t0
	total_gb = total_size / (1024**3)
	agg_gbps = (total_gb / elapsed) if elapsed > 0 else 0.0
	print(
	f"Done: {ok}/{len(inputs)} files in {elapsed:.1f}s — {total_gb:.2f} GB total, avg {agg_gbps:.2f} GB/s"
	)

	if failures:
	print(f"Encountered {len(failures)} failures (Modal handled retries): ")
	for f in failures:
	print(f"[FAIL] {f!s}")

	gharchive.commit()


	@app.local_entrypoint()
	def main():
	download_range.remote(date(2020, 1, 1))

view raw download.py hosted with ❤ by GitHub

You can see there’s actually quite a bit going on, even though it’s a single file with <200 LOC:

We’re using pycurl, Python bindings to cURL, which has been benchmarked to be faster than requests
- We configure it to “fail fast” on flaky downloads, allowing Modal to retry with an exponential back-off. This was key to improving overall performance and reliability - sometimes a download will get “stuck” or a connection gets reset. So instead of waiting out a slow download stream, we just end it quickly and try again (up to 8 times).
We handle and report 404s - it turns out there are some data files (hour chunks) missing from the GH Archive dataset. We retry these downloads, but if they are truly missing, we just report the 404s at the end of the run.
We track download speed, errors, total time, etc. and log as the downloads execute and as a summary at the end of the run.
We use @modal.concurrent(max_inputs=12) to allow for concurrency within a container - Modal will run our download_file call up to 12 times at once in a single container. This allows a single container to do more work, and is a useful knob for tuning performance trade-offs.

Here’s what the output of this code looks like (just the log lines)

And that’s pretty much it! We’re now able to reliably and quickly download data from GH Archive across arbitrary date ranges. In a future post I’ll get this code organized and in a public repo, and show how it can be processed so that DuckDB can query it. Stay tuned!

Query Source Code with Tree-sitter

Patrick DeVivo — Fri, 02 Jul 2021 20:45:20 GMT

Tree-sitter is a really cool project. It’s primarily designed for code syntax-highlighting use cases (in editors and IDEs), but it also exposes a Query API for selecting portions of a parsed syntax tree, using an S-Expression based query syntax.

Tree-sitter is a parser generator tool and an incremental parsing library.

TreeQuery is a new CLI that makes it easier to run Tree-sitter queries against local source code files. It installs as tq and looks something like this:

> tq -q some_file.go "(function_declaration name: (identifier) @method_name)"

init
handleErr
someFunc
main

(prints out all the function names in a go file).

I think this is pretty cool. We’re still exploring use-cases and getting a feel for what’s possible to query, go ahead and give it a try! It’s still rough around the edges and more language support will be added soon. See the open issues for a sense of the roadmap.

It will soon be integrated into askgit.

What Comes After Serverless: How About Codeless?

Patrick DeVivo — Wed, 31 Mar 2021 22:05:05 GMT

Yes, this joke has been made before, but I’m serious. In the same way that serverless lets backend developers “forget” about servers, what if “codeless” did the same, but for your application code.

In other words, what if API requests to your service included the code that the caller wants to be executed on the server?

Your backend could essentially become a “runtime” for client-provided code. GraphQL APIs in some ways approach this - the backend is an “execution environment” for a (well defined and structured) GraphQL query. What if the “query language” became another programming language instead?

Example

A simple example is fairly easy to implement in Deno, which has built-in sandboxing features.

\n \n\n

	import { listenAndServe } from "https://deno.land/std@0.91.0/http/server.ts"
	import { Status } from "https://deno.land/std@0.91.0/http/http_status.ts"
	\n
	const HTTP_PORT = 8080;
	const options = { hostname: "0.0.0.0", port: HTTP_PORT }
	console.log(`HTTP server running on localhost:${HTTP_PORT}`)
	\n
	listenAndServe(options, (request) => {
	if (request.method !== "GET") {
	request.respond({ status: Status.MethodNotAllowed, body: "must GET" })
	return
	}
	\n
	const start = new Date()
	const url = `https://${request.url.substring(1)}`
	let completed = false
	\n
	const p = Deno.run({
	cmd: [ "deno", "run", url ],
	stdout: "piped",
	stderr: "null"
	});
	\n
	// pipe the stdout of the process to the http response
	request.respond({ body: p.stdout })
	\n
	// whenever the process exits, mark it as done
	p.status().finally(() => {
	completed = true
	let elapsed = (new Date()).getTime() - start.getTime()
	console.log({ url, time: `${elapsed}ms` })
	})
	\n
	// once the request is done (or canceled)
	Deno.readAll(request.r).then(async () => {
	// if the process hasn't completed, end it
	if (!completed) p.kill(2)
	})
	\n
	// don't let requests run indefinitely
	setTimeout(() => {
	if (!completed) p.kill(2)
	}, 10*1000)
	});

\n\n\n

\n\n

\n view raw\n main.ts\n hosted with ❤ by GitHub\n

\n","stylesheet":"https://github.githubassets.com/assets/gist-embed-33b98015caf26cfcbee6ce1a5d1fc768.css"}" data-component-name="GitgistToDOM">

      
        import { listenAndServe } from "https://deno.land/std@0.91.0/http/server.ts"
      
        import { Status } from "https://deno.land/std@0.91.0/http/http_status.ts"
      
        const HTTP_PORT = 8080;
      
        const options = { hostname: "0.0.0.0", port: HTTP_PORT }
      
        console.log(`HTTP server running on localhost:${HTTP_PORT}`)
      
        listenAndServe(options, (request) => {
      
          if (request.method !== "GET") {
      
            request.respond({ status: Status.MethodNotAllowed, body: "must GET" })
      
            return
      
          }
      
          const start = new Date()
      
          const url = `https://${request.url.substring(1)}`
      
          let completed = false
      
          const p = Deno.run({
      
            cmd: [ "deno", "run", url ],
      
            stdout: "piped",
      
            stderr: "null"
      
          });
      
          // pipe the stdout of the process to the http response
      
          request.respond({ body: p.stdout })
      
          // whenever the process exits, mark it as done
      
          p.status().finally(() => {
      
            completed = true
      
            let elapsed = (new Date()).getTime() - start.getTime()
      
            console.log({ url, time: `${elapsed}ms` })
      
          })
      
          // once the request is done (or canceled)
      
          Deno.readAll(request.r).then(async () => {
      
            // if the process hasn't completed, end it
      
            if (!completed) p.kill(2)
      
          })
      
          // don't let requests run indefinitely
      
          setTimeout(() => {
      
            if (!completed) p.kill(2)
      
          }, 10*1000)
      
        });

view raw main.ts hosted with ❤ by GitHub

This is just an example to illustrate the point, use at your own risk!

deno run --allow-net --allow-run --unstable main.ts

Will start an HTTP server that responds to GET requests, parses the path, and runs a Deno subprocess (deno run with no additional permissions) pointed at the remote code specified by the request path. In other words, you’ll be able to:

> curl localhost:8080/raw.githubusercontent.com/patrickdevivo/codeless/main/examples/hello_x.ts

hello, world!
hello, patrick!
hello, deno!
hello, reader!

Where the path (with an https:// preceding it) resolves to a file with the following contents (which is a Deno script):

\n \n\n

\n \n\n \n \n \n \n \n \n \n \n \n \n \n \n

	const names = ["world", "patrick", "deno", "reader"]
	\n
	names.forEach(n => console.log(`hello, ${n}!`))

\n\n\n

\n\n

\n view raw\n hello_x.ts\n hosted with ❤ by GitHub\n

\n","stylesheet":"https://github.githubassets.com/assets/gist-embed-33b98015caf26cfcbee6ce1a5d1fc768.css"}" data-component-name="GitgistToDOM">

      
        const names = ["world", "patrick", "deno", "reader"]
      
        names.forEach(n => console.log(`hello, ${n}!`))

view raw hello_x.ts hosted with ❤ by GitHub

If you don’t want to run locally, try:

curl https://codeless-deno-ex-vgetngw32a-uc.a.run.app/raw.githubusercontent.com/patrickdevivo/codeless/main/examples/hello_x.ts

Which will hit an instance of this service running in Google Cloud Run (a container based serverless platform).

Why…?

Why not? As more tools make it easier and possible to sandbox external code, maybe this isn’t such a weird (or dangerous) idea.

You only need to update your API runtime, changes to backend business logic can be made in client implementations, which means that…
Clients can call very specific versions of code since the code-to-run is described with the request (pin to a git SHA, version tag, etc.)
Clients can arbitrarily shape the output to their need (only fetch the data they care about, one of the benefits of GraphQL)
No need to agree on explicit frontend <> backend API contracts (beyond the runtime) - frontends can incrementally change the behavior of their backend calls without synchronizing with the API maintainers

Additional Ideas

Be stricter about API resource consumption (could even be based on request status: auth vs unauthed, paying vs non-paying), timeout calls after N seconds, limit memory usage - treat more like a “cloud service”
Map HTTP headers to ENV variables (or some other place) in the execution environment for contextual information like API keys or auth tokens
Enable (permissioned) access to other backend services - like a database - to use directly in the sandboxed code
Support execution in different languages (WebAssemby?)

Listing Git Authors

Patrick DeVivo — Fri, 03 Jul 2020 23:23:32 GMT

How can I find (unique) contributors to a git repo?

The first answer on this StackOverflow question offers:

git log --format="%an" | sort -u

Which is pretty cool. Simple and easy to understand. The git log command takes a —format flag which lets us supply a format string (where %an indicates author name).

It returns just the author name for every commit in the current history, sort -u then de-duplicates (and sorts) by line (using the piped input).

But here’s another way! And it’s a shameless plug for a project I’ve started called gitqlite, which provides a SQL interface for ad-hoc querying of git information.

gitqlite "SELECT DISTINCT author_name FROM commits"

Returns a similar output. If alphabetical ordering matters, try:

gitqlite "SELECT DISTINCT author_name FROM commits ORDER BY author_name"

Or if you want a sorting by total # of commits:

gitqlite "SELECT author_name, count(*) FROM commits GROUP BY author_name ORDER BY count(*) DESC"

That’s just the beginning! I hope to collect more use cases as I continue working on gitqlite. If you have ideas, please reach out!

Stop caring about your code

Patrick DeVivo — Fri, 19 Jun 2020 23:53:15 GMT

The title of this post is extreme, and maybe clickbait-y, but I’ve come to agree with the sentiment. I think it’s made me a better developer.

A manager I know likes to describe software engineers on a spectrum, between “cowboy” and “astronaut.” Cowboys commit to master frequently, leaving a trail of bugs and TODOs. Astronauts ruminate on open PRs for weeks and over-architect for every possible (unnecessary) scenario. Good developers are somewhere in the middle. I’d say the best are the ones who can play both roles, depending on the requirements at hand.

Sometimes you just need to cowboy 🤠 some code to spike an experiment, other times you need to astronaut 🧑‍🚀 a foundational solution. The extremes are never the answer, but knowing which hat (helmet?) to wear and when is an invaluable skill.

I’d call myself astronaut-leaning, and learning to become more of a cowboy has made me a better developer, which is the sentiment of this post’s title.

Caring less about code doesn’t mean caring less about its outcome. In fact, putting the outcome before the implementation can really unlock you to think about the problems that “matter.” The value your software delivers, not the software itself.

It’s similar to the trap of building what you think your users care about vs what they actually want. No one cares about the abstractions or elegance of the code beneath your product, they care about the product.

Good abstractions, maintainability, elegance, reliability, the “behind the scenes” features of code astronauts like to overthink, can also be the outcome, if that’s what the problem calls for. They just don’t always need to be part of a particular solution.

Learning to care less about code can make you a better developer. It will make you more willing to throw code out to try new ideas, and less likely to get caught up in minutia (though, often it’s the minutia that makes our jobs fun!).