System of Levers

gameboy wifi cart

2021-12-16T00:00:00-04:56

I'm making a WiFi GameBoy Cartridge!

It works, which is cool!

It's not finished, which is ok.

And Sebastian Staacks just posted a nice write up about finishing his own, which makes me a bit sad, but they have enough differences to be interesting.

So here's mine!

I was hoping to have a completed PCB and a demo before writing about it, but since I got scooped maybe I can ride some coattails! Unfortunately I'm trying to get this out fast so that I hopefully don't lose all my motivation and I'm not practiced at writing, soooooo this is going to be rough... Also my site isn't ready. Oh well!

What is it?

It's a GameBoy "cartridge" that can connect to WiFi. When I started this project I also started watching Ben Eater's videos building a breadboard CPU so that's why it looks the way it does.

My goal was to build a cartridge that looked exactly like a normal GameBoy game but could connect to the internet. Exactly like Sebastian did!

How does it work?

In some important ways it works the same as Sebastian's does. I have a ROM chip that has GameBoy code and images, I have an ESP8266 to do the WiFi, I have some hardware logic to map the ESP to a magic GameBoy memory address, and I have a big honkin capacitor to keep the ESP from overloading the GameBoy's power system.

What's different?

There are a few small differences and a few bigger ones.

Small differences

I'm using flash memory for my ROM. I think it was cheaper?
I went with a 2200uF capacitor for power. I'll have to try smaller values now because the 2200uF tantalum capacitor is over $10 (CAD). I got the 2200uF from a YouTube video about evening out the peaks from an ESP8266. The link to the video that I wrote down (on paper) doesn't work so either it's gone or I wrote it wrong.
I'm using a wemos D1 mini for my ESP. My plan is to use a more compact ESP module and a 3.3V regulator once I get to the PCB stage. The wemos is just very convenient for now.
I've mapped the ESP to the GameBoy's cartridge RAM address space instead of a ROM address. I wasn't including RAM in my cartridge and this meant I could use the same address for sending to and reading from the ESP.

Bigger differences

The data channel

The biggest difference is that I'm not connecting 8 of the ESP's GPIO pins directly to the data bus (with a bus transceiver). Instead I'm using 2 shift registers:

A CD74HCT597, parallel-in-serial-out (PISO), to capture data from the GameBoy to feed it to the ESP.
A CD74HC4094, serial-in-parallel-out (SIPO), to get data from the ESP and store it until the the GB is ready to read it.

The reason I went with shift registers is because I wanted to avoid the issue of matching the GameBoy's timing as much as possible and to use fewer of the ESP's pins. From Sebastian's description of the timing issues, and seeing as I don't have an oscilloscope, I think I made the right choice for me.

The reason I chose those particular shift registers is because I didn't know what I was doing and I mostly picked at random!

I have them hooked up to the ESP's SPI bus with 2 additional ESP pins setup as interrupts, one to tell the ESP that the GameBoy wrote some data to the PISO and the other to tell it the GameBoy read from SIPO. This setup gave me reasonable speed. In one test I was able to have the GameBoy copy a value from its memory over to the ESP 10 times in a loop. Something like:

    ld b, 10
    ld hl, $C005
.write10
    ld a, [hli]
    ld [rESP], a
    dec b
    jr nz .write10

(basically a for loop, if you don't read assembly)

Where I have rESP defined as rESP EQU $A000. $A000 is the memory address I use to communicate with the ESP.

Getting fast two-way data was a bit of a problem though since the two shift registers need different SPI modes and switching the modes caused too much of a delay to be work well. To overcome this problem I used one of the ESP's GPIOs as a "ready" flag that the GameBoy could read from the least significant bit at address $B000. A 0 (the ready pin is low) meant the ESP wasn't ready for data. A 1 (the ready pin is high) meant the ESP was ready. In my tests I would also have the ESP load up the SIPO before setting ready to 1.

Surprisingly this didn't work!

At least when I had code that looked like:

    ld [rESP], a
.wait
    ld a, [rESP_READY]
    and 1
    jr nz, .wait
    ld a, [rESP]

It turns out that the ESP couldn't get the ready pin to 0 before the GameBoy read it! I was setting it to 0 as the first step in my interrupt handler. I really didn't expect it to miss the timing.

So I switched to inverting the ready pin to indicate the ESP is ready. This means that the ESP doesn't need to do anything to indicate it isn't ready and it can definitely do nothing faster than the GameBoy! Of course it means the GameBoy needs to keep track of the last value of the ready flag, which could get annoying later.

Logic hardware

In order to control the shift registers and ESP interrupts I need to detect specific values on some of the cartridge's address pins and control pins. Specifically I need to be watching address pins A12-15 (0 based numbering) and the WR, RD, and CS pins. When those pins have the right values I'll either:

latch the data pins into the PISO register and trigger the interrupt pin to tell the ESP it has data to look at
set the output enabled on the SIPO so the GB gets the data from it and trigger the interrupt pin to tell the ESP it can send another byte
get the value of the ESP's ready pin onto the data pin D0

Sebastian solved his similar control problem with logic gate. I almost did that, in fact I bought a bunch of logic gates. I think I got lucky though, because I also bought an ATF16V8B programmable logic device (PLD)! This is a little chip with a bunch of logic gates inside and you can program it to work like an arbitrary logical expression. I bought it because I didn't know any better, it sounded like it would be really handy, and because Ben Eater's videos got me over-confident enough to think I could program anything using an arduino. I couldn't!

But luck struck some more because it turns out that programming a flash memory chip with an arduino is a pain in the butt! So instead of turning that into a project I went and bought a TL866II Plus chip programmer to program my flash memory. And it just so happens that it can also program the ATF16V8B! It took a while to get it all working, but now I have the one chip to do all my hardware logic!

Another detail is that when the GameBoy is reading the ready pin, I'm actually only driving D0 and leaving all the other data pins floating. This isn't a good thing to do, but nothing has broken yet. My feeling/guess is that the GameBoy hardware would have protection against a truly floating data pin but I haven't checked...

Too much power

This is one area where I'm maybe pushing my luck. You see, it turns out that the ESP's GPIO pins are apparently 5V tolerant. So I don't step any of the logic down to 3.3V feeding into the ESP. Also all the chips the ESP talks to will happily register 3.3V as high (I don't remember if I checked this when buying components). It's possible that the GameBoy might not be happy with 3.3V but the ESP never talks directly to the GameBoy. Even the ESP's ready pin goes through the PLD. This saved me a bunch of circuitry and components and so far nothing's exploded!

So what's the state of the project?

Well, it's not done...

It's actually working though, and I've run some simple tests of communicating between the GameBoy and the ESP. I've also done some VERY simple demos using the WiFi (I made a cloud based increment operator).

I was just getting started designing the PCB, but I've never done any PCB design before so who knows how many mistakes I'll make with that.

So best case scenario, if I get enough time to work on the PCB, don't make any mistakes with it, and don't destroy any of my components when soldering them I could have the hardware done in a month. That is very unrealistic!

I'll keep chugging along though. Hopefully this will be a good kick in the pants to get me actually making content about it regularly!

web service api authentication basics

2021-12-24T00:00:00-04:56

Web Service API Authentication Basics

This is a very simplified overview of some Web Service API auth stuff. My main reason for writing this is to have something to point to from a future post. I'm not even remotely an expert in this area but here are some basics.

I'm not going to cover CORS. If you want an introduction to it here's a comic about it from Wizard Zines.

What Are Web Service APIs

I'm talking about APIs that you interact with over the internet using HTTP, not local ones (like a library's API). Basically a website but meant for machines instead of humans. You'll have a URL that you can make a request to and get a response. Often the response will be in JSON. For example if you make a request to https://cat-fact.herokuapp.com/facts/random you'll get a JSON object with a random cat fact! You can try it by navigating to it with your browser or if you're on a machine with curl you can run the following command:

curl https://cat-fact.herokuapp.com/facts/random

Categories of APIs

For the topic of authentication I'm going to say there are three categories of these APIs.

Open/Public
API-Key Based
Token Based

Open/Public

Open/Public APIs are ones that are URLs that you can make requests to without asking anyone for permission. These APIs are unauthenticated and have no sense of who's accessing them, like a user or account. This means these APIs are usually for fetching data and don't give you a way of changing/uploading/posting anything. It's not usually a good idea to let the anonymous public modify your service.

The random cat-fact API is an example of an open API. It's just there, you can get cat facts whenever you want. If you want you could build an app that just serves cat facts fetched from that API. You probably shouldn't make it an important part of your infrastructure though because the cat-facts service probably doesn't have the resources for such a thing.

API-Key Based

API-key based APIs are ones that require you to have a secret value called an API-key that you either include in the request's URL, a request header, or as part of the payload. An API-key is tied to an account you have with the service and you'll usually get it from the service's website. For example IFTTT Webhooks are like this.

Once you have an API-key these APIs are basically work like an open/public API except you'll have to include the key in your request. In fact you can make any API-Key based API into a public API just by publicizing you API-key! That's probably not a good idea though since the service may revoke your key or even charge you for usage. That's one of the downsides of API-keys, they need to stay secret and you need to trust anyone you're sharing it with. Once you've shared it with someone the only way to stop them from using it is to revoke/regenerate your API-key. This will break everyone else who's using the key and you'll need to update it. For example if you've setup a bunch of ESP8266's to send requests to an IFTTT webhook. But you also shared your webhook url (with the key) with your good buddy Jerk-O. In a shocking turn of events Jerk-O turned out to be a jerk and started using it to troll. The only way to stop them is to regenerate your key and go and reprogram/update all your ESPs.

OAuth has a concept of "bearer tokens" that seems similar to an API-key.

Token-Based APIs

I'm going to be very vague here. I don't have deep understanding of this stuff. Auth is an area where mistakes can be bad and thinking you know more than you do can lead to mistakes. So if this stuff is important to you maybe find a better source than some guy's blog with two posts (including this one)!

Access tokens are a piece of data that describes access permissions to a resource. Kiiinda like API-keys! The difference tends to be that an access token has metadata that describes what it can be used to access and for how long. There also tends to be mechanism to for a user, who we'll call Jesse, who has an account on one service, say twitter, to give a token to another service, like a GameBoy twitter client, that will let the GameBoy interact with twitter on Jesse's behalf. For example to like a tweet! If you've ever been using an app or website and suddenly get a little pop-up that says "hey do you want example.com to have access to your google calendar?" with your google account photo and such then this is what's happening. It's using OAuth2 to get an access token so that example.com can access your calendar.

There also tends to be cryptography involved! The tokens will have cryptographic properties, like a signature, so that they can be verified without just needing a list of all the tokens that have been issued.

gameboy twitter client proof of concept

2022-01-20T00:00:00-04:56

GameBoy Twitter Client Proof-of-Concept

I didn't have an interesting demo of my WiFi GameBoy cartridge. I decided that it would be fun to use it to like @diconx's tweet about his WiFi cartridge and tweet a video of it at him!

Of course up to this point I've just been sending handfuls of bytes between the GameBoy and ESP so that I could figure out the hardware and basic communication protocol. Everything seemed to be working reliably though and I figured there was probably a Twitter API library for the ESP8266. How hard could it be? Actually not that hard but I made some silly mistakes and banged my head against the Twitter API a bit.

What Is It

A proof-of-concept Twitter Client for the GameBoy. It's able to load a tweet and render it on the GameBoy. A user can then like that tweet by pressing right and the A-button.

There's no way to change what tweet is viewed. The tweet ID is hardcoded but the tweet content is loaded using the Twitter API. There's no way to unlike the tweet, or to show if the tweet is currently liked. You also can't write a tweet. Also the auth tokens are hardcoded so it can only make me like the tweet!

How Does It Work

There are three parts:

A web service that talks to the Twitter api and formats the tweet text for easier rendering on the GameBoy.
An ESP8266 that talks to the web service.
A GameBoy ROM that renders the tweet and handles user input.

You can find the code for it here.

Web Service

In principle this isn't needed. The ESP could be made to talk directly to the Twitter API. I was trying to get this done quickly though, and I was having trouble working out the right authentication steps so I figured doing it in python with a mature library would be easier. It means that the ESP code can be a lot simpler and I was able to do some pre-processing of the tweet text to make it much simpler for the GameBoy to render.

Talking to Twitter

Twitter has an API! To use it you'll need to setup a new app at https://apps.twitter.com/app/new. Then you'll be able to get credentials that you can use to talk to twitter. If you're not familiar with web API authentication I wrote a brief overview covering some of the basics. You can also check out the twitter docs.

To talk to the Twitter API from my web service I used the Tweepy library. I'm using the Twitter V2 API so I used the tweepy.Client class. You can follow their authentication tutorial to get the right tokens to have the service act on your behalf. You'll have to do the OAuth dance to get them. Using the consumer_key and consumer_secret that you get from the Twitter dev portal for your app.

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
print(auth.get_authorization_url())
# Visit that URL and take tone of the oauth token and verifier code. Fill them
# in in the following lines.

auth.request_token = { 'oauth_token' : "[TOKEN]",
                       'oauth_token_secret' : "[VERIFIER]" }
auth.get_access_token("[VERIFIER]")
print(auth.access_token)
print(auth.access_token_secret)

Save the access_token and access_token_secret and fill those in in the web service code. Now the web service will be able to access twitter on your behalf.

Formatting Tweets

Since I was already using python to fetch the tweet I decided to also do some pre-processing of the tweet to make it easier for the GameBoy to display it. The web service does three things:

Make all lines exactly 20 characters, padding with spaces if needed.
Make all lines start and end with a space to give a blank border on either side when displayed on the GameBoy.
Avoid having a line break in the middle of a word when possible.

This makes the GameBoy code to display the tweet much simpler.

ESP Code

The ESP is a Wemos D1 mini programmed with Arduino. It communicates with the GameBoy using a pair of shift registers, which are normally connected to the ESP's SPI bus but I had them wired up differently while experimenting and using Arduino's shiftIn() and shiftOut() functions. I hadn't realized that at first and spent a bunch of time debugging why nothing was working! A few more details about this hardware setup can be found in my first post.

The ESP takes care of fetching the tweet for the hardcoded tweet-ID, sending it to the GameBoy, and waiting for input to like the tweet. This is managed with a sequence of states:

kConnecting
1. Send the byte value 255 to the GameBoy (the actual value is ignored).
kConnected
1. Wait for the GameBoy to read the sent byte (this is just to synchronize with the GameBoy).
kGettingTweet
1. Request the tweet from the web service.
kSendingToGB
1. Send the tweet to the GameBoy 1 character byte at a time (assumes ASCII).
2. When done, send the value 0 to indicate the end.
kWaitingForLike
1. Wait until the GameBoy sends a byte.
2. When it does, send like request to the web service.
3. When that completes send a byte to the GameBoy to indicate it's done.
kDone
1. do nothing.

Apparently I never used kConnecting, oops! Basically this state is entirely handled by the setup function, and once it's out of there it's assumed to be connected.

GameBoy Code

The GameBoy code is written in assembly. Due to the limited nature of the proof of concept it fairly straight forward. It basically runs in lockstep with the esp code. It also displays some text so I know what step of the initialization it's on. There are two things that help keep the GameBoy and ESP in sync. The GameBoy can read a "status-bit" from the least-significant bit at address $B000. This is how the ESP tells the GameBoy it's ready. The other way is for the GameBoy to read from or write to $A000, which I call the ESP register, or rESP. The ESP receives an interrupt for a read from or write to that address (separate ones for read/write).

The sequence is as follows:

Do some display related initialization.
Copy over the string "Connecting..".
Wait for the ESP to indicate it's ready, while blinking the last "." in "Connecting..".
Display "Connected!!" for a few frames.
Ready a byte from the ESP to tell it to continue.
Display "Loading.." with animated ".".
Wait for the ESP to indicate it's ready.
Copy the tweet from the ESP and display it.
Display a little bar at the bottom with an empty heart in heart.
Wait for the the right-button to be pressed.
Display an arrow icon next to the heart.
Wait for the A-button to be pressed.
Send the byte for the ASCII character "L" to the GameBoy.
Wait for the ESP to indicate it's ready, which the GameBoy assumes means it finished liking the tweet.
Read a byte from the ESP to acknowledge.
Fill in the heart icon to indicate the tweet was liked.
Loop forever.

The text (and any graphics) can only be updated when the GameBoy's display hardware isn't accessing video RAM (VRAM). You can find more details about this in the Pan Docs' rendering section. The section on the LCD status register has a good overview of what parts of VRAM are accessible when and the section on the pixel FIFO gives details about timing.

Since I didn't need precise screen updates I did all my updates during VBlank, which is a chunk of time after the GameBoy has finished drawing an entire frame. The GameBoy has an interrupt to tell you when VBlank is. It also has a halt command that will stop code execution until an interrupt occurs. So VBlank interrupt handler just returns immediately and I have halts whenever I need to wait for a VBlank. There are other types of interrupts but I only have the VBlank one turned on.

For displaying the Tweet, rather than worry about fitting it all in VBlank I just turn off the the display. Only do this during VBlank see here for details. Once the display is off I can copy everything over and turn it on.

Named Pipes to Turn CLI Programs Into Python Functions

2022-03-01T00:00:00-04:56

Tl;dr

A named pipe is like a file that doesn't store anything. It has a path (the name I guess). It can be opened, read from, and written to, but the content is temporary and stored in memory.

Since a named pipe is also a pipe it acts a bit different than a normal file. When you open it you can only open it as read-only or write-only not read-write. The idea is you'd have one process with it open for write (the producer) and one process with it open for read (the consumer). This matches a pipe since they have a read-end and a write-end. At least in Unix.

A call to open for read or write will block until something else opens it for the other. Calls to read or write will also block if the named pipe is empty or full respectively.

Named pipes can be useful if you need to create CLI pipelines with programs that consume or produce multiple inputs and outputs. You have to be careful not to create deadlocks due to the blocking behaviour though. You also can't use this with a program that needs to seek in a file or that reads and writes the same file.

I'm using them to run file based CLI programs from Python without writing to the disk.

What I Was Trying To Do

I'm using two command line programs rgbasm and rgblink. They're part of the RGBDS GameBoy assembler toolchain. If I were running them from the command line I would use the following two commands:

rgbasm A.asm -o A.o
rgblink A.o B.o -o A.gb -m A.map -n A.sym

I want my Python script to be able to provide A.asm and get back the contents of A.gb, A.map, and A.sym.

So I can just have my script write A.asm, use subprocess.run() to run the two commands then read in the three files!

Unnecessary Constraints!

I didn't want to write anything to disk.

Why?

My justification was that I want to run this in a server and having it writing to and reading from the disk felt wrong. That's a bad reason though. This is for a personal toy project. Making scalable, production quality software is not the goal! I'm not saying I achieved scalable, production quality software though, just that it's not the goal.

The actual reason was that I thought it would be possible to do and I wanted to figure out how. That's a good reason! At least for a personal toy project. Even if it is a bit of a detour.

What Did I Know Going In

Linux stuff
- pipes exist
- you can use them to route stdin/out/err from one program to another programs
- files exist
- everything is a file: directories, devices, programs, you, the internet, the computer, your home, the love you feel for others, this list entry
Python
- normal file stuff
- how to use the subprocess module to run programs
- I can yell PIPE to communicate with subprocesses over stdin/out/err
- some thread stuff

Named Pipes / FIFOs

I quickly came across named pipes and how to create them in Python with os.mkfifo or from the command line with mkfifo (I don't know about windows though). After being briefly distracted by going overboard with context managers to create and delete a temp directory and a bunch of FIFOs I ran into my first gotcha. My program froze when I tried to open a FIFO.

The problem was that I was only opening one end of the FIFO/pipe and that will block until the other end is opened.

This was lucky! Confusing, but lucky! You see I didn't know what I was doing and I very well could have accidentally written my code in a way that missed this problem. Then I never would have learned that opening a FIFO/named pipe will block unless the other end of also open. I also wouldn't have realized that I knew less what I was doing than usual and I'd need to pay more attention.

You can use os.open() to open without blocking and that's what I did at first. Something like this (rough memory probably not exactly what I did):

def opener(path, flags):
    return os.open(path, flags | os.O_NONBLOCK)

with open('fifo_in', 'w', opener=opener) as wf, \
         open('fifo_out', 'rb', opener=opener) as rf:
    wf.write(input_data)
    p = subprocess.Popen(['rgbasm', 'fifo_in', '-o', 'fifo_middle'])
    subprocess.run(['rgblink', 'fifo_middle', '-o', 'fifo_out'])
    p.wait()
    output = rf.read()

I was surprised when it worked. But I'm pretty sure this can deadlock in a few ways though. Here's what's happening:

There are three named pipes:
- fifo_in
- fifo_middle
- fifo_out
fifo_in and fifo_out are used to get data into rgbasm and rgblink respectively
fifo_middle is used to get data from rgbasm into rgblink.
fifo_in and fifo_out are opened by the Python code using a specially defined opener function. fifo_out is opened as binary because it contains binary data.
Some data is written to fifo_in. This is the input to rgbasm.
rgbasm is run using subprocess.Popen() so Python doesn't block waiting for it to complete.
- rgbasm itself will be blocked when it tries to open fifo_middle because (presumably) it's trying to open one end of a named pipe with a normal open without the O_NONBLOCK flag. So rgbasm won't finish until something opens fifo_middle for reading.
rgblink is run using subprocess.run(). Python will block (wait) until it completes before moving on.
- rgblink will open fifo_middle for reading and fifo_out for writing. Neither of these blocks because fifo_middle is already opened for writing by rgbasm and fifo_out is already open for reading by the python code.
- This unblocks rgbasm and it can start producing output and everything move along.
I added a p.wait() just to make sure rgbasm was done.
Finally I read the output!

So What's the Problem?

For one thing, it only works because the input isn't very big! Pipes/named-pipes/FIFOs have a maximum capacity. If you try to write to one when it's full then it will block or fail if O_NONBLOCK is set. Reading from one that's empty will have similar results. There's more info in the man(7) page in the I/O on pipes and FIFOs, Pipe capacity, and PIPE_BUF sections. So if the input data were bigger then the python script would crash on wf.write(input_data).

Similarly, if the output were too big then rgblink would block when it tries to write to a full pipe. This means the python script would be blocked from reading anything from the pipe so everything will just be stuck!

Maybe Don't Use `O_NONBLOCK`?

Ok, I could change it to something like:

p_rgbasm = subprocess.Popen(['rgbasm', 'fifo_in', '-o', 'fifo_middle'])
p_rgblink = subprocess.Popen(['rgblink', 'fifo_middle', '-o', 'fifo_out'])
with open('fifo_in', 'w') as wf, \
         open('fifo_out', 'rb') as rf:
   wf.write(input_data)
   p_rgblink.wait()
   output = rf.read()

Now we're not using O_NONBLOCK so at least reads and writes won't fail in the Python code. I even think this might work for this case! Here's what I think is happening:

rgbasm and rgblink are started and running in the background. They're both definitely blocked though since they need to open fifo_in and fifo_out respectively, and their other ends aren't open yet.
I open the other ends offifo_in and fifo_out so rgbasm rgblink should be unblocked.
- They'll both still get blocked when they try to read from empty pipes (rgbasm from fifo_in and rgblink from fifo_middle).
I write to fifo_in. Now rgbasm is unblocked and can read from fifo_in. It should also be writing to fifo_middle so rgblink isn't blocked and it can read and start producing output and writing to fifo_out.
I wait for rgblink to be done.
I read the output from fifo_out. This will unblock rgblink if fifo_out got full...
Oh shoot, I never get to rf.read() when fifo_out is full because I'm still waiting on p_rgblink.wait().

I could probably fix this by getting rid of p_rgblink.wait() but I don't actually know if rf.read() will always return the entire output. I think it will as long as rgblink doesn't do something strange like close the file and reopen it to write some more. The reason I think this is because read() will read until it hits an end-of-file. I think that only happens when all the write-ends of the pipe are closed.

Anyway we have other problems.

Grumble

Maybe there's something in asyncio that could help me?

Ok, no, that's more complicated than I thought it would be and I don't see a magic wand for this anyway.

Fine I'll Use Threads

I already have deadlocks. I can't make it much worse right?

import threading
import subprocess
import io

def doRGBStuff(input_data):
    def do_write(file_name, data):
        with open(file_name', 'w') as wf:
            wf.write(data)

    def do_read(file_name, out_stream):
        with open(file_name, 'rb') as rf:
            out_stream.write(rf.read())

    write_thread = threading.Thread(target=do_write, args=('fifo_in', input_data))
    write_thread.start()

    p_rgbasm = subprocess.Popen(['rgbasm', 'fifo_in', '-o', 'fifo_middle'])
    p_rgblink = subprocess.Popen(['rgblink', 'fifo_middle', '-o', 'fifo_out'])

    out_stream = io.BytesIO()
    read_thread = threading.Thread(target=do_read, args=('fifo_out', out_stream))
    read_thread.start()
    read_thread.join()
    return out_stream.getvalue()

Now we have write_thread to write data into fifo_in and read_thread to read from fifo_out and put it into an io.BytesIO stream. Everything that interacts with the fifos is either a thread or subprocess so when they block it won't block the main Python script. The script blocks at the end with read_thread.join(). This should wait until rgblink finishes writing its output. It still won't work properly if rgblink closes and reopens the output file in the middle.

I can't think of anything that would let this deadlock, but maybe I'm missing something.

To complete the original task of getting three outputs I'll just need two more fifos and reading threads.

Also I haven't tested any of the specific code in this post! It's based on some other code that DOES work though!

A Very Big GameBoy "Game"

2023-05-12T00:00:00-04:56

I made a big GameBoy "game"! Here's the code including the ROM.

All you can do is scroll around to see the level. It only has one level and it's 32,766x16,384 pixels big!

It looks like this:

Ok not very exciting.

Let's see how it works!

How it Works

I'm not using any compression or in-game procedural generation for the level map so the entire 32,768x16,384 pixel level is stored entirely in the game ROM or on the cartridge if you were to put it on a cartridge for some reason. The GameBoy uses tile-based graphics with 8x8 tiles. So the level is 4,096x2,048 tiles big.

The game is designed to use an MBC5 type cartridge. I chose that type of cartridge because it's allows the biggest ROM size for standard cartridge size, Giving me 8MB to work with! It supports 512 ROM banks, each 16KB.

To make things simple bank 0 is only used for code and the tile graphics. The map data is in the other 511 banks.

The map data is made up of 1-byte tile indexes that say what tile goes where. As you move around the level the game copies those indexes from the game ROM to the tile map in the GameBoy's VRAM.

So the real problem is to know where to copy from and where to copy to. I ended up solving that problem by keeping track of what part of the level map was on screen with two 16-bit variables: vWorldColumn and vWorldRow. I used those two variables track what tile in the world/level map is being displayed in the top left corner of the screen. The trick was that I could use a world column and row coordinate to get the ROM address of the needed tile map entry, and the address of where it goes in the VRAM tile map.

Why it's Complicated

If you've ever done any programming where you've had to switch between 2D array indexing and 1D array indexing this might not sound very complicated. If you know the 2D array's dimensions WxH and you want the value at some 2D index x,y then you can do some math:

index = x + y * W

and that gives you the 1D index. So that would let you get the ROM address of the map data you're looking for. To get the VRAM address you could do some more math:

romIndex = (x % 32) + (y % 32) * 32

The first problem is that the GameBoy is an 8-bit computer and those variables are 16-bits. That math isn't coming cheap. Getting the romIndex would be easy enough but the x + y * W would be a pain.

The other problem is ROM banks. If you're not careful then you could end up doing a bunch of bank changes while copying one column of tile map entries. I'd rather avoid that.

How I Map

I got around these problems by assigning meaning to the different bits of vWorldRow and vWorldColumn.

vWorldRow has the structure:


0000 0PQQ QQRr rrrr

and vWorldColumn has the structure:


0000 SSSS TTTt tttt

The first thing to notice is the leading zeros in both. vWorldRow only uses the 11 least significant bits and vWorldColumn uses 12. That lets me have 2,048 rows and 4,096 columns.

The different letters categorize the rest of the bits into their different meanings.

P and Q from vWorldRow and S from vWorldColumn are the 9 bits of the bank number:


0000 000P QQQQ SSSS

The rest are for the ROM address to where the tile map entry is stored:


01Rr rrrr TTTt tttt

The address starts with 01 because that's the banked half of the ROM addresses (controlled by the current bank number). Address that start with 00 are always bank 0. Address that start with 1 aren't for ROM.

The lower case letters give you the tile map entry's address in VRAM:


1001 10rr rrrt tttt

The address starts with 1001 10 because I'm using tile map 0, which starts at $9800.

How Does That Compute

The easy part is getting the ROM address since the needed bits align nicely to 8-bit values. I can just take the rightmost 8 bits from vWorldColumn as is to get the rightmost 8 bits of the ROM address. I can copy the rightmost 8 bits of vWorldRow and set the left 2 bits to 01 to get the leftmost 8 bits of the ROM address.

Since the bank number portion of vWorldRow is spread across 2 bytes it's a bit more trouble. Here's some code:

worldToSource:
  ;; Converts vWorldColumn and vWorldRow into a copy source address and bank.
  ;; Stores the address in vCopySoure and the bank in vSourceBank.
  ;; Modifies:
  ;; A, B, C
  ;; vWorldRow has the form:
  ;;   xxxx xPQQ QQRR RRRR
  ;; vWorldColumn has the form:
  ;;   xxxx SSSS TTTT TTTT
  ;;
  ;; This function produces
  ;; vCopySource with the form:
  ;;   01RR RRRR TTTT TTTT
  ;; vSourceBank with the form:
  ;;   0000 000P QQQQ SSSS

  ;; First get the source address because it's simple.
  ldh a, [vWorldRow + 1]
  ld c, a ; need this later for the bank too
  and %00111111
  set 6, a
  ldh [vCopySource], a
  ldh a, [vWorldColumn + 1]
  ldh [vCopySource + 1], a

  ;; Next get the source bank.
  ldh a, [vWorldRow]
  and %00000111

  sla c ; Get the highest bit of the lower byte of vWorldRow in the carry flag.
  rla ; Rotate the carry flag into the higher byte of vWorldRow and get the
      ; high bit into the carry flag.
  sla c
  rla
  ;; A is now %000PQQQQ
  swap a
  ;; A is now %QQQQ000P
  ld b, a
  res 0, b ; B is now %QQQQ0000

  and %00000001 ; Clear all but P from A.
  ldh [vSourceBank], a

  ldh a, [vWorldColumn]
  and %00001111
  ;; A now has 0000 SSSS
  ;; and B has QQQQ 0000
  or b
  ;; Now A has QQQQ SSSS
  ldh [vSourceBank + 1], a
  ret

Getting the VRAM address involves a bit of bit twiddling. Getting the right bits out of vWorldColumn and vWorldRow is simple enough, but the bits need to be shifted around to get the right VRAM address:

worldToDest:
  ;; Converts vWorldColumn and vWorldRow into a copy destination based on
  ;; _SCRN0. Stores the address in vCopyDest.
  ;; Modifies:
  ;;   A, B
  ;; vWorldRow has the form:
  ;;   xxxx xPQQ QQRr rrrr
  ;; vWorldColumn has the form:
  ;;   xxxx SSSS TTTt tttt
  ;;
  ;; This function produces
  ;; vCopyDest with the form:
  ;;   1001 10rr rrrt tttt
  ldh a, [vWorldRow + 1]
  and %00011111
  sla a
  ;; A now has 00rr rrr0.
  swap a
  ld b, a ; Will use B to calculate the lower byte later.
  ;; Use A to calculate the high byte.
  and %00000011
  or HIGH(_SCRN0)
  ;; A now contains 1001 10rr.
  ld [vCopyDest], a
  ld a, b
  and %11100000
  ld b, a
  ldh a, [vWorldColumn + 1]
  and %00011111
  or b
  ld [vCopyDest + 1], a
  ret

Problem

One problem that this scheme has is that it assumes bank 0 is usable for tile map entries but I said that I avoided putting any of that data in bank 0. To fix that I just created a big dead-zone for the part of the level that would have been stored in bank 0. I render it as just # characters. That area is 256x64 tiles big so the level is actually 32,766x16,384 - 2048x512 pixels. You can still go to that area it's just even more boring than the rest of the level.

Generating the Level

The level is generating with a python script. I made it to help debug and validate the GameBoy code. It splits the level into 8 tile wide chunks and each chunk displays its row and column. That way it helps me figure out if it's loading the right tile map entries as I scroll.

The code is pretty short:

#!/usr/bin/python3

import random

def pad(l, length):
  needed_padding = length - len(l)
  return ([0] * needed_padding) + l


def toHexCharacters(n):
  return [int(d, 16) for d in list(hex(n))[2:]]


def main():
  tile_values_sections = []

  col_start = 0x100
  for i in range(0x4000, 0x800000, 8):
    col_addr = i & 0xff
    row_addr = (i >> 8) & 0x3f
    col_bank = (i >> 14) & 0xf
    row_bank = (i >> 18) & 0x1f
    row = (row_bank << 6) | row_addr
    col = (col_bank << 8) | (col_addr >> 3)
    row_digits = pad(toHexCharacters(row), 3)
    col_digits = pad(toHexCharacters(col), 3)
    tile_values_sections.append(row_digits + [16] + col_digits + [17])
  map_values = [i for c in tile_values_sections for i in c]
  map_bytes = bytes(map_values)
  with open('long_map.tilemap', 'wb') as f:
    f.write(map_bytes)


if __name__ == '__main__':
  main()

A Peak at My YouTube Analytics

2026-01-09T00:00:00-04:56

If you don't know I have a YouTube channel called SystemOfLevers. This is going to be a dive into some of that channel's analytics.

One thing about starting a YouTube channel that I've found challenging is that I had no context for the analytics. For the most part YouTube studio (the admin site for channels) gives you comparisons based on your own past performance. Things like:

how does the view count for the last 28-days compare to the previous 28-days
how does your latest video rank against to the previous 9 videos at the same point in their lifetime
is the video's retention better or worse that your typical retention

Don't get me wrong, I definitely think comparing against yourself is the right way to go. Comparing against well established channels that have already figured things out is probably a great way to kill your motivation. As someone new to YouTube it's going to be VERY hard get close to their performance. You're just starting out, you're going to be bad at it, that's how you learn.

That being said I don't think the "algorithm" cares if your latest video is the best one you've made so far. It cares how the video compares to all the other videos on YouTube. At least I think that's the case.

It would be nice to have something to compare against. So this is a peak into my channel's analytics so that anyone else starting out can have at least one point of comparison.

About My Channel

History

I published my first video in May 2022. It's still a relatively small channel. At the time of writing it has 5981 subscribers and has 13k views in the last 28-days. I've published 31 long-form videos and 11 shorts. I was fully monetized in February 2025 (that means I earn money from ads). I also got a sponsorship last year that I'm not going to discuss further.

My channel has primarily focused on talking about Game Boy technical details.

I got some good luck with my first video, it was picked up by hacker news for a bit. I think my second one was too. But my 2nd-4th videos never got the same traction that the 1st one did. They were also pretty labour intensive to create so I wasn't releasing them very often. It was like 2 a year.

I found another hit of success in February 2023 with a video about the FPGBC, an FPGA based reproduction of the Game Boy Color. I think the success was because it was the right video at the right time. For a while it was my most popular video. It was also a lot less effort than my previous videos!

The Big One

My BIG hit of success came with a video I published at the end of January 2025 about telling you to make a Game Boy emulator. It took off in a huge way. It's currently at 140k views. It accounts for more than a third of my channel's views and a bit less that half of its subscribers. It's also the reason I was monetized!

I have no idea why that particular video did so well. In fact I almost didn't publish it. I thought it was boring and not particularly valuable to anyone. Also the video quality is kinda off in my opinion. The audio isn't my favorite either, but it's fine.

I've tried to reproduce that success but I haven't had much luck. None of my videos since have even cracked 10k views. That's not to say they aren't successful. If you ignore that 140k video then they're definitely strong performers in the context of my channel.

I've tried two ways to reproduce the success:

continuing the topic with followup videos
making a similar video for a different topic

The followup videos had diminishing returns. The 5th one in the series hasn't broken 1k views yet.

The similar video did quite well. As long as you don't compare it to the 140k video.

How I Feel About the Channel

I'm not going to lie, having that one video take off the way it did definitely helped improve my feeling about the channel. I wasn't exactly down about the channel, but I wasn't exactly feeling confident it would go anywhere. Before that big success I was getting nervous I'd miss my chance to get monetized. One of the requirements for monetization is 4,000 watch hours in the last 365 days. Before the big emulator video the bulk of my channel's watch hours came from my video about the FPGBC, published on February 23rd, 2024, and most of the watch time was within a couple month of that publishing. I published the emulator video on January 24th, 2025. By that point I only had 2,690 watch hours and I was getting maybe 5 new hours a day. So I was running out of time. In a few months a huge chunk of my watch time would pass beyond that 365 day mark and it'd be even harder to get monetized!

There are of course other measures of success. I got a bunch of positive comments from people about my videos. I think I was getting better at making them too. My audio certainly improved. And I was still enjoying making them. Being monetized would be nice though...

The Analytics

As of 2026-01-09

Channel views over time since it was created

The two spikes on the left are when I released the first two videos. The little bump in the middle is the FPGBC video. You can see that it did result in an increase in the baseline views per day. The big bump is the big emulator video, and that also resulted in an increase in the baseline. Subsequent bumps are other videos being released.

Subscribers Over Time

Monthly CTR Over Time

Monthly Average View Duration Over Time

Monthly Percent Viewed Over Time

Maybe if I made more videos this graph could be helpful? I don't know.

First Video

First 24 Hours Views

Lifetime Views

Watch Retention

This graph shows the percent of viewers who are watching at a particular point in the video. It primarily captures when people stop watching the video, but it's also affected by people skipping around in the video, which can cause the retention to increase at a particular point and in more extreme cases could cause retention to go above 100%.

Best Video (the one about emulators)

First 24 Hours Views

Lifetime Views

Watch Retention

Retention Comparison

This graph shows the video's retention compared to my other videos. I'm not sure if it's videos publish before this one or all videos I've published. When the line goes into the high region it's saying the retention is better than others, and low means worse.

First 24 Hours CTR

Lifetime CTR

Worst Video

This is a video about using javascript to generate visuals for a video. It did not do well. It's kinda outside of my core topic though so whatever. My thinking was that I was doing the work anyways, might as well have gotten some content out of it. The funny thing is that the stuff I was doing in this video was for my 3rd worst video (I'm not counting my 10 hour fireplace videos btw).

First 24 Hours Views

Lifetime Views

Watch Retention

First 24 Hours CTR

A Clickbait Video

I made a video about why I'm personally disappointed when I buy flash carts for consoles. It was a very low effort video and I made the thumbnail and title a little bit click-baity.

Retention

Lifetime CTR

Lifetime View Percent

Conclusion

Well that's about it. If I had some kind of summary to say here I've completely forgotten it. So, I hope this data helps someone!

Remember to Like and Subscribe!

System of Levers

gameboy wifi cart

I'm making a WiFi GameBoy Cartridge!

What is it?

How does it work?

What's different?

Small differences

Bigger differences

The data channel

Logic hardware

Too much power

So what's the state of the project?

web service api authentication basics

Web Service API Authentication Basics

What Are Web Service APIs

Categories of APIs

Open/Public

API-Key Based

Token-Based APIs

gameboy twitter client proof of concept

GameBoy Twitter Client Proof-of-Concept

What Is It

How Does It Work

Web Service

Talking to Twitter

Formatting Tweets

ESP Code

GameBoy Code

Named Pipes to Turn CLI Programs Into Python Functions

Tl;dr

What I Was Trying To Do

Unnecessary Constraints!

Why?

What Did I Know Going In

Named Pipes / FIFOs

So What's the Problem?

Maybe Don't Use O_NONBLOCK?

Other Problems

Grumble

Fine I'll Use Threads

A Very Big GameBoy "Game"

How it Works

Why it's Complicated

How I Map

How Does That Compute

Problem

Generating the Level

A Peak at My YouTube Analytics

About My Channel

History

The Big One

How I Feel About the Channel

The Analytics

Channel

Channel views over time since it was created

Subscribers Over Time

Monthly CTR Over Time

Monthly Average View Duration Over Time

Monthly Percent Viewed Over Time

First Video

First 24 Hours Views

Lifetime Views

Watch Retention

Best Video (the one about emulators)

First 24 Hours Views

Lifetime Views

Watch Retention

Retention Comparison

First 24 Hours CTR

Lifetime CTR

Worst Video

First 24 Hours Views

Lifetime Views

Watch Retention

First 24 Hours CTR

A Clickbait Video

Retention

Lifetime CTR

Lifetime View Percent

Conclusion

Maybe Don't Use `O_NONBLOCK`?