fitzsim's development log

Here is a retro gaming product idea that I would like to see on CrowdSupply. I do not know if it is actually feasible but I thought I would write up the idea since I would definitely buy this product.

The idea is to create an FPGA-based driver circuit connected directly to an OLED panel’s rows and columns, which simulates the phosphor scanning pattern of a cathode-ray tube.

This table:

https://en.wikipedia.org/wiki/Comparison_of_CRT,_LCD,_plasma,_and_OLED_displays#cite_note-TR-20170112-13

suggests response times of OLED pixels are the same as CRT phosphors. (By contrast, LCD cells switch orders of magnitude too slow.)

In slow motion, the OLED’s output would look like this:

https://www.youtube.com/watch?v=3BJU2drrtCM&t=190s

I looked around for examples of this type of circuit/driver and all I could find is that some small OLED displays use the SSD1351 driver:

https://newhavendisplay.com/content/app_notes/SSD1351.pdf

I wonder what large OLED modules use. In terms of prototyping, how much surgery would a module need such that the the raw pixel row and column lines could be accessed? I could not find anywhere to buy raw panels, i.e., OLED panels without integrated controllers.

If this driver design were implemented, it would enable a product line of OLED screens that could substitute for CRTs for retro gaming. Given OLED panels’ flexibility they could be made with the same shape and curvature of Sony PVMs or arcade monitors. They could accept any retro input type (RGB, composite, component, VGA, 15kHz, 31kHz, etc.), be coated in glass, simulate different CRT shadow masks and phosphor arrangements and so forth.

The most important goal though would be matching a CRT’s zeroish latency. The ultimate “acid test” of this FPGA core would be: does it support Duck Hunt with the NES Light Gun without any modifications to the Light Gun or ROM? This video shows how this setup worked, and why it is so latency-sensitive:

https://www.youtube.com/watch?v=cu83tZIAzlA

If this latency target could not be achieved, then there is no point in doing this project. But if it could, then maybe OLEDs could be the contemporary display technology that finally unseats the venerable CRT for retro gaming.

Request to Google: ungrey the “OEM unlocking” toggle in the factory, before shipping store.google.com devices to customers. Do not make your customers connect the device to the Internet before they are allowed to install the operating system they want.

My wife had a requirement to use Android¹, and she wanted to run GrapheneOS; I experimented with other devices and ROMs to ensure the specific application she needed would run on GrapheneOS.

As part of my research, I read the GrapheneOS installation guide², which stated:

Enabling OEM unlocking

OEM unlocking needs to be enabled from within the operating system.

Enable the developer options menu by going to Settings > About phone and repeatedly pressing the build number menu entry until developer mode is enabled.

Next, go to Settings > System > Developer options and toggle on the ‘OEM unlocking’ setting. On device model variants (SKUs) which support being sold as locked devices by carriers, enabling ‘OEM unlocking’ requires internet access so that the stock OS can check if the device was sold as locked by a carrier.”

None of the many many YouTube videos I watched about bootloader unlocking covered whether or not you need Internet connectivity. Nor did any of Google’s official documentation³. GrapheneOS documentation is the only place on the Internet that documents this requirement, so, well done GrapheneOS documentation team!

GrapheneOS only supports recent Google Pixel phones. Those phones are nice hardware⁴, and I can easily (so I thought) install a different operating system, so I decided to buy one. To be as future-proof as possible, I bought a Pixel 7 Pro from store.google.com (Canada).

I thought (based on the aforementioned GrapheneOS docs) that the device model variant I bought, being sold “unlocked”⁷ by Google, would not need the Internet connection. NOPE; Google sold it to me with “OEM unlocking” greyed out:

I consider this a customer-hostile practice. I should not have to connect a piece of hardware to the Internet, even once, to use all of its features. If I hadn’t connected the Pixel 7 Pro to the Internet, then “OEM unlocking” would have stayed greyed out, thus I would not have been able to unlock the bootloader, thus I would not have been able to install GrapheneOS⁵.

Keep in mind that I bought this phone full price⁶ from store.google.com, where it was advertised right in the FAQ as an “unlocked smartphone”⁷. There is zero carrier involvement here, so carriers cannot be blamed for this policy. Also, I paid full price for the phone, so this is not a case of “if you don’t pay for the product, you ARE the product”.

I probably should have returned the device for a refund. Instead, I set up a network debugging environment to see what activity happens when I connect the Pixel 7 Pro to the Internet.

By tailing some log files and watching them closely, I was able to determine that the final site accessed just before “OEM unlocking” goes from greyed to ungreyed is “afwprovisioning-pa.googleapis.com“. Here is the video of “OEM unlocking” ungreying:

Here is the rest of the network activity, all of which is TLS-encrypted by keys buried in the stock Google operating system, and thus not controlled by the device purchaser:

Only Google knows precisely what all that data is and what it is used for.

As the video shows, the ungreying did happen; I had the Settings application open, then connected the phone to the Internet. I had to close then re-open the Settings application; the access to “afwprovisioning-pa.googleapis.com” seemed to be co-timed with the Settings application restart. After the Settings appliation restart, the “OEM unlocking” option was operable.

I don’t know what subset of the hosts in the above table need to be accessible to the phone for ungreying to take place; I considered firewalling each individually using a script, but I ran out of time. I also don’t know if a factory reset of the phone results in “OEM unlocking” being greyed again. I ended my experimentation when the ungreying took place and I proceeded to install GrapheneOS successfully (the rest of the process was very straightforward, thanks to GrapheneOS’s great documentation and installation scripts).

All in all, cheers to Google for releasing Android as Free and Open Source software, and for selling devices which are (with steps) bootloader-unlockable; both of which make GrapheneOS feasible⁸. Jeers to Google for selling devices from store.google.com that cannot have their bootloaders unlocked without first connecting them to the Internet.

Footnotes

1. One day I hope we can both use PinePhones. ^
2. https://grapheneos.org/install/cli#enabling-oem-unlocking ^
3. https://source.android.com/docs/core/architecture/bootloader/locking_unlocking

   “Devices should deny the fastboot flashing unlock command unless the get_unlock_ability is set to 1. If set to 0, the user needs to boot to the home screen, open the Settings > System > Developer options menu and enable the OEM unlocking option (which sets the get_unlock_ability to 1). After setting, this mode persists across reboots and factory data resets.” ^

4. Google Pixel devices lack several features of my PinePhone; luxuries such as a 3.5mm audio jack, a swappable battery, a microSD card slot, and HDMI output (with a hardware mod). ^
5. The “lock”/”unlock” terminology is hopelessly overloaded; as a result, confusion abounds online, even among phone enthusiasts. The “OEM” term here is also at best confusing and at worst misleading. I hope the screenshot and video make clear the specific context of this post, but here are definitions, and the states of the device I’m discussing:
   • “greyed” => the user interface element is inoperable
   • “ungreyed” => the user interface element is operable
   • “OEM unlocking” toggle is greyed (this is the state of the device after unboxing and before letting it have an Internet connection)
   • “OEM unlocking” toggle is ungreyed (the device must be connected to the Internet for this ungreying to take place (see video))
   • “OEM unlocking” toggle is ungreyed and toggled to “disabled”
   • “OEM unlocking” toggle is ungreyed and toggled to “enabled”
   • “OEM unlocking” toggle is ungreyed and toggled to “enabled” and bootloader is locked
   • “OEM unlocking” toggle is ungreyed and toggled to “enabled” and bootloader is unlocked (this is the state required to install GrapheneOS)

   At this point I don’t care about SIM unlocking or carrier unlocking or any other type of unlocking. There are plenty of horror stories on forums of people having purchased new Pixel phones from carriers at full price and then, via this same mechanism, the carrier never allowing bootloader unlocking (while apparently allowing various forms of SIM and carrier unlocking which are useless for running alternate operating systems like GrapheneOS). ^

6. With a Black Friday discount. ^
7. https://store.google.com/product/pixel_7_pro
   

   “Frequently asked questions

   What is an unlocked smartphone?

   An unlocked smartphone is a phone that isn’t tied to a specific carrier. When you purchase an unlocked Google Pixel phone, you get to choose which carrier or plan works best for you. Most phones in the Google Store come unlocked. Important: Google Pixel phones work with all major carriers. But not all Google Pixel 4a (5G) and later phones have 5G functionality on all 5G networks. See a list of certified carriers to make sure your smartphone works on its 5G network.

   To use a SIM-unlocked phone:
   

   1. Buy an unlocked Google Pixel phone from the Google Store.
   2. Contact a mobile carrier.
   3. Follow their instructions to set up your phone with their service plan.
   4. For 5G, some carriers may require a 5G plan (sold separately). Contact carrier for details. See g.co/pixel/networkinfofor info.” ^ ^
8. Other major phone vendors and operating systems are not in this blog’s Overton window. ^

My wife bought a dress pattern on Etsy that she received as a PDF. It was a large pattern that spanned many pages, meant to be trimmed and taped together into a large continuous sheet. The pattern was labelled “A4-letter“, meaning that it should be printable on A4 or letter paper. For accuracy, the pattern had to be printed at its native scale without cropping. The document contained a calibration box that was intended by the designer to be 5cm x 5cm exactly; I measured it with a ruler after each test print. The pattern’s internal tiles had four alignment chevrons, one on each side, which I could measure to ensure they had not been cropped. Getting the document printed perfectly turned out to be a puzzle, so I thought I’d publish what worked for me.

evince‘s “Properties” dialog showed the document’s native dimensions as “Paper Size:”, “A4, Portrait (8.27 × 11.69 inch)“, but on hand, I only had US letter 8.5 x 11 inch printer paper. I tried printing from several PDF viewers: xpdf 3.04, evince 3.38.2, and Firefox 111.0.1. I did test prints with many settings combinations in the different viewers. All my naive attempts failed in various ways, either scaling the document or cropping the margins. I won’t document all the specific failure modes, I’ll just skip to the method that worked, which combined an external tool and evince.

To change the document’s built-in margins, I used pdfcrop 2020/06/06 v1.40, part of Debian’s texlive-extra-utils 2020.20210202-3 package. First, I removed the document’s built-in margins:

$ pdfcrop pattern_A4-letter.pdf 
PDFCROP 1.40, 2020/06/06 - Copyright (c) 2002-2020 by Heiko Oberdiek, Oberdiek Package Support Group.
==> 23 pages written on `pattern_A4-letter-crop.pdf'.

The cropped document’s right and bottom edges fit on the page, but now the left and top edges spilled off. The final step was to extend the left and top margins of the cropped document by some number of units (arrived at through test-page experimentation), such that no spillover occurred on any side:

$ pdfcrop pattern_A4-letter-crop.pdf --margins "50 20 0 0"
PDFCROP 1.40, 2020/06/06 - Copyright (c) 2002-2020 by Heiko Oberdiek, Oberdiek Package Support Group.
==> 23 pages written on `pattern_A4-letter-crop-crop.pdf'.

In evince, in the “Print” dialog, on the “Page Setup” tab, I set “Scale:” to “100.0” and “Paper size:” to “A4“. On the “Page Handling” tab, I set “Page Scaling:” to “None” and left “Select page size using document page size” unchecked. All of the above steps proved necessary and sufficient to print the pattern at the document-native scale with each tile’s extents fully on the page.

This is a follow-up to my prior post about whisper.cpp. Georgi Gerganov has adapted his GGML framework to run the recently-circulating LLaMA weights. The PPC64 optimizations I made for whisper.cpp seem to carry over directly; after updating my Talos II’s PyTorch installation, I was able to get llama.cpp generating text from a prompt — completely offline — using the LLaMA 7B model.

$ ./main -m ./models/7B/ggml-model-q4_0.bin -t 32 -n 128 -p "Hello world in Common Lisp"
main: seed = 1678578687
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/7B/ggml-model-q4_0.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

main: prompt: 'Hello world in Common Lisp'
main: number of tokens in prompt = 7
     1 -> ''
 10994 -> 'Hello'
  3186 -> ' world'
   297 -> ' in'
 13103 -> ' Common'
 15285 -> ' Lis'
 29886 -> 'p'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000


Hello world in Common Lisp!
We are going to learn the very basics of Common Lisp, an open source lisp implementation, which is a descendant of Lisp1.
Common Lisp is the de facto standard lisp implementation of Mozilla Labs, who are using it to create modern and productive lisps for Firefox.
We are going to start by having a look at its implementation of S-Expressions, which are at the core of how Common Lisp implements its lisp features.
Then, we will explore its other features such as I/O, Common Lisp has a really nice and modern I

main: mem per token = 14828340 bytes
main:     load time =  1009.64 ms
main:   sample time =   334.95 ms
main:  predict time = 86867.07 ms / 648.26 ms per token
main:    total time = 90653.54 ms

The above example was just the first thing I tried; no tuning or prompt engineering — as Georgi mentioned in his README, don’t judge the model by the above output; this was just a quick test. The text is printed as soon as each token prediction is made, at a rate of about one word per second, which makes the generation interesting to watch.

I saw whisper.cpp mentioned on Hacker News and I was intrigued. whisper.cpp takes an audio file as input, transcribes speech, and prints the output to the terminal. For some time I wanted to see how machine learning projects performed on my POWER9 workstation, and how hard they would be to get running. whisper.cpp had several properties that were interesting to me.

First, it is freely licensed, released under the MIT license and it uses the OpenAI Whisper model whose weights are also released under the MIT license. Second, whisper.cpp is a very compact C/C++ project with no framework dependencies. Finally, after the code and the model are downloaded, whisper.cpp runs completely offline, so it is inherently privacy-respecting.

There was one tiny build issue, but otherwise, it just built and ran on PPC64. I was expecting to need dependent libraries and so forth, but the code was extremely portable. However, I knew it was running much slower than it could. A clue: the minor build failure was due to a missing architecture-specific header for vector intrinsics (immintrin.h) that wasn’t available for ppc64le Debian.

I took the opportunity to learn PPC64 vector intrinsics. Thanks to the OpenPOWER initiative, freely-licensed, high-quality documentation was readily downloadable from https://openpowerfoundation.org (no registration, paywalls, click-throughs, JS requirements, etc.).

I did an initial implementation for POWER9 using the IBM Vector-Scalar Extension (VSX) and the transcription speed improved considerably; for the base model, the example transcription ran in about one tenth the time. Meanwhile, the upstream project had re-organized its intrinsics support, so I reorganized my implementation to fit in. This was trickier than I expected, because of how FP32/short packing and unpacking worked in VSX.

Here is a graph of the results:

For the sake of completeness (and for my friends on #talos-workstation) I also added big endian support and confirmed that the example ran on my PPC64BE virtual machine.

I’m sure more optimizations are possible. I may try OpenBLAS (CPU) and/or ROCm (GPU) acceleration later. So far everything is running on the CPU. But I’m glad that, at least for the inference side, the Whisper model can attain reasonable performance on owner-controlled hardware like the Talos II.

One potential downside of Whisper’s trained-model approach (vs other transcription approaches, like Julius) is that for downstream projects, the model is pretty much unfixable if it has an issue. I have run whisper.cpp on real world materials with excellent results, especially with the large model. But if there are bugs, I don’t think fixing them is possible without retraining the model, which at least for Whisper, seems beyond the means of individuals.

I would like to thank Matt Tegelberg for evaluating whisper.cpp’s results against real world audio and for proof-reading this post.