The Mysterious CFexpress Card Specification / Pinout

It seems like CFexpress is going to replace CFast and CF card as the new de-facto DSLR storage medium, at least for Japanese camera manufacturers. The Internet kept its mouth very shut about what this CFexpress Card actually is, with specification no where to be found.

However, piecing a few separate pieces of information together, we can make up pretty much what the specification is: it’s more-or-less M.2 with non PCI-e pins removed, with a new “INS#” pin. It’s also single sided, and utilizes a PCB edge connector.

The connector looks like this:

Here’s the pin assignment table:

PINSignalI/O DirectionRequired on Media?
R/O/NC
Required on Host?
R/O/NC
Notes
1GNDRR
2PERp1OOR
3PERn1OOR
4GNDRRTie to ground if lane 1 present
5PETn1IOR
6PETp1IOR
7ReservedNCR
8ReservedNCR
9PERST#IRR
10+3V3RR
11CLKREQ#OROActive Low.
Pull-up of >5kOhm on Host
12INS#ORRActive Low.
Tie to ground on media side.
Pull-up of 100-200kOhm on Host
13REFCLK-IRR
14REFCLK+IRR
15GNDRR
16PERn0ORR
17PERp0ORR
18GNDRR
19PETn0IRR
20PETp0IRR
21GNDRR
PIN assignment / pinout. Source: http://www.farnell.com/datasheets/2815914.pdf

Host-side dimensions drawing:

(From: http://www.rego.com.tw/_admin/upload/product/rego_15562428745ngxni.pdf)

It seems trivial to make a dummy CFexpress card and just use a NVMe drive instead. Project for another day?

sparsebundle-fuse: Another Implementation for Reading / Writing MacOS Sparsebundles on Linux

There are already a sparsebundle implementation for Linux, but that’s read-only and only supports FUSE. This implementation is read / write, AND supports NBD (network block device) so the sparsebundle actually appears as a block device on the system, without you needing to loop mount the file in FUSE.

There are some useful things that you can do with a sparsebundle, even on Linux. For example you can rsync an entire a file-system (say, XFS), without losing any metadata. Or you can abuse Google Drive by fuse mounting it and creating a full system on you Google Drive. (rootfs on Google Drive, anyone?)

Get it here.

Making a Pittsburgh Map Coaster

Inspired by Dr. Chris Harrison’s Laser-Cut Map project, and since I have a laser cutter myself (because why not), I decided to make my own Pittsburgh Map Coaster, and probably another one for Shanghai. I’ve been living in Pittsburgh for about 6 years now and there’s definitely a part inside me that belongs to Pittsburgh.

Instead of starting directly from OpenStreetMaps, I decided to work with mapbox which takes data from OpenStreetMaps. The major reason behind this is that it provides a lot of styling options, which allows me to create a binary map without much effort.

Pittsburgh Map in Binary Colors

The original PNG is at a resolution of 8000×8000, but I don’t think the software that came along with it can handle that. So I will be supplying a 2000×2000 image to the laser cutter instead.

Now this looks great. Time to engrave.

Engrave away!

The first attempt is at 40% power and 100mm/s speed.

Laser Engraved Pittsburgh Map Take 1

This looks great, but it engraves too deep and burnt the wood a little. So I’m lowering power to 30% and upping speed to 200mm/s. I also circular cut the product with 60% power and 10mm/s speed.

Higher speed + lower power + circular cut
Close up
Used as a coaster

DIY $10 Slack Notification Light

While being isolated at home, my company uses Slack more heavily than ever, and that Slack chime is starting to get old. What if there’s a physical light that can notify me instead?

I have a Smoko ambient light and a few of Digisparks lying around screaming “use me!”. So I started the project using a combination of these two.

Digispark
Smoko Ami Shrimp Sushi Ambient Light

The circuit is really simple, just a NPN and a resistor. I used a NPN with a beta (use hFE on datasheet for a rough estimate of beta) of 40, and with a 1K resistor the NPN gives about 130mA to the LED. If you want to do this project too, change the R1 value to match your desired LED current.

Schematics

I have to make some “battery adapters” to provide power to the light:

“Battery adapter” made with nickel strips
Circuitry wrapped in heat-shrink

On the software side, I don’t really want to reinvent the wheel and build a USB protocol from scratch, so I used code from blink(1) and adapted it for the Digispark.

The rest is relatively easy: use the blink(1) client and set up a slack integration with IFTTT.

Code:

https://github.com/gyf304/usblight

Fixing a Clothing Rack during COVID-19 Lock-down

My clothes hanging rack is broken for a bit of time has developed a crack. It’s obviously time to fix it. I already have a piece of aluminum pipe on hand that should serve as a clothes hanging pole. The only problem is that I don’t have the hardware to mount the pole to the wall, at least not easily during the COVID-19 lock-down.

So, time to home-brew my own solution. First thing would be designing the fastening piece to attach to the wall. I measured the diameter of the pole and the diameter of the screws I have on hand, and designed this in OnShape. (model available here)

The 3D Rendering of the Mounting Bracket
Printed

Works great. A pretty good job given the current circumstances.

Library Occupancy Project Follow Up

The occupancy project is now deployed at Sorrells Library. After running for more than a week, it’s time to see how it performs…

So I dumped the database, feed the dump into a python script for processing, and plotted occupancy for each day. And the results are surprisingly good! Let’s take a look.

Here is occupancy plotted vs. time of the day. We are able to see multiple peaks that sometimes exceeds total capacity of the library. It may sounds strange but there is a scientific explanation. The system detects Wi-Fi devices (and therefore occupancy) present in the last ~3 minutes. So if there are a lot of people moving in and out of the library, the estimated occupancy will increase rapidly, since the system counts everyone in the library during that time, even if the person is leaving / entering the library.

As explained above, large flows of people are shown on graphs as peaks. I marked the most identifiable ones.

Not surprisingly, the peaks coincide start/end times of popular courses. Here’s occupancy data on Saturday for comparison.

Now if we merge these together…

Fixing DjVu eBooks with Problematic Page Sizes

There are a lot of scanned books in the DjVu format, but many of them are not professionally scanned. One of the DjVu books I encounter has different page sizes (page size changes every few pages). There are nice pdf scaling programs, but no such program for DjVu exists.

So I made my own. The python 3 program searches for INFO blocks which specify sizes and DPIs of images and change DPIs to match a constant width. The program needs improvement though – it currently reads the entire file into memory for processing. However, this is not a problem for my files so I’m not going to change it.

DjVu Specification

# usage: python3 djvu.py [FILE] [PAGE_WIDTH(IN)]
import sys
import struct

def main():
    args = sys.argv
    if len(args) != 3:
        print('filename? page_width?')
        return
    filename = args[1]
    page_width = float(args[2])
    file_bytearray = None
    with open(filename, mode='rb') as f:
        file_bytearray = bytearray(f.read())
    for i in range(len(file_bytearray) - 8):
        if file_bytearray[i:i+7] == b'INFO\\0\\0\\0':
            info_start = i+8
            (width, height) = struct.unpack('>HH', file_bytearray[info_start:info_start+4])
            desired_dpi = int(width / page_width)
            file_bytearray[info_start+6:info_start+8] = struct.pack('<H', desired_dpi)
    with open(filename+'mod.djvu', mode='wb') as f:
        f.write(file_bytearray)

if __name__ == '__main__':
    main()


Before

After

The Making of the Library Occupancy Estimation System, Part 5 – the Enclosure

It’s almost time to deploy the ESP8266s. I switched from the previous board to wemos mini lite, a smaller board with a variant of ESP8266 (ESP8285). The ESP8285 includes a on-chip 1M EEPROM, reducing the cost. The most important difference is that the newer boards don’t have pre-soldered pin headers. Desoldering pins from the previous board is proven to be tedious and time-consuming.

While waiting for the boards being shipped from China to me, I designed a enclosure for that board. I can, of course, just stick the boards to walls. But a case is always nice – it makes the project look less cheap (while it actually is pretty cheap).

The design is consisted of 2 parts: the body and the cap. The body is going to be 3D printed while the cap is going to be cut from clear acrylic panel. 4 M1.5 screws hold the components together.

The parts are modeled with Autodesk Fusion 360…

and then made…

and compared to my sketch…

The Making of the Library Occupancy Estimation System, Part 4 – the Frontend

The only part missing was a front-end. It is 2017, and splitting up front-end and back-end is now a must. This is even more true in this project: serving html and other resources from my free Heroku server does not sound like a good plan – with only 1 web instance, the server can be easily saturated. So the plan was to host the front-end on GitHub.

The frameworks I chose for this job is Vue for MVVM and MaterializeCSS for the appearance. The reason for using Vue instead of React and Angular was Vue is more suited for small project like this, and is extremely easy to setup (include a js file, add 5 lines of code and Vue is ready). The reason for using MaterializeCSS was simply because it looks nice.

A few hours later:

The front-end is pretty bare-bones (< 100 SLOC of js), but it does the job. And thanks to MaterializeCSS, it doesn’t look half bad. The page is also responsive (again, thank you MaterializeCSS).

The front-end already looks pretty nice on phones, so I took a step further to make it an iOS Web App (no I don’t want to pay $99/year to make it an actual iOS App).

The first thing to do when developing a mobile app is of course to design an app icon.

It won’t get me any design award but hey, it’s an icon.

The tricky part of making the iOS Web App is trying to make it act like a native app. Some CSS magic was used to make it app-like. Also I spend I lot of time working around an iOS bug that made the entire html scroll and bounce even if I told it not to… A few failed attempts later I gave in and used a library called iNoBounce. That worked well.

How the icon looks on home screen:

Good enough.

Frontend Code on GitHub

The Making of the Library Occupancy Estimation System, Part 3 – the Server

So the sniffer part was complete. The server was the missing piece of the puzzle.

The server needs to do several things

  1. Receive data from the sniffer and decrypt
  2. Save sniffed data (probe requests) to a database
  3. Analyze probe requests and deduce occupancy
  4. Provide public APIs for retrieval of occupancy information

I’m using python3 for the server since it’s one of the better-supported languages on Heroku. Unfortunately, XTEA implementations on python3 do not work well, so I reimplemented it by porting my embedded-C version. (One of the advantages of CTR mode is that the encrypting function is its own inverse. So encrypt == decrypt.) Sniffer MAC address and information of sniffed devices are recovered from decrypted data. These are stored to a DB after some simple processing (filtering out invalid MAC addresses, etc.).

For storage of probe requests, I use sqlalchemy which is based on SQL. I didn’t use one of the now-popular no-SQL databases since I’m not storing GBs of data. Also, enforcing schemas and relationships clears up the logic of the server.

I also made a simple linear model to deduce occupancy from probe requests. The model is run every 30 seconds and output is stored back into the DB. It is not efficient to run the model every time a user asks for occupancy information. When a user requests occupancy, pre-computed occupancy is read from DB and directly sent to the user.

The public API is RESTful and based on simple JSON. I did not bother with pre-existing protocols like JSONAPI 1.0. These protocols are way too complicated and verbose for this project. When the cost of implementing a complicated protocol exceeds the potential benefits of that protocol, it’s generally a good idea to steer away from it.