Yifan Gu's Blog

There are a lot of scanned books in the DjVu format, but many of them are not professionally scanned. One of the DjVu books I encounter has different page sizes (page size changes every few pages). There are nice pdf scaling programs, but no such program for DjVu exists.

So I made my own. The python 3 program searches for INFO blocks which specify sizes and DPIs of images and change DPIs to match a constant width. The program needs improvement though – it currently reads the entire file into memory for processing. However, this is not a problem for my files so I’m not going to change it.

DjVu Specification

# usage: python3 djvu.py [FILE] [PAGE_WIDTH(IN)]
import sys
import struct

def main():
    args = sys.argv
    if len(args) != 3:
        print('filename? page_width?')
        return
    filename = args[1]
    page_width = float(args[2])
    file_bytearray = None
    with open(filename, mode='rb') as f:
        file_bytearray = bytearray(f.read())
    for i in range(len(file_bytearray) - 8):
        if file_bytearray[i:i+7] == b'INFO\\0\\0\\0':
            info_start = i+8
            (width, height) = struct.unpack('>HH', file_bytearray[info_start:info_start+4])
            desired_dpi = int(width / page_width)
            file_bytearray[info_start+6:info_start+8] = struct.pack('<H', desired_dpi)
    with open(filename+'mod.djvu', mode='wb') as f:
        f.write(file_bytearray)

if __name__ == '__main__':
    main()

Before

After

So the sniffer part was complete. The server was the missing piece of the puzzle.

The server needs to do several things

1. Receive data from the sniffer and decrypt
2. Save sniffed data (probe requests) to a database
3. Analyze probe requests and deduce occupancy
4. Provide public APIs for retrieval of occupancy information

I’m using python3 for the server since it’s one of the better-supported languages on Heroku. Unfortunately, XTEA implementations on python3 do not work well, so I reimplemented it by porting my embedded-C version. (One of the advantages of CTR mode is that the encrypting function is its own inverse. So encrypt == decrypt.) Sniffer MAC address and information of sniffed devices are recovered from decrypted data. These are stored to a DB after some simple processing (filtering out invalid MAC addresses, etc.).

For storage of probe requests, I use sqlalchemy which is based on SQL. I didn’t use one of the now-popular no-SQL databases since I’m not storing GBs of data. Also, enforcing schemas and relationships clears up the logic of the server.

I also made a simple linear model to deduce occupancy from probe requests. The model is run every 30 seconds and output is stored back into the DB. It is not efficient to run the model every time a user asks for occupancy information. When a user requests occupancy, pre-computed occupancy is read from DB and directly sent to the user.

The public API is RESTful and based on simple JSON. I did not bother with pre-existing protocols like JSONAPI 1.0. These protocols are way too complicated and verbose for this project. When the cost of implementing a complicated protocol exceeds the potential benefits of that protocol, it’s generally a good idea to steer away from it.