How an Alsa Configuration Change Broke a Game

“LYNE” (lynegame.com) is a neat little puzzle game I’ve been playing. But I recently updated alsa-lib, audio library, and it quit working (well, it quit working, then I tracked down why). It would load the initial screen, then when I selected resolution, etc., it would crash.

I had turned off sound for the first time the last game I had played, so at first I thought maybe that broke it. I looked around and found its configuration file, tried flipping the mute-sound bit, no dice. There was also a Player.log file (part of the Unity3D kit the game uses), but I didn’t look at it initially or it would have saved some time.

Instead, I used gdb to attach to LYNE and see what the crash was (Player.log contained the backtrace, hence it could have saved this step). It showed the problem stemming from libasound.so.2 from the Steam runtime. I ignored the exact calls, which also might have saved some time in figuring it out.

First thing, I downgraded alsa-lib packages, and the game worked. So I knew where the problem was, I thought. I tried a few other Unity3D-using games with the bad alsa-lib, and they worked fine. Weird. Also, how could a system lib be causing a problem when the game uses the steam-runtime version?

I had left libasound2 at the older version and used apt-mark to hold it, but I had left libasound2-data (config files) unheld, so when I saw that I could upgrade it independently, I decided to give it a try. Aha! With the supposedly-working version of the library installed, but with the new configuration files it crashed.

So I got a diff of the config files, which is when I got hinted back to the actual breaking calls: they involved something called device name hinting, and the config file’s only changes were adding (or modifying) device name hints.

A bit more digging, I found Github: Valve Software: steam-runtime: 23 January 2015: “segfault in ALSA snd_device_name_hint(), backport fix?”. A sadly old, known, fixable issue which had suddenly, by a mere config file change, come to bite me. I left a comment and also sent a message to the developer of “LYNE” letting them know that the other Unity3D games avoid the crash, so maybe there’s a way to fix it without Valve fixing the runtime library.

Main lessons? If you see a logfile near a problem, give it a look (kudos to Unity3D for apparently throwing a similar log in with every game that uses it). If you see a call name associated with the problem, look at it first (might not have helped here, but it’s a good practice to follow). If a seemingly harmless change happens to a conf file, it could actually break another piece of software due to an old runtime library. Oy vey.

Messing with TT-RSS in Python

I use the feed reader TT-RSS to read RSS/ATOM feeds. It’s a good reader, and I also use TTRSS-Reader for Android (there are others, which I’ve not tried yet) on my (ancient) Android mobile device to quickly go through incoming feed items. But I have some itches to the web interface that I am considering scratching, so I decided to see how hard it is to start building a simpler interface.

One of the things I knew from setting up the Android application is that TT-RSS has an API, which is where I started. It communicates via JSON, which is great because it is ubiquitous and easy to use.

You have to have a session to talk to the API, so I built a class to handle all requests and the session:

class TTRSSSession():
    def __init__(self, url, user, password):
        self.session_id = None
        self.url = url
        self.user = user
        self.password = password
        self._login()

    def headlines(self, feed):
        data = {
            "feed_id": feed,
            "limit": 10,
            #"skip": 11,
        }
        resp = self._request('getHeadlines', data)
        return resp

    def _login(self):
        login_data = {
            "user": self.user,
            "password": self.password,
        }
        result = self._request("login", login_data)
        self.session_id = result['session_id']

    def _request(self, op, data):
        request_data = {
            "op": op,
        }
        if not self.session_id and op != "login":
            raise OpException("Tried to execute without a session/login.")
            return None
        elif self.session_id and op != "login":
            request_data['sid'] = self.session_id

        request_data = dict_join((request_data, data))
        request_data_j = json.dumps(request_data).encode('utf-8')
        response = urllib.request.urlopen(self.url, data=request_data_j)
        r_data_j = response.read()
        encoding = chardet.detect(r_data_j)['encoding']
        r_data_j = r_data_j.decode(encoding, 'replace')
        r_data = json.loads(r_data_j)
        if r_data['status'] != 0:
            raise OpException(
                "Bad request: {0}".format(r_data['content']['error']))
            return None
        # FIXME handling multiple parts (seq = #)?
        return r_data['content']

That’s the main guts. This isn’t very advanced/functional yet, but it’s enough for me to mess with. I also built a very simple Flask application that uses this class so I can begin sketching some UI I might want (though for now it’s very bare bones, just showing the result of TTRSSSession.headlines()).

I was going to use Django, but I think it’s too heavy for what I want out of it. Flask is very light.

A sample invocation of the class:

feeds = {
    'FRESH': -3,
    'STARRED': -1,
    'ALL': -4,
}
url = 'http://example.com/tt-rss/api/'
user = 'random_hacker'
passw = 'xyzzy'
ttr = TTRSSSession(url, user, passw)
ttr.headlines(feeds['ALL'])

The headlines() method will return the first ten items in the feed containing all items (a special feed that throws everything read and unread together).

As you can see in the class’ _request() method, I haven’t added support for pulling multiple calls together. That will likely require a JavaScript frontend to integrate with my Flask application (or the data would all get pulled by the backend first, which seems wrong). And that begs the question of why not to forgo Python altogether and just write this as a pure client-side JavaScript thing?

Maybe I will. For now I am just messing around. Depending, I might prefer to use a Python backend to help filter or manage some of the items automatically. So at this point the versatility seems useful. Also, I always feel slightly silly when I find myself doing something that feels too close to rewriting part of an application that already (and still will) have that part, so who knows. I may just use this to learn more about Flask.

Numerical Step Function in Python

The classic range() (and in 2.* series, xrange()) is useful for getting an iterator of numbers. Its full signature is: range(start, stop[, step]).

So you can do, e.g., range(5, 10) = [5, 6, 7, 8, 9].

Or you can do, e.g., range(6, 13, 3) = [6, 9, 12].

But as far as I know there’s not an easy, built-in way to iterate over a range-like set of integers defined both by the range and a number of parts desired.

An example: you want five evenly-distributed numbers starting with 1 and ending with 10. So something like [3, 5, 6, 8, 10].

In my case, I had two similar use-cases. The first was the example above: a semi-arbitrary set of values in a range. I didn’t need to strictly include the endpoints in the values, but wanted a decent distribution of the values between start and end.

The second case was a little different, in that I wanted the range to always include the ends (it was a case of covering a whole range), but I also wanted to know how much each “step” over the range amounted to.

In the na├»ve version of this case, you don’t need the magnitude, as you could cheat and throw an extra piece in to account for slight differences (e.g., 11 / 3 => 1, 4, 7, 10 with the last piece being 10 through 11).

But there’s a nice way to evenly distribute the extra pieces: using rounding of the fractional value to distribute the extras.

Example:

10 / 4 = 2.5
0 * 2.5 = 0.0
1 * 2.5 = 2.5; round(2.5) = 2
2 * 2.5 = 5.0
3 * 2.5 = 7.5; round(7.5) = 8
[0, 2, 5, 8]

(In Python, round(number[, ndigits]) of n.5 goes to the even side (when using ndigits=0 or with a single argument).)

In this case, the caller could buffer the previous value and calculate the gap/step itself, but this is Python, so we might as well give it a mode to get that itself.

Without further ado, this is what I came up with:

def equal_parts(start, end, parts, include_step=False):
    part_size = (end - start) / float(parts)
    for i in range(parts):
        part = start + round(part_size * i)
        step = start + round(part_size * (i + 1)) - part
        if include_step:
            yield (part, step)
        else:
            yield chunk

It’s messier than it needs to be, due to its dual-use nature. It’s arguably cleaner to have a second function that would handle the include_step=False case:

def equal_parts_only(start, end, parts):
    for part, step in equal_parts(start, end, parts):
        yield step + part

That function would remove the conditional business at the end of the original equal_parts:

def equal_parts(start, end, parts):
    part_size = (end - start) / float(parts)
    for i in range(parts):
        part = start + round(part_size * i)
        step = start + round(part_size * (i + 1)) - part
        yield (part, step)

In the stepless version, it’s got another nice property: what if you do equal_parts(0, 10, 11)? You get: [1, 2, 3, 4, 5, 5, 6, 7, 8, 9, 10]. That’s a nice property: getting more parts than integers in the range.

I wrote a GIMP plugin to create stepped (or random) gaussian blurs on an image. The stepless version lets me create the set of blur levels, while the step-including version lets me properly select (mostly-)even parts of the image.

Here’s an image that used this plugin containing a dual use of this function: Sample image of gaussian blur in sections

If anyone wants a copy of the plugin, let me know and I’ll put it on Github or such.

Selectively Cleaning Firefox History

Firefox does a nice job of keeping my history. But not a perfect one: certain history gets in my way, and I didn’t find a good extension to automatically clean it. So I wrote a python3 script that lets me do it manually.

GitHub: Gist: clean_places.py: Python 3 script to manually remove some Firefox history items.

#!/usr/bin/python3

# The queries operate only if either:
#  * There are more than 10 visits (for things like my mail, which only bug me
#    once they are trying to clog up my new tab page)
#  * They are > 10 days old (for things like searches; recent searches aren't
#    a big problem; old searches are).

# This should be run only if Firefox is shut down (and I mean really; you
# don't want to corrupt your DB because it was hung!)

# Also, take care about your queries!

import datetime
import sqlite3
import time


PLACES_DB = "/EDITME/path/to/places.sqlite"

conn = sqlite3.connect(PLACES_DB)

cursor = conn.cursor()

select_statement = """select url from moz_places
                      where url like :query and
                      (last_visit_date < :ten_days_ago or
                       visit_count > 10)"""

delete_statement = """delete from moz_places
                      where url like :query and
                      (last_visit_date < :ten_days_ago or
                       visit_count > 10)"""

# The time stored in the DB is in UTC, so we need to be consistent.
old_time_dt = datetime.datetime.utcnow() - datetime.timedelta(days=10)
raw_time = time.mktime(old_time_dt.timetuple())
ten_days_ago = int(raw_time * 1000000)

queries = ["%localhost/roundcube/?%", # Round Cube
           "%.google.com/search?%", # Google Search
          ]

# Set to True if you want to preview what would be deleted
TEST_MODE = False

if TEST_MODE:
    statement = select_statement
else:
    statement = delete_statement

for query in queries:
    variable_bindings = {"query": query, "ten_days_ago": ten_days_ago}
    cursor.execute(statement, variable_bindings)
    if TEST_MODE:
        for row in cursor:
            print(row['url'])

if not TEST_MODE:
    print("Deleted {} rows.".format(conn.total_changes))

conn.commit()
cursor.close()

To use it you first shut down Firefox. You should make sure it’s really shut down, as you don’t want to try to modify the database while something else is. If you want to be extra paranoid, make a fresh backup your places.sqlite file.

You need to edit the PLACES_DB variable to point to your places.sqlite file. It’s usually somewhere like: /home/YOUR_USER/.mozilla/firefox/YOUR_PROFILE_FOLDER/places.sqlite.

You need to edit your query strings (queries variable). Use % as a wildcard match.

You should set TEST_MODE to True and run the script once to make sure it’s hitting your targets. The two statements are the same except one is a SELECT and the other a DELETE.

TEST_MODE will simply print what the script would delete.

If you’re satisfied, you can turn TEST_MODE off and let it rip.

If you’re adventurous you can install the sqlite3 command line tool and open the database and explore. Note that you should also ensure Firefox is not running (at least not on the same profile) when opening your places.sqlite in sqlite3.

Future Software

Occasionally you read interesting ideas for the future of software. Here are a few to ponder.

docopt

docopt is basically a domain-specific language. The language is the standard Usage output of a command-line program. By simply documenting a program’s usage, it aims to provide the functionality of parsing the program’s invocation.

This is a great step forward in finding ways to reduce duplicated effort in an elegant way. Before, you might programmatically add or modify the argument handler, and that might generate the Usage. But it was still tedious, as it required keeping what was a messy bit of code as clean as you could. Messy because you had to do a lot of thinking about the basic form of your command line API, and then translate it to code.

Instead, with docopt you should be able to purely focus on the form of the Usage.

I think that this will be a common theme for future computing, without relying on more complex systems like SOAP.

Why do we need modules at all?

Erlang.org: Erlang-Questions: 24 May 2011: Joe Armstrong: Why do we need modules at all? was an interesting post I remember reading a few years back. It questions the basic assumption of building software hierarchically and statically.

This is akin to the age-old (at least since birth of UNIX) desire to have programming be a set of pipes thrown together at will, with the elegance and wisdom of a seasoned hacker behind it.

What it really seems to suggest though is a move toward a way to let computers program themselves. The desire to let evolution play a key and direct role in how a computing system operates. If some given function is replaced by a more efficient function, all software depending can be faster. If you have some chain of functions A->B->...->ZZ, and someone writes a better chain, the whole can be replaced.

We already get some of this with packages/modules and dynamic linking, so the question (short-/mid-term) is really whether a kind of function-level alternative would replace modules/packages, or work alongside them.

Why do we need GUIs at all?

This is a spin on the previous. It’s the basic question of whether larger elements of the GUI can be easily reused. At present the basic unit of the GUI is the widget. The widget is some semi-small piece of the GUI, like a button or a text input.

If you want a more complex GUI, it’s a matter of assembling widgets together. This is equally true for a native application as it is for a webpage or web application.

Reuse is dependent upon some human brain recognizing the opportunity, and wiring the whole thing up for reuse. That’s opposed to something like the hypothetical functions database above, which in theory would have enough metadata to be able to say, “takes in a list, outputs a sorted list, stable, memory use, complexity.”

But if you can do it with functions (which seems likely enough), you can do it with larger pieces, including GUIs. So you can have a GUI that’s designed separately from the consuming code, and it can say, “presents a list, allows reordering, sorting, adding, deleting,” and you might specify some constraints on the adding/editing portion, so it could select another GUI component (“unique (name, number) pair”, or “unique name, non-unique number pair”, or whatever).

The initial versions of this can probably be written off of docopt, where by the same magic of parsing Usage a GUI could be created for any command line API. This is a project I’ve been meaning to work on, but haven’t done it yet.

Common theme

The common idea is reducing duplication of work, increasing productivity, removing friction to create. Making the computer figure out more of the work. Automating more. Instead of having to spend so much time doing analysis of tradeoffs and pondering why something broke when it shouldn’t, we should be building more.

That leads to the last one: moving away from pure-text programming. That doesn’t mean we write it out to binary; the text layer can persist. But it does mean that to some extent we stop doing everything as text. We can already extract interaction diagrams from text, but we currently do the editing in text and only get the diagrams when we want a conceptual view.

It will take some time before we see these last sorts of changes take place. But it’s likely that they will, if for no other reason than the benefits are there. I’d rather write the Usage once and know everything was wired up for me, just like I’d rather not muck about with every last bit of CSS and markup every time I want to add something to an application.

And as fun as it is to find the extra comma and find out that I wrote it correctly except for that typo, I’d rather that footgun go away entirely.