The site uses cookies that you may not want. Continued use means acceptance. For more information see our privacy policy.

Tag: code

Messing with TT-RSS in Python

Posted in software by Adam
Posted on

Some sample code I wrote to feel my way around the TT-RSS API in Python.

I use the feed reader TT-RSS to read RSS/ATOM feeds. It’s a good reader, and I also use TTRSS-Reader for Android (there are others, which I’ve not tried yet) on my (ancient) Android mobile device to quickly go through incoming feed items. But I have some itches to the web interface that I am considering scratching, so I decided to see how hard it is to start building a simpler interface.

One of the things I knew from setting up the Android application is that TT-RSS has an API, which is where I started. It communicates via JSON, which is great because it is ubiquitous and easy to use.

You have to have a session to talk to the API, so I built a class to handle all requests and the session:

class TTRSSSession():
    def __init__(self, url, user, password):
        self.session_id = None
        self.url = url
        self.user = user
        self.password = password
        self._login()

    def headlines(self, feed):
        data = {
            "feed_id": feed,
            "limit": 10,
            #"skip": 11,
        }
        resp = self._request('getHeadlines', data)
        return resp

    def _login(self):
        login_data = {
            "user": self.user,
            "password": self.password,
        }
        result = self._request("login", login_data)
        self.session_id = result['session_id']

    def _request(self, op, data):
        request_data = {
            "op": op,
        }
        if not self.session_id and op != "login":
            raise OpException("Tried to execute without a session/login.")
            return None
        elif self.session_id and op != "login":
            request_data['sid'] = self.session_id

        request_data = dict_join((request_data, data))
        request_data_j = json.dumps(request_data).encode('utf-8')
        response = urllib.request.urlopen(self.url, data=request_data_j)
        r_data_j = response.read()
        encoding = chardet.detect(r_data_j)['encoding']
        r_data_j = r_data_j.decode(encoding, 'replace')
        r_data = json.loads(r_data_j)
        if r_data['status'] != 0:
            raise OpException(
                "Bad request: {0}".format(r_data['content']['error']))
            return None
        # FIXME handling multiple parts (seq = #)?
        return r_data['content']

That’s the main guts. This isn’t very advanced/functional yet, but it’s enough for me to mess with. I also built a very simple Flask application that uses this class so I can begin sketching some UI I might want (though for now it’s very bare bones, just showing the result of TTRSSSession.headlines()).

I was going to use Django, but I think it’s too heavy for what I want out of it. Flask is very light.

A sample invocation of the class:

feeds = {
    'FRESH': -3,
    'STARRED': -1,
    'ALL': -4,
}
url = 'http://example.com/tt-rss/api/'
user = 'random_hacker'
passw = 'xyzzy'
ttr = TTRSSSession(url, user, passw)
ttr.headlines(feeds['ALL'])

The headlines() method will return the first ten items in the feed containing all items (a special feed that throws everything read and unread together).

As you can see in the class’ _request() method, I haven’t added support for pulling multiple calls together. That will likely require a JavaScript frontend to integrate with my Flask application (or the data would all get pulled by the backend first, which seems wrong). And that begs the question of why not to forgo Python altogether and just write this as a pure client-side JavaScript thing?

Maybe I will. For now I am just messing around. Depending, I might prefer to use a Python backend to help filter or manage some of the items automatically. So at this point the versatility seems useful. Also, I always feel slightly silly when I find myself doing something that feels too close to rewriting part of an application that already (and still will) have that part, so who knows. I may just use this to learn more about Flask.

Better Know a Package: clamz

Posted in linux by Adam
Posted on

A new themed post where I look at some piece of software on my system to explain it in detail. The inaugural package is: clamz, a program to download music from Amazon’s music store.

So I figure it’s good to dig into things at random sometimes, to see what wisdom they will lay upon us.  It’s like when you were a kid and your parents were going to throw away some piece of broken technology (this was before eCycling, of course), they would let you pull it apart (with a hammer, if necessary, though you tried to stick to screwdrivers).  You got to see the guts.  Those little buttons were just proxies for the real buttons (be they plain switches or membranes or whatever).

But, that’s another topic for another day.  The first thing we need is a package to dig into.  That requires either inspiration or randomness, and since I don’t have one in mind, we’ll rely upon the latter.  So, open a shell and put the package manager and other tools to work to pick one.

For the first try, I’ll most restrictions to the search, but if it picks something small like a subpackage of a larger piece of software, I’ll use the main package. I don’t want virtual packages, non-free packages, or libraries to be chosen, so I will ignore them. Also, I only want packages that I’ve got installed, as there’s some chance I’ll have a little familiarity with them.

So:

aptitude search '~i!~snon-free!~slib!~v'
  • ~i means “installed”
  • ! means “not”
  • ~s[expression] means “section contains [expression]”
  • ~v means “virtual packages”

And I got a list of packages back. A little over 1,200 packages (if I pipe the output (command | otherCommand) to wc -l, it counts the lines for me). And now I want to pick one of them, so I’ll rely on shuf to make my life easy. I’ll pipe the search to shuf -n1, which will shuffle the incoming lines and then output the number I’ve chosen, in this case a single line.

So: 

aptitude search '~i!~snon-free!~v!~slib' | shuf -n 1

And we get: 

i   clamz                           - command-line program to download MP3's fro

Well, it gets cut off due to the terminal only being 80 characters wide. So now the investigation begins.

First, I open aptitude proper and type /^clamz$[enter] which searches for that exact name. It highlights the package in the listing:

Actions  Undo  Package  Resolver  Search  Options  Views  Help
C-T: Menu  ?: Help  q: Quit  u: Update  g: Download/Install/Remove Pkgs
aptitude 0.6.4
i     clamz                                                0.4-3      0.4-3     
p     clive                                                <none>     2.2.27-1
p     clive-utils                                          <none>     2.1.6-1
p     clzip                                                <none>     1.2-2
p     cmip5-cmor-tables                                    <none>     1.3.10-1
p     cmospwd                                              <none>     5.0+dfsg-2
p     codfis                                               <none>     0.4.7-1
p     collectd                                             <none>     4.10.1-2.1
p     collectd-core                                        <none>     4.10.1-2.1
p     collectd-dev                                         <none>     4.10.1-2.1
command-line program to download MP3's from Amazon
Clamz is intended to serve as a substitute for Amazon's official MP3           ▒
Downloader, which is not free software. Clamz can be used to download either   ▒
individual songs or complete albums that you have purchased from Amazon.       ▒
Homepage: http://code.google.com/p/clamz/                                      ▒

Now, I press [enter] to examine it. This shows the description again, but tells me the section, priority, maintainer, and so forth.

The interesting part begins with the Depends line:

  --\ Depends (4)                                                               
    --- libc6 (>= 2.3)
    --- libcurl3-gnutls (>= 7.16.2-1)
    --- libexpat1 (>= 1.95.8)
    --- libgcrypt11 (>= 1.4.6)
  --- Packages which depend on clamz (0)
  --\ Versions of clamz (1)
i    0.4-3

This gives us an overview of what the source is probably like. It depends on libc6, which means it’s at least partly written in C. It uses libcurl-gnutls, which is the library that supports the tool curl which downloads things over HTTP (ie, the dominant web protocol). In this case, it’s the version of curl with security support provided by GnuTLS which is the GNU Project‘s implementation of TLS, used to encrypt connections to websites and other Internet/network services.

That makes sense, as the description told us clamz downloads music from Amazon.

Indeed, I was overjoyed to find clamz as a substitute for the seldom-updated and clunky official tool provided by Amazon. It’s a great little tool, does one thing and does it well.

But let’s keep digging. What about libexpat and libgcrypt? I’m not familiar. The latter is almost definitely something to do with encryption, but we can simply select them in turn and read about them.

libexpat‘s information screen says:

i A  --\ libexpat1                                         2.0.1-7    2.0.1-7   
  Description: XML parsing C library - runtime library
    This package contains the runtime, shared library of expat, the C library
    for parsing XML. Expat is a stream-oriented parser in which an application
    registers handlers for things the parser might find in the XML document
    (like start tags).
  Homepage: http://expat.sourceforge.net

I can also read the number of packages that depend on it (directly; indirectly would take some traversal): 292 packages.

And, libgcrypt‘s information screen says:

i    --\ libgcrypt11                                       1.5.0-3    1.5.0-3   
  Description: LGPL Crypto library - runtime library
    libgcrypt contains cryptographic functions.  Many important free ciphers,
    hash algorithms and public key signing algorithms have been implemented:
    Arcfour, Blowfish, CAST5, DES, AES, Twofish, Serpent, rfc2268 (rc2), SEED,
    Camellia, CRC, MD4, MD5, RIPE-MD160, SHA-1, SHA-256, SHA-512, Tiger,
    Whirlpool, DSA, DSA2, ElGamal, RSA, ECC.
  Homepage: http://directory.fsf.org/project/libgcrypt/

So, libgcrypt is used for computing hashes and other cryptographic uses. Neat. How many packages directly depend on it? 278 packages.

Oh, for the record we’ll look at the direct dependent count for the other two libraries:

  • libc6 has 16,978 direct dependents
  • libcurl3-gnutls has 235 direct dependents

All of the libraries that clamz uses are well-used by a lot of other packages. That’s a good thing, as it means their code is highly tested and dependable. It also means that the author(s) of clamz didn’t have to write or maintain any of the code that they depend upon, so that saved them time to work on their own tool.

Now let’s dig a little bit more into clamz itself. To do this, I am going to grab the source that Debian ships (except for the non-free packages, all Debian packages have the source code available for reuse by you or anyone). So (as a regular user):

mkdir clamz
cd clamz
apt-get source clamz

And voila, the source has been downloaded and extracted into clamz/clamz-0.4 with all the necessaries for building it. Okay, not all. It’s possible that I don’t have the development copies of its dependents installed, in which case I would type (either with sudo or as a superuser) apt-get build-dep clamz to tell it I want to have all of the packages that I need to actually build clamz.

But apt-get source [package] does get the source and extract it in a manner that it’s ready to be built into a package. The Debian package parts happen in the [package]/debian folder.

clamz comes with five .c source files and one .h source header file. It also has a man page file, a desktop file, a xml file used for its MIME registration, and its compilation structure (configure files and makefile). And, there’s some folder called .pc which I have no clue about, so let’s look at it.

Ah, I immediately recognize the name quilt that prefixes several of the files, along with several others including the word patch. This folder is obviously there to contain patches to the original source and information about those patches. At present it only contains one patch, and the applied-patches file tells us that it is applied; it’s a patch (fix-clamz-desktop.patch/) that adds the desktop file I mentioned, which is used to add the file to the frontend of GNU/Linux systems, so that users can find it in menus or searches.

I think that’s as far as we go, for now anyway. I could explain the structure of the Debian-specific packaging a bit more, but it’s a simple (ie, good) package so there aren’t a lot of details there. The rules file only contains a command to install the man page.

In other editions of this (if I continue it), I will probably look at source a bit.

If you do use GNU/Linux, and you do download music from Amazon, you should at least consider using clamz. As seen here, the dependencies are a lot nicer than those of the official tool, and it’s definitely more UNIXy.