From one challenge and into another

Collage image of Flickr and Roguelike banner

Well, I didn’t complete the Every Day in May challenge like I was hoping to, but I did do better than last year. Last year I completed 10 of the daily challenges; this year, I finished 17. But even without completing all of the prompts, I am pretty happy with a few of the sketches:

So, what’s this other challenge then? This go around it’s not a sketching challenge, but a programming one instead: r/roguelikedev Does The Complete Roguelike Tutorial. Basically, we’ll follow along with a multi-step tutorial - completing a little bit each week - and by the end, we should have a somewhat full-featured, text-based game. Since the tutorial uses the Python language, that’s another bonus the challenge has.

I had never played many roguelike games in the past, but more recently they have caught my interest. Since I’ve lost most of any skill I had in the past for first-person shooter games (which I played a lot of in the 90s and early 00s), I’ve become more interested in other kinds of video games. One example, is Dwarf Fortress. While it’s not technically a roguelike - though it looks a lot like one - I’ve been diving into the madness that is that game. I’m sure I’ll have a post on that game at some point, too.

~ ~ ~

Giving the Every Day in May challenge another try

Every Day in May 2019, first day entry

If you look at my Flickr album for 2018’s Every Day in May, you’ll see I was only able to complete 10 of the 31 challenges. I can’t recall what led me to stop keeping up with the entries, but I plan to give it a better attempt this year.

I’ll admit that some of this year’s prompts are a little different - like “Your favorite song” - but overall I think they’re all doable. And there are a few I am looking forward to, like “a ruined building,” and “a toy.” I should be able to have fun with those and most of the others on the list.

As I complete each challenge, I’ll post the result to Flickr and my Instagram. Now, if only I could find my watercolor set.

2019-05-01 — #art #sketching
~ ~ ~

Using the Flask-RESTful library

In my last post, I wired up a very simple API server in Python with Flask. This time, I’ll add in a library meant to make building APIs a bit easier: Flask-RESTful.

The goal will be to add in the library and adjust all code necessary to take advantage of this library. If all goes well, then the tests we wrote the last time should still pass. Let’s cross our fingers.

The code for this time will be in the same repository, github.com/billturner/simple_flask_api, but this time the branch used will be 02-flask-restful.

Installing Flask-RESTful

Like you did in the first tutorial, initialize the virtual environment, and then let’s install the library:

pip install flask-restful

Many of the library’s dependencies will already be installed, so the process should go pretty quickly.

Configuration

Since we’ll be updating our app.py file in chunks, we’ll leave the existing library import statements as they are, and then add what we need from flask-restful. Thankfully, there’s a handy quickstart guide on the library’s doc site, which is just about all we’ll need to reference for our changes.

Open up our app.py file, and add the following below the existing from flask import Flask line (again, the [...] signifies code that doesn’t change):

[...]
from flask_restful import Resource, Api
[...]

Then, underneath where we assign the app variable, we’ll add a new api variable that flask-restful will use:

[...]
api = Api(app)
[...]

Updating our first route

Let’s start out updating our useless ‘hello world’ route since it’s the simplest one. Delete - or comment out - the existing code for the hello world route, and then add the following:

[...]
class HelloWorld(Resource):
    def get(self):
        return {'message': 'Hello world!'}
api.add_resource(HelloWorld, '/')
[...]

Save the file, if we start up the app from the command line, we should be able to try out our new ‘hello world’ route. As a reminder, you start up the app at the command line like this:

python app.py

And once that starts up, point your web browser to http://localhost:5000/ and you should still get the familiar JSON ‘hello world’ message:

{
  "message": "Hello world!"
}

Success! Now, just for sanity’s sake, let’s run the tests and see if they all still run clean. Like before, run the pytest command at your prompt to execute the test suite. All the tests should pass just like before.

What we changed, and why

The biggest change here was taking the previous @app.route() decorator and its function away, and replacing it with a class called HelloWorld - which inherits from the flask-restful Resource. Then we specified a get() function in the class to handle our GET request; which in our case, just returns a simple JSON object. Next, we used the add_resource method to add our route to the api object.

To be honest, at this point I’m not sure I’m seeing a major benefit to using the library. Of course, we just updated a basic GET route, but we didn’t really reduce much in the way of lines of code, although we were able to get by without using the jsonify method from the main flask library.

However, one benefit I do see that could become useful in the future is that each route now has its own class for defining what happens. With this, we can easily extract that code into a separate file, one for each route if we wanted to. When you start to build a more complex API service, this could be quite handy.

Updating the remaining routes

Similar to how we replaced the ‘hello world’ root route, let’s do the same for the main list listing route:

class GetLists(Resource):
    def get(self):
        return {'lists': lists}
api.add_resource(GetLists, '/lists')

Before moving forward, quickly run the tests, and see if that route is working as it should. Then, update the single list route:

class GetList(Resource):
    def get(self, list_id):
        _list = {}
        search = [_list for _list in lists if _list['id'] == list_id]
        if len(search) > 0:
            _list = search[0]
        else:
            abort(404)
        _items = [_item for _item in list_items if _item['list_id'] == list_id]
        if len(_items) > 0:
            _list['list_items'] = _items
        return {'list': _list}
api.add_resource(GetList, '/lists/<int:list_id>')

There are a few more changes here. You’ll notice now we add the list_id parameter to the get function. Otherwise, the code is pretty much the same. Let’s run the tests now that we have the single list route added.

Uh oh! If you’re like me, you got an error when running pytest:

>       assert json['error'] == '404 Not Found'
E       KeyError: 'error'

test_app.py:53: KeyError

It appears that the abort(404) we’re using when a list isn’t found does not work like it used to. Luckily, all the other tests did pass without problems.

What happened with abort()

If you add in a print(json) line in the failing test - before the assert line, you’ll see that we do get a ‘not found’ message, but it’s just not the one we set up:

{
  "message": "The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again."
}

So, where is this error coming from? If you search for that error message, we find it comes from the werkzeug library (and werkzeug.exceptions specifically). This library is a requirement of the main flask library. That’s fine, but it’d be nice to know why this is overriding the abort() we’re specifically calling.

From looking at one of the main flask_restful files (init.py), it’s overriding the default flask abort() method with its own version. I don’t really want to spend too much time digging into the chain of order, so I think we’ll just adjust to using the error handling provided by the flask_restful library to make things a bit more simple.

Updating our error handling

To keep things fairly simple, we’re going to change as little as possible to get the tests working again. First, let’s update our imports at the top of app.py to remove importing abort from flask and instead, import it from flask_restful:

from flask import Flask, jsonify
from flask_restful import Resource, Api, abort
[...]

Next, update the get function in the GetList class to use the different abort syntax:

abort(404, error='404 Not Found')

Since the general 404 error handling is working fine, we don’t need to update that at all. After saving, and re-running pytest at the command line, you should have an error-free output.

Wrapping up

That didn’t require too many changes, and having the test suite in place definitely made validating the changes made quite easy. I’m curious if the other libraries are just as simple to use.

Next time, I think I’ll take a look at the Flask-RESTPlus library, and see how it works with our simple API service.

Resources

For this post, the only extra resource needed was the Flask-RESTful documentation, which covered our use basic case.

~ ~ ~

Simple JSON API with Python and Flask

This should be the first of a few blog posts explaining how to get a simple API up and running with Python, and using the Flask framework. I’ll try and link all these together once I get more up. And if you’d like the source code at any time, you can grab it here: github.com/billturner/simple_flask_api. The branch for this part will be 01-bare-bones.

I’ll be using Python 3.x, and I’m going to assume you have it all set up and in your path. If you don’t have it set up, you can find a downloadable installer on the Python web site.

Setting up the virtual environment and installing Flask

We want to create a directory for our Python files, so make one of your own choosing; I used simple_flask_api. Change into that directory, and from here on out all files should go in here. I’ll use the virtual environment ability built into newer versions of Python, and I’ll call the environment venv:

python -m venv venv

To enter the virtual environment, you’ll need to run the appropriate initialization script. Since I’m on Windows, I’ll be executing this command in PowerShell, so what I use may be different than you. For most Linux and OS X users, you’ll use another command which I’ll list below (for other platforms, there are instructions in the Python docs):

# for Windows/PowerShell users, you should run:
.\venv\Scripts\Activate.ps1
# for most Linux and OS X users, you should run:
source ./venv/bin/activate

Now, you should have a prompt that has (venv) at the beginning, like this (the ... is just me shortening the path):

(venv) PS C:\Users\...\simple_flask_api>

For the rest of this tutorial, we’ll assume you’re in this directory and have activated the virtual environmen. Once you have activated the virtual environment, you can install the first of the two libraries we’ll need for this part of the tutorial, Flask:

pip install flask

You should see some text scroll by on the screen as the library (and its dependents) are installed. Now we’re ready to build an API.

Hello world

A tutorial isn’t a tutorial without a “hello world” example, so let’s write up a quick one (a version of which is in a lot of Flask tutorials). Create a new file named app.py and place the following code in it:

from flask import Flask

app = Flask(__name__)

@app.route("/", methods=['GET'])
def hello():
    return 'Hello world!'

if __name__ == '__main__':
    app.run(debug=True)

What we’re doing here is importing from the Flask library, initializing the app with Flask, and then creating a simple route, and a function to be called when that route is called. For this tutorial, we are only going to be fetching data from our API, so the only HTTP method we’ll need is GET.

We’re adding debug=True to app.run() so that when you make changes to the source code, it will automatically reload the app. Now, to start the app running, execute this command where we’ve been building this:

python app.py

You should get some feedback from Flask starting up, and then a URL you can use to access the app in the browser. Should be something like http://127.0.0.1:5000/. If all went well, then you should see the string “Hello world!” in your browser.

But wait, we’re building an API, and a basic string isn’t helpful when building an API, JSON is. So, let’s also import the jsonify utility from Flask and then return a JSON object from our hello() function (I’m only showing the changed lines here, and the [...] signifies code that doesn’t change):

from flask import Flask, jsonify
[...]
@app.route("/", methods=['GET'])
def hello():
    return jsonify({'message': 'Hello world!'})
[...]

With that change, and reloading your browser, you should have a JSON object response, instead of a string:

{
  "message": "hello world!"
}

We now have a JSON API! It doesn’t do much at the moment, but it’s a start.

Adding mock data

Create a file named mock_data.py and place the contents of this gist into it. Since the end goal of teaching myself Flask is so that I can get Lists of Bests up and running again, we’ll use some fake data for possible book and album lists for the site. We’re including it in a separate file just to keep app.py as simple as possible.

At the top of app.py, we’ll pull the mock data into our app:

from flask import Flask
from mock_data import lists

And now that we have the lists, let’s add a new route to handle displaying the lists. Add the following after our hello() function:

[...]
@app.route("/lists", methods=['GET'])
def get_lists():
    return jsonify({'lists': lists})
[...]

Once the app reloads, if we now try to access http://localhost:5000/lists in the browser, we should get a JSON representation of the lists from our mock data file, that looks like this:

{
  "lists": [
    {
      "description": "This is  the description for the first list",
      "id": 1,
      "name": "This is an example list",
      "type": "albums"
    },
    {
      "description": "A description for this list should be different",
      "id": 2,
      "name": "Another list",
      "type": "books"
    }
  ]
}

Getting a specific list by id

In order to get lists by a passed in id, we will need a new route and handling function. Let’s add this below our last one for all the lists:

[...]
@app.route("/lists/<int:list_id>", methods=['GET'])
def get_list(list_id):
    _list = [_list for _list in lists if _list['id'] == list_id][0]
    return jsonify({'list': _list})
[...]

I’m using _list for the variable name above, since list is a reserved word and can cause problems. And since we’re only getting a single list, I’m grabbing the first of the array with [0]. This is brittle, but will work for our simple case where we know exactly what the data looks like.

Once saved, you can now access this by passing a valid id to the URL as the last part of the URL, like http://localhost:5000/lists/1.

We now have a list, but where are the list items that we added to the mock data? Let’s make sure we’re including the list_items in our import statement, and adjust our function just a bit and add the proper list items into the response:

[...]
from mock_data import lists, list_items
[...]
@app.route("/lists/<int:list_id>", methods=['GET'])
def get_list(list_id):
    _list = [_list for _list in lists if _list['id'] == list_id][0]
    _items = [_item for _item in list_items if _item['list_id'] == list_id]
    if len(_items) > 0:
        _list['list_items'] = _items
    return jsonify({'list': _list})
[...]

After saving and reloading, the output should look like this:

{
  "list": {
    "description": "This is  the description for the first list",
    "id": 1,
    "list_items": [
      {
        "creator": "Radiohead",
        "id": 1,
        "list_id": 1,
        "title": "OK Computer"
      },
      {
        "creator": "Miles Davis",
        "id": 2,
        "list_id": 1,
        "title": "Kind of Blue"
      },
      {
        "creator": "Thelonius Monk",
        "id": 3,
        "list_id": 1,
        "title": "Monk Alone"
      }
    ],
    "name": "This is an example list",
    "type": "albums"
  }
}

Nice! Now we have a list and all its items. We could probably clean things up a bit - like removing the list_id attribute from the items - but this works just fine for our basic example.

Testing

Since we’re good little developers, it would probably be a good idea to write some tests to validate that the API is working as expected. We’ll be using pytest as the library, so let’s install that:

pip install pytest

The Flask documentation has some examples of testing using pytest, so we’ll use those as reference while making our own. To start, create a file named test_app.py in the same directory as the other files. First, we’ll need to import pytest and our app:

import pytest
from app import app

In order to reduce a bit of extra typing, let’s write a test fixture for the app client used for testing:

[...]
@pytest.fixture
def client():
    return app.test_client()

We’re not saving a lot of typing here, but later if you have to initialize a database, or set up anything else, this fixture would be a good place to do that. First, let’s write a super-simple test - that makes use of our client fixture - to make sure we’re getting the right kind of HTTP response code, and MIME type back from our JSON API server:

[...]
def test_json_with_proper_mimetype(client):
    response = client.get('/')
    assert response.status_code == 200
    assert response.content_type == 'application/json'

Now, in order to run this our first test, run the pytest command at the console:

pytest

And you should get some output that looks like this (minus a bunch of long lines like ===========):

platform win32 -- Python 3.7.1, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\...\simple_flask_api
collected 1 item

test_app.py .                                [100%]

1 passed in 0.18 seconds

The little dot next to test_app.py should be green, signifying its passing. You’ll get a green dot for each passing test, and if you get an error, it will give you a wordy explanation telling you what went wrong.

With our first test passing, let’s add one where we test the JSON in the response. Here’s one that checks that the JSON from our root URL is correct - and we’ll use the get_json() helper method to parse the JSON for us:

[...]
def test_hello_world(client):
    response = client.get('/')
    json = response.get_json()
    assert json['message'] == 'Hello world!'

If you run pytest again, you should now have 2 green dots. To finish this part up, let’s test that we’re getting the right number lists from our /lists API endpoint, and that the endpoint for a single list also returns the list items:

[...]
def test_lists(client):
    response = client.get('/lists')
    json = response.get_json()
    assert len(json['lists']) == 2

def test_single_list(client):
    response = client.get('/lists/1')
    json = response.get_json()
    assert json['list']['name'] == 'This is an example list'

def test_single_list_items(client):
    response = client.get('/lists/1')
    json = response.get_json()
    assert len(json['list']['list_items']) == 3

Running pytest now should show that we have 5 successful tests.

Catching errors

Right now, if you try to access any URL outside of the ones we’ve specified, you get Flask’s generic 404 HTML error page. It would be nice to customize this a bit to return a JSON error since this is an API service and not a regular web site.

First, let’s write an errorhandler for 404 errors to display them as JSON. In app.py, you can place this before the if __name__ == '__main__' line:

[...]
@app.errorhandler(404)
def not_found(error):
    return jsonify({'error': '404 Not Found'}), 404
[...]

Now, whenever Flask would normally handle a 404 error, instead of returning its built in generic HTML page, it will now return this JSON response. If you try and load a non-existent URL - like http://localhost:5000/fakeurl in our app, you should now get our JSON error.

{
  "error": "404 Not Found"
}

And now that we have a test file all set up, let’s write one to validate what we’re seeing in the browser. You can add this test below all the others:

def test_not_found(client):
    response = client.get('/not/real/url')
    json = response.get_json()
    assert response.status_code == 404
    assert json['error'] == '404 Not Found'

And that should pass like all the other tests. Finally, there is one more case I’d like to test: what if you try and access a list that doesn’t exist. We know we have lists with id of 1 and 2, but what happens if we try for list 3 with http://localhost:5000/lists/3? If you do try it out, you’ll get a nasty Flask error message that says IndexError: list index out of range at the top. We should account for lists not being available.

We can use the built-in abort() method to trigger a not found error if we try and access a list that doesn’t exist. First, we’ll add abort to our import statement, and then update our get_list() function to trigger this when we can’t find the list:

from flask import Flask, jsonify, abort
[...]
def get_list(list_id):
    _list = {}
    search = [_list for _list in lists if _list['id'] == list_id]
    if len(search) > 0:
        _list = search[0]
    else:
        abort(404)
    _items = [_item for _item in list_items if _item['list_id'] == list_id]
    if len(_items) > 0:
        _list['list_items'] = _items
    return jsonify({'list': _list})
[...]

We had to add a bit more code to capture the case where we don’t find a list based on our data, but not too much. Now, if you try accessing a list that doesn’t exist, then you’ll get that handy 404 error we created above. And like before, let’s add a test to validate the changes. Again, just add to the bottom of the test_app.py file:

def test_list_not_found(client):
    response = client.get('/lists/99')
    json = response.get_json()
    assert response.status_code == 404
    assert json['error'] == '404 Not Found'

When you run pytest now, you’ll see a new green dot for a passing test, but also this helps show that the adjustment of our get_list() function still works as expected.

Wrapping up

Even in building our very basic JSON API, it does take a bit of set up to get things started. And at this point we’re not even accessing a database, or allowing updates to the API with POST and PUT requests. However, something like this tutorial was what I initially looked for; I just wanted the absolute basics. And this basic tutorial gives me a good base to build more complex features upon.

And it goes without saying that some of the code above may not be the best, but I’m sure I’ll learn to write more idiomatic Python as I continue.

I think that’s about it for this part of the tutorial. For the next one, I’ll likely pick one of the API libraries - RESTful or RESTPlus - and see how much of a difference they make.

If you have any feedback or questions about this tutorial, please let me know.

Resources

Some resources that helped out in this first part of the tutorial (in addition to the documentation already linked above):

~ ~ ~

How to get started

A wonderful comic by Alex Norris on learning distractions

With the internet, there are so many resources available to help you learn about something new - a new recipe, a new hotel, a new celebrity couple, and a new-to-you programming language. In fact, there are often too many resources and opinions out there, and finding the right place to start learning can be quite daunting. I’ve found this to be the case lately while trying to learn Python.

In addition to the book and tutorial I mentioned last post, I’ve also been going through another tutorial on how to write a roguelike game using Python. Despite the varied resources, I feel like I’m beginning to get a better grip on the language and its syntax.

But then I wanted to start working on an initial Flask API for Lists of Bests. Sounds somewhat simple enough, but once I started looking at documentation and tutorials, I quickly became overwhelmed. Flask by itself could probably do what I want, but I found references to all kinds of libraries to supposedly make things easier. There’s RESTful, and then RESTPlus, and another with the awesome name Marshmallow. And I’m sure that’s not all of them.

I think I may need to take another approach at this. Since I’m still quite new to the language, I think I’ll start with the most simple example first - without any extra libraries or distractions. I think if I can get my head around the basics, it will be much easier to add in those libraries later. And I may not even need them. I’ll attempt to document my learning into a couple of blog posts in the chance that someone else happens to have fallen into the library well like I did.

The excellent and oh-so-true comic by Alex Norris (@dorrismccomics).