Talking Go at SXSW Interactive 201520 March 2015
On March 17, 2015 at 9:30am in Salon B of the JW Marriott in Austin, Texas, I delivered a talk titled "Lessons Learned Building Web Apps with Go" to a crowd. The talk covered much of what the Public Good team has learned over the past two years building publicgood.com.
The slides are online at http://cgansen.github.io/sxsw-go-talk/#/. I look forward to refining the content of the presentation and sharing it again in near future.
2014 Music Top 1014 December 2014
Some friends compile an annual review of their favorite music and asked me to contribute. Here's what I had to say about 2014.
Top 10 albums of the year ranked in order.
- Aphex Twin - Syro: This the is Aphex Twin album everyone wanted. Certainly worth the wait.
- War on Drugs - Lost in the Dream: I don’t think I listened to any album while driving around the Midwest as much as I did this one.
- Tycho - Awake: At first Tycho was a Boards of Canada clone, but with this album they finally shake that off and assert themselves as the real band that they are.
- The Budos Band - Burnt Offering: I love listening to music that makes me feel like I’m in a late-70s police chase scene.
- Jungle - Jungle: These dudes came out of nowhere to remind us all that disco wasn’t terrible.
- The Antlers - Familiars: I listened to this a ton then deleted the Spotify playlist and forgot about it. I’m adding it mostly based on the strength of the amazing show they put on at Lincoln Hall.
- Clark - Clark: Pretty heavy and intimidating end-of-the-world music. I dig it.
- Spoon - They Want My Soul: These guys are flawless.
- Caribou - Our Love: A late addition to the list but incredibly catchy.
- Various - Hyperdub 10.4: Another late entry, but a great cross section of music from the legendary London label.
Top 10 songs of the year ranked in order.
The War on Drugs - An Ocean In Between The Waves
A Sunny Day in Glasgow - MTLOV (Minor Keys)
Jungle - Busy Earning
Aphex Twin - minipops 67 [120.2][source field mix]
The Budos Band - The Sticks
Caribou - Our Love
Tycho - See
(this video is bonkers, you should watch it)
The Antlers - Hotel
The War on Drugs - Red Eyes
Spoon - Do You
The Beyoncé Award for the 2013 album you wish you would have included on last year's list.
nothing comes to mind
Your favorite album you discovered in 2014 that did not come out in 2014.
Disc Two of Derek and the Dominos “The Layla Sessions: 20th Anniversary Edition”. It’s all the session players (including Duane and Gregg Allman, Dickey Betts, and Eric Clapton) jamming for an hour straight. I love it.
Best concert you attended this year.
Slowdive at Pitchfork this past July. Specifically, this:
Honorable mentions: TWEEDY (thanks for taking me on a date, Brian!), and Tycho at Concord (another date night with BB!).
No honorable mentions.
Honorable mentions: Run The Jewels 2, Yo La Tengo - Extra Painful, She & Him - Classics, Arca - Xen, A Sunny Day in Glasgow - Sea When Absent, Sun Kil Moon - Benji, Future Islands.
HealthNear.Me23 March 2014
In November 2013 Melissa and I built HealthNear.Me, a web and phone-based application to help people find public health resources near them in Chicago.
After we surveyed the current tools people have to use to find a free clinic or a place to find free condoms near them, we knew we could build something better. We also built it because we wanted to enter the Making Public Health Data Work Challenge. We won the first place prize in the competition and we're excited to share a bit more about what HealthNear.Me is, how it works, and how we can improve it.
Melissa and I attended the Illinois Public Health Datapalooza to learn more about what was happening at the intersection of the public health and technology fields. Melissa is finishing her Masters degree in Public Health, and I am a software developer, so this was a rare intersection of our professional interests. We sat in on a few interesting talks at the event and decided to build something for the app contest. Our first idea involved an app for folks with asthma to record air quality and get alerts for when they might have bad days. A few minutes of Googling showed us that the market was already pretty saturated with apps that did just that.
Looking for a new angle, Melissa started to survey the available health-related datasets on the City of Chicago and State of Illinois data portals and found many lists of service providers; at the same time I reviewed the City of Chicago and State of Illinois web sites to see how the general public could locate service providers near them. What I found was a fragmented, hard to parse set of pages. Results were broken up by type, search by address wasn't available, and the sites didn't work very well on smart phones.
It was obvious that the experience for finding public health providers was suboptimal, and we could use the raw data published by Chicago Department of Public Health (CDPH) and Illinois Department of Public Health (IDPH) to build our own search tool. I am a big proponent of Tim O'Reilly's argument for government as a platform. Paraphrasing quite a bit, he argues that governments are very good at providing raw data, but not very good at building consumer applications. By focusing on their strengths – collecting, managing, and publishing data about their services – governments can leverage developer communities or hire professional developers to build consumer-facing applications. In this case, the CDPH and IDPH provided immense value by providing relevant, consistent, machine-readable lists of service providers. If they did not publish this raw data in a sane format, we would have never been able to build HealthNear.Me. In the course of a few evenings of coding, I had all of their provider data aggregated, indexed, and available for public searching.
Melissa and I talked about how we thought people might want to look for health resources near them, and what features would be the most useful for them. We didn't spend too much time on design research doing things like interviews or developing personas. Instead, we tried to guess a few common uses cases and build features to support them. For example: "I am a social worker and want to find clinics near where one of my clients goes to school" or "I need to find a place to go warm up" or even "What is the nearest hospital?". One feature that we prioritized was making the application work via SMS. I was inspired by the rich SMS search features available in the Chicago Early Learning site. The City of Chicago has a basic SMS service – CHITEXT – but it doesn't support searching for health providers.
I wrote the backend search index and API in Go. The health provider data lives in an ElasticSearch index. The public-facing application is HTML and AngularJS. Go makes writing APIs very simple. It's fast, handles JSON well, and integrates well with the ElasticSearch. ElasticSearch is fast and does geospatial searches with ease. AngularJS is great at gluing together front end applications and JSON APIs. This technology stack is also what I use for my day job, so I was already very comfortable working with it. The application is running on an EC2 instance provided through Smart Chicago's hosted web space. I signed up for a cheap Twilio plan to handle the SMS integration. Twilio makes it super simple to read and write SMS messages from inside computer programs.
The application is broken down into two main components: a bulk data loader and a HTTP API. The loader app reads the JSON exports of the CDPH and IDPH data and inserts it into the ElasticSearch index. The API app handles requests from Twilio and the AngularJS application, searches the index, and returns data as a blob of JSON. I'm grateful to a a number of open source libraries that made development much faster, especially the Elastigo and go-geo projects. The entire application source code is available at the health-near-me Github project page. It's licensed under the MIT license, meaning anyone can take the code, make improvements, and run their own local instances of the application.
Melissa and I are excited to continue to develop and refine HealthNear.Me. We've had productive conversations with CDPH about how we can work together to make the application available to the community and how we can increase the quantity and quality of the data in our search index. We would like to explore translation and localization to make it easier for users who do not speak or read English to search for providers near them. The user interface is rough around the edges, owing to a short development cycle and a lack of graphic design skills. We would like to improve the look and feel, and make the search results on the web application more actionable, including showing results on a map and seeing what providers in an area provide multiple services.
We're exceptionally grateful and honored to win the first place prize. We hope that HealthNear.Me continues to grow and becomes a useful tool for people in Chicago and beyond to find public health providers in their communities.
Building Chicago Works For You29 September 2013
Note: this essay was originally posted at the Smart Chicago Collaborative blog as part of my work as Civic Innovation Program Manager.
Chicago Works For You (CWFY) is comprised of three components: a AngularJS/Jekyll frontend, a custom Go API and worker layer, and a PostgreSQL/PostGIS database. This post will dive into the mechanics of the system and explain why it works the way it does.
CWFY started with only a handful of requirements. It had to pull service requests from Chicago's Open311 API, save them to a database, do some math, and expose the results to a pretty user interface.
The first sketch of the system was pretty rudimentary, but it matches very closely the end result. We implemented the worker and API in Go, a powerful and fun language that is well-suited for this type of work. We are hosting the Go applications and database on a m1.medium EC2 instance on the Smart Chicago Apps infrastructure.
The worker application finds the timestamp of the most recently updated service request in the CWFY database and asks the Open311 API endpoint for all service requests created or updated since that point. This repeats every thirty seconds. That interval is short enough that we're not clobbering the Open311 API, the number of requests in the response is small (usually less than a dozen), and we still have near-real-time data on our site. We've found that the Open311 API is very reliable — and fast — and can stand sustained load, as it did when we backfilled our database.
The database schema is very simple. There is a single service_requests table with more than three million rows. That's all service data stretching back to January 2008. This table is very similar to the service_requests schema used by Code for America's 311.fm application. In fact, we used that schema to bootstrap this application.
There are two tables of ward boundary data, one for the current ward boundaries, and one for the new ward boundaries, to be instated in 2015. But the city's service request records only include the ward the request is currently in. Using the popular PostGIS PostgreSQL extension, we're able to use the latitude and longitude associated with each service request to geo-locate it within its 2015 ward-to-be and determine if it's in one the 233 areas of the city that are changing from one ward to another. These calculations allow CWFY to compute service delivery information for current and future wards, as well as individual transition areas.
Finally, there's a daily_counts table, populated with counts of service requests opened for each day, ward, and service type combination. This table is populated automatically by a Postgres trigger, and allows CWFY to perform quick aggregate calculations.
The site is hosted in a Amazon Web Service S3 bucket, making it very inexpensive to host, highly available, and simple to update.
We relied heavily on third-party libraries to speed our development. We acknowledge the following and offer our thanks:
- Leaflet, Cloudmade, and OpenStreetMap: powering our beautiful maps
- Gorilla: Golang tools
- SCSS: the only way to write CSS
- PostgreSQL and PostGIS: simple data munging
- lib/pq: Golang + PostgreSQL
- CapistranoRb: simple deployments
- Jekyll: smart static-site creation
- Jim Pravetz: Jekyll page generator
- AngularJS: easy dynamic webapps
- Bootstrap: simple grid framework
- Highcharts: beautiful charts and graphs
Improving Adopt-a-sidewalk08 September 2013
Note: this essay was originally posted at the Smart Chicago Collaborative blog as part of my work as Civic Innovation Program Manager.
TL;DR: Adopt-a-sidewalk is a flawed, under-utilized application with enormous potential. By refocusing the user experience on addressing actual needs of people in Chicago and showing meaningful activity, it could be a powerful tool for engaging citizens in supporting and improving the civic infrastructure in their community.
Winter is officially in Chicago’s rearview mirror, although you would not notice from the chilly temperatures outside. This post is a reflection on one of Chicago’s winter-weather civic applications, Adopt-a-sidewalk, an application I helped bring online over a year ago, and how it can evolve to improve the lives of Chicago residents year-round.
Adopt-a-sidewalk is a Chicago-based version of the Adopt-a-hydrant web application built by Code for America in Boston back in 2011. Developed by Code for America fellow Erik Michaels-Ober, Adopt-a-hydrant lets residents of Boston volunteer to clear fire hydrants when there is a snow storm.
In the fall of 2011, City of Chicago officials, acutely aware of the severity and importance of swift snow removal, saw an opportunity to repurpose the code, and invited a group of civic developers to customize the application for use in Chicago. The key functional difference between the applications is that in Chicago, residents can request help clearing their sidewalk. Adopt-a-sidewalk first went live as part of ChicagoShovels.org in January 2012, and generated a bit of fanfare in local and national media:
- New York Times: Snow Site Lets Chicago See if Plows Are Really in a Rut
- ABC7 News: Mayor’s office launches ChicagoShovels.org
- Chicagoist: City’s Adopt-A-Sidewalk Website Launches
Adopt-a-sidewalk saw moderate adoption, but quickly fell out of use due to a very mild winter, and the fast arrival of spring a few months later. In the fall of 2012, the City of Chicago asked the Smart Chicago Collaborative to assume the responsibility of hosting the application, and development responsibilities were handed over to the Code for America Chicago brigade.
To date, Adopt-a-sidewalk has seen very little adoption in Chicago. There are 557,793 individual sidewalk segments available for adoption, but only 75 registered users. 153 sidewalks have been claimed, either by volunteer shovelers, or people asking for help. That means that only 0.027% of all sidewalk segments in Chicago have been adopted. At its busiest, only 200 people visited the site in a given day.
There are three major issues that impact the usability and adoption of Adopt-a-sidewalk.
First, plainly speaking, the application is boring. In the case of a snow storm, there is a sense of urgency to responding and cleaning up the mess. The City deploys a fleet of snowplows to clear the streets, and neighborhoods are abuzz with residents scraping cars, shoveling steps, and snow-blowing their sidewalks and alleys. On Adopt-a-sidewalk, there is absolutely no perception of activity, urgency, or community. There is no mechanism to show users where activity is happening, or if there is a need for activity. On their first visit to the site, users are presented with a featureless, generic Google map of the city of Chicago, and no clear call to action. If the user does decide to register and adopt a sidewalk, there is little incentive to return or to refer friends to the site.
Second, the path to participating is laden with friction. Users must search using a real Chicago street address and register for an account before they may participate. Registering an account involves giving a name, email address, a password, and completing a captcha. There’s no mechanism to invite your neighbors to join you in shoveling, nor is there a mechanism to share your activity with your social network.
Third, the application is useless when there is no snow on the ground. Adopt-a-sidewalk is irrelevant in the summertime, and, for most of the winter spent between snow storms. There is no incentive to return to the site, and there is no meaningful action to take in between snow storms.
On a conceptual level, the premise of Adopt-a-sidewalk is flawed. Chicago residents are already expected to and, by ordinance, required to, shovel their sidewalks. Adopt-a-sidewalk provides no benefit to users who adopt the sidewalk in front of their house and dutifully shovel it each time snow falls. The steps to register and adopt their sidewalk is busy work.
The real work
Instead of asking users to do monotonous work, Adopt-a-sidewalk should focus on providing a real service: matching people in need of help with people willing to help. In that scenario, there are two key classes of users: people who cannot clear their sidewalks and people who are willing to help shovel sidewalks near them.
By shifting the interaction model from navigating a half million rectangles on a map to a focused, needs-based one, many of the core usability issues can be alleviated. It’s far easier to show activity, in the form of the most recent or most urgent requests for help, and the reward for participating is much more immediate and meaningful. Instead of highlighting what’s expected of people, the focus can be on enabling and rewarding people who want to help their neighbors.
The natural extension of this concept is to move beyond simple sidewalks and instead enable neighborhood adoption of any civic infrastructure. Adopting sidewalks could easily gave way in the spring and summer time to adopting parks and community gardens. In the fall, communities could band together to adopt a local school and fix it up before students return. A baseball team can adopt its ball field and organize events to maintain and improve it.
Fostering community around shared civic infrastructure is not a new concept. However, using technology, it is possible to integrate the real world thing with an online community, and the vast network of people and data that exists there. With the rise of open government data, not only is the civic infrastructure as physical object or place, it’s a continuous stream of data and interactions. The baseball diamond around the corner is not just a sandlot for shagging fly balls, it is a collection of data points: tweets, photos, and events created by community members, and crime reports, 311 requests, park facilities data from the local government.
I look forward to seeing where Adopt-a-sidewalk goes from here, especially if Code for America or one of the brigades takes some of the concepts from Adopt-a-sidewalk and pulls them back into the mainline repository. Adopt-a-sidewalk is, despite its flaws and low adoption, one very small step on a long path to building, enabling, and merging real life and online communities.