Alex Meub

Craigslist Alerts with Python

I love Craigslist, but trying to buy certain things (like Subarus in Portland) can be ridiculously competitive. I wanted a simple way to be alerted immediately when an item I was looking for was posted.

Craigslist Search Alerts

I tried using Craigslist’s built in Search Alert feature but I found some major limitations with it. For instance, it’s not clear how often the alerts run. There is no official answer for how often postings are checked, but from my experience, it takes at least 5-10 minutes after something is posted. Also, alerts are sent when postings are edited or reposted which creates a lot of false positives.

IFTTT Alerts

IFTTT is a really cool service but currently the only way for it to get up-to-date Craigslist data is to check manually. I ran some tests and it seems that IFTTT sends alerts out about 1 hour after items are posted on Craigslist.

Adapting an Existing Project

After being disappointed with Saved Search Alerts and IFTTT, I found an existing Python project with a similar goal called apartment finder (created by Vik Paruchuri). It’s a clean and readable codebase that looked to be the perfect place to start, I just needed to make a few changes for it to suit my needs.

First, I needed to adapt apartment finder to be used in Craigslist’s “For Sale” category rather than “Housing”. That was relatively easy but I didn’t like the use of the python-craigslist module because it’s super limited in terms of what filters can be used. I ended up swapping that out in favor of just using BeautifulSoup to do the Craigslist scraping myself.

Second, I wanted to be able to easily change what the bot was searching for. Instead of hardcoding the criteria, I put the searches I wanted on S3 so I can change the items I’m looking for on the fly without having to re-deploy anything.

Third, I wanted to have the option to be alerted by email in addition to Slack.

How it works

After adding some error handling and running it through its paces on my machine, I had a working Craigslist alerting system. The bot does the following every 3 minutes on a continuous loop:

  1. Pulls the latest list of desired searches (including all filters) from a JSON file hosted on S3

  2. Scrapes Craigslist for all items that match any of the desired searches

  3. Dedupes those items against a local database to prevent duplicate notifications

  4. For all new items, sends an email and slack message with the posting details and link

For more details, check out the code on github.

Deployment

To make sure the script was running 24/7 I copied the files up to my $5/month Digital Ocean instance. Then I simply ran it in the background as a service (as suggested here) and I was good to go.

I also added a quick line to make the bot periodically check in with Dead Man’s Snitch. This way I can be alerted if it stops running for some reason.

Up and Running

I’ve had the alerting bot running for a few weeks now and it’s working really well. On average I get alerted just a few minutes after things are posted which is great for competitive items. Also, I love being able to easily update the file on S3 to manage alerts on the fly.