How we built an AI recycling app in a week for a hackathon and won $11k
This was one of my most intense projects to date, but also one of the most rewarding
Issue #31
Hey! I’m Alyssa X, a serial entrepreneur building 12+ products by myself. This email is part of my maker journey, which you’re subscribed to. Feel free to forward it to others if you find it interesting! You can support me through GitHub Sponsors, I’d really appreciate it! ❤️
Hey all,
Happy holidays, hope you’re all doing well this time of the year. I’ve been very quiet for the past months, but I can assure you I’ve been keeping busy.
I recently participated in a really fun (and intense) hackathon, and I wanted to share my experience with you. Here goes!
The hackathon
A couple weeks ago a friend of mine told me about Ben’s Bites AI Hackathon, asking if I wanted to participate. I was skeptical at first, as I hadn’t built anything with Artificial Intelligence before, but thankfully Leo, an excellent data scientist (and even better person 😘), decided to join me for the challenge.
The rules were simple: build any sort of project using AI within a week. I’ve built projects with tighter time constraints before, so that was not a problem. We just had to come up with an idea that involved AI somehow.
We quickly realized that most of the other hackathon participants were going to build projects using tools like ChatGPT3, Stable Diffusion, DALL-E 2, which have recently gained a lot of popularity. Willing to distinguish ourselves, and with the advantage of having an actual data scientist in the team, we decided we would do something completely different, and build our own AI model from scratch.
In order to build an AI model, we needed a training data set. We felt it would take too long if we gathered the data ourselves, so instead we searched on Kaggle for existing datasets that we could use in our project. We managed to stumble upon a dataset with the 7 plastic resin codes, which could be used to identify the different kinds of plastic.
From there we came up with a project idea: we would create an app that let the user take a picture of a plastic code within an item they wanted to dispose of, and it would tell them how to recycle it. In order to provide the user with the information, we used guidance from londonrecycles.co.uk (and later on from some other websites as well), since we’re based in London. We called the app EcoSnap.
Developing the app
While Leo started working on building the model in Python (she explained a bit more on the GitHub repo README, if curious), I decided to create a prototype to see if I could get a barebones version of our project working, using Next.js with Vercel.
I found a library I could use to display the user’s camera (both on a laptop and on a phone, including a way to switch between the back and front camera), and I was able to create a simple UI to take pictures from it. I used HTML5 Canvas to resize and crop the pictures to 240x240 in order to work for the AI model, and I quickly put together a set of possible responses.
With the basic prototype working, I jumped into Figma to design some screens. I wanted it to be very intuitive and user-friendly, as I think this is one of the major pitfalls of AI-related products, using buzzwords and feeling too technical and impersonal.
I designed a simple onboarding flow to set the region (to show different recycling advice), plus show to the user how they’re supposed to use the app, with 3 steps paired with animated illustrations (some of which I made from zero, others using the Stubborn Generator).
I worked on the camera UI next, displaying the scan region and just enough buttons for the user not to get confused (including the option to upload an image). Then we decided to add a feedback loop, where the app would display to the user what symbol it thinks it recognized, and then lets the user accept or rectify that value. The idea was to then use that data with the image for re-training the model, but we ran out of time (we still kept the system in place, it just doesn’t save the images).
After the user has confirmed the symbol, a small modal appears saying whether the item is recyclable or not, and after a few seconds I made it so it animated to full size to reveal the rest of the information (to avoid information overload), including steps to recycle or anything else the user should note.
The dashboard design was actually last, as we were mainly focused on shipping the MVP with just the camera. When we realized we had more time to polish, we ended up adding some more things, including a box with the number of recyclable plastics the user has scanned (tracked using local storage), and a search area to find recycling information about lots of other items based on the user’s post code (UK only). I also made a simple design for a settings page to review the onboarding, and change the region.
The challenges
While it seems like everything was going very smoothly, the reality couldn’t be further from that. We actually had a lot of obstacles in the way, impeding progress.
Once Leo had trained the model in Python, we decided it was time to connect it to the app. We thought of a couple of approaches, we could either keep using Python to do the image classification bit and find somewhere to host it, or we could use Node.js and use serverless functions in Vercel to perform the task, keeping everything in one ecosystem. We decided on the latter, which in retrospect might not have been the best idea.
We first had to use for a serverless version of Tensorflow.js that could run in Vercel serverless functions (otherwise it would be too big). Once it was all set up, we had to figure out how to get the AI model in node, as it had been built and developed for Python. This was probably the worst part of the whole hackathon, and we spent several days stuck on this problem.
First we found an article that said we could use the SavedModel Python format on Tensorflow.js directly through a simple function. This worked locally, but didn’t when deployed. After a lot of investigation, we realized the serverless function was running the entire library in development, and only using the “compressed” version in production. As luck would have it, the library was out of date: the latest update was on 2019, and the native SavedModel execution was added on 2020. So we had to find another solution.
Looking around, we found a very concise (and deceptively simple) article by Google CodeLabs about converting a SavedModel into a tfjs format, which we again followed with no success. Every time we attempted to run the command, Python would crash. This was basically a dead end, we couldn’t find any alternative ways to convert it, not even a way to export the tfjs format from Python directly (it also wouldn’t work). It wasn’t after spending hours and hours of doing research and trial and error, that I somehow managed to get it work. Here’s how:
The first problem was that we were developing with M1 MacBooks. For it to work, it needed to be in x86_64 architecture (Intel), using a Rosetta terminal. But even then, the rest was pretty unintuitive. I had to install MiniConda, set up an environment using Python 3.8 (newer versions of Python don’t work, I tried), activate it, install Tensorflow using Conda’s package manager as opposed to Python’s pip/pip3 (which is extremely confusing), then install macOS and metal Tensorflow plugins, install TensorflowJs itself, and finally run the command
We also ended up having to switch the serverless library to this other one, because the one we had wasn’t working properly. Just great 🙃
Final bits
After a lot of headaches, we finally managed to get it all working.
It was time for the last few tasks: I created some meta images, Leo worked on the copy, then I decided to make a quick video showcasing the app.
This was annoying as we kept updating the app, so we had to wait until the last minute to record. I ended up deciding to do some After Effects magic to get it looking just right, animating things a bit around. Here’s the video:
We also spent ages fixing all sorts of bugs that friends were telling us about, and Leo made a bunch of different improved models until it got incredibly accurate at identifying the different symbols.
We stayed up after 2AM just to have everything prepared and ensure to meet the deadline (which was at ~4AM), all while fighting to stay awake. Once we finished up our submission, we practically just collapsed out of exhaustion and fell asleep 🫠
The next few days were filled with uncertainty, we got a lot of encouragement and the app got a positive reception from our friends, but we didn’t know how it would be received by the judges.
It wasn’t until Friday that we learnt of the results:
Not only did we get 3rd place, but we also won in the category Best in Open-Source (so glad we put it all on GitHub!). This meant that we won a total prize of $11k, which is just incredible!
Anyway, there’s so much more I’d love to talk about, but this is getting too long. We plan on possibly polishing this up and launching it on January. I might make a more in-depth article at some point, maybe more on the AI side of things. I’ve also got some more things to share about a new project, but I will keep it for the next issue. You can try out EcoSnap here.
See you,
Alyssa X