when I have 2 weeks of holidays with no place to go and nothing to do in a lockdown situation, there is a high chance I will be working on a random project. This time it is a bot to check when GPU’s and PS5 are back in stock in online stores.
a couple of weeks ago I met my friend (completely legal! only 1 household) and we spend most of the time having no topics to talk about (as expected thanks to covid19), one of his biggest topics was: how bad is the availability situation of certain devices and Graphics Cards due to scalping problem.
So basically some bad guys are buying all the available stocks for these items and then put them for resale for a price that is many times more than the actual retail price. To them, this is reselling and they see no moral problem with it, the actual problem lays with how they acquire this stock.
seemingly, the way they get hold of these items faster than anyone else is by using bots and automated tools! so basically they check the stores for availability of these items non-stop and then buy whatever is available in the blink of an eye, thus the only ways for consumers to be able to shop these items are:
- Be super lucky! Open the product page at a given time where nobody else is on it!
- Subscribe to bots (Like the one I built) to know when an item is in stock.
- Pay a lot more than the retail price and buy from scalpers who call themselves resellers.
The idea of a bot that checks these products and lets people know when they are available seemed pretty fun to me! I’ve previously done a lot of work-related to scraping websites, doing web automation, and similar things!
So I decided to get into the business and make my own! although I could make a small amount of money for the affiliate sales I make with my bot, I was more interested in learning how to do integrations with Telegram and Discord.
That is how InStock was born. In order to use this simply open https://instock.ml and join the respective notification channel for the product, you’re eyeing to buy! For example, you can subscribe to the Telegram or Discord channel.
How did I do it?
Basically, I have a list of products that have a relationship to a set of product links, these links can be from Amazon or other providers.
I have an instance of a browser that is controlled remotely via my code that opens these links in a small interval and checks if the item is available to be purchased. To do this I needed to write a small parser for each website that contains the logic that is needed to determine whether the product is in stock on that website or not.
Here are some of the problems I faced:
My IP is keeps getting blocked 🙁
Well, I get blocked by Amazon and some other sites from time to time! I literary below up my server IP in the first 1 week of working on the project! So I needed a cheap way of rotating my IPs. to do so I needed to create my own Proxy Provider, a simple proxy scrapper. I then try to use all those proxies in multiple async requests and then cherry-pick the ones which are functional, cache them for 30 minutes and randomly pick one for making the request.
Although this has been pretty efficient, I still sometimes have problems with Amazon showing me dog photos instead of the product page, well at least I got to know the Dogs of Amazon.
Twitter Blocking my Account!
One of the channels I use to update my visitors about stock availability is Twitter, I got blocked on Twitter by sending my first 3 automated tweets! They basically revoke the write permission from my application, which is crap! I’m now trying to contact them and argue the problem, hopefully, I can be up and running on Twitter again.
Future of instock.ml
So far I’ve been enjoying dealing with the challenges I have while scraping. I will definitely be working on making this better and add more sources to check for stock availability. I will be keeping this project online as long as the scalping problem exists and I do not run out of money for my servers.
Well, that was pretty much it, an official blog post to call it a day. if you have checked my bot and find it interesting let me know what you think and drop me a message if you have any suggestions on what I can implement next.