Mahdi Hazaveh

Parsing ndjson stream API’s with PHP

Readable Streams are API responses that are broken into small chunks and sent to the client. As the client should be able to parse the chunks of received data as they arrive the conventional JSON responses are not ideal to be used for streams.

So In most cases, streamable API endpoints should send the data line by line. ndjson is a good option for sending JSON structured data in a streamable way. if you have not heard about ndjson before, it stands for new line delimited json.

First thing first we should call the API via Guzzle, by default guzzle uses PSR-7 stream object to represent the responses.

Here is a sample call to a streaming API that returns a stream response object:

<?php
$stream = $guzzleClient->get("http://stream-api.com/api",
    [
        'stream' => true,
    ]
)->getBody();

Now that we have stored the stream object in $stream variable we can use the “eof” method to verify we have not reached the end of the streamed data. by utilizing the helper readline we will read the output line by line. which should give us a single JSON object.

while (! $stream->eof()) {
  $line = Utils::readLine($stream);    
}

as simple as that we can read chunked received data from a streaming API with php.

Ok Google, Please do not play “Someone You loved by Lewis Capaldi”

I don’t know who Lewis Capaldi is, I don’t really wanna know either. but recently he has turned into a problem in my life, and I certainly want him off my Speaker.

I use Google Home speakers, both in my bedroom and living room. I’ve been using Google Speakers for a couple of years now. recently, I feel that their AI experience has degraded, a lot. I don’t know if it is because of me forgetting how to speak English (blame it on being stuck at home for over a year), or the more people are using it is blowing its training models up. whatever is going on, the fact that it almost plays “Someone You Loved by Lewis Capaldi” is becoming intolerable for me.

I’m a pretty extremist when it comes to music, I listen to it every day, everywhere but not everything. I only listen to Post Rock, and sometimes when it gets too dark in my head, I play some random Jazz songs to change the mood.

So dear Google, Spotify, or whoever is to blame, please do not play “Someone You loved by Lewis Capaldi” the next time I ask for: some music or something to play.

I even started to look up ways to block Google from playing a specific song and found this guy claiming on Reddit that saying “hey Google, never play that song again” seemed to help with his wife’s taste of music.

I don’t know if this works yet or not, as a matter of fact right before writing this blogpost I asked it from my Speaker, so I guess fingers crossed to me 🤞🏻

Update 15 May 2021: it’s now 2 days passed since I’ve tried asking my speaker to “never play that song again”, and it seems the solution was working! I have not heard Lweis since then.

pwdreset.com a Password Reset Service

pwdreset.com – Password Reset as Service

Over the past week, I spent my time after work on building a password reset service that allows website owners to reset their WordPress passwords quickly without technical knowledge, and that is how pwdreset was created.

WordPress comes with its own password reset functionality, but there are scenarios where the “Forget Password” does not work.

How to Reset WordPress Password using FTP

Reset WordPress Password with FTP

in this post, I am going to show you the way that you can reset your WordPress admin Password via FTP.

WordPress stores user credentials along with other contents in the database, to reset WordPress password using FTP we need to make be able to read and write in the database.

Continue reading “How to Reset WordPress Password using FTP”

I’ve created Bot that sends notifications when PS5/GPU’s are in stock

when I have 2 weeks of holidays with no place to go and nothing to do in a lockdown situation, there is a high chance I will be working on a random project. This time it is a bot to check when GPU’s and PS5 are back in stock in online stores.

a couple of weeks ago I met my friend (completely legal! only 1 household) and we spend most of the time having no topics to talk about (as expected thanks to covid19), one of his biggest topics was: how bad is the availability situation of certain devices and Graphics Cards due to scalping problem.

So basically some bad guys are buying all the available stocks for these items and then put them for resale for a price that is many times more than the actual retail price. To them, this is reselling and they see no moral problem with it, the actual problem lays with how they acquire this stock.

seemingly, the way they get hold of these items faster than anyone else is by using bots and automated tools! so basically they check the stores for availability of these items non-stop and then buy whatever is available in the blink of an eye, thus the only ways for consumers to be able to shop these items are:

Be super lucky! Open the product page at a given time where nobody else is on it!
Subscribe to bots (Like the one I built) to know when an item is in stock.
Pay a lot more than the retail price and buy from scalpers who call themselves resellers.

The idea of a bot that checks these products and lets people know when they are available seemed pretty fun to me! I’ve previously done a lot of work-related to scraping websites, doing web automation, and similar things!

So I decided to get into the business and make my own! although I could make a small amount of money for the affiliate sales I make with my bot, I was more interested in learning how to do integrations with Telegram and Discord.

That is how InStock was born. In order to use this simply open https://instock.ml and join the respective notification channel for the product, you’re eyeing to buy! For example, you can subscribe to the Telegram or Discord channel.

How did I do it?

Basically, I have a list of products that have a relationship to a set of product links, these links can be from Amazon or other providers.

I have an instance of a browser that is controlled remotely via my code that opens these links in a small interval and checks if the item is available to be purchased. To do this I needed to write a small parser for each website that contains the logic that is needed to determine whether the product is in stock on that website or not.

Here are some of the problems I faced:

My IP is keeps getting blocked 🙁

Well, I get blocked by Amazon and some other sites from time to time! I literary below up my server IP in the first 1 week of working on the project! So I needed a cheap way of rotating my IPs. to do so I needed to create my own Proxy Provider, a simple proxy scrapper. I then try to use all those proxies in multiple async requests and then cherry-pick the ones which are functional, cache them for 30 minutes and randomly pick one for making the request.

Although this has been pretty efficient, I still sometimes have problems with Amazon showing me dog photos instead of the product page, well at least I got to know the Dogs of Amazon.

Twitter Blocking my Account!

One of the channels I use to update my visitors about stock availability is Twitter, I got blocked on Twitter by sending my first 3 automated tweets! They basically revoke the write permission from my application, which is crap! I’m now trying to contact them and argue the problem, hopefully, I can be up and running on Twitter again.

Future of instock.ml

So far I’ve been enjoying dealing with the challenges I have while scraping. I will definitely be working on making this better and add more sources to check for stock availability. I will be keeping this project online as long as the scalping problem exists and I do not run out of money for my servers.

Well, that was pretty much it, an official blog post to call it a day. if you have checked my bot and find it interesting let me know what you think and drop me a message if you have any suggestions on what I can implement next.