puppeteer Taming real browser for web scraping: Chrome (Puppeteer) as a cloud API service Okay, let's admit it - web scraping via Puppeteer and Playwright is the most versatile and flexible way of web scraping nowadays. Unfortunately it's also the most cumbersome, time consuming way of scraping,
sports Morning sports is my happiness magic pill I am an indie hacker and CTO in my mid-thirties, and my life improved so much a few years ago when morning activity magic clicked for me. After a period of burnout, reduced
How to do web scraping in PHP Web scraping is a big and hot topic now, and PHP is a pretty fast language which is convenient for rapid prototyping and is wildly popular across web developers. I have pretty extensive
ScrapeNinja: never handle retries and proxies in your code again I am glad to announce that ScrapeNinja scraping solution just received major update and got new features: RetriesRetries are must have for every scraping project. Proxies fail to process your request, the target
Simple proxy checker script via CURL While working on the ScrapeNinja scraping solution, I often need to verify if particular proxy is alive and if it is performing well. Since I don't want to use various online services, especially
Sending Requests in Web Scraping: cURL, Chrome, Firefox, REST.client, netcat Contents:Chrome Dev ToolsCopy as cURLcURL options: proxy, show only headersFirefox: edit&resend; multi-account containerscURL to Python scraper converterVS Code REST.client extensionHTTP server one-liner for debuggingWhile working with scraping, I have
Making PDF look like scanned. Top 4 tools to apply scanner effect, reviewed. Some bigger companies still require wet signatures on documents, which was a source of constant hassle for me during recent years. My workflow was:Receive email with the PDF documentDownload the documentPrint the
scraping ScrapeNinja: bypassing CloudFlare 403 (code:1020) errors I've recently started getting Cloudflare 1020 (403) errors when scraping some random e-commerce website. At first, I thought that the website didn't like my scraper IP address, but changing IP addresses to clean
VS Code Remote for Node.js caveat: dealing with detached nodemon process I develop all my new projects on a remote Hetzner Cloud machine, using wonderful and almost too-good-to-be-true VS Code Remote. I recommend this setup for everyone who does not like spinning fans of
Clickhouse as an alternative to ElasticSearch and MySQL, for log storage and analysis, in 2021 In 2018, I've written an article about Clickhouse, this piece of content is still pretty popular across the internet, and even was translated a few times. More than two years have passed since,
inWidget proxified: Free Instagram widget for your website, in 2021 For one of my projects, I needed a widget which would render posts from an Instagram hashtag. It turned out to be very cumbersome to implement nowadays, because Instagram shut down its legacy
Best way to daemonize node.js process in 2021: forever, pm2, nodemon, docker, supervisor, systemd and what to choose During my development career I used a lot of different solutions to daemonize processes (mostly, node.js scripts), and I decided to do a quick writeup with very short description of each approach
[UPDATED] Scraping Instagram in 2021: avoiding 302 and 429 errors Instagram is a tough target for scraping. For one of my side projects, I needed to get information from several public accounts, on a daily basis – for example, their followers counts, and their
esbuild Frontend development in Docker is pain in 2020. But it gets better I've just started building a dashboard for my new project, which is an opionated Node.js API gateway (still in its infancy), with Clickhouse for logging: https://github.com/restyler/api-gateway. Here is
Clickhouse as a replacement for ELK, Big Query and TimescaleDB UPD 2020: Clickhouse is getting stronger with each release. We are using Clickhouse as an ELK replacement in our ApiRoad.net project - API marketplace with ultimate observability and analytics of HTTP requests.
Good thing in PHP nobody talks about Do you know what I like about PHP?It is designed to die after execution.You should not fear code duplication, or bad formatting of code, of bad variables naming - mentioned artefacts
vuejs Vue.js vs React in 2017: state of art One year ago I've published a post about reasons why our team chose Vue.js over React for our qwintry.com project rewrite. I've made some predictions back then:I expect Vue to
vuejs Why we chose Vue.js over React Qwintry team recently started active migration to Vue.js as a frontend framework in all our legacy and new projects:in legacy Drupal system (qwintry.com)in our new, completely rewritten qwintry.com
Rotating original file for image field in Drupal 7 and dealing with browser cache While working on new Qwintry.com tasks we needed to provide our operators the interface to rotate uploaded images (and I wanted to rotate the original image file). Surprisingly, I could not find
Building scalable IT system for delivery from US to Russia: Drupal, Symfony2 and Yii2 compared I was not posting to the blog for a long time, and finally it’s time to share my experience with new project. This post will also cover some badly structured thoughts about
Packt Publishing book - Premium Drupal themes In 2013 I was invited by Packt Publishing to play a role of technical reviewer in one of their books.Just got my sample of the book Premium Drupal Themes by mail.It
Drupal 7 vs Symfony 2: overview after 1 year of Symfony development We've decided to switch to Symfony2 development in July 2012, if I'm not mistaken - after 7 years of (mostly) Drupal development. There were reasons to do that:1. Not-too-great experience with high
Storing monetary amounts in db? Use decimals, not floats! Lot's of people will be surprised (or shocked, when it happens in production) when they see that mysql can work like this:Query: SELECT transaction_id, amount FROM transactions WHERE amount>20.
Rules won't work properly when run during cron, if you use node access restrictions I've recently created USPS tracking module for Drupal, so Qwintry.com users could get notifications when their international packages change state. I've used queue operations to build requests to USPS API by cron,
Image title as text description for colorbox image formatter (Drupal 7) When you have core "image" field on your node type and you want to output "title" (or "alt") of this image just under the image thumbnail as text - you can use imagecaptionformatter
![[UPDATED] Scraping Instagram in 2021: avoiding 302 and 429 errors](/blog/content/images/size/w600/2021/01/IMG_20210112_114636.jpg)
