Most people don’t understand online automation. They don’t even realize that almost all their cyber-chores can be automated inexpensively. They think that Automation is too expensive and/or meant for larger enterprises. But fact is if any part of your business depends on the internet (And let’s face it, this is 2018. Everything does) chances are automation can save you a lot of time. Whether it’s automatically processing orders, keeping an eye on your competitors or just some cyber-chores.
Disclosure: I own two small businesses and also work as a freelance automation developer. Both of my businesses are highly automated and I’ve helped over 30 clients save more than a combined 100+ hours every day.
It’s hard to explain exactly what can be automated so I’ll instead give you an intuition by giving you a few examples:
Online car rental – One of my clients rented out cars via several online car rental websites. Each day he’d log into each website, browse various pages (Some websites with multiple accounts) and create an Excel spreadsheet of all the cars that have been booked, updated locations of each car etc. This took him about 1-2 hours per day. For $300, he now gets an updated spreadsheet in his Google Drive every 30 minutes with no action required by him.
Form generation – Another SMB client provided legal services. They would access data from an Excel sheet, fill it out on a PDF form then print it and mail it to a government office. He would then track the application online on their website to know the status of the application every day. Now a script automatically reads the Excel sheet, fills and prints out the form and also automatically tracks the status of every application and updates it in another Google Drive sheet.
Competitor watch – Another client had to check their competitor’s e-commerce websites regularly to keep an eye on their prices, this took them about 3-6 hours of work every week. Instead they now have a script that E-Mails them every time a price change is detected on a competitor’s website within 5 minutes of the price change happening.
This should give you an intuition for the kind of things that online automation can do for you. If you have any questions feel free to comment and I’ll try to give you as thorough an answer as possible!
Most projects i work on have little to no maintenance cost. It’s usually a script that you run on your computer, you just click it on it and watch it do it’s magic.
Maintenance costs may come in one of two ways:
- When the script breaks – This happens sometimes, mostly in the case of web scrapers. A website may get a new design and some piece of info that the script is reading may not be where it used to be which confuses the script. In cases like these it’s usually a quick fix. I usually offer to fix it for free in case of very minor issues, otherwise a very small fee (Usually $10-30).
- Servers – Sometimes we have to rent servers in the cloud (So that the script can run on your server instead of your computer). Depending on how much firepower your script needs the server’s charges can be anywhere between $2 a month for a weak server to hundreds of dollars a month (For when you need a really powerful cluster of servers. This is an extremely rare case though). Most projects usually don’t need servers though and when they do, there are also some free-of-cost options like the Amazon Lambda free tier.
Kofax, Connotate and Mozenda are three service-based automation SaaS-based products that anyone, including non-technical people, can use to build and execute automation scripts to run on a schedule, on demand or as a part of an “if-this-then-that’ workflow. As a person who has built automation software as a part of my business for the last 8 years, these services are impressive, cost-effective and reliable for the average use case. Plus, they are massively parallel and include built-in IP masking as well as a simple user interface to design and maintain scrape templates.
I have no affiliation with any of these services but just passing it along as it’s something I had considered at one time or another.
An addendum as well: In probably 80% of the cases I’ve seen, Excel is not the right tool for the job for any given data needs. Spending a little on a database architect to properly come up with a plan to store your data can save you tons of time down the road. It makes automating tasks that involve your data even easier to implement. Source: I’m a database architect.
I’m by no means a database architect, but I’m a competent user of Access. I watch some of our data analysts build huge Excel spreadsheets with all sort of complexity, essentially trying to recreate database functionality in a spreadsheet. It takes forever, is prone to errors, and incredibly difficult to audit. The same task in a database takes seconds.
MS Access is my dirty little secret. I throw data into a database, analyse it, and spit out the results in minutes rather than days. No one else around me is familiar with access, and they’re blown away by how quickly I can do the number crunching and come up with a compelling story about what the data means.
So people usually use Excel for everything from storing data, munging, doing pivots, joins, and data visualizations. Often times, I see a single spreadsheet contain multi-dimensional data (e.g. they have cells A1:E40 as a ‘table’, G2:G30 as another ‘table’, etc.) and then similar data from the pull from 6 months ago will be in a separate Excel file with similar, but different storage convention.
A much better approach would be to store everything in a normalized format such that there is no redundancy. For example, if you are looking at survey data, you would ideally put all of your questions in one table, people in another table, and then a third table that joins the questions, people, and answers together. By doing this, you can easily compare, say, the same persons answer from the last 6 months instead of needing to go into two spreadsheets, figure out how both of them store the data, and then manually determine which questions they answered, etc. etc.
You don’t even need anything complicated to do this either. While you can setup a database like PostgreSQL (free), you could also use database containers like SQLite or a hybrid like Access (if the data are <2GB) or LibreBase.
That said, there are situations where Excel is great. Quick and dirty munging is where it excels (pun intended) at in particular. The import process into say, SQLite takes like 30 seconds to go through all the steps. In Excel, that process can be like, seconds. So if you’re doing a simple, add columns 1 and 2 together and that’s all, Excel definitely wins. BUT the cool part about Excel is that if you aren’t using it for data storage, you can still use it to access your data, which gives you the best of both worlds.
As far as visualization tools are concerned, it’s fine. Again, quick and dirty. PowerBI (also MS product) makes better visualizations, especially if the data are more complicated. Tableau makes better live visualizations (embedded, interactive visualizations are possible very easily). And something like Matplot (Matplotlib in Python) gives you much more fine grained control over the appearance of your visualizations. But the latter three have more of a learning curve too.
All about the right tool for the job, which sometimes Excel can be, but there are so many great technologies that are out there that do things better than Excel it’s definitely worth it to branch out!
Yes. Python can do pretty much everything. Reddit is written in Python, for example. Python is also the most popular language for machine learning and data science. You can even program some of the more powerful microcontrollers with it. For automation in particular, the book Automate the Boring Stuff is highly recommended, though I haven’t read it myself. There’s also r/learnpython.
Be careful of the split between Python 2 and Python 3; not everyone was eager to update when 3 came out, so some libraries only support 2. On the other hand, some only support 3. Everyone’s gradually moving toward 3, so that’s what you should start with as a newcomer, so that you don’t have to switch to 3 after learning 2 (not that the differences are huge).
Interpreted vs. compiled and scripting vs. ‘real programming’ aren’t really meaningful distinctions anymore, if they ever were.
Screen scrapers to monitor competitors’ pricing, applications for automating tasks that were repetitive and monotonous (and therefore highly prone to user error), etc. It cannot be overstated just how much time can be saved through automating the simple little things.
For simple tasks I used to just write the scrapers in C#. For more complicated actions I used the iMacros API (I wouldn’t recommend it). These days I just use CefSharp for the complicated stuff.
I’d like to talk about a client who worked hard for their business. 10 hours a day! Their job? They would scour over 14 different mediums for trade alerts (Alerts for potentially good trades). These 14 were through different services. Some were over E-Mail, some over SMS, some on Slack chatrooms and others posted on websites.
Their day job was to filter through these alerts, find the most worthwhile ones and forward them on to their own users. On top of this they’d end up spending 1-2 hours a day just managing overhead. Subscriptions, adding them to their own E-Mail, SMS lists, verifying that the people already on the lists have paid for their subscription etc. Quite a tiresome process!The automation
One problem with the automation here was that the client needed to hand-pick the alerts that finally went out. They couldn’t provide a simple algorithm to do it, their customers were paying for 20 years of experience!
The job was simple. Build a bot that automatically: Reads E-Mails, SMS, Slack channels & constantly updating websites. Okay, maybe not so simple. An algorithm would then filter out the worst of these leads based on a few objective criteria. This’d eliminate about 70-80% of the alerts.
For the rest? They would get a notification on their mobile phone. The notification would have an ‘accept’ and ‘reject’ button right there in the notification bar. Accept it and the alert is forwarded to all their subscribers.
Their entire subscription management was also automated. A script would automatically add & remove subscribers and verify payments received.Lessons learned
Many of the times i work with clients, i end up automating 10-30% of their workload. Maybe some tool that allows them to pursue a new line of work that was previously too time consuming. Rarely do i have the opportunity to automate 80-90% of the work!
With one tool, this man got his life back. An entire workday spent monitoring different websites, E-Mail, slack and SMS changed to just going about your day and responding to mobile notifications every 5-10 minutes (Remember, most of the alerts are automatically filtered out, only a few actually go to him for review).
That’s the real power of automation. It gives you your life back. One day you’re working hard on your business. The next day you’re thinking hard about your next side hustle. Automation doesn’t just give you wings, it gives you an entire jet engine. Think hard about how much work you do for your business. Unless it requires that ‘insight’ garnered over years of experience, i could probably automate it. And even if it does, automation can take away most of the work as in this example.
F(AWXM) 