Metamorphosis from Link Broker to Enterprise Data Mining
In order to solve the problem of scaling without compromising safety we developed crawling and link graph extraction infrastructure. Comprising of a modular family of routable sub systems:-
- Streisand (Multi-lingual, keyword suggestion tree)
- Garland (Google crawler, SERP extraction)
- DeGeneres (56 link metrics – filtering & boolean operators)
- Elton (Contact extraction)
- Dolly Parton (Data pass-through – end to end workflow)
Engineered by Wade Womersley, the unique technology runs on a scalable NoSQL platform, Amazon Elastic Cloud (EC2) and uses an automated captcha solving system. Recent additions include VPN servers located in mainland China to enable secure and reliable encrypted access to mainland China.
CURRENT CRAWLING CAPABILITY
We’re proud to say that today our crawling and extraction infrastructure allows us to mine and extract, filter and perform, webmaster outreach in a business-critical, secure and scalable process. Further to the technological advantage and safety measures of business functions, including three levels of client sign-off, which provide an additional layer of transparency along with further reduction in risk from human error.
No longer a pure link broker, driven by necessity and client demand, we quickly found that reporting and SEO intelligence was vital for success, with our clients’ increasing appetite for aggression on the battleground in which we operate.
Our policy has always been to recycle bi-products of preceding solutions – which in our case is usually data (which carries a cumulative storage cost) and so, that is exactly what we did.
Since we were tracking the performance of our clients, we simply increased the frequency of the recrawl to six-hourly intervals, segmented each keyword or phrase into an industry, or product category and began calculating the standard deviation and volatility of this ever-increasing result set. This was the most exciting progression in our evolution, and is largely attributed to our sustained growth and widespread reputation for avoiding trouble with penalties, even when working on the most aggressive of campaigns.
SERP Deviation (now defunct) the first algorithm monitoring system with sector and industry granularity. Visualising algorithm updates in under six-hours (maximum) of them being applied by Google.
The technology later became part of my.reach.ai – currently is in private beta. To join the waiting list click here.