How to Survive an AI Apocalypse and Win at Screen Scraping

In today’s rapidly evolving technological landscape, AI advancement is creating unprecedented uncertainty in our world. Understanding how to navigate this changing environment is crucial, especially for developers and product managers trying to plan for the future.

Planning in an Age of AI Uncertainty

The concept of the technological singularity – the theoretical point where we create AI slightly smarter than ourselves, leading to a recursive explosion of intelligence – was popularized by Ray Kurzweil in “The Singularity is Near.” Kurzweil predicted human-level artificial general intelligence by 2029, a forecast that once seemed absurd but now appears increasingly plausible.

However, attempting to prepare for superintelligence is largely futile. Much like prehistoric creatures couldn’t have meaningfully prepared for human dominance, we can’t effectively prepare for a world of superintelligence. Instead, we should focus on the spectrum of outcomes between our current state and that extreme scenario.

Disruptive vs. Sustaining Innovations

Clayton Christensen’s “The Innovator’s Dilemma” distinguishes between disruptive innovations (like PCs, which started as underpowered but rapidly improved) and sustaining innovations (like GUIs, which enhance existing products). While AI appears highly disruptive, technologies typically have both disruptive and sustaining aspects.

The key is to evaluate whether your current projects will be sustained or disrupted by AI advancements. Products that automatically improve when underlying AI gets better will thrive, while those that might be incorporated into foundation models will struggle.

A useful mental exercise: imagine a future where most of your customers are AI agents rather than humans. How would your product fare in that environment?

Leveraging AI Tools Today

The major AI labs (OpenAI, Anthropic, etc.) are currently burning through unprecedented amounts of investment capital. OpenAI and Anthropic reportedly burned $5 billion last year, with projections suggesting OpenAI will burn $14 billion and Anthropic $3 billion this year – far exceeding even Uber’s record-setting cash burn.

This situation won’t last forever. Developers should take advantage of these heavily subsidized tools while they’re available. The productivity gains can be substantial – personal data suggests a 3x increase in productivity (measured by git commits) after incorporating AI tools into development workflows.

Effective Ways to Use AI in Development

Here are key strategies for leveraging AI in your development process:

1. Treat AI as junior engineers: Provide clear interfaces and function declarations, let AI handle implementation details, then carefully review the results
2. Code review assistance: Have AI review your critical code, catching bugs you might miss and providing valuable checklists of considerations
3. Pair programming: Talk through problems with AI to overcome development blocks that might have previously taken days or weeks
4. Test automation: Generate comprehensive test cases to improve coverage, especially valuable for startups with limited testing resources
5. Documentation assistance: Create and maintain thorough documentation at scale

However, AI tools currently struggle with truly novel R&D challenges (where there’s little existing training data) and with genuine creativity that transcends pattern recognition.

Real-World AI Application: Building a Search Service

When building a complex search service with multiple components (browser cluster, results parser, and API), AI tools provided varying levels of benefit:

• For the browser cluster (highly specialized with limited documentation), AI offered minimal benefit
• For standard API components, AI was helpful but not transformative
• For classification of diverse search result types, AI provided massive acceleration by rapidly understanding and categorizing numerous result formats
• For CSS selectors, AI provided initial implementations but often created brittle solutions requiring human refinement

This mixed pattern of usefulness reflects the current state of AI tools – exceptionally powerful in some areas while limited in others.

Looking Ahead

The most prudent approach is to assume AI development will follow an S-curve rather than exponential growth – rapid advancement that eventually plateaus. Focus on creating products complementary to AI rather than those at risk of being substituted by it.

While we can’t predict exactly how AI will transform our technological landscape, we can position ourselves to thrive in the wide range of possible futures between our current state and a theoretical singularity.