Based in London, united kingdom. working as a software engineer for palantir technologies. thoughts and ideas expressed are my own.

Modern software engineering

Modern software engineering

Immutable laws are rare in software engineering. When an engineer claims to have found one, that engineer is usually regarded with skepticism.

The intellectual tradition is one of servility to power, and if I didn’t betray it I’d be ashamed of myself
— Noam Chomsky

Immutable laws are rare in software engineering, and when an engineer claims to have found one, that engineer is usually regarded with skepticism.

General principles are more welcome.

1. You do not have to prove yourself.

Software is a new field and nobody knows how to do it. If someone says you are unqualified and therefore you must do maintenance work, you should question that person. We have an upside down system where the people who are paid the least do the crappiest work. They tend to be young and naive.

This is not an axiom.

The narrative that is sold to young engineers by giant companies is the following:

“take your $80k/yr job, do the software maintenance which makes the company $1 million, and hate your life.”

“After you have spent enough time in the first tier of the intellectual strip mine, we will make you an SDE 2, where you can do slightly higher level refactoring for $150k a year, which will make the giant company $5 million.”

This is what is called an arbitrage.

2. You are not a commodity.

We have an assembly line mindset left over from the industrial age. Software is more of an artisanship. Don’t believe that you are a replaceable cog. Don’t believe the one-size-fits-all interview process with whiteboarding problems. These serve to grind away your individuality and make you feel like an assembly line worker.

3. Software engineering is an art and a science — but rarely both at once.

The planning and design process is an art, but once the requirements are in place you can proceed more deterministically. The same is true for the other quantitative activities like poker, music, and writing.

4. You are not your job.

If you make your job the same thing as who you are, then your self-worth is defined by those who are judging you in your job.

Your job is a means to service your own higher purpose.

5. The world is a distributed system.

When you take an action on your smartphone, there is latency before that action is ingested. Servers sometimes will lie to you, but servers tend towards eventual consistency. The world works the same way. In the short term, human systems lie to us all the time, but the world tends toward eventual consistency–the truth eventually presents itself. A decent analogy is the efficient market hypothesis: slowly efficient markets are an eventually consistent process.

The world is a distributed system–what is the consequence of this? We have to do the arduous risk and reward calculations that are mandatory for every distributed systems programmer.

Long-tail failures can and do occur.

In a distributed system, we often prioritize safety over liveness. In a distributed operating system, the programmer takes all precaution to avoid data loss. Similarly, if a real life scenario presents a small probability of giant downside risk, you should take huge precaution. If someone offers you to roll a die with 1000 sides, and if you roll a 1–999, you get $100 million, but if you roll a 1000, you get your head chopped off, the upside is capped, but the downside is not.

I can estimate how significantly $100 million will improve my life, but it is impossible to quantify how bad it is to get my head chopped off. I would almost never take that offer, because the expected value is infinitely negative.

Coming back to the consequences of living in a distributed system, there is a huge range of potential outcomes in a given transaction, and assessing these long tail, massive risks takes a lot of deep thought and calculation.

6. You are not a lottery ticket.

This is a quote from Peter Thiel. He argues that we should think in terms of calculus rather than statistics because if you believe the world is a lottery, then you will give yourself permission to lose.

Example: a software engineer might say “I am applying to a job with 1000 applicants, so the chances I will get the job are pretty low.” That is looking at a game of skill as a game of luck, and if you don’t optimize with lots of preparation and sleep for that interview, then it’s your own fault.

If you play poker or Magic or Dominion, it is easy to blame a loss on luck, but the best players are the ones who optimize in spite of bad luck. Personally, I like playing games under circumstances when I get unlucky because it makes for more of an interesting challenge and a better learning experience.

Luck is an idea that rescues us when we don’t do enough due diligence. We live in a society that loves to talk about luck because it gives us an excuse when we lose. That is not to say that there isn’t luck in the world–plenty of people are unlucky. But if you are reading this, chances are you are in the luckiest .01%.

There is a long-running philosophical argument about whether or not we control our own fate. In actuality, it doesn’t really matter.

We should assume that we are in control of our own fate.

If we aren’t in control, it doesn’t change anything whether or not we assume that we are in control. But if we are in control, it would be very hazardous to our well-being to assume that we are not in control.

7. Action over planning

Talk is cheap and execution is scarce.

This is why nobody cares about your ideas, they care about seeing your prototype. John Mayer said that all he had to do to become successful is finish his songs.

At Amazon, they called this “bias for action”. At Facebook, they “move fast and break things”.

That said, usually you can choose both. Long-term planning is underrated because few people take the aggressive short term actions necessary to iterate towards it. If you have never seen long-term planning bear fruits, it’s hard to see the point.

8. Software engineering is full of lies and people who will try to take advantage of you.

You have to figure out what to believe for yourself.

There are also lots of great people, but it is up to you to develop intuition and strong reasoning.

There is a scene from The Big Short where Mark Baum is standing in front of bankers saying “the world is full of fraud, our food is fraudulent, our banking system is fraudulent, our medicine is fraudulent”. If you look around and investigate each of these things, you do find that these systems have layers of deception and lies. Software engineering is the same.

For example, if you get to a place that gives you lots of perks like free lunch or back massages, understand that they are doing that because they are underpaying you. If you are getting $130k salary, $70k in stock options, free lunch, and free laundry, that may sound like a pretty good deal, but what does that say about how much money the company is making off of you?

Apple: $1,865,306 per employee

Google: $1,154,896 per employee

Microsoft: $732,224 per employee

Amazon: $577,482 per employee

Software economics are unintuitive: It takes $0 to make new copies of a commodity that only has to be written once. This contrasts with the assembly line where each new unit takes new effort and human friction to reproduce. With software, only the first unit takes significant labor to produce.

As software engineers, we have to rethink our value as workers in a marketplace with the high leverage of software economics. Giant tech companies can pull off this arbitrage of underpaying engineers because engineers let themselves be seduced into a set of myths.

“The intellectual tradition is one of servility to power, and if I didn’t betray it I’d be ashamed of myself.”

-Noam Chomsky

There is a narrative of a programmer who is incapable of doing anything except programming. Some programmers talk about this with pride, saying things like “I am just an engineer, I don’t want to think about the business side of things, I don’t understand the business side of things”.

Engineers have been seduced by the industrialist’s perspective that we cannot lead ourselves, we cannot evaluate opportunity cost, and we don’t understand the market as a whole. All of these are lies, and the world will be more efficient and utilitarian if engineers take control of their careers and start evaluating the options outside of their immediate, narrow context.

9. You are not your credentials nor your past.

If you started in sales, you can learn to become an engineer. If you started as an engineer and don’t like it, you can become a podcaster. You don’t need a degree–if you can do the work, you can get a job as an engineer.

Don’t let yourself be defined by your past, and the labels and messaging that society applies to you. You can take control of your life and define your future.

10. As a software engineer, you can take career risks aggressively because your downside is fundamentally capped.

You should be skipping to work. We spend most of our time at work. The job market is really good right now because our economy is being completely reshaped by software. Massive economic opportunity is being generated:

  • AWS (still young) takes startup costs from $50k+ to ~$0

  • Mobile (still young) puts a supercomputer in everybody’s pocket

  • Emerging markets still aren’t quite online, hinting at huge latent demand

  • Each of the above has compounding effects on each other

These are economic fundamentals, not spurious technical signals. Don’t believe the ranting and raving venture capitalists arguing for lower valuations and screaming about a bubble. Don’t believe the luddite anti-technologists who sneer into their lattes.

By any logical projection, the high demand for engineers isn’t dissipating anytime soon, unless the entire economy collapses in some ghastly black swan catastrophe — in which case your current job will likely vaporize anyway.

It is worth considering how often your job puts you in a state of flow and tranquility. If your work is stressful and entirely unpleasant, finding a new job should be a huge priority.

The most sophisticated software written by humankind

The most sophisticated software written by humankind

Is Cambridge Analytica a threat to democracies?

Is Cambridge Analytica a threat to democracies?