Based in London, united kingdom. working as a software engineer for palantir technologies. thoughts and ideas expressed are my own.

Software Engineering Bottlenecks

Software Engineering Bottlenecks

People who are not software engineers simply do not understand how bad are tools we are using every day compared to what we actually need and what we actually could have. Universally praised IDE’s, at the top level, are not that far from the text editor inside which I write this article.

 

Tools

From my standpoint, the biggest bottleneck and inefficiency in the software development is, without any doubt, tools we have to use. Especially the text editors and IDEs.

Since 70’s we have almost 50 years of progress in the hardware and software, and it’s still hard to navigate code, hard to get meaningful code completion, hard to type efficiently, hard to visualize program structure, hard to get any static analysis, hard to run code in isolation, hard to debug — it’s still a problem to do anything we are expected to do consistently across all the possible languages and runtimes.

Pinnacle of general-purpose text editors is Emacs+Evil which has horrible outdated UI concept as the foundation and no sane modern defaults. First spend half a year programming it, then you probably start to be productive.

While everyone in the industry understands that code does not read as a prose and is non-linear, and every token in source code has meaning attached to it, everything still relies on plaintext input and plaintext output. All the truly useful tools are command-line which emit plaintext to stdout. Visualization is non-existent.

Pinnacle of IDEs targeted specifically for programmer’s convenience is IntelliJ IDEA set of IDEs for a couple of largest mainstream languages, which are leagues ahead of their competition, except Visual Studio for C++/C# development on Windows, and even IDEA is still a bloated Java-based equivalent of Word for programmers, where you actually use 5% of the included buttons and even this 5% works so-so, because at the foundation it still works with text, not target language concepts. It’s non-modular, as base package is enormous, and compared to Emacs it’s essentially non-configurable.

People who are not software engineers simply do not understand how bad are tools we are using every day compared to what we actually need and what we actually could have. Universally praised Visual Studio Code feature-wise, at the top level, is not that far from the text editor inside which I write this article!

Debugging

The biggest inefficiency is in how we deal with bug fixing. For the past 50 years, we’ve used the same basic approach to fixing bugs:

1. test the software

2. detect a bug

3. diagnose the bug (find out where it comes from)

4. fix the bug (edit the source code)

5. save and compile the source code

go to #1

This edit-compile-test-debug cycle is very inefficient and time-consuming. It’s the reason why we can’t break out of the productivity trap. Any improvement in productivity that we’ve gained over the past couple of decades has been incremental. We’ve not seen 2–3X gains or higher.

The key to overcoming this bottleneck, I believe, is live coding, whereby you can inspect and modify code and data while the program is running. Detect a bug? No problem. Immediately inspect the code and data to determine the cause. Make the appropriate changes. Continue execution. No need to save the code, compile the code and rerun the program from the beginning.

This technique is not widely used for some reason. It’s been adopted in half-measure by languages like Java and Python, but live coding is never standard practice.

There is, however, one language where live coding is central to the way it’s used. Its implementation of live coding is convenient and easy to use and fully baked into the programming environment. It’s extremely efficient and the principal reason the language is twice as productive, on average, as JavaScript, C++, Go, Java, PHP, Python, and C#, according to Namcook Analytics. This is on average. In some instances, it enjoys much higher productivity like 5X.

What is this remarkable language? It’s called Smalltalk (Pharo). For more information, read Why Pharo Might be the Future of Software Development

DevOps

The real lack in current systems is that computers have very little ability to observe or reflect on their own state or resources. In today’s state-of-the-art IDEs, a compiler will fail because a library isn’t on the build path. But then just throws the problem back to you.

Imagine if that IDE could look on your computer and say “your path isn’t right but over in this directory there’s something that looks like the library, I just tried building against it and it seems to work like this. Is that right? Shall I just update your path?”

Or your web-server can’t find the database. Why can’t the computer tell you the database was moved to different address yesterday? There’s probably logs somewhere containing events sufficient to work this out. But PHP just says “connection failed”, it doesn’t go off to those logs to investigate.

Dev-Ops with scriptable infrastructure is a step in the right direction. But, really, operating systems could be so much better at reflecting on what resources they have and what the status of those resources is. A future operating system should provide an API for other tools to query that information, so that the tools could be more proactive in helping the programmer track down and solve resource related problems.

Reusability

We are unable to support generic medium-grained reuse. We got large-grained reuse (e.g., operating systems, databases systems) and we can excel at small-grained reuse (e.g., sort functions), but as soon as we enter medium-grained reuse (e.g., reusing a solution for a design pattern and configuring for your needs), then there are problems. There is domain-specific medium-grained reuse in frameworks such as .NET or Java Beans, but these technologies has a steep learning curve and there are multiple ways of doing the same thing. Developers can choose to use something that does not have as steep learning curve and ending up reinventing the wheel again.

Our inability to hinder or reducing the effect of adding technical debt to technical solutions. We make a lot of decisions in a project, all which are reflected in the technical solutions. Making incorrect or less useful decisions early on may lead to catastrophes at the end (e.g. Software Runaways).

The top 1%

The top 1%

Software Engineering Myths Debunked

Software Engineering Myths Debunked