Has everyone forgotten the art of creating or using performance profiling tools
I was in a meeting with a senior software engineer, who makes a shit ton more than me. While sharing my screen he saw flame graph I had for one program I was working on. They asked “what is that?”, I was taken back, and said it’s for looking at which functions are taking more time. Then I was asked “where did you find it?” I was shell shocked, didn’t know how to answer that. Similar things have happened before where I taught them certain things, and they go “great find”, like dude, I didn’t find it just now. That’s basic knowledge if you work with database.
I am not even a developer hire, I’m self learned, and because the company doesn’t have many developers they gave me some of the tasks and I’m doing it because it’s more fun than my job responsibilities, and when I don’t have enough tasks.
You see, performance is rarely a talking point in retrospectives unless its egregious. If you say a piece of compute is struggling the solution is to always buy more compute. It is rare that anyone would think of optimization unless you’re writing libraries.
Which is exactly why western AI companies are bankrupting themselves building data centers while China is easily keeping pace with a fraction of the resources. They’re not magic. They just do basic things like make things more efficient.
Or if its employee facing only let it get slower and slower to the point where it takes a minute to update one record
I think that’s one of the problem with people that are only of a “finish this task cheaply for now” mind.
Having something run in 0.1 second vs 1 second is fine when a user is running it for one thing. So they consider it good or instantaneous. And later they have to do it for 1000 other things, and they don’t think that saving time in that “instantaneous result” there would save so much more time. They just go “it’s 1000 of these so it takes time”. I’ve found functions that open a db file, read one entry and close it. Then that function is called for each entry they have to read later. In a preprocessing step where they know all the entries they need beforehand. And even the data tyoe to store it is nor preallocated, they just append it on every data read; again on a fixed number of entries they need.
I understand performance not being the most important thing, but I’d expect least a basic understanding of design that has unnecessary processing.
Flame graphs have really only been a thing for the last decade or so. The free tooling for profiling code has been shitty and difficult to understand for most of my career. And most companies aren’t willing to pay for it if the product runs “good enough”. These days companies lock down their shit so tight that you can’t even install an unapproved app no matter how free or useful.
I think that’s true for many tools. Programming tools are the one thing that has most advancement because so many programmer are there to improve on them. And even if you don’t use it, not knowing about it completely makes it a bit odd. Although most tools I have used to make flamegraph were as easy as just running the program as a argument to another program that tracks the function calls.
Ive used them to help a lot in the past, but they often didnt give enough detail. Like I would know the area something was a problem, but pinpointing exactly what it was was still up to me, and would take some time. It worked but it could have been better.
Turns out there was a library you can add which adds that extra missing detail, and with that it was possible to highlight the exact issue. Wish id known about that sooner. That was on me though.
Pay me to do it and I’ll gladly do it. Most companies, however, don’t really care much about whether the customers take an extra 100 milliseconds for every request or not. And when it’s promotion time, you’re often judged by how many tasks you get done, not how fast your code is.
Yes, people care way more about which cool new features were delivered than which technical debts were solved by a proposed initiative. Sometimes it’s frustrating to see that performance improvement work only gets prioritized after a big incident.
I’d add that most of the time nobody actually measured anything. at work we inherit slow sites all the time and the first guess is always “the stack”, then you profile it and it’s one query running per item in a loop or something like that. I had the same with my own side project, was sure I needed a bigger server and after measuring the problem was somewhere else completely. measuring first is boring but it’s cheaper than any rewrite I guess
Profiling complex systems can be quite difficult. That said, I believe in the adage “Profile twice, optimize once.”, adapted from the old carpenters adage of “Measure twice, cut once.”
Slowness is rarely essential, but I can be far away from where you expect it.
That’s what Open Telemetry is for, honestly, specifically tracing. I’m in the process of adding tracing for all of our web handlers, http client calls, database transactions/individual queries, and asynchronous jobs. Should give us a pretty good view into the performance of the system, and allow us to monitor for things like performance regressions without any dedicated profiling needed (though we certainly could add additional profiling information through Otel down the road). Since I know you’re a Haskeller (recognize your handle), the
hs-opentelemetryecosystem is quite solid and has integrations with a lot of libraries in the wider ecosystem, likewai,persistent,postgresql-simple,conduit, and a bunch more. I haven’t needed to write too much of our own instrumentation, aside from our job system since that’s a bespoke system.Yeah we used hs-opentelemetry at my last paid position. The company was an MS shop, tho so we were using ODBC to connect to MS SQL Server which doesn’t have great in-process profiling. But, at least we could figure out what quer(y/ies) were running long and then we could use the MS tools to look at the plan and fiddle with that. It was also a small org, so it’s not like there was a lot of internal services in one flow – basically everything either stayed in the Web Yesod app or in the IVR app. All our profiling could have been instrumented process profiling instead of having to use otel at all.
So, I haven’t seen how easy/difficult it is to actually use otel across multiple internal service calls to satisfy one external request.
yeah that matches what I see, most of what we run at work is one app and one database, so the slow query log plus looking at the plan gets you like 90% of the way. I never really had the multi service case where otel pays off, I guess at some point it does, but for a small team it felt like a lot of plumbing to end up finding the same n+1 query lol
What if the language has a bad design?
Then use it poorly and it’s a double negative.
Every method just returns an empty string. Runs fast as hell. Just don’t expect it to do anything.
I worked at a place where a site had to show the number of members, but for some reason it didn’t show the correct number. The reason was the query to the database was wrong, I corrected it in five minutes and the boss told me to put numbers manually, increasing them until we could display the correct query result because apparently the problem was there a long time before I started working there and it would make a bad image for the company if it was so easily corrected.
On other project a friend of mine told me about things were so slow and getting slower with time. Database didn’t have keys, indexes, nothing you should expect a relational database to have.
I haven’t worked in a while because of my health issues and I can assure you I’d be happier getting paid for watering plants than working with computers.
Both matter…
deleted by creator
I hate these kinds of comments that have no other purpose than to sling shit and start arguments.
Since this is a programming community (and giving you the benefit of the doubt, I assume you’re not a vibe coder), you know your argument is nonsense. Writing something in Python doesn’t automatically make it slow or use a lot of RAM or whatever. Synapse is shitty because Synapse is shitty. You can write slow and inefficient software in any language, and it feels really tedious to type this out because it’s obvious and common sense.
You can write slow and inefficient software in any language
Yes, but you can’t write snappy fast software in any language, high level languages like python, lua and php carry alot of overhead from the get go, because of dynamic typing, heavy abstractions, and heavy garbage collectors.
deleted by creator
I don’t think you’re making the kind of argument you intended to make. Those are two different programs with two different development teams behind them. The design behind them (the literal thing the article is talking about) are likely completely different from each other. Comparing the languages based on two different implementations of the same spec is a bit more than disingenuous. If you want to actually compare you need to recreate the same implementation as closely as feasible. Something like this https://youtu.be/WB7v-ezqcJM
deleted by creator
Optimising too early is the root of all evil or something like that
Too many abstraction layers
He is right in many aspects. However, for example Guile, Racket or SBCL Lisp are a lot faster than Python, and they are more powerful than Python while being much simpler, which makes them very interesting for prototyping algorithms.
The ideal combo for such things would be a powerful interactive language with debugging capabilities combined with Rust, perhaps even a language similar to Rust. I do not know of any easy solution for that, but it could be made work with Rust’s macro system (since the above languages all provide C ABI FFI).
Also,most of design happens when a program is written first. It is hard to re-design a larger program later, which might be one reason to re-write it. So maybe the main reason for rewriting is to get a better design, not the faster language - the faster language might be an additional reason.
Also, LLMs can produce terribly bloated Rust code - not only by a factor ten larger, but also much slower.
Also, CRUD apps are uninteresting to many programming enthusiasts, and for companies, optimizing them is only interesting if it saves them money.
Never understood the idea of, “if it needs to be fast, that part is in a faster language.” Why not just use the faster language? Most of the libraries that have Python wrappers also come with c headers. Sometimes you wind up in the reverse situation where your dependency doesn’t have a Python wrapper and now you’re stuck managing silly boilerplate. Python just feels like an extra thing in the way to wrangle for little gain.
Also, hotspot-only optimization doesn’t seem to have held up in reality, right? We can all agree that software is getting slower every day. The infrastructure is slow.
All feels like cope, through and through.
Why not just use the faster language?
Because the other language is more developer-friendly or faster to write/read even if slower to execute. Some code doesn’t need to be fast because it is always waiting on other things anyway.
Some languages effectively prevent whole classes of runtime failures / bad code. But, they do that by doing more which takes longer, at compile-time, run time, or both.
Some programs can’t avoid graph-structured data dependencies. In that case, the simplest solution is a garbage collector, and one built into the runtime is capable of being more efficient (and less buggy) than one written for a specific program.
Anyway, there are many factors that determine which language is “best”, and not all of them are objective. That’s why we don’t just use the fastest language. (Currently, I do all my coding in either Idris or Purescript, but I’m also unemployed and just programing for myself, for now.)
IMHO it’s easier to quickly iterate with Python than with C++. If you aren’t even sure what you want and you’re just writing stuff, then Python is great. Is “4” true? Ofc it is.
All programming is a trade-off between what’s easy for the developer and what’s fast for the machine.
Sure, you could theoretically write everything in Assembly and it’d be blazingly fast; but unless you’re a particularly elite developer, that process would take forever for any program more complicated than a calculator, the result would be super buggy, and those bugs would be very hard to fix.
On the other end of the spectrum, you might imagine a natural language machine learning compiler which translated directly from human prose to binary. It would be dead simple to “program” anything in; but such an application would probably be unbelievably slow, and again, have a staggering number of difficult-to-fix bugs.
Somewhere in the middle are modern programming languages that abstract away all of the stuff that’s necessary but hard for humans to wrap their minds around (memory management, API interaction, network stack, etc) and let the human write clear and unambitious code to handle all of the business logic and displaying it to the user. The result won’t run as blisteringly fast as writing directly in Assembly, and it won’t be as quick to market as Vibes-to-ASM, but the application will be developed in a reasonable amount of time and be reasonably easy to maintain.
There are NP problems and you should know how to assess them.
deleted by creator
Is slow because of both. Using bad languages is one form of bad design itself, in fact.
muh io bound
🤣
If the DB is IO-bound, and “slow languages” don’t matter because “IO bound”, then show me the DB you’re using implemented in a slow language.
It’s hilarious how op is being this stupidly unaware, even within his stupid false argument which is provided as a supposed given.
How are people seriously engaging with this blog fart full of false premises and contradictions?
Python copium.
Bullshit. My experience of Python is that only a small subset of programs are actually IO bound.
