IMO the author is mixed up on his definitions for concurrency.
replies(4):
Example: python allows concurrency but not parallelism. Well not really though, because there are lots of examples of parallelism in python. Numpy both releases the GIL and internally uses open-mp and other strategies to parallelize work. There are a thousand other examples, far too many nuances and examples to cover here, which is my point.
Example: gambit/mit-scheme allows parallelism via parallel execution. Well, kindof, but really it's more like python's multiprocess library pooling where it forks and then marshals the results back.
Besides this, often parallel execution is just a way to manage concurrent calls. Using threads to do http requests is a simple example, while the threads are able to execute in parallel (depending on a lot of details) they don't, they spend almost 100% of their time blocking on some socket.read() call. So is this parallelism or concurrency? It's what it is, it's threads mostly blocking on system calls, parallelism vs concurrency gives literally no insights or information here because it's a pointless distinction in practice.
What about using async calls to execute processes? Is that concurrency or parallelism? It's using concurrency to allow parallel work to be done. Again, it's both but not really and you just need to talk about it directly and not try to simplify it via some broken dichotomy that isn't even a dichotomy.
You really have to get into more details here, concurrency vs parallelism is the wrong way to think about it, doesn't cover the things that are actually important in an implementation, and is generally quoted by people who are trying to avoid details or seem smart in some online debate rather than genuinely problem solving.
I can do a lot of things asynchronously. Like, I'm running the dishwasher AND the washing machine for laundry at the same time. I consider those things not occurring at "the same time" as they're independent of one another. If I stood and watched one finish before starting the other, they'd be a kind of synchronous situation.
But, I also "don't care". I think of things being organized concurrently by the fact that I've got an outermost orchestration of asynchronous tasks. There's a kind of governance of independent processes, and my outermost thread is what turns the asynchronous into the concurrent.
Put another way. I don't give a hoot what's going on with your appliances in your house. In a sense they're not synchronized with my schedule, so they're asynchronous, but not so much "concurrent".
So I think of "concurrency" as "organized asynchronous processes".
Does that make sense?
Ah, also neither asynchronous nor concurrent mean they're happening at the same time... That's parallelism, and not the same thing as either one.
Ok, now I'll read the article lol
> Asynchrony: the possibility for tasks to run out of order and still be correct.
> Concurrency: the ability of a system to progress multiple tasks at a time, be it via parallelism or task switching.
> Parallelism: the ability of a system to execute more than one task simultaneously at the physical level.
its like the whole flammable/inflammable thing
You can write to one file, wait, and then write to the second file.
Concurrency not required.
Example 2
You can NOT do Server.accept, wait, and then do Client.connect, because Server.accept would block forever.
Concurrency required.
For more I'd look up Rob Pike's discussions for Go concurrency.
Wikipedia had the wrong idea about microkernels for about a decade too, so ... here we are I guess.
It's not a _wrong_ description but it's incomplete...
Consider something like non-strict evaluation, in a language like Haskell. One can be evaluating thunks from an _infinite_ computation, terminate early, and resume something else just due to the evaluation patterns.
That is something that could be simulated via generators with "yield" in other languages, and semantically would be pretty similar.
Also consider continuations in lisp-family languages... or exceptions for error handling.
You have to assume all things could occur simultaneously relative to each other in what "feels like" interrupted control flow to wrangle with it. Concurrency is no different from the outside looking in, and sequencing things.
Is it evaluated in parallel? Who knows... that's a strategy that can be applied to concurrent computation, but it's not required. Nor is "context switching" unless you mean switched control flow.
The article is very good, but if we're going by the "dictionary definition" (something programming environments tend to get only "partially correct" anyway), then I think we're kind of missing the point.
The stuff we call "asynchronous" is usually a subset of asynchronous things in the real world. The stuff we treat as task switching is a single form of concurrency. But we seem to all agree on parallelism!
Asynchrony is when things don't happen at the same time or in the same phase, i.e. is the opposite of Synchronous. It can describe a lack of coordination or concurrence in time, often with one event or process occurring independently of another.
The correctness statement is not helpful. When things happy asynchronously, you do not have guarantees about order, which may be relevant to "correctness of your program".
> Unfortunately this code doesn’t express this requirement [of concurrency], which is why I called it a programming error
I gather that this is a quirk of the way async works in zig, because it would be correct in all the async runtimes I'm familiar with (e.g. python, js, golang).
My existing mental model is that "async" is just a syntactic tool to express concurrent programs. I think I'll have to learn more about how async works in zig.
I don't know how many "monad tutorials" I had to read before it all clicked, and whether it ever fully clicked!
Asynchrony means things happen out of order, interleaved, interrupted, preempted, etc. but could still be just one thing at a time sequentially.
Parallelism means the physical time spent is less that the sum of the total time spent because things happen simultaneously.
Okay, but don't go with this definition.
Concurrency is writing code with the appearance of multiple linear threads that can be interleaved. Notably, it's about writing code. Any concurrent system could be written as a state machine tracking everything at once. But that's really hard, so we define models that allow single-purpose chunks of linear code to interleave and then allow the language, libraries, and operating system to handle the details. Yes, even the operating system. How do you think multitasking worked before multi-core CPUs? The kernel had a fancy state machine tracking execution of multiple threads that were allowed to interleave. (It still does, really. Adding multiple cores only made it more complicated.)
Parallelism is running code on multiple execution units. That is execution. It doesn't matter how it was written; it matters how it executes. If what you're doing can make use of multiple execution units, it can be parallel.
Code can be concurrent without being parallel (see async/await in javascript). Code can be parallel without being concurrent (see data-parallel array programming). Code can be both, and often is intended to be. That's because they're describing entirely different things. There's no rule stating code must be one or the other.
This is an artifact of wanting to write async code in environments where "threads" and "malloc" aren't meaningful concepts.
Rust does have a notion of autonomous existence: tasks.
But... that's everything, and why it's included.
Undefined behavior from asynchronous computing is not worth study or investment, except to avoid it.
Virtually all of the effort for the last few decades (from super-scalar processors through map/reduce algorithms and Nvidia fabrics) involves enabling non-SSE operations that are correct.
So yes, as an abstract term outside the context of computing today, asynchrony does not guarantee correctness - that's the difficulty. But the only asynchronous computing we care about offers correctness guarantees of some sort (often a new type, e.g., "eventually consistent").
in other contexts these words don't describe disjoint sets of things so it's important to clearly define your terms when talking about software.
It will IMO also be quite difficult to combine stackless coroutines with this approach, especially if you'd want to avoid needless spawning of the coroutines, because the offered primitives don't seem to allow expressing explicit polling of the coroutines (and even if they did, most people probably wouldn't bother to write code like that, as it would essentially boil down to the code looking like "normal" async/await code, not like Go with implicit yield points). Combined with the dynamic dispatch, it seems like Zig is going a bit higher-level with its language design. Might be a good fit in the end.
It's quite courageous calling this approach "without any compromise" when it has not been tried in the wild yet - you can claim this maybe after 1-2 years of usage in a wider ecosystem. Time will tell :)
One issue with the definition for concurrency given in the article would seem to be that no concurrent systems can deadlock, since as defined all concurrent systems can progress tasks. Lamport uses the word concurrency for something else: "Two events are concurrent if neither can causally affect the other."
Probably the notion of (a)causality is what the author was alluding to in the "Two files" example: saving two files where order does not matter. If the code had instead been "save file A; read contents of file A;" then, similarly to the client connect/server accept example, the "save" statement and the "read" statement would not be concurrent under Lamport's terminology, as the "save" causally affects the "read."
It's just that the causal relationship between two tasks is a different concept than how those tasks are composed together in a software model, which is a different concept from how those tasks are physically orchestrated on bare metal, and also different from the ordering of events..
All the pitfalls of concurrency are there - in particular when executing non-idempotent functions multiple times before previous executions finish, then you need mutexes!
Stated another way: if we just didn't talk about concurrent vs parallel we would have exactly the same level of understanding of the actual details of what code is doing, and we would have exactly the same level of understanding about the theory of what is going on. It's trying to impose two categories that just don't cleanly line up with any real system, and it's trying to create definitions that just aren't natural in any real system.
Parallel vs concurrent is a bad and useless thing to talk about. It's a waste of time. It's much more useful to talk about what operations in a system can overlap each other in time and which operations cannot overlap each other in time. The ability to overlap in time might be due to technical limitations (python GIL), system limitations (single core processor) or it might be intentional (explicit locking), but that is the actual thing you need to understand, and parallel vs concurrent just gives absolutely no information or insights whatsoever.
Here's how I know I'm right about this: Take any actual existing software or programming language or library or whatever, and describe it as parallel or concurrent, and then give the extra details about it that isn't captured in "parallel" and "concurrent". Then go back and remove any mention of "parallel" and "concurrent" and you will see that everything you need to know is still there, removing those terms didn't actually remove any information content.
Therefore I think this definition makes the most sense in practical terms. Defining concurrency as the superset is a useful construct because you have to deal with the same issues in both cases. And differentiating asynchrony and parallelism makes sense because it changes the trade-off of latency and energy consumption (if the bandwidth is fixed).
This is one of those "in practice, theory and practice are different" situations.
There is nothing in the async world that looks like a parallel race condition. Code runs to completion until it deterministically yields, 100% of the time, even if the location of those yields may be difficult to puzzle out.
And so anyone who's ever had to debug and reason about a parallel race condition is basically laughing at that statement. It's just not the same.
In that case, asynchronous just means the state that two or more tasks that should be synchronized in some capacity for the whole behavior to be as desired, is not properly in-sync, it's out-of-sync.
Then I feel there can be many cause of asynchronous behavior, you can be out-of-sync due to concurent execution or due to parallel execution, or due to buggy synchronization, etc.
And because of that, I consider asynchronous programming as the mechanisms that one can leverage to synchronize asynchronous behavior.
But I guess you could also think of asynchronous as doesn't need to be synchronized.
Also haven't read the article yet lol
So I guess you could define this scenario as asynchronous.
For single threaded programs, whether it is JS's event loop, or Racket's cooperative threads, or something similar, if Δt is small enough then only one task will be seen to progress.
They're just different names for different things. Not caring that they're different things makes communication difficult. Why do that to people you intend to communicate with?
No thanks.
Indeed so, but I would argue that concurrency makes little sense without the ability to yield and is therefore intrinsic to it. Its a very important concept but breaking it out into a new term adds confusion, instead of reducing it.
readA.await
readB.await
join(readA, readB).await
In ecosystems with good distributed system stories, what this looks like in practice is that concurrency is your (the application developers') problem, and parallelism is the scheduler designer's problem.
I think there needs to be a stricter definition here.
Concurrency is the ability of a system to chop a task into many tiny tasks. A side effect of this is that if the system chops all tasks into tiny tasks and runs them all in a sort of shuffled way it looks like parallelism.
No, because async can be (quote often is) used to perform I/O, whose time to completion does not need to be deterministic or predictable. Selecting on multiple tasks and proceeding with the one that completes first is an entirely ordinary feature of async programming. And even if you don't need to suffer the additional nondeterminism of your OS's thread scheduler, there's nothing about async that says you can't use threads as part of its implementation.
The abstraction makes it possible to submit multiple requests and only then begin to inquire about their results.
The abstraction allows for, but does not require, a concurrent implementation.
However, the intent behind the abstraction is that there be concurrency. The motivation is to obtain certain benefits which will not be realized without concurrency.
Some asynchronous abstractions cannot be implemented without some concurrency. Suppose the manner by which the requestor is informed about the completion of a request is not a blocking request on a completion queue, but a callback.
Now, yes, a callback can be issued in the context of the requesting thread, so everything is single-threaded. But if the requesting thread holds a non-recursive mutex, that ruse will reveal itself by causing a deadlock.
In other words, we can have an asynchronous request abstraction that positively will not work single threaded;
1 caller locks a mutex
2 caller submits request
3 caller unlocks mutex
4 completion callback occurs
If step 2 generates a callback in the same thread, then step 3 is never reached.
The implementation must use some minimal concurrency so that it has a thread waiting for 3 while allowing the requestor to reach that step.
Many other languages could already use async/await in a single threaded context with an extremely dumb scheduler that never switches but no one wants that.
I'm trying to understand but I need it spelled out why this is interesting.
await Join(f1(), f2())
Promise1 = f1(); Promise2 = f2();
await Join(Promise1, Promise2);
Asynchrony means that the requesting agent is not blocked while submitting a request in order to wait for the result of that request.
Asynchronous abstractions may provide a synchronous way wait for the asynchronously submitted result.
No, the definition provided for asynchrony is:
>> Asynchrony: the possibility for tasks to run out of order and still be correct.
Which is not dependence, but rather independence. Asynchronous, in their definition, is concurrent with no need for synchronization or coordination between the tasks. The contrasted example which is still concurrent but not asynchronous is the client and server one, where the order matters (start the server after the client, or terminate the server before the client starts, and it won't work correctly).
Quote from the article where the exact opposite is stated:
> (and task switching is – by the definition I gave above – a concept specific to concurrency)
Quote from the post where the opposite is stated:
> With these definitions in hand, here’s a better description of the two code snippets from before: both scripts express asynchrony, but the second one requires concurrency.
You can start executing Server.accept and Client.connect in whichever order, but both must be running "at the same time" (concurrently, to be precise) after that.
Alright, well, good enough for me. Dependency tracking implies independency tracking. If that's what this is about I think the term is far more clear.
> where the order matters
I think you misunderstand the example. The article states:
> Like before, *the order doesn’t matter:* the client could begin a connection before the server starts accepting (the OS will buffer the client request in the meantime), or the server could start accepting first and wait for a bit before seeing an incoming connection.
The one thing that must happen is that the server is running while the request is open. The server task must start and remain unfinished while the client task runs if the client task is to finish.
Synchronous logic does imply some syncing and yielding could be a way to sync - which is what i expect you mean.
Asynchronous logic is concurrent without sync or yield.
Concurrency and asynchronous logic do not exist - in real form - in von Neumann machines
It's true that it's possible - two async tasks can be bound together in sequence, just as with `Promise.then()` et al.
... but it's not necessarily the case, hence the partial order, and the "possibility for tasks to run out of order".
For example - `a.then(b)` might bind tasks `a` and `b` together asynchronously, such that `a` takes place, and then `b` takes place - but after `a` has taken place, and before `b` has taken place, there may or may not be other asynchronous tasks interleaved between `a` and `b`.
The ordering between `a`, `b`, and these interleaved events is not defined at all, and thus we have a partial order, in which we can bind `a` and `b` together in sequence, but have no idea how these two events are ordered in relation to all the other asynchronous tasks being managed by the runtime.
If asynchrony, as I quoted direct from your article, insists that order doesn't matter then the client and server are not asynchronous. If the client were to execute before the server and fail to connect (the server is not running to accept the connection) then your system has failed, the server will run later and be waiting forever on a client who's already died.
The client/server example is not asynchronous by your own definition, though it is concurrent.
What's needed is a fourth term, synchrony. Tasks which are concurrent (can run in an interleaved fashion) but where order between the tasks matters.
try io.asyncConcurrent(Server.accept, .{server, io});
io.async(Cient.connect, .{client, io});
As written, `asyncConcurrent(...)` is confusing as shit, and unless you memorize this blog post, you'll have no idea what this code means. I get that Zig (like Rust, which I really like fwiw) is trying all kinds of new hipster things, but half the time they just end up being unintuitive and confusing. Either implement (async-based) commutativity/operation ordering somehow (like Rust's lifetimes maybe?) or just use what people are already used to.
From the article:
> Like before, the order doesn’t matter: the client could begin a connection before the server starts accepting (the OS will buffer the client request in the meantime), or the server could start accepting first and wait for a bit before seeing an incoming connection.
When you create a server socket, you need to call `listen` and after that clients can begin connecting. You don't need to have already called `accept`, as explained in the article.
So I think it's nice when type systems let you declare the environments a function supports. This would catch mistakes where you call a less-portable function in a portable library; you'd get a compile error, indicating that you need to detect that situation and call the function conditionally, with a fallback.
I don't mean "promise.then", whereby the issuance of the next request is gated on the completion of the first.
An example might be async writes to a file. If we write "abc" at the start of the file in one request and "123" starting at the second byte in the second requests, there can be a guarantee that the result will be "a123", and not "abc2", without gating on the first request completing before starting the other.
async doesn't mean out of order; it means the request initiator doesn't synchronize on the completion as a single operation.
Yes yes, in theory they're the same. That's the joke.
And with green threads, you can have a call chain from async to sync to async, and still allow the inner async function to yield through to the outer async function. This keeps the benefit of async system calls, even if the wrapping library only uses synchronous functions.
But even with that definition, it seems like the idea of promises, task tracking, etc is well tread territory.
Then they conclude with how fire and forget tasks solve coloring but isn't that just the sync-over-async anti-pattern? I wouldn't be excited that my UI work stops to run something when there are no more green threads but they seem excited by it.
Anyway, I guess I got too distracted by the high concept "this is a fundamental change in thinking" fluff of the article.
If you need to synchronize stuff in the program you can use normal plain variables, since it's guaranteed that your task will be never interrupted till you give control back to the scheduler by performing an await operation.
In a way, async code can be used to implement mutex (or something similar) themself: it's a technique that I use often in JavaScript, to implement stuff that works like a mutex or a semaphores with just promises to syncronize stuff (e.g. you want to be sure that a function that itself does async operations inside is not interrupted, it's possible to do so with promises and normal JS variables).
Currently my best answer for this is the bind (>>=) operator (including, incidentally, one of its instances, `.then(...)`), but this is just fuzzy intuition if anything at all.
Edit: maybe it's actually implication? Since the previous line(s) logically imply the next. L_0 → L_1 → L_2 → L_n? Though this is non-commutative. Not sure, it's been a few years since my last metalogic class :P
If I launch 2 network requests from my async JavaScript and both are in flight then that’s concurrent.
Definition from Oxford Dictionary adjective 1. existing, happening, or done at the same time. "there are three concurrent art fairs around the city"
For example, it might be partial ordering is needed only, so B doesn't fully depend on A, but some parts of B must happen after some parts of A.
It also doesn't imply necessarily that B is consuming an output from A.
And so on.
But there is a dependency yes, but it could be that the behavior of the system depends on both of them happening in some partial ordering.
The difference is with asynchronous, the timing doesn't matter, just the partial or full ordering. So B can happen a year after A and it would eventually be correct, or at least within a timeout. Or in other words, it's okay if other things happen in between them.
With synchronous, the timings tend to matter, they must happen one after the other without anything in-between. Or they might even need to happen together.
[1] https://math.stackexchange.com/questions/785576/prove-the-co...
I also wrote a blog post a while back when I did a talk at work, it's Go focused but still worth the read I think.
[0] https://bognov.tech/communicating-sequential-processes-in-go...
await foo()
await bar()
Since any function can be turned into a coroutine, is the red/blue problem being moved into the compiler? If I call:
io.async(saveFileA, .{io});
Furthermore, I guess if you are dealing with pure zig, then its fine, but if you use any FFI, you can potentially end up issuing a blocking syscall anyways.
1. Zig plans to annotate the maximum possible stack size of a function call https://github.com/ziglang/zig/issues/23367 . As people say, this would give the compiler enough information to implemented stackless coroutines. I do not understand well enough why that’s the case.
2. Allegedly, this is only possible because zig uses a single compilation unit. You are very rarely dealing with modules that are compiled independently. If a function in zig is not called, it’s not compiled. I can see how this helps with point 1.
3. Across FFI boundaries this is a problem in every language. In theory you can always do dumb things after calling into a shared library. A random C lib can always spawn threads and do things the caller isn’t expecting. You need unsafe blocks in rust for the same reason.
4. In theory, zig controls the C std library when compiling C code. In some cases, if there’s only one Io implementation used for example, zig could replace functions in the c std library to use that io vtable instead.
Regardless, I kinda wish kristoff/andrew went over what stackless coroutines are (for dummies) in an article at some point. I am unsure people are talking about the same thing when mentioning that term. I am happy to wait for that article until zig tries to implement that using the new async model.
That being said, I agree we don’t need a new term to express “Zig has a function in the async API that throws a compilation error when you run in a non-concurrent execution. Zig let’s you say that.” It’s fine to so that without proposing new theory.
For lamport concurrent does not mean what it means to us colloquially or informally (like, "meanwhile"). Concurrency in Lamport's formal definition is only about order. If one task is dependent or is affected by another, then the first is ordered after the second one. Otherwise, they are deemed to be "concurrent", even if one happens years later or before.
I don't think it's sufficient to say that just because another term defines this concept means it's a better or worse word. "commutativity" feels, sounds, and reads like a mess imo. Asynchrony is way easier on the palette
Maybe there will be unforeseen problems, but they have promised to provide stackless coroutines; since it's needed for the WASM target, which they're committed to supporting.
> Combined with the dynamic dispatch
Dynamic dispatch will only be used if your program employs more than one IO implementation. For the common case where you're only using a single implementation for your IO, dynamic dispatch will be replaced with direct calls.
> It's quite courageous calling this approach "without any compromise" when it has not been tried in the wild yet.
You're right. Although it seems quite close to what "Jai" is purportedly having success with (granted with an implicit IO context, rather than an explicitly passed one). But it's arguable if you can count that as being in the wild either...
Commutivity is a very light weight pattern, and so is correctly applicable to many things, and at any level of operation, as long as the context is clear.
For example, C# uses this syntax:
await readA();
await readB();
Is Zig any different?
Only for n = 0, I think. Otherwise, generalizing associative binary f_2 to f_n for all positive integers n is easily done inductively by f_1(x) = x and f_{n + 1}(x_1, ..., x_n, x_{n + 1}) = f_2(f_n(x_1, ..., x_n), x_{n + 1}), with no need to refer to an identity. (In fact, the definition makes sense even if f_2 isn't associative, but is probably less useful because of the arbitrary choice to "bracket to the left.")
I can't think of anything in practice that's concurrent but not parallel. Not even single-core CPU running 2 threads, since again they can be using other resources like disk in parallel, or even separate parts of the CPU itself via pipelining.
Please don't implement this yourself
Subtraction for instance is not commutative. But you could calculate the balance and the deduction as two separate queries and then apply the results in the appropriate order.
I can't agree. It is confusing, because you need to remember the blog post, it wouldn't be confusing in the slightest if you internalized the core idea. The question remains: is it worth it to internalize the idea? I don't know, but what I do know is some people will internalize it and try to do a lot of shit with this in mind, and after a while we will be able to see where this path leads to. At that point we will be able to decide if it is a good idea or not.
> "Asynchrony" is a very bad word for this and we already have a very well-defined mathematical one: commutativity.
It is risky to use "commutativity" for this. Zig has operators, and some of them are commutative. And it will be confusing. Like if I wrote `f() + g(). Addition is commutative, then Zig is free to choose to run f() and g() in parallel. The order of execution and commutativity are different things. Probably one could tie them into one thing with commutative/non-commutative operators, but I'm not sure it is a good idea, and I'm sure that this is the completely different issue to experimenting with asynchrony.
Exactly, but why would anyone think differently when the goal is to support both synchronous and async execution?
However, if asynchrony is done well at the lower levels of IO event handler, it should be simple to implemcent by following these principles everywhere — the "worst" that could happen is that your code runs sequentially (thus slower), but not run into races or deadlocks.
Oxford dictionary holds no relevance here, unless it has took over a definition from the field already (eg. look up "file": I am guessing it will have a computer file defined there) — but as it lags by default, it can't have specific definitions being offered.
...this seems like a long way round to say "JS code is not parallel while C code can be parallel".
Or to put it another way, it seems fairly obvious to me that parallelism is a concept applied to one's own code, not all the code in the computer's universe. Other parts of the computer doing other things has nothing to do with the point, or "parallelism" would be a completely redundant concept in this age where nearly every CPU has multiple cores.
This isn't even remotely true; plenty of languages have both async and concurrency, probably more than ones that don't. C# was the language that originated async/await, not JavaScript, and it certainly has concurrency, as do Swift, Python. Rust, and many more. You're conflating two independent proprieties of JavaScript as language and incorrectly inferring a link between them that doesn't actually exist.
Promise1 = f1(); Promise2 = f2();
v1,v2 = await Join(Promise1, Promise2);
return v1 + v2
On the other hand, it is necessary becase some of underlying async calls can be order dependend.
for example
await sock.rec(1) == 'A' && await sock.rec(1) == 'B'
Except for loops which allow going backwards, and procedures which allow temporarily jumping to some other locally linear operation.
We have plenty of syntax for doing non-forwards things.
One might say "Rust's existing feature set makes this possible already, why dedicate syntax where none is needed?"
(…and I think that's a reasonably pragmatic stance, too. Joins somewhat infrequent, the impediments that writing out a join puts on the program relatively light… what problem would be solved?
vs. `?`, which sugars a common thing that non-dedicated syntax can represent (a try! macro is sufficient to replace ?) but for which the burden on the coder is much higher, in terms of code readability & writability.)