Klipper is a 3d-Printer firmware. It combines the power of a general purpose computer with one or more micro-controllers. See the features document for more information on why you should use Klipper.
To begin using Klipper start by installing it.
Klipper is Free Software. Read the documentation or view the Klipper code on github. We depend on the generous support from our sponsors.
Analysis Intel is ending its Optane product line of persistent memory and that is more disastrous for the industry than is visible on the surface.
The influence of ideas from the late 1960s and early 1970s is now so pervasive that almost nobody can imagine anything else, and the best ideas from the following generation are mostly forgotten.
Optane presented a radical, transformative technology but because of this legacy view, this technical debt, few in the industry realized just how radical Optane was. And so it bombed.
To get to the heart of this, let's step back for a long moment and ask, what is the primary function of a computer file?
The first computers didn't have file systems. The giant machines of the 1940s and 1950s, built from tens of thousands of thermionic valves, only had a few words of memory. At first, programs were entered by physically wiring them into the computer by hand: only the data was in memory. The program ran, and printed out some results.
As capacities grew and we arrived at the von Neumann architecture in which the computer program is stored alongside the data in the same memory. In some early machines, that "memory" was magnetic storage: a spinning drum.
To get it into the memory, it was read off paper: punch cards, or paper tape. When computer memories got big enough to store several programs at once, operating systems appeared: programs that managed other programs.
Still no file systems, though. There was RAM and there was I/O: printers, terminals, card readers and so on, but all the storage directly accessible to the computer was memory. In the 1960s, memory often meant magnetic core storage, which had one great advantage that's sometimes forgotten now: When you turned the computer off, whatever was in core store stayed there. Turn the computer back on, and its last program was still there.
Around this time, the first hard disk drives started to appear: expensive, relatively slow, but huge compared to working memory. The early operating systems were given another job: the problem of managing that vast secondary storage. Indexing its contents, finding those sections that were wanted and loading them into working memory.
Once operating systems started managing disk drives, a distinction appeared: primary and secondary storage. Both directly accessible to the computer, not loaded and unloaded by a human operator like reels of paper tape or decks of punched cards. Primary storage appears right in the processor's memory map, and every individual word is directly readable or writable.
Secondary storage is a bigger, much slower, pool that the processor can't directly see, and can only access by requesting, or sending, whole blocks to another device, a disk controller, which fetches the contents of the specified blocks from a big pool of storage, or places them into that pool.
This split continued down into the eight-bit microcomputers of the 1970s and 1980s. The author fondly remembers attaching a ZX Microdrive to his 48K ZX Spectrum. Suddenly, my Spectrum had secondary storage. The Spectrum's Z80 CPU had a 64kB memory map, of which a quarter was ROM. Each Microdrive cartridge, even though it was just 100kB or so, could store about twice the machine's entire usable memory. So there had to be a level of indirection: it was impossible to load the whole cartridge's contents into memory.
It wouldn't fit. So cartridges had an index, and then named blocks containing BASIC code, or machine code, or screen images, or data files.
Since microcomputers, we still call primary storage "RAM" and we still call secondary storage "disks" or "drives", even though in many modern end-user computers, it's all just different types of electronics with no moving parts or separate media.
You start the computer by loading an OS from "disk" into RAM. Then, when you want to use a program, the OS loads it from "disk" into RAM, and then that program probably loads some data from disk into RAM. Even if it's a Chromebook and it doesn't have any other local apps, its single app loads data from another computer over the internet, which loads it from disk into RAM and then sends it to the laptop.
Since UNIX was first written in 1969, this has become a mantra: "Everything is a file." Unix-like OSes use the file system for all kinds of things that aren't files: access to the machine is governed by metadata on files, I/O devices are accessed as if they were files, you can play sounds by "copying" them to a sound device, and so on. Since UNIX V8 in 1984, there's even a fake file system, called /proc, that displays information about the memory and processes of the running system by generating pretend files that users and programs can read, and in some cases write.
Files are a powerful metaphor of sorts, which have proved versatile to a degree unimaginable in 1969, when Unix was written on a minicomputer with a maximum of 64k words of memory and no sound, graphics or networking. Files are ubiquitous now.
But files, and file systems, were only a crutch.
The concept of the "computer file" was invented because memory was too expensive, too big, and too slow. The only way to attach millions of words of storage to a 1960s mainframe was a disk drive the size of a filing cabinet, and too much storage to fit into the computer's memory map.
So instead, mainframe companies designed disk controllers, and built a form of database into the OS. Imagine, for instance, a payroll program, maybe only a few thousand words in size, that could handle a file for tens of thousands of employees, by doing it in tiny chunks: read a row from the personnel file, and a row from the salaries file, compute a result, and write a row to the paycheck file, then repeat. The OS checks the indexes and converts this into instructions to the disk controller: "here, fetch block 47 track 52, head 12, sector 34, and block 57 from track 4, head 7, sector 65… now, write 74.32 into this block…"
SSDs appeared in the 1990s, and by the first decade of this century they were getting affordable. SSDs replace magnetic storage with electronic storage, but it's still secondary storage. SSDs pretend to be disk drives: the computer talks to a disk controller, and sends and receives sectors, and the drive converts them and shuffles around blocks of storage which can only be erased in chunks, typically of a megabytes or more, to emulate hard-disk-style functionality that writes 512-byte sectors.
The trouble is, flash memory has to be accessed this way. It's too slow to be mapped directly into the computer's memory, and it's impossible to rewrite flash byte-by-byte. In order to modify a byte in a block of flash, the rest of contents of that whole block must be copied elsewhere, and then a whole block wiped. This is not how computers' memory controllers work.
Optane made it possible to eliminate that. Like core store, it is working memory: primary storage. Optane kit is as big and as cheap as disk drives. It shipped in the hundreds of gigabytes size range, the same sort of size of a modest SSD, but it could be fitted directly into a motherboard's DIMM slots. Every byte appeared right there in the processor's memory map, and every byte could be rewritten directly. No shuffling around blocks to erase them, like flash. And it supports millions of write cycles, rather than tens of thousands.
Many hundreds of gigs, even terabytes, of dynamic non-volatile storage, thousands of times faster and thousands of times more robust than flash memory. Not secondary storage on the other side of a disk controller, but right there in the memory map.
Not infinitely rewritable, no. So your computer needs some RAM as well, for holding variables and fast-changing data. But instead of "loading" programs from "disk" into "RAM" every time you want to use them, a program loads once, and then it's there in memory forever, no matter if there's a power cut, no matter if you turn your computer off for a week's holiday. Turn it back on, and all your apps are still right there in memory.
No more installing OSes, no more booting up. No more apps. The OS sits in memory all the time, and so do your apps. And if you have a terabyte or two of nonvolatile memory in your computer, what do you need SSDs for? It's all just memory. One small section is fast and infinitely rewritable, but its contents disappear when the power goes. The other 95 per cent holds its contents forever.
Sure, if the box is a server, you can have some spinning disks so you can manage petabytes of data. Data centers need that, but very few personal computers do.
Linux, of course, supported this. This particular vulture wrote the documentation for how to use it on a prominent enterprise distro. But Linux being Linux, everything must be a file, so it supported it by partitioning it and formatting it with a filesystem. Using primary storage to emulate secondary storage, in software.
No current mainstream OS understands the concept of a computer that only has primary storage, no secondary storage at all, but it's split between a small volatile section and a large nonvolatile section. It's hard to even describe it to people familiar with how current computers work. I have tried.
How do you find a program to run if there are no directories? How do you save stuff, if there's nowhere to save it to? How do you compile code, when there is no way to #include one file into another because there are no files, and where does the resulting binary go?
There are ideas out there for how to do this. The Reg wrote about one of them 13 years ago. There is also Twizzler, a research project investigating how to make it look enough like a Unix system for existing software to use it. When a lab boffin at HP invented the memristor, HP got very excited and came up with some big plans… but it takes a long time to bring a new technology to the mass market, and eventually, HP gave up.
But Intel made it work, produced this stuff, put it on the market… and not enough people were interested, and now it is giving up, too.
The future was here, but when viewed through the blurry scratched old lenses of 1960s minicomputer OS design, well – if everything is a file, this Optane was just a sort of really fast disk drive, right?
No, it wasn't. It was the biggest step forward since the minicomputer. But we blew it.
Goodbye, Optane. We hardly knew you. ®
As of now, Capy is NOT ready for use in production as I'm still making breaking changes
Capy is a graphical user interface library for Zig. It is mainly intended for creating applications using native controls from the operating system. Capy is a declarative UI library aiming to be easy to write for and versatile.
It has been made with the goal to empower standalone UI applications, integration in games or any other rendering process is a non-goal.
A simple application using capy:
const capy = @import("capy"); const std = @import("std"); pub fn main() !void { try capy.backend.init(); var window = try capy.Window.init(); try window.set( capy.Column(.{ .spacing = 10 }, .{ // have 10px spacing between each column's element capy.Row(.{ .spacing = 5 }, .{ // have 5px spacing between each row's element capy.Button(.{ .label = "Save", .onclick = buttonClicked }), capy.Button(.{ .label = "Run", .onclick = buttonClicked }) }), // Expanded means the widget will take all the space it can // in the parent container capy.Expanded( capy.TextArea(.{ .text = "Hello World!" }) ) }) ); window.resize(800, 600); window.show(); capy.runEventLoop(); } fn buttonClicked(button: *capy.Button_Impl) !void { std.log.info("You clicked button with text {s}", .{button.getLabel()}); }
It is easy to add something like a button or a text area. The example can already be used to notice a widget's parameters are usually enclosed in anonymous structs (.{ .label = "Save" }). You can also see that simply wrapping a widget with capy.Expanded( ... ) will tell it to take all the space it can.
If you're starting a new project, simply clone capy-template and follow build instructions
Otherwise if you're adding capy to an already existing project:
Before proceeding, you must first install the zigmod package manager. Then, in the folder of your project, you can execute the following commands:
In your build.zig, add:
diff --git a/usr/bin/ziglang/lib/zig/init-exe/build.zig b/build.zig index 29b50b5..ccbb74b 100644 --- a/usr/bin/ziglang/lib/zig/init-exe/build.zig +++ b/build.zig @@ -1,6 +1,7 @@ const std = @import("std"); +const deps = @import("deps.zig"); -pub fn build(b: *std.build.Builder) void { +pub fn build(b: *std.build.Builder) !void { // Standard target options allows the person running `zig build` to choose // what target to build for. Here we do not override the defaults, which // means any target is allowed, and the default is native. Other options @@ -11,7 +12,9 @@ pub fn build(b: *std.build.Builder) void { // between Debug, ReleaseSafe, ReleaseFast, and ReleaseSmall. const mode = b.standardReleaseOptions(); - const exe = b.addExecutable("$", "src/main.zig"); + const exe = b.addExecutable("capy-template", "src/main.zig"); + const pathToCapy = ".zigmod/deps/git/github.com/zenith391/capy/"; exe.setTarget(target); exe.setBuildMode(mode); + try deps.imports.capy.install(exe, pathToCapy); exe.install();
And in your zigmod.yml file, add:
diff --git a/default_zigmod.yml b/zigmod.yml index e39f6f1..4774adb 100644 --- a/default_zigmod.yml +++ b/zigmod.yml @@ -2,4 +2,6 @@ id: Random ID name: Your app name license: Your license description: A description. +build_dependencies: + - src: git https://github.com/zenith391/capy root_depedencies:
Finally, run
For more information, please look in the wiki
A platform is considered supported only if it can be built from every other OS.
✅ Windows x86_64
✅ Windows i386
✅ Linux x86_64
✅ Linux i386
✅ Linux aarch64 (PinePhone, PineBook...)
✅ FreeBSD x86_64
✅ WebAssembly
🏃 macOS M1
🏃 macOS x86_64
Note: As there's no "official" GUI library for Linux, GTK 3 has been chosen as it is the one that works and can be configured on the most distros. It's also the reason Libadwaita won't be adopted, as it's meant for GNOME and GNOME only by disallowing styling and integration with other DEs.
Programming language from down under, inspired by this Reddit post.
View live demo here.
Special thanks to MarkWhyBird, louis100, and others who came up with the language spec.
G'DAY MATE!
THE HARD YAKKA FOR fibonacci IS ( x ) <
YA RECKON x <= 1 ? BAIL x;
BAIL fibonacci(x - 1) + fibonacci(x - 2);
>
GIMME fibonacci(30);
Check out DOCS.md
Check out CONTRIBUTING.md
Thanks goes to these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind welcome!
In this issue:
Note: Today's post is pretty long. Consider reading it on the site, since your email client may truncate it.
The highest-stakes gambling events in the world are typically very discreet, invite-only affairs. One that might be close to the top in terms of available winnings happens at the end of Jane Street internships: interns get a stack of 100 poker chips and spend half a day getting asked brainteasers and then betting on their confidence in the answers. Some of these questions might be pure math and probability questions, some might be more abstract bets on making a market in some outcome, and apparently one of the questions is a tough probability question where part of the prompt is to bet on how long it will take to get the answer.
This would actually be a pretty fun way for a math-minded person to spend a few hours, if it weren't so high-stakes: the winners get a job from which people routinely retire rich in their 30s, and the losers... don't. Jane Street is a quantitative prop-trading firm, i.e. they use their internal capital to make markets in liquid assets around the world, balancing completely automated strategies with human intervention. In terms of expected value, this might be the single best field for a smart and technical person to work in. Starting compensation can be up to $425k. (Don't feel too bad for interns who don't get an offer, either; they're apparently getting $16.5k/month.)
The point of the poker chip challenge is not just to test out people's probability chops in a thematic way. It's to measure what's arguably the most important bundle of traits for a trader to have: a sense of when to risk what, and enough self-awareness to calibrate confidence. It's also a way to filter out one particular form of overconfidence: no trade is good enough that it's worth the risk of losing everything. For what shall it profit a man, if he shall make a bet with unbounded positive expected value, and lose his entire bankroll? The history of finance in general and quantitative finance in particular is littered with examples of very smart people who had good track records and then blew up. The brilliant people are table stakes. Building a lasting institution means embedding risk awareness into everything.
Jane Street is not the only big prop trading firm, and not the only firm that thinks this way and tries to hire from the same pool. They're worth focusing on for three reasons: first, the combination of secrecy about some things (and more on this below) plus aggressive recruiting means that they have a bigger surface area for outside research than most firms; second, a few of their decisions, like systematically buying black swan insurance and using the obscure Ocaml language, make them an interesting outlier; and third, it's hard to argue with success: Jane Street earned $6.3bn in the first half of 2020, up more than 10x from the year before ($, FT).
In one sense, every investor is a market maker and the only difference is their timeline. Jane Street is far along the continuum towards strict market-making: being willing to buy and sell assets at a price close to, but not exactly at, the market price.
Two truisms in folk economics are that liquidity is free and that payment, which doesn’t involve credit risk, is also basically free. Both are much more true than they used to be, but not quite true—both of these things are expensive, best provided with the kind of scale that only humans working closely with computers can offer, and (especially as they get cheaper) catastrophic to offer on the wrong terms.
The price of liquidity reflects basically three factors:
Even if there are buyers and sellers, they may not be willing to transact and precisely the same terms and at exactly the same moment. A market-maker is a wholesaler, willing to store inventory between when a seller needs to get rid of it and when the next buyer comes along.
Prices drift around somewhat randomly, and the pain of losses exceeds the joy of wins. (That's true both for psychological reasons and because sufficient losses can wipe you out.)
Informed traders may keep trading in the same direction; the more eager someone is to accept what looks like an unfavorable price, the more likely they are to a) be planning to trade in larger volume in the very near future, and b) to be well-informed about the likely profits of that trade.
So market-makers are, in the average case, getting a steady return in exchange for taking some big tail risks. Jane Street's early specialization was in exchange-traded funds, but they also trade equities, bonds, options, commodities, and crypto.
The exchange-traded fund market is a great case study in how these kinds of firms can make a profit and provide liquidity. ETFs are a nice, simple structure where a tradable asset represents a basket of securities (which could be an index fund like the S&P, or oil futures, or a set of positions returning triple the inverse return of an index, or whatever). "Authorized participants" can exchange a block of the relevant securities for a block of 50,000 ETF shares, or vice-versa. So a market-maker can continuously make bets on the gap between the price of the ETF and the price of its constituents, and if they’re an AP they can periodically close the arbitrage by delivering ETF shares or the underlying.
That's the trivial part, which you could do manually in excel if you had quick reaction time, and also a time machine that would take you back to the late 90s. Today, things are more complicated. And Jane Street has grown up along with those complications: their original system was, in fact, written in Excel (with lots of VBA and C#). Today, their systems are written in Ocaml, a language that would be fairly obscure if it weren't for Jane Street itself.
The tricky part is finding the million tricks that help you spot adverse selection, detect patterns in the market, determine when a 99%-hedged position is a better deal than paying the transaction cost for a 100%-hedged position, and reselling some of this liquidity-provision skill to other investors. One example of this is NAV trading (Jane Street has a paper here), where an investor wants to place a large trade in an ETF and agrees to buy it at some future point at whatever its net asset value is, less some small fee. The fee example they give is five basis points, so for their hypothetical $30 million trade, that's a fee of $15,000. It's a slim margin for a low-risk trade, but it's not a no-risk trade. In this trade, what the client is paying Jane Street to do is a fun act of financial alchemy: transmute the liquidity of an ETF's constituents into liquidity in the ETF itself. Jane Street can do this by just buying the underlying assets of the ETF, by buying the ETF itself, by buying things that correlate to the value of the ETF such that they'd be comfortable being long the ETF and short the other assets as a hedge, or any combination of these. So the real deal is that the counterparty is paying Jane Street 0.05% to find the most efficient way to let them take the risk they want to take, and also paying them for the possibility that selling this risk will be done imperfectly.
Some parts of this process can get automated, and some should require manual intervention. And this gets to what's fundamentally so interesting about the business: Jane Street is in a constant process of finding and extending the efficient frontier of where computers can replace humans in finance. The firm has researchers, programmers, and traders, but the exact definitions of these jobs seem quite porous. A trader's job is partly to take risk, and partly to accumulate data on what kinds of risk can be systematically measured and hedged away.
Full automation is tempting, but it's critical to do it in a judicious way. As Jane Street's Yaron Minsky once put it, "There is no faster way for a trading firm to destroy itself than to deploy a piece of trading software that makes a bad decision over and over in a tight loop." (This was written in November 2011. About a year later, Knight Capital deployed a new trading system that accidentally activated some old code it wasn't supposed to. The company engaged in a whirlwind of trading with itself, turning 212 orders into 4 million accidental orders covering 397 million shares—making a bad decision over and over again in a very tight loop—and, within a few hours, had destroyed itself.)
The best thing for an algorithmic trader is another algorithmic trader who is overconfident but not as good. Let's say one both traders are using one signal, but the smarter one is using a second signal that sometimes suggests the opposite trades. In cases where the signals point in opposite directions, the worse trader is a wonderful source of liquidity for the smart one.
Human discretion is a great safeguard against this: for n-of-1 events, you want humans in the loop to quickly update when a system is going to make bets that no longer make sense. But for n-of-more-than-1 events, human traders are basically training their replacement, finding underlying patterns and then identifying ways to automate exploiting them. This is a great way for a company to approach trading if it has a deliberately flat hierarchy with vague distinctions between roles; a trader is always implicitly doing research on how to effectively implement strategies and how to spot their edge cases, and is ideally going to sketch out an automated fix once the nature of the fix is obvious.
If you think about systematic trading, the fun thing to think about is picking up a continuous stream of income from taking the other side of less sophisticated traders. But what you actually want to spend most of your time on is thinking about ways things can go wrong. It's not a short list.
Correlations can break, or reverse. They're especially prone to reversing if people trusted them and bet accordingly (if Ford and GM always correlated, but GM launches a new vehicle that starts taking serious share from the F150, a naïve systematic investor will be long Ford and short GM because of the growing valuation gap, even though that gap reflects a change in the real world. When they exit the trade, the gap will get even bigger.)
Prices move overnight, and there isn't time to exit illiquid positions.
Rules change. If you were arbitraging the difference between prices of Russian stocks and US-traded ADRs representing those stocks, you might find that one side of your trade was suddenly illegal.
And the rules don't just change because of geopolitics. Sometimes, a financial institution will decide that one of its customers is too big to fail, and good luck to everyone else. Jane Street, incidentally, is suing the London Metals Exchange over voiding trades ($, FT). This happens in other cases, too. Some of the funds that made good bets against mortgage-backed securities in 2007 found that their brokers were very reluctant to mark those trades accurately.
More prosaically, but just as expensively: code sometimes has bugs. Market data feeds are not always 100% prompt or 100% accurate. Keyboards are not necessarily coffee-proof. And when margins are low and turnover is high, the impact of even minor technical errors can quickly be magnified.
As with all jobs, you can make a selection effects-based generalization about traders: the experienced ones are obsessed with risk, because the ones who aren't obsessed with risk get the experience of being fired.
Market makers have a special reason to care unreasonably about risk: a market-making strategy with no edge is exactly equivalent to writing put and call options on the underlying security. That's worth unpacking in two senses, one more impressionistic and one more theoretical:
The impressionistic one is that you should imagine what happens to a market-maker's P&L if they continuously provide liquidity to a trader who is making a big trade in one direction. If a big trader starts selling, the market-maker's position rises as the stock price goes down. So, like the seller of a put option, their exposure goes up as their position moves against them. Meanwhile, the higher the volatility of the stock, the more valuable it is for an informed seller to get a good price when selling. If there isn't such a seller, the market maker collects continuous profits. In other words, the market-maker's return is a function of the gap between expected volatility (what liquidity cost traders will pay to put on a position right now) and realized volatility (how much the stock subsequently moves). Just like an option seller.
The more theoretical view is that any standard option can be replicated through continuous delta hedging, i.e. if you sell someone a put option on 100 shares, and that option moves up 0.5% for every 1% the stock moves, you are perfectly hedged right now if you sell 50 shares short. When the price moves, the option's sensitivity to the price moves, too; as the stock goes down, the put's delta goes up, so you'd sell more shares to compensate. At zero transaction costs, the delta-hedging strategy perfectly replicates the option's returns, so any trading strategy that requires you to behave like an options hedger is economically equivalent to just making the options trade directly.
So a market-maker is, by default, in a position where they're selling insurance against extreme market moves. This is a tough place to be. Every time a trade successfully executes, there's some probability that it's because the trader on the other side just wanted liquidity, and some other probability that every other market maker saw that it was a bad trade and you're the one who got stuck making it.
But there are ways to mitigate this. One of them is to devote a lot of effort to finding every possible indicator of adverse selection, continuously model them out, seek out counterparties who are less likely to be well-informed, etc. In fact, this is one advantage of human traders: they're humans, dealing with other humans, and can get a sense of one another's personalities, propensity to bluff, etc. If liquidity provision is a market for lemons, where any big counterparty can represent a colossal risk, then traders' relationships are one way to mitigate that.
You can view the entirety of market making as a grand effort to write lots of theoretical options, collect a juicy premium, and develop an endless toolkit for dealing with adverse selection.
The other mitigation strategy is: just buy some puts. Markets usually don't crash upwards, but they do have a habit of crashing downwards. And for a market-maker, a crash is a uniquely interesting situation: volume is high, spreads rocket up because people are afraid to trade or don't have the liquidity, so an active participant can make a staggering amount of money. (I liked this Reddit AMA: "Yeah, 08-09 was insane. I've heard stories. No one knew what the fuck was going on and everyone was on edge. Then it all turned out fine and everyone got PAID.")
Jane Street has been in the habit of buying out-of-the-money put options and rolling them over. AQR has a nice paper on how this typically works; the chart on page two shows that buying puts loses money almost all the time and has little blips of outperformance during crashes. For most investors, it's a bad deal. But for a market-maker, it means that they're better-capitalized than ever at exactly the time when they have the most opportunities for profitable trades. They've been looking at tail risk for a long time. At year-end 2007, they had the largest exposure of any institutional investor to Bear Stearns puts ($, WSJ), though that may have been part of a different strategy.
A market-maker might be the part of the private sector best-equipped to profit from tail risk insurance. They have an immediate way to put it to good use for themselves, and meanwhile for the rest of the market participants, it's handy to have someone willing to trade—transaction costs won't be the bulk of your losses during a market crash, but they do add insult to injury, and one reason markets crash is that liquidity disappears. So firms that make markets and own out-of-the-money puts are probably saving investors tens of basis points on transaction costs when those investors trade during a panic, but might also be saving them percentage points of capital if the crash is less severe because there's a natural well-capitalized buyer.
Finance is intimately tied to language: money is the high-order bit, and finance is one of the easiest ways to convert a communications slip-up into a sizable loss. One fun place this shows up is in the industry's insistence on verbal disambiguation, at least in the parts of the industry that are still phone-centric: people and stocks get nicknames because you don't want to sell the wrong "Liberty" on behalf of the wrong "Bill" (an example not chosen at random; The Bond King apologizes for how many people in the book are named "Bill"). It's also why stocks have tickers; a lot can happen in the time it takes your ticker tape to print out UNITED STATES STEEL CO so you might as well call it X instead. Traders, shell scripting wizards, and haiku writers are at the top of the heap for compressing as much information as possible into the minimum number of characters or syllables.
At a firm focused on automation, that means thinking carefully about programming languages.
Ironically, "thinking carefully" is not the origin story of Jane Street's highly idiosyncratic decision to use the Ocaml language. The origin story is that they had a crufty system built on Excel, and hired a part-time researcher to build some analytical systems. That researcher, Yaron Minsky, chose Ocaml because he liked it, and because he didn't expect anyone else to have to maintain it afterwards. But then he decided to stick around to run a research group that used Ocaml, and a few years later convinced the rest of the company to move to Ocaml, too.
Programming language choices lead to endless debates, and in fact one of the reasons to use a popular language is that if everyone slightly hates it, at least you'll avoid factions where one group loves it and one loathes it. One solution to the political problem is to make an esoteric choice early on, and then filter hires in part by whether or not they think it's a bad one. That can be tricky, because a company can end up wedded to a subpar technology, but Jane Street has found a novel solution to that, too; they contribute lots of open-source libraries, and help keep the language alive and up-to-date. The more people there are who can throw together side projects in Ocaml, or who are writing other libraries for their own convenience, the better-off Jane Street is. Other companies can free-ride on this somewhat, but there are two likely cases here: first, if they're not prop traders, it's good for Jane Street to increase the population of Ocaml-fluent programmers and the number of contributors to new libraries. Second, if they are prop traders, Jane Street can probably outbid them for talent, since returns in systematic trading tend to compound.
Given the endless fractal nature of programming debates, and given that the readers most interested in these debates already have strong opinions, I'll focus on two areas where using Ocaml makes a difference.
First, Ocaml's design requires programmers to be very explicit about what data they're using and what they're trying to do with it. Excel is happy to tell you that the square root of today's date is 2:22pm on July 29th, 1900; Ocaml will insist that you figure out what you meant to take the square root of first. This has the obvious advantage that it spots certain kinds of bugs before code goes live; it simply won't work if you make certain categories of mistakes, even if what you're doing just consists of known mathematical operations on numbers. It won't catch every bug, of course, but this is helpful.
More subtly, this system makes it easier to reason about complex systems, since it's easier to see what code is executing when, and what it's trying to accomplish. Complex systems generate edge cases, and in trading edge cases generate bankruptcies. The more a trading system resembles a theorem instead of a recipe, the easier it will be. And code that explicitly declares types also has a form of documentation that's automatically audited by the compiler—you can predict often that the code will do what it says because if it doesn't do that, it won't do anything.
A second reason is consistent performance. A trading strategy needs many things to go right at the right time. Suppose there are four different components: a data feed, a pricing model, a trading system, and a risk monitor. The entire strategy's latency is ideally determined by the slowest—and I say "ideally" because if your trading system is acting right this second on what the risk system said ten minutes ago, and that's a ten-minute period during which volatility exploded and it overwhelmed your risk system, you can find yourself in a very bad situation. Pure speed matters, all else being equal, but one of the things that is not necessarily equal is the predictability of that speed.
Exchanges run a sort of bug bounty program where the prize is not necessarily a positive number. Their code is imperfect, and those imperfections are more likely to show up in extreme market events, both because of unanticipated price changes and because the high volume of trades can lead some systems to fail. (One admirable event in the chaotic spring of 2020 was CME's decision, on April 15th, to warn traders that oil prices could go negative and suggest that they test how their system would react in that event. Five days later, it happened.) Prop trading firms need to work around the deficiencies of the platforms they interact with, and having confidence in their own systems is a good way to speed up triage.
Using an unusual language is also a way for a firm to stay truly technical, when the temptation is to drift in the direction of a more traditional finance culture. A sufficiently technical firm can find ways to appropriately compensate people who do boring but necessary work like fine-tuning and refactoring old code. Even if they don't get a performance improvement, reducing future maintenance costs is valuable, and when traders, researchers, and developers are all somewhat fungible, you can pay someone appropriately for cleaning up an existing program, eliminating some redundancy, and making the code shorter and more readable. That's right: they can reverse the classic finance formula and pay a big bonus after someone obliterates a bunch of lines.
One of the meta questions to ask about an automated trading firm is: what do they want to ensure is the most reliable piece of code they have? Yaron Minsky has answered this in an interview: "We have lots of ways of turning things off, including a literal physical big red button." There are many ways to avoid blowups, but the future contains plenty of unprecedented events, and the only way to deal with them is to find a way to stop doing anything else until it's clear what to do next. And this is a case where readability, clarity, and reason-about-ability are all critically important: as anyone who has worked for a big government agency, corporation, or other bureaucracy knows, the hardest feature to engineer is an off-switch.
One thing you'll notice about quants is that they do not like to talk in detail about what they work on. Depending on the context, you might hear about an asset class, or a trade frequency, but actual strategies are kept very close to the vest. This is a notable contrast with non-systematic investors, who are all too happy to tell you what they like or don't like (at least once they're done putting on a trade). But this secrecy extends beyond talking about what they're doing—quants are even reluctant to talk about what strategies they used to use, even if those strategies no longer work, or about what they've looked into, even if it didn't produce a profit. This love of secrecy can reach truly pathological levels: when Bloomberg did a story on a (retired) founder of Jane Street, the company said it would comment and then declined to after four requests and the article's subject refused to disclose the names of his cofounders (information that's publicly available).
The secrecy about live strategies makes sense: if you have a source of alpha, you're splitting it with everyone else trading on the same signal, and the most pleasant split is to keep 100% for yourself. But the core of the trading business is not just executing current strategies, but constantly replacing them; most signals decay over time, whether it's because other traders catch on to the opportunity or because the traders who create that opportunity figure out what they're doing wrong. Avoiding information leakage even happens at the trade level. Every trade expresses a view, and one constraint on traders is that they want to be careful not to tip their hands. There can be times when it's optimal to take a position more slowly, and perhaps get a smaller one, rather than trading aggressively and prompting your counterparties to figure out what you know. Secrecy at every level of abstraction is a good way for a firm to get the most value out of the ideas it develops and the people it hires.
This creates some fascinating unit economics for prop trading firms. Every employee they hire has a high opportunity cost—when they're doing multiple rounds of interviews with multiple employees in each, they're burning lots of six- and seven-figure time vetting someone. Hiring well would be expensive even if the new employees weren't getting paid so well.
So when the firm looks at new opportunities, whether it's trading a new asset class or adding a new signal to an existing strategy, they have to run some basic calculation: expected alpha from the signal; expected upfront time spent on research and software implementation; the expected ongoing cost in terms of capital, time, and distraction from having human traders help implement the strategy; and the risk that either the research won't pan out or that it will produce a signal that looks good in theory and doesn't work in practice.
In this model, knowledge about which strategies don't work is incredibly valuable! It saves everyone a lot of time, and in a world where the value of existing signals is persistently decaying, that time is enormously valuable. Some strategies work for a while, fade out, and then start working again—perhaps the trade migrated from specialized hedge funds to banks in the 2000s, but banks were too risk-constrained to pursue it after the financial crisis, so the trading avenue opened up again. And some strategies might work by analogy; perhaps a market anomaly shows up in any country that hits middle-income status and develops a domestic base of equity investors, and knowing that it worked in Japan in the 70s can tell you something about how well it will work in Poland today.
This has another even more interesting takeaway. When unit economics are this visible, and when specific strategies are likely to become obsolete, it means the scope for strategic decisions is radically reduced. A strategy is either positive-EV or not; a new data source is either going to produce incremental profits that pay for it, or it won't; adding a new asset class might lead to some mild spillover effects—perhaps getting into currencies will add some alpha in pricing US-based exchange-traded funds tracking foreign companies—but a firm that's hedging out that part of the risk is basically outsourcing sophistication to their counterparties.
This doesn't mean that prop trading is a completely strategy-free business. It just means that the overwhelmingly important strategic choices are around hiring, company culture, and supporting Ocaml. Those are the cases where there are potential nonlinear returns, and complementary opportunities whose upside exceeds the sum of their parts. So one reason Jane Street spends so much energy on finding hiring the right people, running job ads on programming theory blogs or posting puzzles, holding guest lectures and after-hours chess tournaments, etc. is that every company reaches the point where "do something strategic!" is an itch that's worth scratching, but in prop trading there are very few legitimate ways to scratch it, so companies are forced to excel.
Is "liquidity provision" a socially useful thing, or is it a way that millionaire options traders rationalize the fact that they're not building something socially useful but less lucrative? This is a genuine, and difficult, question. If a firm asks employees to constantly think about edge cases, novel risks, and whether or not their behavior makes sense from first principles, it's only a matter of time before they go meta and start worrying that the big risk is that they're wasting their lives.
Maybe this is why so many of them retire in their 30s: the utility of money is roughly log(wealth), but the disutility of guilt seems more like a linear relationship. And if one of your hiring screens focuses on introspection, you may accidentally filter out the sociopathic personality traits that minimize guilt.
But there are two moral cases to be made for why someone who's good at programming and math should take the offer from Jane Street and not, say, go work for a fusion startup or try to cure cancer:
Market liquidity is actually useful. A tight bid/ask spread for S&P futures doesn't directly lead to more houses and chip fabs getting built, or spur the creation of massive healthcare breakthroughs. But liquidity increases the present value of every financial asset, and if our system for funding things is driven by financial investors, an implicit subsidy to their upside means more of them competing to provide it. That liquidity will have to be provided by someone, and if a small number of programmers and a bunch of cheap servers can replace hundreds of people yelling into phones in order to do the same work less efficiently, that's a social win. And if the traders are constantly working to automate their own efforts, then either a) we determine that liquidity provision is a fundamentally unsolvable problem, which is interesting to know and may have other profound implications, or b) we solve it, and the entire financial industry is replaced with server farms while its employees go off to do other kinds of work.
Trading is a lottery operated by market makers, and like the lottery its social function is to convert mass innumeracy into funding for better causes. There is a weirdly high overlap between quant finance and Effective Altruism in general, and between Jane Street and EA in particular (it is emphatically not 100%). Some of this may be because of where they advertise jobs, but some of it is an overlap in outlooks. Systematic thinkers who worry about tail events when they're pricing crude oil options may go home and think about existential risks to humanity instead of their portfolio. In a way, effective altruism and quantitative finance are inspired by the same core set of traits: a tendency to systematize and a keen sense that we live in a broken world. Human suffering and nonzero transaction costs are both symptoms of the world's imperfection, cast in a harsh glare by reasoning straightforwardly from first principles. And if fixing one of those problems translates directly into money that can be used to fix the other, it’s a perfect match. Building a better risk management system for fixed income ETF arbitrage strategies and spending the proceeds on mosquito nets are two sides of the same coin.
Either way, pushing out those efficient frontiers is a fun and engaging job. And that may be its own justification. We don't demand that chess champions use their skills in something with more real-world applications, or that concert pianists find a more practical outlet for their manual dexterity and attention to detail. Some jobs take a field with a general category of problem and create the cleanest, most abstract, and most challenging form of it. Sometimes it's not until you push a problem to its limits that you figure out what problem you're really solving.
Further reading: The author of The Laws of Trading worked at Jane Street, and it's a good book on the philosophy that systematic trading inculcates. This interview with Yaron Minsky is a great look at their decision to use Ocaml. If you're interested in more about the mechanics of exchanges and trading, the Hide not Slide Substack is good. As a starting point, here's their writeup on Jane Street. Max Dama on Automated Trading is old, but a very helpful overview of the industry for technical people. If you want to learn Ocaml, Jane Street's Yaron Minsky has coauthored a good book on it.
Tegus is the first port of call for M&A professionals and institutional investors ramping up on an industry or company.
Get access to a database of 35,000+ expert call transcripts, spanning 5+ years, or schedule expert calls through the platform for a fraction of the usual cost.
When thousands of research analysts are pooling their expert calls into an on-demand database, using Tegus is table stakes. It's the leading platform for due diligence and primary research.
See the power of a Tegus subscription, and get up to data parity with your competitors, with a two week free trial through the Diff.
US public pensions mostly report annual returns as of June 30th, and so far the results don't look great. Pension funds have been a looming problem for a long time, and early in the pandemic it looked like they might finally reach the point where they'd need a bailout and restructuring. For now, they're still targeting higher returns than they can likely achieve, which forces them to move into riskier asset classes to meet their benchmarks. This means that events that reduce asset valuations, i.e. increase expected future returns, hit them even harder because their portfolios are more-than-optimally concentrated in the riskiest assets.
Amazon has announced plans to shut down its Amazon Drive service at the end of next year. The classic Hacker News thread about the announcement of Dropbox includes some skepticism from users who see it as a marked-up version of Amazon S3. There's a lot of value in bridging the gap between cheap storage and a product people will actually pay for, and bridging that gap is nontrivial even for companies that make a point of customer obsession.
Disclosure: long AMZN.
Drone delivery is booming as a way to deliver contraband to prisons. New technologies often have a comparative advantage in the military or crime, because those are two cost-insensitive fields where unit economics are strongly influenced by physical risk. One response is to restrict their use, but that's a good way to ensure that crime still has a comparative advantage (the person with the lowest marginal cost for flying a drone illegally is the one who's flying a drone to get away with something else illegal).
Uber is showing drivers how much they'll be paid if they accept a trip. Ride-sharing is a business that scales to the point where churn is the dominant concern, and as it's matured, it's gotten harder for companies to compete on coverage and arrival time. Making concessions to driver preferences will mean that some users won't get as much service, but it also provides another dimension for the company to determine the optimal pricing tradeoff.
Early in the Ukraine war, I argued that the run-up in energy prices due to concerns about the war made it more likely ($), by increasing the cost of energy sanctions. This is still happening: a recent plan to ban insurance for Russian oil shipments has stalled because the UK is reluctant to sign on ($, FT), partly because higher energy costs would be so hard to bear. This presents the interesting possibility that the length of the war is partly a function of the world's economic health: a severe recession, and an ensuing drop in oil prices, would reduce the relative cost of sanctions and make continued conflict untenable for Russia.
Diff Jobs is our service matching readers to job opportunities in the Diff network. We work with a range of companies, mostly venture-funded, with an emphasis on software and fintech but with breadth beyond that.
If you're interested in pursuing a role, please reach out—if there's a potential match, we start with an introductory call to see if we have a good fit, and then a more in-depth discussion of what you've worked on. (Depending on the role, this can focus on work or side projects.) Diff Jobs is free for job applicants. Some of our current open roles:
A mid-frequency systematic fund is looking for quant developers. (New Jersey)
A company building a wildly ambitious decentralized future is looking for a principal product designer. (US, remote)
A company solving a major pain point for developers is looking for their Lead ML engineer. (Washington D.C. area)
A new emerging market bank is looking for people to support with payment and account operations. (Worldwide, remote)
A company building an AI-powered investment platform is looking for a data engineer. (NYC)
In a setback for Visa in a case alleging the payment processor is liable for the distribution of child pornography on Pornhub and other sites operated by parent company MindGeek, a federal judge ruled that it was reasonable to conclude that Visa knowingly facilitated the criminal activity.
On Friday, July 29, U.S. District Judge Cormac Carney of the U.S. District Court of the Central District of California issued a decision in the Fleites v. MindGeek case, denying Visa’s motion to dismiss the claim it violated California’s Unfair Competition Law — which prohibits unlawful, unfair or fraudulent business acts and practices — by processing payments for child porn. (A copy of the decision is available at this link.)
In the ruling, Carney held that the plaintiff “adequately alleged” that Visa engaged in a criminal conspiracy with MindGeek to monetize child pornography. Specifically, he wrote, “Visa knew that MindGeek’s websites were teeming with monetized child porn”; that there was a “criminal agreement to financially benefit from child porn that can be inferred from [Visa’s] decision to continue to recognize MindGeek as a merchant despite allegedly knowing that MindGeek monetized a substantial amount of child porn”; and that “the court can comfortably infer that Visa intended to help MindGeek monetize child porn” by “knowingly provid[ing] the tool used to complete the crime.”
“When MindGeek decides to monetize child porn, and Visa decides to continue to allow its payment network to be used for that goal despite knowledge of MindGeek’s monetization of child porn, it is entirely foreseeable that victims of child porn like plaintiff will suffer the harms that plaintiff alleges,” Carney wrote.
In a statement, a Visa spokesperson said: “Visa condemns sex trafficking, sexual exploitation and child sexual abuse materials as repugnant to our values and purpose as a company. This pre-trial ruling is disappointing and mischaracterizes Visa’s role and its policies and practices. Visa will not tolerate the use of our network for illegal activity. We continue to believe that Visa is an improper defendant in this case.”
A rep for MindGeek provided this statement: “At this point in the case, the court has not yet ruled on the veracity of the allegations, and is required to assume all of the plaintiff’s allegations are true and accurate. When the court can actually consider the facts, we are confident the plaintiff’s claims will be dismissed for lack of merit. MindGeek has zero tolerance for the posting of illegal content on its platforms, and has instituted the most comprehensive safeguards in user-generated platform history.”
The company’s statement continued, “We have banned uploads from anyone who has not submitted government-issued ID that passes third-party verification, eliminated the ability to download free content, integrated several leading technological platform and content moderation tools, instituted digital fingerprinting of all videos found to be in violation of our Non-Consensual Content and CSAM [child sexual abuse material] Policies to help protect against removed videos being reposted, expanded our moderation workforce and processes, and partnered with dozens of non-profit organizations around the world. Any insinuation that MindGeek does not take the elimination of illegal material seriously is categorically false.”
In June, MindGeek CEO Feras Antoon and COO David Tassillo resigned. The Montreal, Quebec-based company also laid off an unknown number of employees. That came in the wake of a June 20 New Yorker exposé that found Pornhub hosted sexually explicit nonconsensual videos including those with children.
The plaintiff in the case against MindGeek and Visa is Serena Fleites, who, when she was 13, was pressured by her then-boyfriend into making a sexually explicit video — which he allegedly uploaded to Pornhub (with the title “13-Year Old Brunette Shows Off For the Camera”) without her knowledge or consent. Fleites’ attorneys say the video, which was alleged to have been viewed millions of times on MindGeek sites, destroyed her life: “While MindGeek profited from the child porn featuring plaintiff, plaintiff was intermittently homeless or living in her car, addicted to heroin, depressed and suicidal, and without the support of her family,” her lawsuit, filed in June 2021, states. Fleites’ story was featured by New York Times columnist Nicholas Kristof in December 2020, who detailed how MindGeek “monetizes child rapes.”
In his July 29 decision, Carney ruled partly in Visa’s favor. He wrote in the opinion that Fleites “simply has no basis for claiming Visa directly participated in the sex trafficking ventures that harmed her.” In addition, he ordered Fleites to provide “a more definite statement with respect to her common law civil conspiracy cause of action against Visa.”
In a second ruling (available at this link), Carney compelled MindGeek to undergo jurisdictional discovery, which attorneys for Fleites said will reveal MindGeek’s “shadowy operations and those controlling it” by exposing the defendant’s financial relationships. “Where the money flows in the MindGeek web, which may relate to ownership of the porn sites that generate revenue, matters to the court’s jurisdictional analysis,” the judge said in the opinion. “As the court sees it, financially benefitting from the sexual exploitation of minors is the core of this case.”
On Saturday, activist investor Bill Ackman of Pershing Square Holdings, who has previously called out the role of Visa and Mastercard in enabling MindGeek’s ability to make money from child pornography, posted a thread on Twitter about the ruling in the case.
“Visa’s conduct here is inexcusable, likely to cause the company incalculable financial and reputational damage” as well as “create serious… personal liability and potential criminal liability for the board,” Ackman wrote in part. According to Ackman, neither he nor Pershing Square have any economic interest, long or short, in Visa, Mastercard or any other payments company, bank or financial institution.
According to Ackman, after he read the Times’ story about Fleites and Pornub, he reached out to the CEOs of Visa and Mastercard to express concerns about their part in enabling MindGeek’s business. Shortly afterward, both companies cut off consumer payment processing to MindGeek’s sites; within “a day or so, MindGeek removed >10m illegal videos, 80% of its content,” the hedge fund manager said. However, they both soon reactivated business-to-business payments for the purchase of ads on MindGeek sites and for subscriptions to “premium” content, representing about 90% of the company’s revenue, per Ackman.
Ackman wrote that Visa CEO Alfred Kelly “should know that the majority of child trafficking victims are from lower-income families including Black and Brown families. I would recommend that Visa’s board, and separately Mr. Kelly, should hire independent white collar and criminal counsel.” He concluded the thread with, “Et tu, @Mastercard?”
Michael Bowe, partner at Brown Rudnick and lead attorney representing Fleites in the lawsuit, said in a statement, “The court’s holding that our detailed complaint adequately pleads Visa was engaged in a criminal conspiracy to monetize child porn means Visa and other credit card companies are finally going to face the civil and perhaps criminal consequences of this unconscionable and illegal activity.”
The case, Serena Fleites v. MindGeek S.A.R.L. et al., is Docket No. 2:21-cv-04920-CJC-ADS in the U.S. District Court for the Central District of California.
Fleites is one of 34 individual plaintiffs who last year sued Pornhub and MindGeek, alleging exploitation and monetization of child pornography, rape videos, trafficked content, stolen content and other nonconsensual content. The litigation is the first application to date of Racketeer Influenced and Corrupt Organizations (RICO), child pornography and trafficking laws seeking to hold financial institutions accountable for illegal conduct monetized by and through the systems of companies whose payments they process.
We will always provide free access to the current law. In addition, we provide special support for non-profit, educational, and government users. Through social entrepreneurship, we’re lowering the cost of legal services and increasing citizen access.