Monday, November 30, 2020

Raycast (YC W20) Is Hiring macOS Software Engineers (EU, Remote)

Software Engineer, macOS

Redefine how developers experience productivity tools on Macs.

Raycast makes it simple, fast and delightful to control your tools. We're looking for a Software Engineer to join our small team to craft, shape and improve our macOS app that redefines productivity tools for developers.

Raycast is a native Mac application with a significant focus on top-notch UI and interactions. We genuinely care about user experience, performance, privacy and ease of use. We had enough of dealing with slow and clunky tools. Raycast is our attempt to let you enjoy your daily workflows without all the distractions.

Terminals are outdated: Rich media elements aren't supported, ASCII characters are too limiting and commands are hard to discover. Raycast combines the benefits of a command line with those of a modern user interface. It's simple, responsive and extensible. At the core of this, is a rock-solid desktop client that handles plugins and connects to a store full of productivity tools. We need engineers that are eager to build a tool that improves their own day-to-day life to realise this vision with us.

We're a fully remote company and looking for candidates in the UTC timezone (London, Lisbon, Dakar, ...) ± 3 hours. This position is for a full-time employment.

In this role you will...

  • Build user-facing features in the core product. You work on our macOS application which is our primary surface. You build new features that enrich the core user experience in the app and expand our library of custom and keyboard-driven UI components.
  • Co-own the product and work self-directed. We're a small team of individual contributors. Everybody has autonomy and responsibility. You work on features from ideation over release to maintenance and shape the roadmap of the product.
  • Solve interesting technical and UX problems. Raycast fits many tools into a single user interface. For this, you solve hard problems like optimistic updates, database indexing and standardized user interactions with like-minded team-mates.
  • Improve performance and care about quality. You dogfood your own features with our nighlty builds, respond to user feedback in our community and jump into the profiler to make every interaction feel snappy. You automate tedious tasks. At best, directly in Raycast.
  • Wear many hats. In such a small startup, we're all generalists. You might add a new page to our Vue.js website, set up an internal endpoint in our Node.js backend or write a blog post about a new feature.

You will be a great fit if...

  • You're highly productive while writing quality code. We release updates every 1-2 weeks. You can break down big projects in small deliverables. You care about the outcome and take a pragmatic approach to deliver it. You keep things simple and rather write a little bit more code than adopting a third-party dependency. You know when you want a code review or need to write tests.
  • You're product-minded. Raycast is designed to keep developers focused. You have a sense for great user experience and feel when something is off. You prefer polishing a feature instead of shipping soul-less pixels. You add analytics to reason about changes in the app and pro-actively iterate.
  • You put our users first. Throughout Raycast you find many little details. You pay attention to detail and provide our users a delightful experience. You can't stand bugs and don't shy away from user feedback.
  • You're an empathetic communicator. You treat technology as tradeoffs. You may be opinionated but you're not ideological. When disagreeing, you communicate thoughtfully your perspective and compromise when needed. You're thriving in a team environment and hold your colleagues accountable.

Our stack

  • Swift for our macOS app. Everything is written in Swift using AppKit with Swift Packages. We don't use SwiftUI because it's too premature for rapid development. We rarely adopt third party dependencies. Some are GRDB for our database, Sentry for crash reporting, Nuke for image fetching and a few C libraries to crank up performance.
  • JavaScript/TypeScript for everything on the web. We use Vue.js with Gridsome for our website and deploy it with Vercel. Our backend is a simple Node.js app written with Express and hosted on Heroku.
  • Modern work tools for everything. Linear for issue tracking, GitHub as source control, Sentry for error reporting, Segment and Amplitude for analytics, Slack for internal communication, Figma for pixels, Notion as knowledge base and a few others. As a team, we enjoy using CleanShot, Bear, Things and more Mac apps.

How to apply?

Now that you know about us, we would like to learn more about you. Send us an engaging message at jobs@raycast.com with information you think is relevant. Tell us why you want to join us, what excites you about the problem we're solving and how you envision your role at Raycast. Read about our application process.



from Hacker News https://ift.tt/3fRwQzE

Simple Math to Set Up a Sales Team

Let’s say you're a SaaS founder who’s looking to build a sales team for the first time. How do you structure quotas and compensation for the initial sales reps and their manager? Oftentimes, the biggest hurdle in hiring the first sales rep is not knowing how to incentivize them. This post provides simple rules that you can use to set up a sales team. Structured properly, the process is surprisingly mathematical.

Individual Plans

Let’s start with the sales plan for an individual sales rep, or “account executive” (AE). The key elements of an AE sales plan are:

  • Base Salary;

  • Quota;

  • Commission Rate;

  • Variable Pay (Quota times Commission Rate); and

  • OTE (“on-target earnings”), the sum of the AE’s Base Salary and Variable Pay. 

This sounds like a lot of variables, but two industry standards allow us to simplify greatly:

  1. The standard commission rate for SaaS products is 10%. For example, closing a $100,000 ARR deal generates a $10,000 commission.

  2. In constructing an OTE for an AE, a 50/50 split between base and variable compensation is typical. For an AE who’s hitting quota, their variable pay will equal their base salary. 

Taken together, these standards generate a third rule: the typical quota will equal 10x base salary. I call this the Rule of 10. This generates the following pay scale for AEs:

For simplicity, I’ve presented annual numbers, but quarterly sales plans are actually ideal. (I explain why in The Cadence.) Just divide the numbers by 4 to put your AEs on a quarterly plan.

Once an AE hits quota in a quarter, their commission rate should increase to 15% on incremental sales. This “accelerator” incentivizes the best AEs to keep pushing past their target, instead of slowing down at a perceived finish line.

For new AEs, the industry-standard time to hit productivity is 4 months, but this can be longer or shorter depending on how long it takes to learn the product and develop a pipeline. Most sales orgs set a ramping schedule, during which time new AEs have a lower quota and higher guaranteed comp (known as a “draw”) to make up for lack of commission. For example, a 50% draw would mean that the AE is granted half their variable comp for a quarter. If an AE is still not productive after 2 quarters, you may need to let them go or put them on a performance improvement plan.

It’s better for all AEs at the same level to be on the same plan. One-off promotions or raises can become a slippery slope. That said, if an AE consistently exceeds quota, the Rule of 10 provides a basis for giving them a raise. For example, if an AE reliably generates $600k in new ARR, then it’s fine to increase their base salary to $60k. However, their quota would now need to increase from $500k to $600k. Be careful to avoid raises that turn a successful AE who’s hitting quota into an underperforming AE who’s missing quota. 

Team Plans

Quota Capacity” (QC) refers to the total team quota. For example, if you have 10 AEs with quotas of $500k each, the team’s quota capacity will be $5 million. If you are forecasting $5 million in new ARR for next year, but have only 2 AEs ramped ($1M of quota capacity), you are very unlikely to hit your goal. Hence the value of the QC concept.

The average team attains about 70% of their QC, so some over-capacity is needed to hit your goal. In planning, make sure you also allow enough time for new AEs to ramp. 

A manager’s quota should be set at 80% of their team’s QC. The manager’s OTE (also 50/50 between base and variable) will be set by market rates for a manager, director, or VP in their position. Commission rate will be the dependent variable. For example:

  • Let’s assume a manager of 10 junior AEs ($5M QC) makes $200k OTE.

  • $200k OTE means $100k base and $100k variable.

  • 80% of quota would be $4M New ARR.

  • Their commission rate at $4M New ARR needs to generate $100k of variable. 

  • Therefore, their commission rate is $100k/$4M = 2.5%.

If the team is hitting over 80% of QC, that’s a sign you can hire more AEs. In fact, anything above 70% could warrant additional hiring as long as you have enough leads being generated at the top of funnel. Do not hire more AEs unless you can “keep them fed.” In my experience, AEs are rarely self-feeding and need marketing support to generate inbound leads. A great AE will generate at most 50% of their leads through their own outbound effort. 

Outbound motions are much harder than inbound so don’t assume production from any outbound channel until it is proven. Bottom-up SaaS products are a special case where all the leads are generated by the product, making growth much more predictable and efficient. In general, bottom-up SaaS vendors shouldn’t waste their time on outbound; they should focus on creating greater awareness.

Territories

The last part of an AE’s sales plan is their territory. A territory describes the boundaries within which the AE can sell. To avoid channel conflict and ensure adequate coverage, the world should be divided into territories that are “M.E.C.E.” (mutually exclusive and collectively exhaustive). There may be some parts of the world that you’re not selling into yet. That’s fine – inbound leads can go into an inbox until those territories are ready to be assigned. 

There are a few different ways to divide up the world:

  1. Geos. Most commonly, leads are assigned to geographic territories based on their location (typically company headquarters). APIs connected to Salesforce (such as Clearbit or Zoom Info) can do this assignment automatically. Territories should be roughly the same size, not in geographic area, but in terms of the number of leads they generate. If the world is divided up fairly, AEs should be largely indifferent to the territory they get. 

    As more AEs get hired, existing territories need to be broken up to create new patches for the new hires to work. As a SaaS vendor gets bigger and bigger, the territories get smaller and smaller. Offsetting this challenge for the AE is the fact that sales get easier as the vendor becomes more established and successful. Continued growth means that the vendor is able to extract more from less.

  2. Verticals. Instead of using geos, some SaaS vendors will create territories based on industry verticals. This makes sense when the product is highly specific to a particular type of customer. Specializing by vertical allows the AE to master those use cases.

  3. Round-Robin. Finally, early-stage startups will often divvy up the leads round-robin. This makes sense for small teams that don’t want the overhead of maintaining geos or verticals. This is a good place to start.

New territories, quotas, and plans should be rolled out quarterly at Sales Kickoff. This is also a time to retrain the sales team on product changes and best practices. You should figure out what your top AEs are doing right and seek to replicate those behaviors across the team.

Expansion and Renewals

A perennial question is who should get credit for expansion and renewals.

I like to let an AE keep the account for 12 months from the initial close to incentivize “land and expand” deals. Any expansion in the account during that time is considered New ARR for which the AE receives full quota credit. 

After 12 months, the account becomes a renewal, which is subject to a separate quota and commission rate and may be worked by a different rep. Commission rates on Existing ARR should be much lower than for New ARR.

There are several different options for assigning renewals, and each has its advantages and disadvantages:

  1. Original owner. Renewals can stay with their original owner – the sales rep who is most familiar with the account. The advantage is obvious, but the problem is that tenured reps will build a “book of business” and spend less and less time closing new business. This turns the best “hunters” into “farmers.” Eventually there will be a need for some kind of reassignment. 

  2. Territories. Renewals can be assigned based on geographic territories just like new business. If an AE expands the deal, the expansion is considered New ARR and applies to their new business quota. This approach is the most democratic, because it assigns juicy opportunities for expansion to the entire team, but it can also result in important renewals being assigned to newer and less experienced reps who could mishandle them. 

  3. Specialists. Renewals can be handled by specialists, typically elite sellers selected for the significant potential of these opportunities to impact ARR. This is what we did at my company Yammer, and it proved effective at maximizing these opportunities. 

  4. CSMs. Finally, in some companies, CSMs handle the renewals. I am not a fan of this approach. While it can be fine for mature vendors, in the early years of a startup, AEs are much more likely to maximize the value of the renewal or to save a troubled deal. Thus, I prefer to see CSMs partnered with AEs and bonused based on customer retention, rather than owning the renewal themselves. 

Sales Productivity

Finally, the Rule of 10 yields some insights about sales productivity. 

If an AE is unable to generate at least $400k/year in New ARR (implying $80k-100k OTEs), a sales-driven distribution strategy may not pencil. Moreover, if your product requires very high-paid, experienced sellers, then it also needs to generate the contract sizes that justify their quotas. If your product generates only small contract values, that’s fine as long as sales velocity is high and junior reps can be hired to sell it.

Since OTE is 20% of sales, the fully-loaded cost of an AE will represent about a 25% cost of sales. Sales overhead (management, operations, sales development) will typically cost another 15-25% of sales. That means you can spend about 50% of New ARR on Marketing CAC (lead gen) and still keep your payback under 12 months. Of course, these are just rules of thumb – be sure to check the actual numbers. 

I see a lot of companies implementing sales ad hoc, and they usually come around to some version of these principles, after experiencing some confusion and pain. If you stick to these simple math-based rules, it will be much easier to build out your sales function.


Glossary

  • Accelerator – the increase in Commission Rate that an AE receives once they’ve hit quota in a given period.

  • AE (Account Executive) – a sales rep.

  • Base Salary – the AE’s guaranteed compensation.

  • Commission Rate – the percent of New ARR paid to the AE as an incentive, typically 10%.

  • Geos – geographic territories, a process for assigning leads based on location.

  • M.E.C.E. (“mutually exclusive and complete exhaustive”) – a principle for dividing up the entire world into non-competing territories. 

  • OTE (“on-target earnings”) – the sum of the AE’s Base Salary and Variable Pay. 

  • Quota – the New ARR that an AE is expected to close within a given period.

  • Quota Attainment – the percent of Quota actually hit by an AE (as opposed to the amount specified in plan).

  • Quota Capacity – the sum of the individual quotas on a team; the amount of new sales generated if everyone on the team hits quota.

  • Ramp – the period where a new AE becomes productive. Ramp times vary based on your product and industry. Sales comp should factor in a ramp schedule so new reps have time to hit full productivity without their OTE being penalized. 

  • Round Robin – lead assignment process where all sales reps are assigned leads in a rotational order.

  • Rule of 10 – Typical AE quota should equal 10x base salary. Most other compensation math hinges on this rule. 

  • Territory – the boundaries describing where an AE can sell. 

  • Variable Pay – the incentive compensation that the AE receives for hitting quota (i.e. Quota times Commission Rate).



from Hacker News https://ift.tt/37lSo3I

QEMU Advent Calendar 2020

About

The QEMU Advent Calendar 2020 features a QEMU disk image each day of December until the 24th. Each day a new package becomes available for download.

Every download contains a little 'run' shell script that starts the QEMU emulator with the recommended parameters for the disk image. Disk images are either contained directly in the download or are downloaded by the 'run' script (you need to have installed 'curl' or 'wget' in that case).

The disk images contain interesting operating systems and software that run under the QEMU emulator. Some of them are well-known or not-so-well-known operating systems, old and new, others are custom demos and neat algorithms.

The 'run' scripts (and disk images if included in the download) were created by volunteers from the QEMU community to showcase cool software that QEMU can run.


Back to top

QEMU logo by BenoƮt Canet under Creative Commons Attribution 3.0 Unported, Santa hat added by Stefan Hajnoczi

Website © 2020 QEMU contributors · MIT License. Design based on the 2014 website created by Stefan Hajnoczi, and modified for 2020 by Thomas Huth.

Source code for software that is licensed under the GPL is available on request. Please write an e-mail to Eldon Stegall (see contact information above).

This website uses cookies to store the state of the doors. We do not use cookies to track you or to collect any personally identifiable information.



from Hacker News https://ift.tt/39B25hv

DevRelOMeter

Comments

from Hacker News https://ift.tt/39xVmoj

The biggest hacks, data breaches of 2020

Cybersecurity may be far from many of our minds this year, and in light of a pandemic and catastrophic economic disruption, remembering to maintain our own personal privacy and security online isn't necessarily a priority. 

However, cyberattackers certainly haven't given anyone a break this year. Data breaches, network infiltrations, bulk data theft and sale, identity theft, and ransomware outbreaks have all occurred over 2020 and the underground market shows no signs of stopping.

As a large swathe of the global population shifted to work from home models and businesses rapidly transitioned to remote operations, threat actors also pivoted. Research suggests that remote workers have become the source of up to 20% of cybersecurity incidents, ransomware is on the rise, and we are yet to learn that "123456" is not an adequate password

Many companies and organizations, too, have yet to practice reasonable security hygiene, and vulnerabilities pose a constant threat to corporate networks. As a result, we've seen a variety of cyberattacks this year, the worst of which we have documented below.

January:

  • Travelex: Travelex services were pulled offline following a malware infection. The company itself and businesses using the platform to provide currency exchange services were all affected.
  • IRS tax refunds: A US resident was jailed for using information leaked through data breaches to file fraudulent tax returns worth $12 million. 
  • Manor Independent School District: The Texas school district lost $2.3 million during a phishing scam.
  • Wawa: 30 million records containing customers' details were made available for sale online. 
  • Microsoft: The Redmond giant disclosed that five servers used to store anonymized user analytics were exposed and open on the Internet without adequate protection.
  • Medical marijuana: A database backing point-of-sale systems used in medical and recreational marijuana dispensaries was compromised, impacting an estimated 30,000 US users.

February:

  • EstĆ©e Lauder: 440 million internal records were reportedly exposed due to middleware security failures. 
  • Denmark's government tax portal: The taxpayer identification numbers of 1.26 million Danish citizens were accidentally exposed.
  • DOD DISA: The Defense Information Systems Agency (DISA), which handles IT for the White House, admitted to a data breach potentially compromising employee records.
  • UK Financial Conduct Authority (FCA): The FCA released sensitive information belonging to roughly 1,600 consumers by accident as part of an FOIA request.
  • Clearview: Clearview AI's entire client list was stolen due to a software vulnerability.
  • General Electric: GE warned workers that an unauthorized individual was able to access information belonging to them due to security failures with supplier Canon Business Process Service.

March:

  • T-Mobile: A hacker gained access to employee email accounts, compromising data belonging to customers and employees. 
  • Marriott: The hotel chain suffered a cyberattack in which email accounts were infiltrated. 5.2 million hotel guests were impacted. 
  • Whisper: The anonymous secret-sharing app exposed millions of users' private profiles and datasets online.
  • UK Home Office: GDPR was breached 100 times in the handling of the Home Office's EU Settlement Scheme.
  • SIM-swap hacking rings: Europol made arrests across Europe, taking out SIM-swap hackers responsible for the theft of over €3 million.
  • Virgin Media: The company exposed the data of 900,000 users through an open marketing database.
  • Whisper: Millions of users' private profiles and datasets were left, exposed and online, for the world to see.
  • MCA Wizard: 425GB in sensitive documents belonging to financial companies was publicly accessible through a database linked to the MCA Wizard app.
  • NutriBullet: NutriBullet became a victim of a Magecart attack, with payment card skimming code infecting the firm's e-commerce store.
  • Marriott: Marriott disclosed a new data breach impacting 5.2 million hotel guests.

April:

  • US Small Business Administration (SBA): Up to 8,000 applicants for emergency loans were embroiled in a PII data leak.
  • Nintendo: 160,000 users were affected by a mass account hijacking campaign.
  • Email.it: The Italian email provider failed to protect the data of 600,000 users, leading to its sale on the Dark Web.
  • Nintendo: Nintendo said 160,000 users were impacted by a mass account hijacking account caused by the NNID legacy login system.
  • US Small Business Administration (SBA): The SBA revealed as many as 8,000 business emergency loan applicants were involved in a data breach.

May:

  • EasyJet: The budget airline revealed a data breach exposing data belonging to nine million customers, including some financial records.
  • Blackbaud: The cloud service provider was hit by ransomware operators who hijacked customer systems. The company later paid a ransom to stop client data from being leaked online.
  • Mitsubishi: A data breach suffered by the company potentially also resulted in confidential missile design data being stolen.
  • Toll Group: The logistics giant was hit by a second ransomware attack in three months. 
  • Pakistani mobile users: Data belonging to 44 million Pakistani mobile users was leaked online.
  • Illinois: The Illinois Department of Employment Security (IDES) leaked records concerning citizens applying for unemployment benefits.
  • Wishbone: 40 million user records were published online by the ShinyHunters hacking group.
  • EasyJet: An £18 billion class-action lawsuit was launched to compensate customers impacted by a data breach in the same month.

June:

  • Amtrak: Customer PII was leaked and some Amtrak Guest Rewards accounts were accessed by hackers.
  • University of California SF: The university paid a $1.14 million ransom to hackers in order to save COVID-19 research.
  • AWS: AWS mitigated a massive 2.3 Tbps DDoS attack. 
  • Postbank: A rogue employee at the South African bank obtained a master key and stole $3.2 million.
  • NASA: The DopplePaymer ransomware gang claimed to have breached a NASA IT contractor's networks. 
  • Claire's: The accessories company fell prey to a card-skimming Magecart infection.

July:

  • CouchSurfing: 17 million records belonging to CouchSurfing were found on an underground forum.
  • University of York: The UK university disclosed a data breach caused by Blackbaud. Staff and student records were stolen.
  • MyCastingFile: A US casting platform for actors exposed the PII of 260,000 users.
  • SigRed: Microsoft patched a 17-year-old exploit that could be used to hijack Microsoft Windows Servers.
  • MGM Resorts: A hacker put the records of 142 million MGM guests online for sale.
  • V Shred: The PII of 99,000 customers and trainers was exposed online and V Shred only partially resolved the problem.
  • BlueLeaks: Law enforcement closed down a portal used to host 269 GB in stolen files belonging to US police departments.
  • EDP: The energy provider confirmed a Ragnar Locker ransomware incident. Over 10TB in business records were apparently stolen.
  • MongoDB: A hacker attempted to ransom 23,000 MongoDB databases.

CNET: Russian and North Korean hackers are targeting COVID-19 vaccine researchers | The best outdoor home security cameras for 2020 | Android and iPhones are all about privacy now, but startup OSOM thinks it can do better

August:

  • Cisco: A former engineer pleaded guilty to causing massive amounts of damage to Cisco networks, costing the company $2.4 million to fix.
  • Canon: The photography giant was struck by ransomware gang Maze.
  • LG, Xerox: Maze struck again, publishing data belonging to these companies after failing to secure blackmail payments.
  • Intel: 20GB of sensitive, corporate data belonging to Intel was published online.
  • The Ritz, London: Fraudsters posed as staff in a clever phishing scam against Ritz clients.
  • Freepik: The free photos platform disclosed a data breach impacting 8.3 million users. 
  • University of Utah: The university gave in to cybercriminals and paid a $457,000 ransom to stop the group from publishing student information.
  • Experian, South Africa: Experian's South African branch disclosed a data breach impacting 24 million customers. 
  • Carnival: The cruise operator disclosed a ransomware attack and subsequent data breach.

See also: Black Hat: When penetration testing earns you a felony arrest record

September:

  • Nevada: A Nevada school, suffering a ransomware attack, refused to pay the cybercriminals -- and so student data was published online in retaliation. 
  • German hospital ransomware: A hospital patient passed away after being redirected away from a hospital suffering an active ransomware infection.
  • Belarus law enforcement: The private information of 1,000 high-ranking police officers was leaked. 
  • NS8: The CEO of the cyberfraud startup was accused of defrauding investors out of $123 million.
  • Satellites: Iranian hackers were charged for compromising US satellites. 
  • Cerberus: The developers of the Cerberus banking Trojan released the malware's source code after failing to sell it privately. 
  • BancoEstado: The Chilean bank was forced to close down branches due to ransomware.

October: 

  • Barnes & Noble: The bookseller experienced a cyberattack, believed to be the handiwork of the ransomware group Egregor. Stolen records were leaked online as proof. 
  • UN IMO: The United Nations International Maritime Organization (UN IMO) disclosed a security breach affecting public systems.
  • Boom! Mobile: The telecom service provider became the victim of a Magecart card-skimming attack.
  • Google: Google said it mitigated a 2.54 Tbps DDoS attack, one of the largest ever recorded.
  • Dickey's: The US barbeque restaurant chain suffered a point-of-sale attack between July 2019 and August 2020. Three million customers had their card details later posted online.  
  • Ubisoft, Crytek: Sensitive information belonging to the gaming giants was released online by the Egregor ransomware gang.
  • Amazon insider trading: A former Amazon finance manager and their family were charged for running a $1.4 million insider trading scam.

November: 

  • Manchester United: Manchester United football club said it was investigating a security incident impacting internal systems.
  • Vertafore: 27.7 million Texas drivers' PII was compromised due to "human error."
  • Campari: Campari was knocked offline following a ransomware attack.
  • $100 million botnet: A Russian hacker was jailed for operating a botnet responsible for draining $100 million from victim bank accounts. 
  • Mashable: A hacker published a copy of a Mashable database online.
  • Capcom: Capcom became a victim of the Ragnar Locker ransomware, disrupting internal systems.
  • Home Depot: The US retailer agreed to a $17.5 million settlement after a PoS malware infection impacted millions of shoppers.

TechRepublic: How remote working poses security risks for your organization | How phishing attacks are exploiting Google's own tools and services | Linux and open source: The biggest issue in 2020

December:

  • As new cybersecurity incidents occur, we will update for the month of December.

Previous and related coverage


Have a tip? Get in touch securely via WhatsApp | Signal at +447713 025 499, or over at Keybase: charlie0




from Latest Topic for ZDNet in... https://ift.tt/3octdqZ

Advent of Code 2020 has begun

Comments

from Hacker News https://ift.tt/2Jrvlfs

Micro Magic RISC-V Core Claims to Beat Apple M1 and Arm Cortex-A9

Comments

from Hacker News https://ift.tt/36rgpHo

Csidh Reference Implementation

To build with default parameters, simply run make and then sudo make install to install the corresponding tools to the system. The default parameter BITS is set to 512.

To build a shared library and the corresponding csidh-p*-util program, simply run:

make shared-library

To install:

sudo make install

Generate one CSIDH private and public key per user:

$ csidh-p512-util -g -s demo-512-0.secret  -p demo-512-0.public
$ csidh-p512-util -g -s demo-512-1.secret  -p demo-512-1.public

Perform a key agreement from the perspective of each respective user:

$ csidh-p512-util -d -s demo-512-0.secret  -p demo-512-1.public
c7cee77cb809e99a3d38c661f53fe1a1be0f0b6071f490e6a260f7c0b28774c548e09158d5193f118a418a800683ec51be6814d156fdb34b2b3310d3ae8f8b4a
$ csidh-p512-util -d -s demo-512-1.secret  -p demo-512-0.public
c7cee77cb809e99a3d38c661f53fe1a1be0f0b6071f490e6a260f7c0b28774c548e09158d5193f118a418a800683ec51be6814d156fdb34b2b3310d3ae8f8b4a

It is also possible to use stdin and stdout:

$ cat demo-512-0.secret | csidh-p512-util -d -p demo-512-1.public
c7cee77cb809e99a3d38c661f53fe1a1be0f0b6071f490e6a260f7c0b28774c548e09158d5193f118a418a800683ec51be6814d156fdb34b2b3310d3ae8f8b4a
$ cat demo-512-0.secret demo-512-1.public | csidh-p512-util -d
c7cee77cb809e99a3d38c661f53fe1a1be0f0b6071f490e6a260f7c0b28774c548e09158d5193f118a418a800683ec51be6814d156fdb34b2b3310d3ae8f8b4a

To build a static library and a statically linked csidh-p*-util program, simply run:

make static-library

The build may be configured with other arguments:

  • BITS is one of the supported parameter sets.
  • UINT_IMPL is an unsigned integer implementation.
  • FP_IMPL is a finite field implementation.

For example:

make BITS=1024

builds the code with a 1024-bit parameter set, and

make UINT_IMPL=ui.c FP_IMPL=fp.c

uses generic C arithmetic instead of assembly.

Other parameter sets may be used by creating a directory modelled after the p512/ and p1024/ folders, optionally including specialized ui.s and fp.s files for assembly arithmetic. Else the generic C implementation will be used.

To build the benchmark code, run "make bench"; it also supports the arguments listed above. To configure benchmarking, use the following options:

  • BENCH_ITS is the number of iterations.
  • BENCH_VAL benchmarks public-key validation.
  • BENCH_ACT benchmarks the group action.

For example,

make bench BENCH_ITS=1000 BENCH_ACT=1 BENCH_VAL=0

builds a benchmark for 1000 iterations of the group action without public-key validation.

Benchmarks

A simple benchmark and CSIDH shared secret agreement demonstration on use with a Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz:

$ csidh-p512-bench
doing 1000 iterations of validation and action.
iterations: 1000
clock cycles: 125700756 (125.7*10^6)
wall-clock time: 48.493 ms
stack memory usage: 4368 b

Shared secret derivation with timing information:

$ csidh-p512-demo

Alice's private key   (  0.500 ms):
  0c0c010d4dfeecd0025ecf1f2b11bed02be3422f1feddbec5dbb301e1315f02f4c0124e55f

Bob's private key     (  0.404 ms):
  ecd0b4f23b3e33d2e1345c45ee4502def2bf122114ef41b053eee04425cdb05d1d43e2e5c1

Alice's public key    ( 70.244 ms):
  3a6e03ff0cfe3fcfac6f4b4ea69fa62366a8a20051061814926b535c38fc4a015dc6f0e6bc3dc449e5a4f9117e7696470f96488e9fab164d6ce0c39d5ea6fdac

Bob's public key      ( 37.351 ms):
  54ae89e13e63dc86065f96a210598dbe81f790e4c519735bea6e703804677931462f7c50e2411a9a515f26a519159a935e4f3db96feef0543396171050fc7322

Alice's shared secret ( 36.721 ms):
  2dfee422f92e7ee93a2ecd102300d4a4b67c66fd451244fb444f5e9675c6f4dc6e1a55d81818194019b63efc9dcb67b518b344ade7476e2acf870186cd26421b

Bob's shared secret   ( 42.306 ms):
  2dfee422f92e7ee93a2ecd102300d4a4b67c66fd451244fb444f5e9675c6f4dc6e1a55d81818194019b63efc9dcb67b518b344ade7476e2acf870186cd26421b

    equal.

The same benchmark and key agreement on a Raspberry Pi 4 Model B Rev 1.1:

$ csidh-p512-bench
doing 1000 iterations of validation and action.
iterations: 1000
clock cycles: 29717321 (29.7*10^6)
wall-clock time: 549.508 ms
stack memory usage: 4768 b

Shared secret derivation with timing information:

$ csidh-p512-demo

Alice's private key   (  0.371 ms):
d5311d13ddb14c3ebf52bc4020fd5fc34cfc1fef223512df3d25343d55ed445d4fb0c115f2

Bob's private key     (  0.415 ms):
dd0eed2dfd0401eeb4454b41bf2ce3bccf1423c0232cec02bb3ff2e20bee15bdded511fb2e

Alice's public key    (571.734 ms):
50721ee9067ece5fbae5d6864017352d0b3fb892daa828cbb618baeb4288475b74a60bd5341c2893025549fb6a0288330d8eb056a69be5a6d0822245558b1b82

Bob's public key      (529.899 ms):
1b3d030d38f70ee2aa5083d3681aa4a13cd0f6469f7ac37033c4dc9c66c1fa164aff569fafda029bb48d7d2a279895ef151372f3f7c9cc236d9a6f0fd0223570

Alice's shared secret (578.907 ms):
5a075d7f4f5e37fa95bd6103bc71bdd2e98173078e26dd2aeaf3f9581ff847056e6e781cea986991e43ffeb4e68fc2763a9059088296ded0a0df415ffd4913fa

Bob's shared secret   (534.403 ms):
5a075d7f4f5e37fa95bd6103bc71bdd2e98173078e26dd2aeaf3f9581ff847056e6e781cea986991e43ffeb4e68fc2763a9059088296ded0a0df415ffd4913fa

  equal.

The same benchmark and key agreement on a Raspberry Pi 3 Model B Rev 1.2:

$ csidh-p512-bench
doing 1000 iterations of validation and action.
iterations: 1000
clock cycles: 22086467 (22.1*10^6)
wall-clock time: 1148.395 ms
stack memory usage: 4768 b

Shared secret derivation with timing information:

$ csidh-p512-demo

Alice's private key   (  0.301 ms):
  4d004fb0fe1be55c050c03b03f31d30f0d2b245bd2e21103fe40b4f520345cc5b554e5ecde

Bob's private key     (  0.240 ms):
  0d3f0ccb504c2debe14c51b21db4c0f22dccf5e2ee5cbbe1ed20f322c414f2d23440d00223

Alice's public key    (1175.329 ms):
  4f6c8630d59f524651d0caa010605528df9e8588a6560b94bde3ab276745ebb164ec65d9c6f7ac04d8787d7084118ce48d36eba3f4d8afcf25bdee60a5dbcf49

Bob's public key      (1063.863 ms):
  621279da5aab0140668e4db8d0ad021e92fdb7119a7950a69cd98337bd31366d5385238f75f58fc0a28efc71e5b0a0f798a474c91048843253e05acf33cfdc8e

Alice's shared secret (1158.707 ms):
  1eb8091dbc7e3949a22f159676506a5de136ca3912624b0a1fb2d24dce5a5b65bb3724d66834c166125f0192aa5260b338ae156e28251d9309a67d297c722684

Bob's shared secret   (1077.129 ms):
  1eb8091dbc7e3949a22f159676506a5de136ca3912624b0a1fb2d24dce5a5b65bb3724d66834c166125f0192aa5260b338ae156e28251d9309a67d297c722684

    equal.


from Hacker News https://ift.tt/3o7kBC4

I Hired a Freelance Editor for My Blog

A year in blogging šŸ”—︎

I started this blog in May of last year. I don’t mean to brag, but by last April, after less than a year of blogging, I was pulling in upwards of 20 visitors per day, several of whom were not spam bots. That number reached as high as 50 visitors on days when I made a new post and begged for readers through every social media channel at my disposal.

The size of my audience changed a bit in May 2017. The sharp-eyed reader may be able to spot the subtle shift in my traffic graph:

Blog traffic graphs

Number of unique readers visiting mtlynch.io per week

As you notice from the chart above, my numbers went from “too insignificant to appear in a graph” for most of the first year of the blog’s existence to over 9,000 readers per week starting last May. From that point on, when I published a new post, the blog received up to 40,000 visitors per week.

If I compare by month, the blog grew by more than 20x month over month, and by more than 450x over a two month period:

Month Unique visitors Growth (one month) Growth (two months)
April 2017 251
May 2017 5,572 2,119% / 22x
June 2017 113,121 1,930% / 20x 44,968% / 450x

What changed? šŸ”—︎

Just before I went to bed on May 13, 2017, I posted a job for a blog editor on Upwork, a site I use regularly for hiring freelancers. Not a permanent position, just a one-time gig to read through my already posted articles, identify weak patterns in my writing, and suggest improvements.

When I woke up the next morning, I had already received applications from 10 different freelancers. Several of the matches seemed strong. The applicant pool included PhDs, journalists, and writing teachers.

Some of the matches were… not so strong. The lowest bidder, at $8/hr, told me she was, “well-equipped with knowledge how to teach writing.” Another freelancer mentioned in her cover letter that, “editing and writing are [her] forte’s [sic].”

The strongest applicant was an editor named Samantha Mason. At $55/hr, she was almost twice as expensive as the next highest bidder. But her response was the only one that showed that she had actually read any of my writing. In her cover letter, she pointed out that I had a habit of diluting my sentences with helper verbs. “As you can see in the graph below,” instead of simply, “As you see in the graph below.”

Samantha was also a great match because she had a background in engineering. She told me that she was a programmer as well, so I thought this would give her additional context into my more software-heavy posts.

I offered Samantha the job that same morning. By 10 AM, she accepted. The entire process from writing the job posting to completing the hire took less than 12 hours.

Note: Because this entire post is about editing and Samantha’s work specifically, it felt appropriate for her to edit this piece , so if all of my negative comments about her have been removed, that’s why.

Why hire an editor? šŸ”—︎

When I publish a blog post and it flops, I don’t get much feedback about why it was unsuccessful. To date, none of my readers have written to say, “Hey, you had great ideas in that post, but I never read them because your repetitive sentence structure lost my attention, and I closed the tab.” An editor actually can give me that kind of feedback.

I felt that my writing was pretty good but definitely had room for improvement. Changes to my style could have tangible benefits like a larger audience and readers gaining a better understanding of the content.

Beyond my blog, investing in my writing would pay dividends in many aspects of my life. Writing is a highly transferable skill, much like public speaking, time management, or knife juggling. Techniques I learned to better my blog writing would likely carry over into design documents I write at work or even emails I send to friends.

Colbert with eyebrow raised

“Your editor, you say?”

My other reason for hiring an editor was the most important: to give my friends, family, and co-workers a misleadingly grandiose perception of my blog. Before, when I told someone that I wrote a new blog post about using Selenium to test Ansible roles, they nodded politely and changed the subject. But when I started saying things like, “I’m rewriting my new post because my editor thinks that the introduction is too weak,” people became intrigued. “Your editor?”

Learning from my mistakes šŸ”—︎

A few days after I hired her, Samantha sent me detailed feedback on two of my blog posts. As soon as I read her notes, I knew that hiring an editor was a good decision. She identified several mistakes that I had no idea I was making. It started with small things, like incorrect use of commas or overuse of parentheticals (I used to have a habit of inserting them into sentences to cram in information that wasn’t worth mentioning).

One surprising insight was how frequently I made the assumption that my readers shared my frame of reference. Samantha noted that in one post, I mentioned Seamless without explaining what it was. Seamless is a food delivery service: extremely popular in Manhattan, not so popular in Wisconsin, where Samantha lives. When I mentioned it in the post, it never occurred to me that someone might not know the company. Samantha’s note made me realize how frequently I was alienating my audience when I could easily avoid this by being conscious of potentially unfamiliar references.

What are you trying to accomplish with this story? šŸ”—︎

The most valuable advice Samantha gave me was in response to a post where I described using an odd job service to hire a private chef:

What are you trying to accomplish with this story? If it’s just the facts, you’ve done that, but I think you can do more. This is creative writing. Have fun with it. Make the reader laugh. Make the reader want to keep reading.

My first thought was, “What story? This is just an explanation of what I did.” I wrote the post because I had told a few friends about how I hired my chef. People seemed amused by the process, and they were interested in the logistics of it: “How much did it cost? How did you pick the chef?” It seemed natural to me that a blog post would simply answer those questions.

After letting the suggestion roll around in my head for a few days, I thought, “Why can’t it be a story? People love stories.” I had a backlog of ideas for article topics, so I thought about whether I could write any of them in the form of a story.

Telling it like a story šŸ”—︎

A few weeks later, I was browsing reddit when I saw a user post a corrupted version of their password. The password was protecting a large sum of money they held in cryptocurrency, so they posted it in frustration because a corrupt password effectively meant they lost their money. I figured out a clever way of reconstructing the correct password and used it to steal their money. Don’t worry; I gave it back.

The technique I used was unusual, so over the following week, I thought about how to write it up for the blog. Then it hit me: this could be one of those “stories” that Samantha had been telling me about.

I wrote the article on a Thursday night in a non-stop, four-hour writing session. I never had so much fun writing a blog post. It was silly and self-effacing and included tongue-in-cheek pop culture references — all qualities that my previous blog posts lacked. I knew that presenting it in story format might make it more compelling to read, but I hadn’t anticipated how much easier it would make the article to write.

The next morning, I published the article, “How I Stole Your Siacoin," and posted the link to reddit and Hacker News, two popular link-sharing sites. By the end of the day, it was the most upvoted story of all-time on two of reddit’s subforums, /r/siacoin and /r/cryptocurrency (though a few days later, I was shamefully bumped from the #1 spot on /r/CryptoCurrency by a picture of a sign). It had also gained enough traction on Hacker News to make it to their front page, an enviable accomplishment for tech bloggers.

Visitor stats for How I Stole Your Siacoin post

Blog visitor statistics on the day that “How I Stole Your Siacoin” was published.

I also noticed something interesting about the comments on the article. Many of them specifically praised the style of my writing:

Reddit comments

Positive comments on the writing style of “How I Stole Your Siacoin.” Creating this collage does not count as narcissism because I’m doing it for a blog post.

I browse reddit frequently, and I don’t recall seeing many users compliment submissions on their writing — though it should be noted that I don’t recognize praise unless it’s directed at me. The feedback seemed like such clear validation of my decision to hire an editor that I almost had to wonder if Samantha was surreptitiously posting these comments herself. [Editor’s note: There is no evidence to support this allegation.]

Before and after šŸ”—︎

You’re probably thinking, “That’s just one article. He probably got lucky.” Or at least that’s what I thought after the article got much more attention than I was expecting. But the next article I published received a similarly positive response. I certainly don’t think every article I write will be a hit, but it’s clear to me that receiving expert feedback has made such popularity possible, when it previously was not.

If you compare the visitor statistics for the three articles I published before working with Samantha to the three I wrote after, the difference is unmistakable:

Before working with an editor šŸ”—︎

After working with an editor šŸ”—︎

Cost šŸ”—︎

As I mentioned earlier, I didn’t set out to hire someone to edit my articles on an ongoing basis. At ~$110 per article, the cost was too high to have every new blog post I write professionally edited. Instead, I wanted a one-time review of my writing so that I could apply the techniques myself.

I feel that this plan worked. Of the three articles I wrote after I started working with Samantha, she only edited the GreenPiThumb post. The others, I edited myself based on what I had learned from her previous feedback.

With Samantha’s permission, I’ve included below the exact cost of the work she did with me and the notes that she provided for each article:

The twist is that Samantha tricked me. I didn’t anticipate how effective her feedback would be, so now I do want to work with her on a recurring basis. I can’t afford to hire her for every article, but I do plan to check in with her regularly. My strategy remains the same. I don’t want to offload editing to Samantha, but rather learn from her feedback on each article so that I can focus on different areas to improve.

Suggestions for working with editors šŸ”—︎

If you’re a blogger and are considering hiring an editor, here are some recommendations based on my experience:

Pay for quality šŸ”—︎

If you post to a freelancing site like Upwork, you will invariably receive cheap offers from people willing to take any job they can get, regardless of their ability to deliver results. Do not be tempted to save money by hiring a cut-rate editor.

If you go to the trouble of hiring someone to critique your writing, hire an expert who can give you excellent guidance. If you needed surgery, would you hire the cheapest person to approach you with a scalpel? An investment in expert feedback on your writing will pay dividends for a long time, so invest well.

Screen carefully šŸ”—︎

Freelancer sites show you ratings and reviews of potential freelancers from their past clients. Read through these reviews to see if the editor has the qualities that are important to you. Prefer applicants who have completed at least 10 previous jobs with a success rate of 90% or higher. Samantha, for example, has a success rate of 99% and 39 completed jobs, which are excellent indicators of her quality. [Editor’s note: This is an astute observation.]

Require applicants to submit a cover letter, and scrutinize it carefully. For an editor, it’s essentially a sample of their work. Did they send you a form letter that they blast out to everyone? Or did they customize it to address the areas where you need help? The grammar in their cover letter should be impeccable, and the wording should be clear and easy to understand.

Look for subject matter familiarity šŸ”—︎

Find an editor who can understand and appreciate your writing. They don’t have to have the same level of expertise that you do, but they should have familiarity with the subject on par with your potential audience — someone who might read your blog even if you weren’t paying them.

If you have a blog about pop music, you don’t need to hire a professional music critic, but you should look for someone with enough appreciation for music to understand your terminology and references. Similarly, if you write a mommy blog, don’t require potential editors to hold a graduate degree in child development, but they should at least be familiar with the concept of children.

Catch the easy stuff yourself šŸ”—︎

You’re paying a premium for an expert’s time, so there’s no sense in squandering that time on simple mistakes you could identify yourself. Before sending your writing to an editor, run it through a tool like Grammarly or Microsoft Word to catch spelling and grammatical errors.

Part of your proofreading process should also be reading your posts aloud. My editor encouraged me to do this, and I was amazed at how effectively it catches careless errors and unnatural wording.

Don’t take it personally šŸ”—︎

Your editor is critiquing your writing, not you. If your writing is very personal, the two can feel one and the same, but you’ll get the most out of your editor’s notes if you can separate yourself from your writing and approach their feedback without defensiveness or ego.

You don’t have to accept every note šŸ”—︎

Notwithstanding the previous suggestion, remember that it’s ultimately your writing, and you have to decide what feedback to accept and decline.

[Editor’s note: Pay no attention to this suggestion.]

There have been several instances where my editor suggested a change that I recognize is clearer or more eloquent, but it doesn’t sound like my voice. In those cases, I try to rewrite the passage to move closer to the suggestion. But occasionally, I’ll wrestle with the note and reach the conclusion that what I wrote is what I want.

Make a checklist šŸ”—︎

Every time I complete the first draft of a new blog post, I check my editor’s notes on the last article she reviewed. For mistakes I find myself repeating, I keep a separate checklist that I run through at the end of my writing process. My checklist has reminders like, “make sure you’re not overusing the word ‘really’,” but you’ll find patterns to add to your own list. Really.

Special thanks to my editor, Samantha Mason, for volunteering her time to edit this piece.



from Hacker News https://ift.tt/2uwLGUM

ARM and Lock-Free Programming

I was inspired by the release of Apple’s M1 ARM processor to tweet about the perils of lock-free programming which led to some robust discussion. The discussion went pretty well given the inanity of trying to discuss something as complicated as CPU memory models in the constraints of tweets, but it still left me wanting to expand slightly on the topic in blog form.

This is intended to be a casual introduction to the perils of lock-free programming (which I last wrote about some fifteen years ago), but also some explanation of why ARM’s weak memory model breaks some code, and why that code was probably broken already. I also want to explain why C++11 made the lock-free situation strictly better (objections to the contrary notwithstanding).

Mandatory disclaimer: if your code always uses locks An actual lock (with link to an eleven second lock-picking video)when sharing data between threads then you don’t need to worry about any of this – properly implemented locks avoid all of these data races (unlike the horrible lock in this video).

The basic problems of lock-free programming are best explained through the example of a lock-free producer/consumer pattern, with the producer thread looking like this (C++ pseudo-code with data/function boundaries omitted):

// Producer thread:
Data_t g_data1, g_data2, g_data3;
bool g_flag

g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
g_flag = true; // Indicate that the data is ready for consumption

And here is the consumer thread which retrieves and uses the data:

// Consumer thread:
if (g_flag) {
  DoSomething(g_data1, g_data1, g_data2, g_data3);

Details? More like duck tails, am I right (thousands of ducks being dumped into the Thames, circa 2006)This omits a ton of details (when does g_flag get cleared? How do the threads avoid spinning?) but it suffices for my purposes. The question is, what is wrong with this code, in particular the producer thread?

The basic problem is that the code relies on the three data variables being written to before the flag, but it does not enforce that. If the compiler rearranges the writes then g_flag may be set to true before all of the data is written and the consumer thread may see incorrect values.

Optimizing compilers can be very aggressive about rearranging operations – it’s one of the ways they make their generated code run fast. They may do this in order to reduce the use of registers, to improve the use of CPU pipelines, or just because of some random heuristic added to make Office XP load slightly faster. It’s not worth thinking too much about why a compiler might rearrange things, it’s just important to realize that they can and do.

Compilers are “allowed” to rearrange the writes because of the “as-if” rule which says that they have done their job as long as the program that they generate behaves “as-if” they hadn’t optimized it. Since the C/C++ abstract machine has long assumed a single-thread of execution – with no external observers – all this rearrangement of writes has been correct and reasonable, and has been done for decades.

The question then, is what must be done to stop the compiler from breaking our beautiful code? Let’s pretend for a moment that we are a circa 2005 programmer trying to make this work. Here are some bad ideas:

  1. Declare g_flag as volatile. That prevents the compiler from omitting the reads/writes of g_flag but, to the surprise of many, it does not prevent the problematic rearrangement. Compilers are not allowed to reorder volatile reads/writes with respect to each other, but they are allowed to rearrange them relative to “normal” reads/writes. Adding volatile does nothing to solve our reordering problem (/volatile:ms on VC++ does, but it is a non-standard extension to the language that may generate slower code).
  2. If declaring g_flag as volatile is insufficient then let’s try declaring all four variables as volatile! Then the compiler can’t rearrange the writes and our code will work… on some computers.

It turns out that compilers aren’t the only things that like to rearrange reads and writes. CPUs like to do this as well. This is separate from out-of-order execution (always invisible to your code), and in fact there are in-order CPUs that reorder reads/writes (Xbox 360 CPU) and there are out-of-order CPUs that mostly do not reorder reads/writes (x86/x64 CPUs).

So, if you declare all four variables as volatile then you have code that will only run correctly on x86/x64. And, this code is potentially inefficient because no reads/writes to those variables can be optimized away, potentially leading to redundant work (as when g_data1 is passed twice to DoSomething).

A big barrier (Hadrian's wall?)If you are satisfied with inefficient non-portable code then feel free to stop here, but I think we can do better. But let’s continue to constrain ourselves to the options available in 2005. We now have to make use of… memory barriers.

On x86/x64 we need a compiler memory barrier to prevent reordering. This does the trick:

g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
_ReadWriteBarrier(); // VC++ only and deprecated, but okay in 2005
g_flag = true; // Indicate that the data is ready for consumption.

This tells the compiler not to rearrange the writes across that barrier which is exactly what we need. Another barrier may be needed after the write to g_flag to ensure that value gets written but the details are too uncertain for me to want to discuss. A similar barrier should be used in the consumer thread, but I’m ignoring that thread for now to keep things simple.

The problem is that this code is still broken on CPUs with a weak memory model. A “weak” memory model indicates CPUs that can reorder reads and writes (for greater efficiency or simplicity of implementation) and this includes ARM, PowerPC, MIPS, and basically every in-use CPU except for x86/x64. The solution to this is also a memory barrier, but this time it needs to be a CPU instruction which tells the CPU not to reorder. Something like this:

g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
MemoryBarrier(); // Windows only, and an expensive full memory barrier.
g_flag = true; // Indicate that the data is ready for consumption.

The actual implementation of MemoryBarrier depends on the CPU. In fact, as the comment suggests, MemoryBarrier is not really the ideal choice here because we just want a write/write barrier instead of a much more expensive full memory barrier (which makes reads wait for writes to fully complete) but this is good enough for our purposes today.

I assume that the MemoryBarrier intrinsic is also a compiler memory barrier, so we only need one or the other, so our awesome/efficient producer thread now becomes:

#ifdef X86_OR_X64
#define GenericBarrier _ReadWriteBarrier
#else
#define GenericBarrier MemoryBarrier
#endif
g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
GenericBarrier(); // Why did I have to define this myself?
g_flag = true; // Indicate that the data is ready for consumption.

More barriers (Cusco, Peru) - these should do the trickIf you have circa-2005 code without these memory barriers then your code is broken, and has always been broken, on all CPUs, because compilers have always been allowed to rearrange writes. With these memory barriers (implemented as needed for different compilers and platforms) your code is beautiful and portable.

It turns out that ARM’s weak memory model really doesn’t make things any more complicated. If you are writing lock-free code and not using any sort of memory barriers then your code is potentially broken everywhere due to compiler reordering. If you are using memory barriers then it should be easy to extend them to include hardware memory barriers.

The code above is error prone (where do the barriers go?), verbose, and inefficient. Luckily when C++11 came along we got better options. Prior to C++11 the language didn’t really have a memory model, there was just the implicit assumption that all code was single threaded and if you touched shared data outside of locks then god have mercy on your soul. C++ 11 added a memory model that acknowledged the existence of threads. This made it more explicit that the no-barrier code above was broken, but also gave new options to fix it, like this:

// Producer thread:
Data_t g_data1, g_data2, g_data3;
std::atomic<bool> g_flag // Look at this!

g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
g_flag = true; // Indicate that the data is ready for consumption.

The change is subtle and easy to miss. All I did was change the type of g_flag from bool to std::atomic<bool>. This tells the compiler not to elide reads and writes of this variable, not to rearrange reads and writes across reads and writes to this variable, and to add appropriate CPU memory barriers as needed.

We can even optimize this code slightly:

// Producer thread:
Data_t g_data1, g_data2, g_data3;
std::atomic<bool> g_flag

g_data1 = calc1();
g_data2 = calc2();
g_data3 = calc3();
g_flag.store(true, std::memory_order_release);

By using memory_order_release we are telling the compiler exactly what we are doing so that it can use the appropriate (cheaper) type of memory barrier instruction, or no memory barrier instruction in the case of x86/x64. Our code is now relatively clean and perfectly efficient.

At this point writing the consumer thread is easy. In fact, with the new declaration of g_flag the original version of the consumer thread is now correct! But, we can optimize it slightly:

// Consumer thread:
if (g_flag.load(std::memory_order_acquire)) {
  DoSomething(g_data1, g_data1, g_data2, g_data3);

The std::memory_order_acquire flag tells the compiler that we don’t need a full memory barrier – a read-acquire barrier just ensures that the data values don’t come from shared storage before g_flag without blocking other reordering.

Finishing the code so that the threads can avoid busy-waits and other problems is left as an exercise for the reader.

If you want to learn these techniques then start by carefully reading Jeff Preshing’s introduction to lock-free programming or This is Why They Call It a Weakly-Ordered CPU, and then consider joining a monastery or nunnery instead. Lock-free programming is the most dangerous hammer in the C++ toolkit and that is saying a lot, and it is rarely appropriate.

Note: writing an x86 emulator on ARM forces you to deal with this in a particularly messy way because you never know when reordering will be a problem so you typically have to insert a lot of memory barriers. Or you can follow Apple’s strategy and add a CPU mode that enforces x86/x64 memory ordering, turning that on when emulating.

References:

Most discussion is on reddit but there is some discussion on twitter and some discussion on hacker news.

Like this:

Like Loading...

Related



from Hacker News https://ift.tt/39pudnp

AWS Announces macOS on EC2

Throughout the course of my career I have done my best to stay on top of new hardware and software. As a teenager I owned an Altair 8800 and an Apple II. In my first year of college someone gave me a phone number and said “call this with modem.” I did, it answered “PENTAGON TIP,” and I had access to ARPANET!

I followed the emerging PC industry with great interest, voraciously reading every new issue of Byte, InfoWorld, and several other long-gone publications. In early 1983, rumor had it that Apple Computer would soon introduce a new system that was affordable, compact, self-contained, and very easy to use. Steve Jobs unveiled the Macintosh in January 1984 and my employer ordered several right away, along with a pair of the Apple Lisa systems that were used as cross-development hosts. As a developer, I was attracted to the Mac’s rich collection of built-in APIs and services, and still treasure my phone book edition of the Inside Macintosh documentation!

New Mac Instance
Over the last couple of years, AWS users have told us that they want to be able to run macOS on Amazon Elastic Compute Cloud (EC2). We’ve asked a lot of questions to learn more about their needs, and today I am pleased to introduce you to the new Mac instance!

The original (128 KB) Mac

Powered by Mac mini hardware and the AWS Nitro System, you can use Amazon EC2 Mac instances to build, test, package, and sign Xcode applications for the Apple platform including macOS, iOS, iPadOS, tvOS, watchOS, and Safari. The instances feature an 8th generation, 6-core Intel Core i7 (Coffee Lake) processor running at 3.2 GHz, with Turbo Boost up to 4.6 GHz. There’s 32 GiB of memory and access to other AWS services including Amazon Elastic Block Store (EBS), Amazon Elastic File System (EFS), Amazon FSx for Windows File Server, Amazon Simple Storage Service (S3), AWS Systems Manager, and so forth.

On the networking side, the instances run in a Virtual Private Cloud (VPC) and include ENA networking with up to 10 Gbps of throughput. With EBS-Optimization, and the ability to deliver up to 55,000 IOPS (16KB block size) and 8 Gbps of throughput for data transfer, EBS volumes attached to the instances can deliver the performance needed to support I/O-intensive build operations.

Mac instances run macOS 10.14 (Mojave) and 10.15 (Catalina) and can be accessed via command line (SSH) or remote desktop (VNC). The AMIs (Amazon Machine Images) for EC2 Mac instances are EC2-optimized and include the AWS goodies that you would find on other AWS AMIs: An ENA driver, the AWS Command Line Interface (CLI), the CloudWatch Agent, CloudFormation Helper Scripts, support for AWS Systems Manager, and the ec2-user account. You can use these AMIs as-is, or you can install your own packages and create custom AMIs (the homebrew-aws repo contains the additional packages and documentation on how to do this).

You can use these instances to create build farms, render farms, and CI/CD farms that target all of the Apple environments that I mentioned earlier. You can provision new instances in minutes, giving you the ability to quickly & cost-effectively build code for multiple targets without having to own & operate your own hardware. You pay only for what you use, and you get to benefit from the elasticity, scalability, security, and reliability provided by EC2.

EC2 Mac Instances in Action
As always, I asked the EC2 team for access to an instance in order to put it through its paces. The instances are available in Dedicated Host form, so I started by allocating a host:

$ aws ec2 allocate-hosts --instance-type mac1.metal \
  --availability-zone us-east-1a --auto-placement on \
  --quantity 1 --region --us-east-1

Then I launched my Mac instance from the command line (console, API, and CloudFormation can also be used):

$ aws ec2 run-instances --region us-east-1 \
  --instance-type mac1.metal \
  --image-id  ami-023f74f1accd0b25b \
  --key-name keys-jbarr-us-east  --associate-public-ip-address

I took Luna for a very quick walk, and returned to find that my instance was ready to go. I used the console to give it an appropriate name:

Then I connected to my instance:

From here I can install my development tools, clone my code onto the instance, and initiate my builds.

I can also start a VNC server on the instance and use a VNC client to connect to it:

Note that the VNC protocol is not considered secure, and this feature should be used with care. I used a security group that allowed access only from my desktop’s IP address:

I can also tunnel the VNC traffic over SSH; this is more secure and would not require me to open up port 5900.

Things to Know
Here are a couple of fast-facts about the Mac instances:

AMI Updates – We expect to make new AMIs available each time Apple releases major or minor versions of each supported OS. We also plan to produce AMIs with updated Amazon packages every quarter.

Dedicated Hosts – The instances are launched as EC2 Dedicated Hosts with a minimum tenancy of 24 hours. This is largely transparent to you, but it does mean that the instances cannot be used as part of an Auto Scaling Group.

Purchase Models – You can run Mac instances On-Demand and you can also purchase a Savings Plan.

Apple M1 Chip – EC2 Mac instances with the Apple M1 chip are already in the works, and planned for 2021.

Launch one Today
You can start using Mac instances in the US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), and Asia Pacific (Singapore) Regions today, and check out this video for more information!

Jeff;

 



from Hacker News https://ift.tt/2JyFruR

Write your own arbitrary-precision JavaScript math library


Javascript has its fair share of ‘wat’ moments. Even though most of them have a logical explanation once you dig in, they can still be surprising. But JavaScript doesn’t deserve all the indignant laughter. For example, you’ll sometimes see jokes like this:

In what language does 0.1 + 0.2 not equal 0.3?

console.log(0.1 + 0.2 === 0.3);
// ⦘ false
console.log(0.1 + 0.2);
// ⦘ '0.30000000000000004'

In JavaScript! Hahahaha. What a stupid language.

In this case the criticism is entirely undeserved. JavaScript, like nearly every other popular programming language, represents numbers using a standard. To be precise, the IEEE 754 standard for double-precision 64-bit binary format numbers. Let’s try that same joke in some other languages:

How about Ruby?

In what language does 0.1 + 0.2 not equal 0.3?

$ irb
irb(main):001:0> 0.1 + 0.2 == 0.3
=> false
irb(main):002:0> 0.1 + 0.2
=> 0.30000000000000004

In Ruby! Hahahaha. What a stupid language.

Or Clojure?

In what language does 0.1 + 0.2 not equal 0.3?

$ clj
Clojure 1.10.1
user=> (== (+ 0.1 0.2) 0.3)
false
user=> (+ 0.1 0.2)
0.30000000000000004

In Clojure! Hahahaha. What a stupid language.

Or how about the mighty Haskell?

In what language does 0.1 + 0.2 not equal 0.3?

$ ghci
GHCi, version 8.10.1: https://www.haskell.org/ghc/  :? for help
Prelude> 0.1 + 0.2 == 0.3
False
Prelude> 0.1 + 0.2
0.30000000000000004

In Haskell! Hahahaha. What a stupid language.

You get the idea. The issue here isn’t JavaScript. It’s the bigger problem of representing floating point numbers in binary. But I don’t want to get into the details of IEEE 754 for the moment. Because, if we need arbitrary precision numbers, JavaScript now makes that possible. Since October-ish 2019, BigInt is officially part of the TC39 ECMAScript standard.

Why bother?

We’ve gotten by with IEEE 754 for ages. It doesn’t seem to be a problem most of the time. That’s true. It isn’t a problem most of the time. But on occasion, it is. And in those moments, it’s good to have options.

For example, I was working on a chart library earlier this year. I wanted to draw candlestick charts in SVG. And SVG has this neat feature called a transform. You can apply it to a group of elements, and it will change the coordinate system for those elements. So, with a little care, you can simplify generating the chart area. Instead of calculating chart coordinates for each candlestick, you specify a single transform. And then specify each candlestick using raw data values. It’s neat. At least, in theory.

But in my property tests, I was running into problems. If the chart was small, and the data values were large, I’d get rounding errors. And most of the time, that’s OK. But in a chart, certain pixels have to line up. Otherwise it doesn’t look right. So I started to look into BigInt. The outcome was a library I’ve called ‘Ratio’. And I’m going to show you how you could write it too.

The Ratio class

The problem with floating point numbers is binary representation. Computers do all their calculations in binary. And binary is fine for integers. The trouble comes when we want to represent decimal numbers. For example, in English-speaking countries like Australia, we write decimal numbers like this:

\(3.1415926\)

The bit to the left the dot ( \(.\) ) is the integer part. And the bit to the right of the dot is the fractional part. But the trouble is, some numbers have fractional parts that don’t divide easily into two. So they’re hard to represent in binary. But we even have the similar problems working in base 10. For example, consider. the fraction \(\frac{10}{9}\). You can try to write it something like this:

\(1.11111111111111111111111111111111111\)

That’s an approximation, though. To represent \(\frac{10}{9}\) with full accuracy, those ones have to go on forever. So we have to use some other notation to represent the repeated ones. Like the dot notation:

\(1.\dot{1}\)

That dot over the one indicates the ones keep on going. But we don’t have dot notation in most programming languages.

Notice though, that \(\frac{10}{9}\) has perfect accuracy. And all it takes is two pieces of information. That is a numerator and a denominator. With a single BigInt value we can represent arbitrarily large integers. But if we create a pair of integers, we can represent arbitrarily large or small numbers.

In JavaScript, that might look like this:

// file: ratio.js
export default class Ratio {
  // We expect n and d to be BigInt values.
  constructor(n, d) {
    this.numerator = n;
    this.denominator = d;
  }
}

And with that, we’ve done the trickiest bit. We’ve ‘invented’ a way to represent numbers with near-infinite accuracy. (We’re still limited by the amount of memory in our devices). All that remains is to apply some mathematics. Stuff you might have studied in school.

So let’s add some features.

Equals

The first thing we want to do is compare two ratios. Why? Because I like to write my code test-first. If I can compare two ratios for equality, then it’s much easier to write tests.

For the simple case, writing an equality method is pretty easy:

// file: ratio.js
export default class Ratio {
  constructor(n, d) {
    this.numerator = n;
    this.denominator = d;
  }

  equals(other) {
    return (
      this.numerator === other.numerator &&
      this.denominator === other.denominator
    );
  }
}

That’s fine. But it would be nice if our library could tell if, say, \(\frac{1}{2}\) was equal to \(\frac{2}{4}\). To do that, we need to simplify our ratios. That is, before we test for equality, we want to reduce both ratios to the smallest possible integers. So, how do we do that?

A naĆÆve approach is to run through all the numbers from 1 to \(\min(n,d)\) (where \(n\) and \(d\) are the numerator and denominator). And that’s what I tried first. It looked something like this:

function simplify(numerator, denominator) {
    const maxfac = Math.min(numerator, denominator);
    for (let i=2; i<=maxfac; i++) {
      if ((numerator % i === 0) && (denominator % i === 0)) {
        return simplify(numerator / i, denominator / i);
      }
    }
    return Ratio(numerator, denominator);
}

And, as you’d expect, it’s ridiculously slow. My property tests took ages to run. So, we need a more efficient approach. Lucky for us, a Greek mathematician figured this one out a couple of millennia ago. The way to solve it is using Euclid’s algorithm. It’s a way of finding the largest common factor for two integers.

The recursive version of Euclid’s algorithm is beautiful and elegant:

function gcd(a, b) {
    return (b === 0) ? a : gcd(b, a % b);
}

It can be memoized too, making it quite snappy. But alas, we don’t have tail call recursion in V8 or SpiderMonkey yet. (At least, not at the time of writing). This means that if we run it with large enough integers, we get stack overflow. And large integers are kind of the point here.

So, instead, we use the iterative version:

// file: ratio.js
function gcd(a, b) {
    let t;
    while (b !== 0) {
        t = b;
        b = a % b;
        a = t;
    }
    return a;
}

Not so elegant, but it does the job. And with that in place, we can write a function to simplify ratios. While we’re at it, we’ll make a small modification so that denominators are always positive. (That is, only the numerator changes sign for negative numbers).

// file: ratio.js

function sign(x) {
  return x === BigInt(0) ? BigInt(0)
       : x > BigInt(0)   ? BigInt(1) 
       /* otherwise */   : BigInt(-1);
}

function abs(x) {
  return x < BigInt(0) ? x * BigInt(-1) : x;
}

function simplify(numerator, denominator) {
  const sgn = sign(numerator) * sign(denominator);
  const n = abs(numerator);
  const d = abs(denominator);
  const f = gcd(n, d);
  return new Ratio((sgn * n) / f, d / f);
}

And with that it place, we can write our equality method:

// file: ratio.js -- inside the class declaration
  equals(other) {
    const a = simplify(this);
    const b = simplify(other);
    return (
      a.numerator === b.numerator &&
      a.denominator === b.denominator
    );
  }

We’re now able to compare two ratios for equality. It might not seem like much, but it means we can write unit tests and make sure our library works as expected.

Converting to other types

Now, I won’t bore you by writing out all the unit tests for this library. But something that would be nice is to convert these ratios into other formats. For example, we might want to represent them as a string in debug messages. Or we might want to convert them to numbers. So let’s override the .toString() and .toValue() methods for our class.

The .toString() method is easiest, so let’s start with that.

// file: ratio.js -- inside the class declaration
  toString() {
    return `${this.numerator}/${this.denominator}`;
  }

Easy enough. But how about converting back to a number? One way to do it is to just divide numerator by denominator:

// file: ratio.js -- inside the class declaration
  toValue() {
    return  Number(this.numerator) / Number(this.denominator);
  }

That works, most of the time. But we might want to tweak it a little. The whole point of our library is that we use large integers to get the precision we need. And sometimes these integers will be too large to convert back to a Number. But, we want to get the number as close as we can, wherever possible. So we do a little bit of arithmetic when we convert:

// file: ratio.js -- inside the class declaration
  toValue() {
    const intPart = this.numerator / this.denominator;
    return (
      Number(this.numerator - intPart * this.denominator) /
        Number(this.denominator) + Number(intPart)
    );
  }

By extracting the integer part, we reduce the size of the BigInt values before we convert them to Number. There are other ways to do this that have fewer range problems. In general, they are more complex and slower though. I encourage you to look into them further if you’re interested. But for this article, the simple approach will cover enough cases to be useful.

Multiply and divide

Let’s do something with our numbers. How about multiplication and division? These are not complicated for ratios. For multiplication, we multiply numerators with numerators, and denominators with denominators.

// file: ratio.js -- inside the class declaration
  times(x) {
    return simplify(
      x.numerator * this.numerator,
      x.denominator * this.denominator
    );
  }

Division is similar. We invert the second ratio, then multiply.

// file: ratio.js -- inside the class declaration
  divideBy(x) {
    return simplify(
      this.numerator * x.denominator,
      this.denominator * x.numerator
    );
  }

Add and subtract

We now have multiplication and division. The next logical thing to write is addition and subtraction. These are slightly more complicated than multiplication and division. But not too much.

To add two ratios together, we first need to manipulate them so they have the same denominator. Then we add the numerators together. In code, that might look something like this:

// file: ratio.js -- inside the class declaration
  add(x) {
    return simplify(
      this.numerator * x.denominator + x.numerator * this.denominator,
      this.denominator * x.denominator
    );
  }

Everything is multiplied by denominators. And we use simplify() to keep the ratios as small as possible.

Subtraction is similar. We manipulate the two ratios so that denominators line up as before. Then we subtract instead of adding the numerators.

// file: ratio.js -- inside the class declaration
  subtract(x) {
    return simplify(
      this.numerator * x.denominator - x.numerator * this.denominator,
      this.denominator * x.denominator
    );
  }

So we have our basic operators. We can add, subtract, multiply and divide. But we still need a few other methods. In particular, numbers have an important property: we can compare them to each other.

Less than and greater than

We’ve already discussed .equals(). But we need more than just equality. We’d also like to be able to tell if one ratio is larger or smaller than another. So we’ll create a method .lte() that will tell us if a ratio is less than or equal to another ratio. Like .equals(), it’s not obvious which of two ratios is smaller. To compare them, we need to convert both to have the same denominator. Then, we can compare numerators to see which is larger. With a bit of simplification, it might look like so:

// file: ratio.js -- inside the class declaration
  lte(other) {
    const { numerator: thisN, denominator: thisD } = simplify(
      this.numerator,
      this.denominator
    );
    const { numerator: otherN, denominator: otherD } = simplify(
      other.numerator,
      other.denominator
    );
    return thisN * otherD <= otherN * thisD;
  }

Once we have .lte() and .equals() we can derive all the other comparisons. We could have chosen any comparison operator. But once we have equals() and any one of \(>\), \(<\), \(\geq\) or \(\leq\), then we can derive the others with boolean logic. In this case, we’ve gone with lte() because that’s what the FantasyLand standard uses. Here’s how working out the others might look.

// file: ratio.js -- inside the class declaration
  lt(other) {
    return this.lte(other) && !this.equals(other);
  }

  gt(other) {
    return !this.lte(other);
  }

  gte(other) {
    return this.gt(other) || this.equals(other);
  }

Floor and ceiling

We can now compare ratios. And we can also multiply and divide, add and subtract. But if we’re going to do more interesting things with our library, we need more tools. Some of the handy ones from JavaScript’s Math object include .floor() and .ceil().

We’ll start with .floor(). Floor takes a value and rounds it down. With positive numbers, that means we just keep the integer part and throw away any remainder. But for negative numbers, we round away from zero, so it needs a little extra care.

// file: ratio.js -- inside the class declaration
  floor() {
    const one = new Ratio(BigInt(1), BigInt(0));
    const trunc = simplify(this.numerator / this.denominator, BigInt(1));
    if (this.gte(one) || trunc.equals(this)) {
      return trunc;
    }
    return trunc.minus(one);
  }

With that in place, we can leverage it to help us calculate ceiling values. This is where we round up.

// file: ratio.js -- inside the class declaration
  ceil() {
    const one = new Ratio(BigInt(1), BigInt(0));
    return this.equals(this.floor()) ? this : this.floor().add(one);
  }

We now have most of what we’d need for lots of math operations. And with .toValue() we can easily convert our calculations back to decimal numbers. But what if we want to convert a floating point number to a ratio?

Numbers to Ratios

Converting a number to a ratio is more involved than it might seem at first glance. And there are many different ways to do it. The way I’ve done it is not the most accurate, but it’s good enough. To make it work, we first convert the number to a string we know will be in a consistent format. For this, JavaScript gives us the .toExponential() method. It gives us the number in exponential notation. Here’s some examples so you get the idea:

let x = 12.345;
console.log(x.toExponential(5));
// ⦘ '1.23450e+1''

x = 0.000000000042;
console.log(x.toExponential(3));
// ⦘ '4.200e-11'

x = 123456789;
console.log(x.toExponential(4));
// ⦘ '1.2346e+8'

It works by representing the number as a normalised decimal value and a multiplier. We call the normalised decimal bit the significand. And the multiplier, the exponent. Here, ‘normalised’ means the absolute value of the significand is always less than 10. And the exponent is always a power of 10. We indicate the start of the multiplier with the letter ‘e’, short for ‘exponent’.

The advantage of this notation is that it’s consistent. There’s always exactly one digit to the left of the decimal point. And .toExponential() lets us specify how many significant digits we want. Then comes the ‘e’ and the exponent (always an integer). Because it’s so consistent, we can use a cheeky regular expression to parse it.

The process goes something like this. As mentioned, .toExponential() takes a parameter to specify the number of significant digits. We want maximum digits. So we set the precision to 100 (which is as many as most JavaScript engines will allow). For this example though, we’ll stick with a precision of 10. Now, imagine we have a number like 0.987654321e0. What we want to do is move that decimal point 10 digits to the right. That would give us 9876543210. Then we divide it by \(10^{10}\), and we get \(\frac{9876543210}{10000000000}\). This, in turn, simplifies to \(\frac{987654321}{10000000000}\).

We have to pay attention to that exponent though. If we have a number like 0.987654321e9, we still move the decimal point 10 digits to the right. But we divide by ten to the power of \(10 - 9 = 1\).

$$ \begin{align} 0.987654321\times10^{9} &= \frac{9876543210}{10^{1}} \\ &= \frac{987654321}{1} \end{align} $$

To make all this happen, we define a couple of helper functions:

// Transform a ‘+’ or ‘-‘ character to +1 or -1
function pm(c) {
  return parseFloat(c + "1");
}

// Create a new bigint of 10^n. This turns out to be a bit
// faster than multiplying.
function exp10(n) {
  return BigInt(`1${[...new Array(n)].map(() => 0).join("")}`);
}

With those in place we can put the whole fromNumber() function together.

// file: ratio.js -- inside the class declaration
  static fromNumber(x) {
    const expParse = /(-?\d)\.(\d+)e([-+])(\d+)/;
    const [, n, decimals, sgn, pow] =
      x.toExponential(PRECISION).match(expParse) || [];
    const exp = PRECISION - pm(sgn) * +pow;
    return exp < 0
      ? simplify(BigInt(`${n}${decimals}`) * exp10(-1 * exp), BigInt(1))
      : simplify(BigInt(`${n}${decimals}`), exp10(exp));
  }

We now have most of the basic functions covered. We can go from numbers to ratios, and back again. For my particular application though, I needed more. In particular, I needed to find exponents and logarithms.

Exponentiation

Exponentiation is where you multiply something by itself repeatedly. For example \(2^3 = 2 \times 2 \times 2 = 8\). For simple cases where the exponent is an integer, we already have a built-in BigInt operator: **. So, if we’re taking our rato to the power of an integer, we’re good to go. The power law for ratios looks like so:

$$ \left(\frac{x}{y}\right)^{n} = \frac{x^n}{y^n} $$

Hence, a first cut of our exponentiation method might look something like this:

// file: ratio.js -- inside the class declaration
  pow(exponent) {
    if (exponent.denominator === BigInt(1)) {
        return simplify(
            this.numerator ** exponent.numerator,
            this.denominator ** exponent.numerator
        );
    }
  }

That works fine. Well… mostly fine. Things start to get tricky from here. Because of the limits of hardware and mathematics, we have to make some compromises. We might have to sacrifice precision for the sake of getting an answer in a reasonable amount of time.

With exponentiation it’s not hard to generate very large numbers. And when numbers get large, everything slows down. While writing this article I created calculations that ran for days without finishing. So we need to be careful. But that’s OK. It comes with the territory for BigInt.

There is another problem though. What do we do if the denominator of the exponent is not 1? For example, what if we wanted to calculate \(8^{\frac{2}{3}}\)?

Fortunately, we can split this problem up into two parts. We want to take one ratio to the power of another. For example, we might take \(\frac{x}{y}\) to the power of \(\frac{a}{b}\). The laws of exponentiation say that the following are equivalent:

\[\left(\frac{x}{y}\right)^\frac{a}{b} = \left(\left(\frac{x}{y}\right)^\frac{1}{b}\right)^a = \left(\frac{x^\frac{1}{b}}{y^\frac{1}{b}}\right)^a\]

We already know how to take a BigInt to the power of another BigInt. But what about the fractional power? Well, there’s another equivalence we can bring in here:

\[x^\frac{1}{n} = \sqrt[n]{x}\]

That is, taking \(x\) to the power of \(\frac{1}{n}\) is equivalent to finding the nth root of \(x\). This means, if we can find a way to calculate the nth root of a BigInt, then we can calculate any power.

With a well-crafted web-search or two, it doesn’t take long to find an algorithm for estimating the nth root. The most common is Newton’s method. It works by starting with an estimate, \(r\). Then we make the following calculation to get a better estimate:

$$ \begin{align} r &\approx x^{\frac{1}{n}} \\ r^{\prime} &= \frac{1}{n}\left((n-1)r + \left(\frac{x}{r^{n-1}}\right)\right) \end{align} $$

We keep repeating that calculation until we reach the desired precision. Unfortunately, there are some roots that can’t be represented as a finite fraction. To put it another way, to get perfect precision we’d need infinitely long BigInt values. In practice, this means we have to pick an arbitrary limit on how many iterations we’ll do.

We’ll come back to this point. For now, let’s work out how we can calculate a good estimate of the nth root. Because the estimate \(r\) will be a ratio, we can write it as:

$$ r = \frac{a}{b} $$

And that allows us to rewrite the estimate calculation as:

\[\frac{a^{\prime}}{b^{\prime}} = \frac{(n - 1)a^{n} + x b^{n}}{n b a^{n - 1}}\]

This puts it in a form where everything is in terms of integer calculations suitable for use with BigInt. Feel free to plug \(\frac{a}{b}\) into the equation for \(r'\) above and check my derivation. Putting that into JavaScript looks something like the following:

const estimate = [...new Array(NUM_ITERATIONS)].reduce(r => {
  return simplify(
    (n - BigInt(1)) * r.numerator ** n + x * r.denominator ** n,
    n * r.denominator * r.numerator ** (n - BigInt(1))
  );
}, INITIAL_ESTIMATE);

We just repeat that calculation until we reach a suitable accuracy for our nth root estimate. The trouble is, we need to come up with suitable values for our constants. That is, NUM_ITERATIONS and INITIAL_ESTIMATE.

A lot of algorithms start with their INITIAL_ESTIMATE as 1. It’s a reasonable choice. Most of the time we have no real good way of guessing what the nth root might be. But in our case, we can cheat. Let’s assume (for the moment) that our numerator and denominator are in the range allowed by Number. We can then use Math.pow() to get an initial estimate. That might look like so:

// Get an initial estimate using floating point math
// Recall that x is a bigint value and n is the desired root.
const initialEstimate = Ratio.fromNumber(
    Math.pow(Number(x), 1 / Number(n))
);

So we have a value for our initial estimate. But what about NUM_ITERATIONS? Well, in practice, what I did was start with a guess of 10. And then I would run my property tests. I kept dialling the number back until they finished in a reasonable amount of time. And the figure that finally worked was… 1. One iteration. Which makes me a little sad, but we are at least a little more accurate than floating point calculations. In practice, you can tune this number number up if you’re not calculating a lot of fractional powers.

To keep things simple, we’ll pull the nth root calculation out into its own function. Putting it all together it might look like the following:

// file: ratio.js -- inside the class declaration
  static nthRoot(x, n) {
    // Handle special cases
    if (x === BigInt(1)) return new Ratio(BigInt(1), BigInt(1));
    if (x === BigInt(0)) return new Ratio(BigInt(0), BigInt(1));
    if (x < 0) return new Ratio(BigInt(1), BigInt(0)); // Infinity

    // Get an initial estimate using floating point math
    const initialEstimate = Ratio.fromNumber(
      Math.pow(Number(x), 1 / Number(n))
    );

    const NUM_ITERATIONS = 1;
    return [...new Array(NUM_ITERATIONS)].reduce((r) => {
      return simplify(
        n -
          BigInt(1) * (r.numerator ** n) +
          x * (r.denominator ** n),
        n * r.denominator * r.numerator ** (n - BigInt(1))
      );
    }, initialEstimate);
  }

  pow(n) {
    const { numerator: nNumerator, denominator: nDenominator } = n.simplify();
    const { numerator, denominator } = this.simplify();
    if (nNumerator < 0) return this.invert().pow(n.abs());
    if (nNumerator === BigInt(0)) return Ratio.one;
    if (nDenominator === BigInt(1)) {
      return new Ratio(numerator ** nNumerator, denominator ** nNumerator);
    }
    if (numerator < 0 && nDenominator !== BigInt(1)) {
      return Ratio.infinity;
    }

    const { numerator: newN, denominator: newD } = Ratio.nthRoot(
      numerator,
      nDenominator
    ).divideBy(Ratio.nthRoot(denominator, nDenominator));
    return new Ratio(newN ** nNumerator, newD ** nNumerator);
  }

It’s not perfect, and it’s slow. But it gets the job done. Well, mostly. There’s still the issue of how to get an estimate if we have integers larger than Number.MAX_VALUE. I’ll leave that as an exercise to the reader though, as this article is already far too long.

Logarithms

I have to admit, logarithms stumped me for weeks. For the thing I’m building, I need to calculate logarithms in base 10. So I went searching for algorithms to calculate logs. And there’s plenty of them. But I couldn’t find one that worked well enough to be included in a math library.

Why is it so hard? My goal was to calculate logarithms to be more accurate than floating point. Otherwise, why bother? The floating point log function, Math.log10(), is fast and built-in. So, I looked at algorithms that provided ways to iteratively calculate logarithms. And they work. But to get accuracy higher than floating point, they’re slow. Not just a little bit slow. Very slow.

What happens is that as we go through the iterations, the fraction we build gets more and more accurate. But that accuracy comes at a cost. The BigInt values in our fraction become larger and larger. And as they get larger, multiplying them together starts to take a long time. At one point I left a calculation running for three days. But while that calculation was running, I remembered something.

I remembered that I wanted the log10() method so that I could calculate nice scale values for charts. And for those calculations, every time I called .log10(), I would immediately call .floor(). Which means I only need the integer part of the log. Calculating the logarithm to 100 decimal places was just a waste of effort.

Better yet, there’s a simple way to calculate the integer part of a base 10 logarithm. All we need to do is count the digits. A naĆÆve attempt might look like the following:

// file: ratio.js -- inside the class declaration
  floorLog10() {
    return simplify(BigInt((this.numerator / this.denominator).toString().length - 1), BigInt(1));
  }

Unfortunately, that doesn’t work for values less than one. But even then, we can use some logarithm laws to work around it.

$$ \begin{align} \log_{10}\left(\frac{a}{b}\right) &= \log_{10}(a) - \log_{10}(b) \\ \log_{10}\left(\frac{1}{x}\right) &= \log_{10}(1) - \log_{10}(x) \\ &= -\log_{10}(x) \end{align} $$

Therefore:

$$ \log_{10}\left(\frac{b}{a}\right) = -\log_{10}\left(\frac{a}{b}\right) $$

Putting it all together, we get a more robust floorLog10() method:

// file: ratio.js -- inside the class declaration

  invert() {
    return simplify(this.denominator, this.numerator);
  }

  floorLog10() {
    if (this.equals(simplify(BigInt(0), BigInt(1)))) {
      return new Ratio(BigInt(-1), BigInt(0));
    }
    return this.numerator >= this.denominator
      ? simplify((this.numerator / this.denominator).toString().length - 1, 1)
      : simplify(BigInt(-1), BigInt(1)).subtract(this.invert().floorLog10());
  }

Again. Why bother?

At this point the library has all the functions I need for my charting application. But you might still be wondering, why go to all this trouble? There’s already several arbitrary precision libraries around. Why not just use one of those and be done with it?

To be fair, most of the time I would use an existing library. Especially if I’m in a hurry. There’s no point in doing all this work if someone else has already done a superior job.

The key word there is ‘superior’ though. And that’s where my motivations for wanting to write my own library come into play. The floorLog10() method above is the perfect case study. For what I want to do, it provides the precise calculation I need. It does it efficiently, in about six lines of code.

If I were to use someone else’s library I’d face one of two scenarios:

  1. They don’t implement a log10() or any other logarithm methods; or
  2. They do implement a log10() method (or equivalent).

In the first scenario, I’d end up having to write floorLog10() anyway. In the second scenario, I’d probably end up using their logarithm method. And my code would have been slower and more complex than it needed to be.

Writing my own library allows me to tailor it to the application. Sure, other people might find it useful, but I’m not beholden to their needs. So my application doesn’t have to carry around complex code it never uses.

Besides all that, I learned a lot writing my own library. I now understand the practical limitations of BigInt much better than before. I know that I can tune the performance of my nth root method. I can tweak it depending on how many calculations I’m running and what accuracy I need.

Sometimes it’s worth writing your own general purpose library. Even if you don’t plan to open-source it. Even if nobody else ever uses it. You can learn a lot and, besides, it can be fun.

Finally, if you’re interested in finding out more about the problems with floating point numbers, check out https://0.30000000000000004.com. And if you want to see the library all together and making some calculations, you can check out this code sandbox.



from Hacker News https://ift.tt/3fNNgJn