Create your own JavaScript Engine

In this post we'll walk through creating a custom JavaScript runtime. Let's call it runjs. Think of it as building a (much) simplified version of deno itself. A goal of this post is to create a CLI that can execute local JavaScript files, read a file, write a file, remove a file and has simplified console API.

Let's get started.

Pre-requisites

This tutorial assumes that the reader has:

• a basic knowledge of Rust
• a basic knowledge of JavaScript event loops

Make sure you have Rust installed on your machine (along with cargo) and it should be at least 1.62.0. Visit rust-lang.org to install Rust compiler and cargo.

Make sure we're ready to go:

$ cargo --version
cargo 1.62.0 (a748cf5a3 2022-06-08)

Hello, Rust!

First off, let's create a new Rust project, which will be a binary crate called runjs:

$ cargo init --bin runjs
     Created binary (application) package

Change your working directory to runjs and open it in your editor. Make sure that everything is set up properly:

$ cd runjs
$ cargo run
   Compiling runjs v0.1.0 (/Users/ib/dev/runjs)
    Finished dev [unoptimized + debuginfo] target(s) in 1.76s
     Running `target/debug/runjs`
Hello, world!

Great! Now let's begin creating our own JavaScript runtime.

Dependencies

Next, let's add the deno_core and tokio dependencies to our project:

$ cargo add deno_core
    Updating crates.io index
      Adding deno_core v0.142.0 to dependencies.
$ cargo add tokio --features=full
    Updating crates.io index
      Adding tokio v1.19.2 to dependencies.

Our updated Cargo.toml file should look like this:

[package]
name = "runjs"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
deno_core = "0.142.0"
tokio = { version = "1.19.2", features = ["full"] }

deno_core is a crate by the Deno team that abstracts away interactions with the V8 JavaScript engine. V8 is a complex project with thousands of APIs, so to make it simpler to use them, deno_core provides a JsRuntime struct that encapsulates V8 engine instance (so called Isolate) and allows integration with an event loop.

tokio is an asynchronous Rust runtime that we will use as an event loop. Tokio is responsible for interacting with OS abstractions like net sockets or file system. deno_core together with tokio allow JavaScript's Promises to be easily mapped onto Rust's Futures.

Having both a JavaScript engine and an event loop allows us to create a JavaScript runtime.

Hello, runjs!

Let's start by writing an asynchronous Rust function that will create an instance of JsRuntime, which is responsible for JavaScript execution.

use std::rc::Rc;
use deno_core::error::AnyError;

async fn run_js(file_path: &str) -> Result<(), AnyError> {
  let main_module = deno_core::resolve_path(file_path)?;
  let mut js_runtime = deno_core::JsRuntime::new(deno_core::RuntimeOptions {
      module_loader: Some(Rc::new(deno_core::FsModuleLoader)),
      ..Default::default()
  });

  let mod_id = js_runtime.load_main_module(&main_module, None).await?;
  let result = js_runtime.mod_evaluate(mod_id);
  js_runtime.run_event_loop(false).await?;
  result.await?
}

fn main() {
  println!("Hello, world!");
}

There's a lot to unpack here. The asynchronous run_js function creates a new instance of JsRuntime, which uses a file-system based module loader. After that, we load a module into js_runtime runtime, evaluate it, and run an event loop to completion.

This run_js function encapsulates the whole life-cycle that our JavaScript code will go through. But before we can do that, we need to create a single-threaded tokio runtime to be able to execute our run_js function:

fn main() {
  let runtime = tokio::runtime::Builder::new_current_thread()
    .enable_all()
    .build()
    .unwrap();
  if let Err(error) = runtime.block_on(run_js("./example.js")) {
    eprintln!("error: {}", error);
  }
}

Let's try to execute some JavaScript code! Create an example.js file that will print "Hello runjs!":

Deno.core.print("Hello runjs!");

Notice that we are using the print function from Deno.core - this is a globally available built-in object that is provided by the deno_core Rust crate.

Now run it:

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/runjs`
Hello runjs!⏎

Success! In just 25 lines of Rust code we created a simple JavaScript runtime, that can execute local files. Of course this runtime can't do much at this point (for example, console.log doesn't work yet - try it!), but we have integrated a V8 JavaScript engine and tokio into our Rust project.

Adding the console API

Let's work on the console API. First, create the src/runtime.js file that will instantiate and make the console object globally available:

((globalThis) => {
  const core = Deno.core;

  function argsToMessage(...args) {
    return args.map((arg) => JSON.stringify(arg)).join(" ");
  }

  globalThis.console = {
    log: (...args) => {
      core.print(`[out]: ${argsToMessage(...args)}\n`, false);
    },
    error: (...args) => {
      core.print(`[err]: ${argsToMessage(...args)}\n`, true);
    },
  };
})(globalThis);

The functions console.log and console.error will accept multiple parameters, stringify them as JSON (so we can inspect non-primitive JS objects) and prefix each message with log or error. This is a "plain old" JavaScript file, like we were writing JavaScript in browsers before ES modules.

To ensure we are not polluting the global scope we are executing this code in an IIFE. If we didn't do that, then argsToMessage helper function would be available globally in our runtime.

Now let's include this code in our binary and execute on every run:

let mut js_runtime = deno_core::JsRuntime::new(deno_core::RuntimeOptions {
  module_loader: Some(Rc::new(deno_core::FsModuleLoader)),
  ..Default::default()
});
+ js_runtime.execute_script("[runjs:runtime.js]",  include_str!("./runtime.js")).unwrap();

Finally, let's update example.js with our new console API:

- Deno.core.print("Hello runjs!");
+ console.log("Hello", "runjs!");
+ console.error("Boom!");

And run it again:

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `target/debug/runjs`
[out]: "Hello" "runjs!"
[err]: "Boom!"

It works! Now let's add an API that will allow us to interact with the file system.

Adding a basic filesystem API

Let's start by updating our runtime.js file:

};

+ globalThis.runjs = {
+   readFile: (path) => {
+     return core.opAsync("op_read_file", path);
+   },
+   writeFile: (path, contents) => {
+     return core.opAsync("op_write_file", path, contents);
+   },
+   removeFile: (path) => {
+     return core.opSync("op_remove_file", path);
+   },
+ };
})(globalThis);

We just added a new global object, called runjs, which has three methods on it: readFile, writeFile and removeFile. The first two methods are asynchronous, while the third is synchronous.

You might be wondering what these core.opAsync and core.opSync calls are - they're mechanisms in deno_core crate for binding JavaScript and Rust functions. When you call either of these, deno_core will look for a Rust function that has an #[op] attribute and a matching name.

Let's see this in action by updating main.rs:

+ use deno_core::op;
+ use deno_core::Extension;
use deno_core::error::AnyError;
use std::rc::Rc;

+ #[op]
+ async fn op_read_file(path: String) -> Result<String, AnyError> {
+     let contents = tokio::fs::read_to_string(path).await?;
+     Ok(contents)
+ }
+
+ #[op]
+ async fn op_write_file(path: String, contents: String) -> Result<(), AnyError> {
+     tokio::fs::write(path, contents).await?;
+     Ok(())
+ }
+
+ #[op]
+ fn op_remove_file(path: String) -> Result<(), AnyError> {
+     std::fs::remove_file(path)?;
+     Ok(())
+ }

We just added three ops that could be called from JavaScript. But before these ops will be available to our JavaScript code, we need to tell deno_core about them by registering an "extension":

async fn run_js(file_path: &str) -> Result<(), AnyError> {
    let main_module = deno_core::resolve_path(file_path)?;
+    let runjs_extension = Extension::builder()
+        .ops(vec![
+            op_read_file::decl(),
+            op_write_file::decl(),
+            op_remove_file::decl(),
+        ])
+        .build();
    let mut js_runtime = deno_core::JsRuntime::new(deno_core::RuntimeOptions {
        module_loader: Some(Rc::new(deno_core::FsModuleLoader)),
+        extensions: vec![runjs_extension],
        ..Default::default()
    });

Extensions allow you to configure your instance of JsRuntime and expose different Rust functions to JavaScript, as well as perform more advanced things like loading additional JavaScript code.

Let's update our example.js again:

console.log("Hello", "runjs!");
console.error("Boom!");
+
+ const path = "./log.txt";
+ try {
+   const contents = await runjs.readFile(path);
+   console.log("Read from a file", contents);
+ } catch (err) {
+   console.error("Unable to read file", path, err);
+ }
+
+ await runjs.writeFile(path, "I can write to a file.");
+ const contents = await runjs.readFile(path);
+ console.log("Read from a file", path, "contents:", contents);
+ console.log("Removing file", path);
+ runjs.removeFile(path);
+ console.log("File removed");
+

And run it:

$ cargo run
   Compiling runjs v0.1.0 (/Users/ib/dev/runjs)
    Finished dev [unoptimized + debuginfo] target(s) in 0.97s
     Running `target/debug/runjs`
[out]: "Hello" "runjs!"
[err]: "Boom!"
[err]: "Unable to read file" "./log.txt" {"code":"ENOENT"}
[out]: "Read from a file" "./log.txt" "contents:" "I can write to a file."
[out]: "Removing file" "./log.txt"
[out]: "File removed"

Congratulations, our runjs runtime now works with the file system! Notice how little code was required to call from JavaScript to Rust - deno_core takes care of marshalling data between JavaScript and Rust so we didn't need to do any of the conversions ourselves.

Summary

In this short example, we have started a Rust project that integrates a powerful JavaScript engine (V8) with an efficient implementation of an event loop (tokio).

A full working example can be found on denoland's GitHub.

from Hacker News https://ift.tt/b3IHurU