Sunday, February 12, 2023

The program is the database is the interface

The program is the database is the interface

!!! DRAFT !!!

I do my accounts each year with a simple script. Something like this:

(ns accounts
  (:require 
    [clojure.string :as str]
    [clojure.pprint :as pp]))

;; converted from statement.csv
(def txs
  [{:date #inst "2022-05-13T11:01:56.532-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T10:41:56.843-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T00:01:03.264-00:00"
    :amount -72.79
    :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"}
   {:date #inst "2022-05-10T10:33:04.011-00:00"
    :amount -20.00
    :text "e-Transfer to: John Smith"}
   {:date #inst "2022-05-11T17:12:43.098-00:00"
    :amount -90.00
    :text "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"}])

(def date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things})

(def text->tag 
  {"Coffee" :eating-out
   "Range Physio" :medical})

(defn tx->tag [tx]
  (or
    (date->tag (:date tx))
    (first 
      (for [[text tag] text->tag
            :when (str/includes? (:text tx) text)]
        tag))))

(def txs-with-tags
  (vec 
    (for [tx txs]
      (assoc tx :tag (tx->tag tx)))))

(def total-per-tag
  (reduce 
    (fn [totals tx]
      (update-in totals [(:tag tx)] #(+ (:amount tx) (or % 0))))
    {}
    txs-with-tags))

(def untagged 
  (vec
    (for [tx txs-with-tags
         :when (nil? (:tag tx))]
      tx)))

(pp/pprint 
  [[:untagged untagged]
   [:total-per-tag total-per-tag]])

There are many things about this which are nice.

  • It's just a single file - easy to backup and version control.
  • It's composable - I can easily write code to answer unexpected questions (eg how much did I spend on currency conversions this year) or use the computed data in other calculations (eg runway projections).
  • It's easy - I just pretty-printed the data-structures I was already using instead of having to build a UI.

But the workflow isn't always great. When I run the code above, I see:

> clj accounts.clj
[[:untagged
  [{:date #inst "2022-05-10T10:33:04.011-00:00",
    :amount -20.0,
    :text "e-Transfer to: John Smith",
    :tag nil}]]
 [:total-per-tag
  {:eating-out -3.3, :things -72.79, :medical -90.0, nil -20.0}]]

That transfer to John Smith isn't covered by any of the tagging rules. So I select the date, switch to the editor window and paste the date into the date->tag definition:

(def date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things
   #inst "2022-05-10T10:33:04.011-00:00" :eating-out})

Now I see:

> clj accounts.clj
[[:untagged []]
 [:total-per-tag
  {:eating-out -23.3, :things -72.79, :medical -90.0}]]

Multiply this by a thousand transactions and it becomes tedious.

Pretty-printing also makes it difficult to decide the amount of detail I should print. Sometimes I want to see which transactions contribute to each tag total. But if I always print them all then it's hard to see the totals themselves without a lot of scrolling.

It's also hard to share this workflow with someone non-technical. I have to setup and maintain the correct environment on their machine, teach them how to use a text editor to change the tagging rules, how to interpret syntax errors, how to use git to share changes etc.

I could solve these kinds of problems by writing a web app and storing txs, date->tag and text->tag in a database.

Then I could put controls on the transaction itself that allow changing the tag in place. And I could add an expandable section next to each total, so that it's easy to see the transactions for that tag when I want to but they don't take up space by default.

Plus sharing becomes trivial - everyone has a web browser.

But this is a big leap in effort:

  • A database introduces a different data model - I have to translate between database data-types and my programming languages native data-types.
  • A database introduces a different language, or at least a new api that differs from the way I access native data-structures.
  • Rather than using a familiar text editor to read and change data, I can only access it via some query language or api.
  • I have to shuffle data back and forth between database and program at the correct times.
  • It isn't easy to version control the contents of the database.
  • Before I can add interactive elements, I have to map my existing data-structures to gui elements (eg to html and css) instead of just printing them.
  • I have to add an extra layer of state management to manage the state of the gui itself.
  • All of this work has to be repeated whenever I have a new question I want to answer.

None of this is insurmountable, but it's definitely much more work than the original script.

So a simple script is low-effort but produces a low-quality experience. And a custom app can produce a high-quality experience but is high-effort.

So I made a thing.

It is very hacky and easy to break. But it will hang together just long enough to convey the idea.

It's kind of like a notebook. There are cells where you can write code and the resulting values will be nicely rendered:

;; converted from statement.csv
(def txs
  [{:date #inst "2022-05-13T11:01:56.532-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T10:41:56.843-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T00:01:03.264-00:00"
    :amount -72.79
    :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"}
   {:date #inst "2022-05-10T10:33:04.011-00:00"
    :amount -20.00
    :text "e-Transfer to: John Smith"}
   {:date #inst "2022-05-11T17:12:43.098-00:00"
    :amount -90.00
    :text "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"}])

(defs date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things})

(defs keyword->tag
  {"Coffee" :eating-out
   "Range Physio" :medical})

Funtions render a little differently.

(defn text->tag [text]
  (first
    (for [[keyword tag] keyword->tag
          :when (clojure.string/includes? text keyword)]
      tag)))

If you type "Joe's Coffee Hut" (including the "!) into the textbox above and hit the text->tag button, you'll see the result of running the text->tag function on that input.

Functions aren't limited to just returning values though. They can modify the values stored in other cells:

(def txs-with-tags
  (vec 
    (for [tx txs]
      (assoc tx 
        :tag (or
               (date->tag (:date tx))
               (text->tag (:text tx)))
        :actions [(fn ignore []
                    (edit! 'date->tag assoc (:date tx) :ignore))
                  (fn tag [tag] 
                    (edit! 'date->tag assoc (:date tx) tag))]))))

If you type :eating-out into the one of the textboxes above and then hit the tag button, it will change the date->tag cell to contain an entry for the date of that transaction with the tag :eating-out.

And then any downstream cells will update, so you'll see the tag for that transaction change to :eating-out.

These actions are just values so they can be passed around like any other value. For example, if I make a list of untagged transactions then I'll still have access to the same actions:

(def untagged
  (vec
    (for [tx txs-with-tags
         :when (nil? (:tag tx))]
      tx)))

We can also attach metadata to values to control how they render:

(def hidden-vec
  (with-meta
    [1 2 3]
    {:preimp/hidden true}))

If you click on the + above it will reveal the contents of the vec. This is useful for controlling the default level of detail.

(defn hidden [v]
  (with-meta v {:preimp/hidden true}))

(def total-per-tag
  (reduce 
    (fn [totals tx]
      (if (= :ignore (:tag tx))
        totals
        (update-in totals [(:tag tx)] 
          (fn [total+txs]
            (let [[total txs] (or total+txs [0 (hidden [])])]
              [(+ total (:amount tx))
               (conj txs (update-in tx [:actions] hidden))])))))
    {}
    txs-with-tags))

You can click on a + above to reveal the transactions for that tag.

The demo on this page is ephemeral - you could do something similar in many notebook environments using mutable data-structures.

But the preimp repo contains a server which persists the entire history of the notebook to disk, and also syncs changes between different clients to allow (coarse-grained) collaborative editing.

The server also allows reading and writing cell values over http. Here's the script that I use to upload my bank statements:

(ns wise
  (:require [clj-http.client :as client]
            [clojure.data.json :as json]))

(def endpoints {
  ;; FILL ME IN
})

(defn api-get [user path]
  (let [domain (get-in endpoints [user :wise-domain])
        url (str domain "/" path)
        response (client/get url {:headers {"Authorization" (str "Bearer " (get-in endpoints [user :wise-token]))}})]
    (assert (= 200 (:status response)))
    (json/read-str (:body response))))

(def now (java.time.Instant/now))

(defn get-transactions [user]
  (into []
        (for [profile (api-get user "v2/profiles")
              :let [profile-id (get profile "id")]
              balance (api-get user (str "v4/profiles/" profile-id "/balances?types=STANDARD"))
              :let [balance-id (get balance "id")
                    statement (api-get user (str "/v1/profiles/" profile-id "/balance-statements/" balance-id "/statement.json?intervalStart=2022-01-01T00:00:00.000Z&intervalEnd=" now "&type=COMPACT"))]
              transaction (get statement "transactions")]
          transaction)))
          
(defn update-preimp [user]
  (let [transactions (get-transactions user)
        cell-name (symbol (str (name user) "-wise-transactions"))
        cell-value (pr-str `(~'defs ~cell-name ~transactions))
        body (json/write-str
              {:cell-id (get-in endpoints [user :cell-id])
               :value cell-value})]
    (client/put
     (get-in endpoints [user :preimp-domain])
     (merge (get-in endpoints [user :preimp-headers]) {:body body}))))

(defn update-preimp-dev [_]
  (update-preimp :sandbox))

(defn update-preimp-prod [_]
  (update-preimp :jamie)
  (update-preimp :cynthia))

Finally, you can export a preimp notebook to produce a perfectly valid clojurescript program.

You can run this program in the repl:

> clj -M -m cljs.main -i ./exported.cljs --repl --repl-env node 

preimp.exported=> (require 'clojure.pprint)
nil

preimp.exported=> (clojure.pprint/pprint txs-with-tags)
[{:date #inst "2022-05-13T11:01:56.532-00:00",
  :amount -3.3,
  :text
  "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER",
  :tag :coffee,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-12T10:41:56.843-00:00",
  :amount -3.3,
  :text
  "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER",
  :tag :coffee,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-12T00:01:03.264-00:00",
  :amount -72.79,
  :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA",
  :tag :things,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-10T10:33:04.011-00:00",
  :amount -20,
  :text "e-Transfer to: John Smith",
  :tag :eating-out,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-11T17:12:43.098-00:00",
  :amount -90,
  :text
  "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER",
  :tag :medical,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}]

nil

This tiny extension to the notebook model allows writing simple ugly crud apps with very little effort. You simply provide the logic and preimp gives you storage, sharing and UI for free. But the result is not an opaque image, nor is it tied to the preimp environment - you can export everything into a regular script and run it in the repl.

TODO rest of this section

required: rich data model, which can be roundtripped through text some way of attaching metadata to values, to control their rendering without interfering with execution in the repl

needed: no declaration order no mutable environment no side-effects (other than edit!) integrity constraints perf (compiler latency) (thoughput only needs to be better than spreadsheet)

extensions: much more ui options selection (+ actions on the side) dropdowns understand types / destructuring render values as copyable text (cf ?) undo/vc (have whole history in db) live repl, sandboxing distribution - single binary, single-file db, optional server (like fossil)

research: finer-grained collaboration (what data model?) bidirectional editing / provenance

We have to do the other demo too. You know the one.

(defs next-id 2)

(defs todos 
  {0 {:text "make a cool demo" :status :done}
   1 {:text "step 2: ???" :status :todo}})

(defn new-todo [text] 
  (edit! 'todos assoc next-id {:text text :status :todo})
  (edit! 'next-id inc))

(defn named [name fn]
  (with-meta fn {:preimp/named name}))

(defn status-toggle [id]
  (let [status (get-in todos [id :status])
        other-status (case status
                       :done :todo
                       :todo :done)]
    (named (name status)
      (fn [] (edit! 'todos assoc-in [id :status] other-status)))))

(defs filter :all)

(def toggle-filter
  (named (str "viewing " (name filter))
    (fn [] (edit! 'filter #(case filter 
                             :all :todo
                             :todo :done
                             :done :all)))))

(def filtered-todos
  (vec
    (for [[id todo] todos
          :when (#{:all (:status todo)} filter)]
      [(:text todo)
       (hidden [(fn set-text [text]
                  (edit! 'todos assoc-in [:id :text] text))
                (status-toggle id)
                (fn delete []
                  (edit! 'todos dissoc id))])])))

(def app
  (vec
    (apply list
      new-todo
      toggle-filter
      filtered-todos)))

TODO spreadsheets don't have state problem, but not interactive (at least in same way)



from Hacker News https://ift.tt/z4D5pY2
Posted by at

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)