The last few months I really been heavy into software development and, honestly, that is my happy place. I started to get into the habit of writing down my nits or annoyances of my workflow or my Emacs setup to try to streamline things. When I get a chance, I go through the list and start figuring out how to tune things. The following is a list of what things I have figured out in the last few months. Most of them are small, but the small things add up.

dotenv creds

I'm on a project using AWS and my AWS creds refresh twice a day. Luckily we are using dotenv so I just need to paste the values into a .env file in my project. But AWS gives me export statements while dotenv just wants VAR=value . And I have to do this twice a day! So a little Emacs and Hyperbole to the rescue.

I worked with my favorite AI and we came up with the following ELisp:

(defun my-clean-env ()
  "Clean environment variables and format text below the button."
  (interactive)
  (let ((kill-content (current-kill 0)))  ; Get content first
    (save-excursion
      ;; Move to end of current line (after the button)
      (end-of-line)
      (insert "\n")
      (let ((start-point (point)))
        ;; Delete everything from start-point to end of buffer
        (delete-region start-point (point-max))
        ;; Insert and clean the new content
        (insert kill-content)
        (let ((end-point (point)))
          ;; Remove lines starting with AWS_
          (goto-char start-point)
          (while (re-search-forward "^AWS_.*\n" end-point t)
            (replace-match ""))
          ;; Replace all "export " instances
          (goto-char start-point)
          (while (search-forward "export " end-point t)
            (replace-match "")))))))

;; Create the Hyperbole button
(defact clean-env ()
  "Hyperbole action to clean environment variables."
  (my-clean-env))

Then make a Global Button to run the Lisp code when activated. I called this button clean-env. The bottom part of my .env file looks like:

## other stuff
# <(clean-env)>
AWS_TOKEN=<old>
AWS_SECRET=<old>
AWS_SESSION=<old>

When I need to refresh my creds:

1. goto the AWS console, copy them into the system clipboard (which is a button on the AWS interface)
2. Open the .env file in my project
3. Go to the clean-env in the comment (which is a Hyperbole button) and hit M-enter.
4. The old values are replaced by the new values (without the export charaters).
5. Save the file

This seems simple but has saved me a lot of brain cycles.

Using eshell to run tests

I am on Windows for my current project and mostly because I use native Python in Msys2 Emacs, I can't run my Python test with project-run-test. This makes it hard to go exactly to the failed tests. But I remembered that I can use eshell, which I generally forget about. Then I can run the pytests within eshell and then use Hypberbole's action key to go to the failed test. Easy!

Navigating between variables, function call, etc.

This doesn't have anything to do with Hyperbole, funny enough. I stumbled onto someone talking about highlight-symbol, which does what it states -- will highlight all instances of a symbol that you are one. But it also has highlight-symbol-nav-mode, which let's you use M-p and M-n to goto the next instance of that symbol in a file. This is the kind of thing I've been looking for but didn't know how to describe it. Before used helm-occur for this type of navigation but that was very quickly replaced.

Editing the results of helm-occur

Speaking of helm-occur, I still use it and generally for tweaking a value or a pattern in my file. You can edit the search results directly, but I don't do it every day so I tend to forget the step. I finally wrote them down:

1. Find candidates with helm-occur (it's C-c h o in my setup)
2. Save the buffer with C-x C-s
3. C-c C-p to change to wgrep mode
4. Now it's a normal buffer! So I can make my changes
5. C-x C-s to save the changes
6. q to quit
7. You may want to kill the *occur* buffer or Emacs will keep it around and ask you about it later

It's amazing what kind of little things can really add up to make you more productive.

I've been doing a lot of work in SQLAlchemy lately and I'm continually impressed with how easy it is for the basic things but also how it can also handle complicated items. One of the things I have had to do again and again is to have a complicated query with a lot of joins. I could fetch the top-items and the all the other rows and then all the other tables... but then I would have a lot of round trips and a mess of objects. I could use a session.query(Table1,Table2...).join(...) type of structure but then I wouldn't necessarily the object that I wanted. What I discovered is that you can map a SQLAlchemy subquery to an object, just like you do with tables. This is very much like a view in a database...only that you are doing all the work in SQLAlchemy, and not have to let the DBA know!

I have a contrived example on Github that shows how this work. The tables.py file defines the tables... there is a Person table, and Authors and Fans point to Person. Author's have Books and a Fan can have many Books (favorite_books). I wanted to create a Fandom query, which will show what each Fan likes (even if they like more than one! The query.py file shows that:

from tables import *
from sqlalchemy import select
from sqlalchemy.orm import aliased

FanPerson = aliased(Person, name="fan_person")
AuthorPerson = aliased(Person, name="author_person")

fandom_q = (
    select(
        Book.id,
        FanPerson.name.label("fan_name"),
        Book.title,
        AuthorPerson.name.label("author_name"),
    )
    .select_from(Fan)  # explicitly say where to start
    .join(FanPerson, Fan.person_id == FanPerson.id)  # Join to fan's person details
    .join(Fan.favorite_books)  # Join to books
    .join(Book.author)  # Join to author
    .join(
        AuthorPerson, Author.person_id == AuthorPerson.id
    )  # Join to author's person details
    .subquery()
)


class Fandom(Base):
    __table__ = fandom_q
    __mapper_args__ = {
        "primary_key": [Book.id, Fan.id]
    }  # using Book.id and Fan.id as the primary ensures we see all the rows

The fandom_q is the actual subquery and you can see that it isn't trivial. You have to make aliases for the Fans and the Authors to get their names and manually do joins. The SQLAlchemy-table object is the one that we are familiar with but note that it's slightly different -- instead of a __tablename__ attribute that takes the name of the table, we have a __table__ attribute that we set to the fandom_q subquery. We also set the primary key to make sure we get all the people back. You can download the project, install the deps with poetry and run the table_query.py file to see it in action (and play around with it.)

Now, in our code, we can use Fandom as table to do queries (ie filters in SqlAlchmey speak) on it, just like a normal table:

    with Session(engine) as session:
        for x in (
            session.query(Fandom).filter(Fandom.author_name == "Stephen King").all()
        ):
            print(x.fan_name, x.author_name, x.title)

So from an API perspective, the Fandom object is just another table that you query from but an insert, update, or delete operation because it's a select not a table. If you try one of those, you will get an error:

sqlalchemy.exc.ArgumentError: subject table for an INSERT, UPDATE or DELETE expected, got <sqlalchemy.sql.annotation.AnnotatedSubquery at 0x105babbb0; anon_1sqlalchemy.exc.ArgumentError: subject table for an INSERT, UPDATE or DELETE expected, got <sqlalchemy.sql.annotation.AnnotatedSubquery at 0x105babbb0; anon_1

Regardless, I find this way to use and re-use a long, complicated query really powerful, because I use all the power of a SQLAlchemy table object.

I am not sure how I discovered git-worktrees but I'm sure glad that I did. It has made a big difference in how many branches I can work on at once, while still remaining simple

Git uses worktrees internally but you can use them to make clone from your local repo instead of clone from the origin. However, the big feature is that it shares history with your local repo. So you act on it independently (even push and pull from remotes ) but it's kind of an extension from the local "main" repo. You can think of it as a local fork, if you will, but with a shared history. A good explanation from the main git documentation:

A git repository can support multiple working trees, allowing you to check out more than one branch at a time. With `git worktree add` a new working tree is associated with the repository, along with additional metadata that differentiates that working tree from others in the same repository. The working tree, along with this metadata, is called a "worktree".

This new worktree is called a "linked worktree" as opposed to the "main worktree" prepared by git-init or git-clone. A repository has one main worktree (if it is not a bare repository) and zero or more linked worktrees.

My common use is this: I have a develop branch checked out locally. I then make new worktree and make a new feature branch in that new worktree to work on that feature. If a bug comes up that needs fixing quickly, I either make a new bugfix branch from my main develop worktree or I make a new fix worktree, push and merge it from that folder. Once it is merged into develop, I can use git merge <bugfix-branch> in my feature worktree. I'll give a step-by-step example below.

Note that I don't push, or pull develop from my branches around develop. That is because they are sharing the same history and metadata files. So it simulates doing everything in that folder...but you are actually in different folders. This allows you to have different branches in different folder yet still have the name metadata/history in each. One interesting quirk is that you can't have have the same branch checked out by two different worktrees. So if you are working on BranchA on one worktree repo, you can't work on BranchA in another worktree folder . This seems weird but worktrees are not stand-alone - they all share the same git history.

I like Tomups git-worktree scripts. They not only manage creating the folders, but they also copy over ~.env~ files, node_modules folders, etc. I'll give an example of how I use them with my general workflow:

1. I have a repo cloned locally and make sure that the develop branch is checked-out.
2. I run git wtadd ../proj-new-feature which creates a worktree in the parent folder above my project called proj-new-feature
3. I change to the proj-new-feature folder and make a feature branch just like normal: git checkout -b feature/fancy-new-thing
4. I start doing my normal development work.
5. Soon I'm told about a bug in the dev environment. I go to the folder where my develop branch is. It's a one-line fix, so I make a new branch there : git checkout -b bugfix/easy-bug
6. Once I fix the bug and push it up, I can do back to my proj-new-feature folder.
7. There I can merge in my bugfix/easy-bug branch easily with git merge bugfix/easy-bug. No pulling from remote, etc because the worktrees share the same history
8. When I'm done I push the feature/fancy-new-thing branch to remote
9. After the PR is merged in, I go back to development, pull in remote and then I can delete the feature worktree with git wtremove ../proj-new-feature

The cycle then continues.

With git worktrees I can have different features/branches active and they don't get in the way. It seems like a simple idea but really has made things easier in my workflow

I saw a few posts on r/emacs about Hyperbole, including some videos, and I -put it in a tab- save the URL to look at it later. When I finally got down to it and watched some of the videos and read some of the documentation... well I was intrigued but still confused on what it was and how I could use it. So I installed it and have used it for a while. I have to say I like it and it has improved my workflow, but I don't know if I can describe Hyperbole succinctly. So I'm writing this up, in case someone else is confused and perhaps I can get some clarity as to what the Hyperbole package is.

Let me make a quick disclaimer -- there is a lot of functionality in Hyperbole and I'm only covering the things I have found useful in the few weeks I've been experimenting with. There are things that I keep uncovering (even in the middle of writing this post).

The Magic Key Combination

Hyperbole introduces the key combo of M-return that becomes magic. The easiest way to describe it is "M-return does what you want in the context that you are currently in`". Hyperbole calls this the Smart Key and it's assigned both the "Action Key" and the "Assist Key". I'm assuming they did this because you can configure the "Magic Key" to be what you want, and the Smart and Assist Keys could be different. But what do I mean about "does what you want"? Here are examples from the Hyperbole documentation.:

    Smart Key - Magit Mode
     If pressed within a Magit buffer and not on a button:
      ACTION KEY
       (1) on the last line, quit from the magit mode ({q} key binding);
       (2) at the end of a line, scroll up a windowful;
       (3) on an initial read-only header line, cycle visibility of diff sections;
       (4) anywhere else, hide/show the thing at point ({TAB} key binding)
          unless that does nothing in the mode, then jump to the thing at point
          ({RET} key binding) but display based on the value of
          hpath:display-where.
      ASSIST KEY
      (1) on the last line, quit from the magit mode ({q} key binding);
      (2) at the end of a line, scroll down a windowful;
      (3) on an initial read-only header line, cycle visibility of all sections;
      (4) anywhere else, jump to the thing at point ({RET} key binding)
         but display based on the value of hpath:display-where."
q

    Smart Key - Org Mode
     When in an Org mode context and hsys-org-enable-smart-keys is non-nil:
     ACTION KEY
     (1) If on an Org todo keyword, cycle through the keywords in
         that set or if final done keyword, remove it.

     (2) If on an Org agenda view item, jump to the item for editing.

     (3) Within a radio or internal target or a link to it, jump between
         the target and the first link to it, allowing two-way navigation.

     (4) Follow other internal links and ID references in Org mode files.

     (5) Follow Org mode external links.

     (6) When on a Hyperbole button, activate the button.

     (7) With point on the :dir path of a code block definition, display the
         directory given by the path.

     (8) With point on any #+BEGIN_SRC, #+END_SRC, #+RESULTS, #+begin_example
         or #+end_example header, execute the code block via the Org mode
         standard binding of {C-c C-c}, org-ctrl-c-ctrl-c.

     (9) When point is on an Org mode heading, cycle the view of the subtree
         at point.

     (10) In any other context besides the end of a line, invoke the Org mode
          standard binding of {M-RET}, org-meta-return.

So these are just examples from the docs but it goes with what I am saying: - Are you in a Magit buffer online that can cycle visibility? M-return will do that. - Are you at the end of Magit buffer? Then M-return will quit the buffer - Are you on a OrgMode Todo item? Then M-return will change status - Want to execute code in a org-src section? Get on the begin_src and hit M-return

But probably the most important things that M-return executes are buttons -- both Implicit and Explicit.

Button Pushing

I think Buttons are the most important concept in Hyperbole and yet they took me a while to get my brain around. They aren't really that confusing but they are abstract concepts because a button can be literally anything. I will present some ways I have utilized them that have helped my workflow more in a short time than I would have thought. It's easier to start with Implicit Buttons and then talk about Explicit ones.

Implicit Buttons

These are by far the easiest ones to understand... you have some in your codebase now! It's pretty basic - this, for example, is an Implicit Button:

~/Projects/my-project/ReadMe.md

Just put your cursor on that file path, hit M-return and if the file exists, it will open it in Emacs. "Wait," you say, "that's just a text filepath?" And you are right! This is what makes it implicit because it's just text link! Hyperbole tries to open up or follow any string that looks like a filepath. But what if just used a directory?

~/Projects/my-project

If you hit M-return on that, it will open up Dired in that folder, if the path exists! So, as I said above, M-return does what you expect it to do.

So "Well great, hyperlinks! So sorta like a wiki." and that is true! But wait... there is more! Let say you have this:

"!poetry run pytest"

So that seems like a shell command... and you are 100% correct! If you put your cursor between the quotes (which are important), hit M-return, Hyperbole will start a terminal in Emacs (ansi-term, in my case), cd to the folder that file is in, and run the command. But here is another one, with < >.

<describe-function 'describe-function>

"Wait, " you think, "is that Elisp?" Yes, yes it is. Hit M-return and watch the magic happen. Yes you can run arbitrary Elisp functions with Hyperbole anywhere in your file. This opens up a lot of possibilities. I quickly came up with a solution to something that always causes me some frustration: in Django, a View can return an HTML template, but that template file could be under the current directory or in the main template folder. So there can be a journey to figure out where it is. So I used Direnv to set TEMPLATE_PATH to a list of locations where my templates could be:

export TEMPLATE_PATH=$HOME/Projects/my-project/project/templates:$HOME/Projects/my-project/apps/app1/templates

and then I wrote this little Elisp function

(defun mh/dj-open-template-file (filename)
  "Use Emacs to find and open a file named FILENAME under TEMPLATE_PATH."
  (interactive "sEnter filename to search for: ")
  (let ((template-path (getenv "TEMPLATE_PATH")))
    (if template-path
        (let* ((files (directory-files-recursively template-path (regexp-quote filename)))
               (file-to-open (car files))) ; Just take the first match
          (if file-to-open
              (find-file-other-window file-to-open)
            (message "File not found under %s" template-path)))
      (message "TEMPLATE_PATH environment variable is not set"))))

Now in my Django views, I can put in this implicit button:

# <mh/dj-open-template-file 'home.html'>
return render(request, "home.html")

Then I can hit M-return in that comment, and another window opens in the frame with the home.html open. Similarly, you can do keystrokes:

{C-x 3 C-x o }

So that will just simply run those keystrokes to split vertically and switch to that other window.

Explicit Buttons

The Explicit Buttons are similar only they are formal and look like buttons. They are really made to run Elisp function easily (or, at least that is what I think). To make a button, you have to use the Hyperbole menu, which I haven't had to mention yet. C-h h e will bring up the Explicit Button Menu. From there, c will be created. I will use my mh/dj-open-template-file as an example.

So I would create a button with a label called home because that is the file that I want to load. Next, I would choose my elisp function (mh/dj-open-template-file) and then it will ask me for a parameter (because the function requires one) and I put in home.html. Then where I was in the file says to write <(home)>. There! That is the Explicit Button! Now, anywhere in that file, you can type <(home)> that button will be a way to run mh/dj-open-template-file to find home.html. Note that I said anywhere in that file... from what I have found the explicit buttons only exist in the current file. So if you want to use the same explicit button in another file, you have to re-create it with the same process. It's because of this I find explicit buttons to be less useful than implicit ones (why remake the same buttons over and over again?).

From a utility standpoint, the information about the explicit button is written to $PWD/.hypb in the same directory. It's not editable (and not meant to be) but that enables the buttons to persist after you restart Emacs.

Window/Frame control

This is something that I didn't think I would use much, but I found that I really like it. C-h h s w will bring up a menu to tweak all kinds of things about the current window -- size, position in the frame, change frame, and more! The similar C-h h s f then you have a similar thing for the frame. I find the Frame controls a lot more useful than the Window one. You can also create a grid of windows in the frame with the @

A lot is going on in this so check out the docs. But what is nice about this is that it makes small adjustments very quick and easy without taking your hands off the keyboard. Also makes it easy to load the file in a new frame, which is a command I can never remember.

Window Config

This name is similar to the above but it's very different. In essence you can save the opened buffers and the window formation to use later. So, if you are like me and get 3 different windows just so and then mess that all up when debugging, this is the fix for you! You can save it into a "window-ring" (that is my term) where you can just pop off the configs. Or you can save it by name and then just restore it by name later. You get to this by C-h h w.

There is a lot to be said when you have a bunch of windows saved in a circular ring and you just pop them off until you find the right one. Or save one by name so you can get at it again later. The big caveat to this, however, is that it doesn't persist. So you can set up a bunch of saved windows but lose them when you restart Emacs. Generally not a huge deal but I was excited to save some window configs in my Emacs configuration but no cigar.

My Config

There is a lot that I liked here and some that I didn't. And then there are some that I tweaked. :)

One of the things that didn't work is M-return didn't follow the definition in my Python code. I figured out that it had to do with Hyperbole assuming that I was using Etags and my Prelude config is using anaconda-mode. I did some digging and figured out what the function in Hyperbole is smart-python-tag and then I put some elisp advice around it:

(defun mh/override-smart-python-tag (original-function &rest args)
  "Conditionally override `smart-python-tag` to use `anaconda-mode-find-definitions` in Python mode."
  ;; Check if we're in python-mode and anaconda-mode is active
  (if (and (eq major-mode 'python-mode)
           (bound-and-true-p anaconda-mode))
      (call-interactively 'anaconda-mode-find-definitions) ;; Call anaconda-mode-find-definitions interactively
    (apply original-function args))) ;; Otherwise, call the original function


(advice-add 'smart-python-tag :around #'mh/override-smart-python-tag)

Now M-return works in Python like expected.

The next things are less drastic but also important to me. I liked the functionality with Windows and Frames but I didn't want to have to dig into the Hyperbole menu each time (C-h h w s to save a window? No thanks!). Of course, this is Emacs so I can configure things how I want. I do like Magit's Transient interface (and Hyperbole's menus essentially work the same way) I decided to dip my toes into making my own Tranient menu. And it was a lot easier than I thought:

(transient-define-prefix mh/hyperbole-transient ()
  ["Hyperbole commands"
   ("s" "save window" hywconfig-ring-save)
   ("y" "yank window" hywconfig-yank-pop)
   ("w" "window control" hycontrol-enable-windows-mode)
   ("f" "frame control" hycontrol-enable-frames-mode)
   ])

(global-set-key (kbd "C-h j") 'mh/hyperbole-transient)

Yes, I assigned it to C-h j, because it's close to Hyperbole. And it was an open keystroke on my system. Now when I use that, I have a list of Hyperbole options that I use the most.

Related, I wanted to be able to save a lot of windows and just move through them. Both my Transient and the Hyperbole menus quit after you get a window off the ring. What I needed was a Hydra -- something I had tried before but couldn't figure out a use. But I did now!

(defhydra mh/window-config-hydra (:color blue :hint nil)
  "
^Window Configurations^
-----------------------
_s_: Save Window Config  _y_: Yank Window Config
"
  ("s" hywconfig-ring-save "save" )
  ("y" hywconfig-yank-pop  "yank" :color red))


(global-set-key (kbd "C-c m") 'mh/window-config-hydra/body)

I have a hard time finding open key sequences but C-c m is fast and was unused. Now I can simply type that and hit y until I find the window that I want.

You can see my Hyperbole specific things (including my two Django-based functions I use for buttons) on my Gitlab page. You can look at the rest of my configuration as well, if that is useful.

Epilogue

When I started this post, I didn't plan on a 2,000+ words about Hyperbole but there is so much to talk about and there are things that I didn't even touch on like Koutliner, which is an outliner system sorta like OrgMode, and Hyrolo is a record management system (think Rolodex but apparently for anything). Most of these are because I haven't needed to dive into it.

If nothing else, I think the Hyperbole is a big package but you can use whatever piece you want. I use Hyperbole as a way to reduce some cognitive load while working, like making it easy to link filenames to their locations, quickly jump between windows, etc.

I now have a pretty structured way to configure my projects -- I use Nix and direnv and then the language-specific files. But when I setup a new project, it's been a bit of a mess. What I used to do is something like:

1. Find a project that is similar to the new one I'm making...Python, Node, Java, etch
2. copy that old project's .envrc and shell.nix file to the new project. And probably the docker-compose.yml and maybe assorted other files
3. tweak as necessary
4. Find other file and tweak as necessary

I really got tired of doing this. Sometimes there was nothing to tweak but I looked anyway and sometimes I didn't have quite what I wanted. I said to myself "Surely someone else has solved this problem!". I used Lazybones many years ago...and it hasn't been updated in a long time (maybe since I used it last!) so I decided to keep looking and somehow I stumbled onto CookieCutter, which pretty much solves all my problems.

There are a lot of CookieCutter templates out there but I wanted something of my own. Luckily it's easy to make your own templates. You can check out my Github Repo. I found them pretty easy to make though there are some things I don't like about it -- like using a Jinja template to generate a folder name, which means you have folders named {{cookiecutter.project_name}}. For me that is weird to see on the command line but it's a small nit and one I've chosen to live with.. Anyway, not only did I put in Nix and Direnv configs but also docker-compose.yml with Postgresql (my db of choice) and default script names, with the db name and the project name configurable.. I also do a git init at the end, because I will put it in git eventually (though not necessarily push it anywhere).

Basically, you store your templates, er, cookie cutters, in $HOME/.cookiecutters and set each project type as a folder.. Or fork mine and check it out into that folder. I installed cookiecutter through my package manager and that makes the exe really long (cookiecutter-3.10), so I made two aliases:

alias new_proj="cookiecutter-3.10"
alias list_proj="cookiecutter-3.10 -l"

So if I want to know the names of my template types, I type list_proj. When I want a new project, I use new_project [type], and then follow the prompts. It now takes me 3 seconds to start a new project source instead of a few minutes and, even better, less cognitive load. Just find what template that I have that works best and new_proj type and I'm ready to get started.

My Solution Log History

As a software engineer, I have been a believer in a solutions log -- a place where you record the solution to a really sticky problem you had ,or a surprise that you didn't expect. My solution log started in Evernote, became a bunch of OrgMode files, then got filtered down and I started writing them into large OrgMode files based on subject, and then I discovered the idea of a Zettelkasten and put everything in org-roam. More recently I discovered Obsidian and converted all my files from org to Markdown and moved them into an Obsidian folder.

Note that these files are used a bit differently than a "normal" Zettelkasten note. Most of the time a Zettelkasten system is used as an insight for research and putting a paper or article together. Most of my notes are used for reference -- like dealing with the BOM character when parsing CSV files in Python! I have it written down so I don't have to memorize it. I.e. use my internal brain space for something else.

Regardless my Knowledge Base (what I call my personal solutions log) was mixed in with my other Zettelkasten notes all in one folder. It got really unwieldy. To me, the Knowledge Base was a different -- it was mostly reference material and, sure, things linked to each other were mixed up with insights I was documenting. I needed a folder structure and felt that I wasn't going to come up with a good one on my so I started searching. And I found Nick Milo's LYT Kit which gave me some new things to think about and a workable arrangement.

Linking Your Thinking (LYT) is Nick Milo's paradigm for Zettelkasten and has some great videos to show you his ideas. Anyway I decided go from my "everything in my folder" idea to something more rigid like "LYT".

My Setup

I won't go into too much detail on the set up of LYT, but I will say that I created a folder under "Spaces" called "Knowledge Base" and started to copy all the files from my old vault into my LYT vault and then tweak them. After a bit of doing that I decided to stop and instead I made a new space called 'Unruly Forest', copied all the remaining Knowledge Base files in there. And then, when I read one again, I can take a minute to move it to the Knowledge Base space. This makes things a lot easier and I still can find the note that I need when I need it.

One concept that really drew me into LYT was the Map of Content, or MOC in LYT-speak. Basically it's a common file that all you link to move up. But really they are so much more than that. Bob Doto describes it well:

MOCs behave as work stations, places to examine how notes interact. They're far more than records of already established connections. Instead, MOCs are places to challenge notes, to see whether notes need to change, cleave, or resolve. Rather than being a representation of previously made connections, MOCs are where connections are made anew.

My Knowledge Base folder has a file called "Knowledge Base MOC" that all my knowledge base files link to in the top as an "UP" category. This makes navigation easy -- I can click 'up' to navigate to the MOC. What do you put in that MOC file? Anything you want! I'm not to the "change, cleave, or resolve" in my Knowledge Base yet -- I simply put in a DataView table in my Knowledge Base MOC that displays my post used tags in my Knowledge Base and displays the count of each one. It looks like:

TABLE length(rows.file.name) as numfiles  FROM "Spaces/Knowledge Base" 
flatten file.tags as tag group by tag 
sort length(rows.file.name) desc limit 10

As I started using my setup, I found a lot of notes still in the large files that I originally wrote in Org in Emacs. Again — instead of breaking those big files up, I instead use those file as kinda Subject MOC. For example all the Django notes I make link back to the main Django note. That helps keeps things “linked” together. In that file, I have a DataView table created with a tag, like so:

table file.mtime as Updated from #django  sort file.mtime desc

I haven't refactored my "large" files yet.. and that's ok! I will when I get to them.

The next issue was search... I mean this is a Knowledge Base so I need to search the contents of the files to find what I need. I was struggling with search for a long time but recently discovered the Omnisearch plugin which happens to work really great for finding the right file.

Actually writing the note

I used to write really short snippets and then try to link to notes that this item reminded me of. But I stopped that and tried to make things more conversational - like explaining the situation, or if I think this solution could be better. The guideline I follow is "write" a blog post to yourself". So I kinda write my note like I would have liked to find in a blog post but I'm coming at it assuming that someone knows the context. I got this idea from Andy Matuschak. He didn't' state it like I do but perhaps better

When it’s a topic I understand well, I can write notes for both myself and an audience simultaneously. But that sometimes produces the false impression that I can pull this off all the time! To avoid that false impression, I’ll write notes for myself “by default,” and only “opt into” writing notes for an audience explicitly.

Even though I don't necessarily write these notes for connectedness and find-by-exploring, I do try to make it so I could. I make sure that I do these things:

• tag with the tag of the project I'm working on when I get stuck
• Link it to the Project MOC of what I am working on
• Otherwise link heavily -- for example, make a link to Python if the note is anything to do with Python. Link it to JavaScript if is it javascript related. Both of these are really Subject MOCs
• Add thoughts to the note. I.e don't copy straight out of StackOverflow

This seems like a lot of rules but really they are very simple to follow. Why? Because my note is conversational but having a conversation with myself. I avoid the easy method simply of linking to and pasting from the Internet and walking away. I mean, I do that but I wrap some context and opinions to it -- and link to them freely. I feel that I can have a deeper understanding of what I just did as well as having methods to find it when I need it -- even if it's not the exact solution I'm working for! Sometimes I find something similar enough in my knowledge base to a solution to a problem I didn't think was similar.

Like most of my Zettelkasten workflows, this is always a work in progress. Always more tweaks or perhaps removing some layers. But now I have something that works for me, even after carrying files from one place to another. I do plan on keeping Obsidian for a long, long while however.

About a year ago I started playing with Nix and it has become as essential as Docker and Emacs for my local development environments. I switched from a world of direnv and virtualenv based environments to one with direnv and nix. The Nix part seems complicated but it's really not. I really just uses a sliver of what Nix does and I'm fine with it. See what my friend Marty wrote for a good discussion on what Nix is, how to install it and several reasons why it's awesome.

As I said above, my setup is pretty basic but it does everything I hoped it could do.

In the Direnv file I have usual environment variables, maybe a couple PATH settings for scripts and then at the end:

use nix

and... that is it for direnv.

Then I make a file called shell.nix that creates the python environment

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {

  buildInputs = [
    pkgs.python38
    pkgs.poetry
    pkgs.black

  ];

}

That is a simple one. Here is some that is Python and a few more things:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {

  buildInputs = [
    pkgs.python3
    pkgs.poetry
    pkgs.python3Packages.pip
    pkgs.pgcli
    pkgs.postgresql
    pkgs.libxml2
    pkgs.libxslt
  ];

}

This is a lot like previous one, but with some Postgresql helpers and the XML libraries are there because some Python libraries I use needed them.

Then do the direnv allow ... and you have your environment setup. Nix handles the install and maintance of what I specified in my shell file. Which you can see when you look at the Python executable:

➜ which python
/nix/store/df2vs5yxfwj2flslf6ng0a0w8swdwgp7-python3-3.9.9/bin/python

After that I go into the folder and/or open my project with Emacs.. and my environment is set up. I can use plain Poetry commands and it will be a "normal" Python project... because it is a normal Python project! With just the executables setup for Nix.

My examples above were all for Python but it's really for any type of dev platform. Here is a React project that I have setup:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {

  buildInputs = [
     pkgs.yarn
     pkgs.nodejs-14_x

  ];

}

No messy rvm or anything -- just set your version and let Nix do all the work

Every Saturday, I try to wake up before anyone else. With the house quiet, I make some coffee, shuffle to my office and shut the door. I open my bullet journal, my Obsidian notes, and Todoist list and start the most important 20 to 30 minutes of my week. I call it "my Weekly Review" but it's really more of a "Weekly Review and Planning session". I helps me figure out if what I did last week helps me with my goals and then what do I need to do for the next week to get closer to my goals.

I know that a few people will read this and think that I am really good at this process and, to be frank, I feel like I am really bad at it. I really started it because I felt I wasn't getting enough done or not working on the right things. I get into this state of being that I call The Rut but what I am learning is that the The Rut is really a sign of burnout called productivity dysmorphia, which is simply defined as "a decreased sense of accomplishment". So, in my quest to accomplish more, I'm realizing that I'm really accomplishing enough. Regardless, it's a process and I'm still in the midst of it. This planning process has helped me feel that I'm getting enough done and am focusing on the right things.

Let me be clear about something else -- I didn't come up with this. Most of this is from this GTD Review checklist (pdf) which I got from this great comment on Reddit. It's really those things that helped me piece this together

I'm pretty specific about what software tools I use for this. I have used and struggled a lot and have a pretty good system going now. But hey this is always evolving. Right now, as I write this I am using

• Todoist as the todo-list
• Obsidian is where I put notes -- both project descriptions and quick notes/links for later. I used a plugin to easily make a Todoist item from a note
• Fantastical for an overview of my events as well as my tasks
• Friday.app -- the newest one to the list. I use mostly as a daily dashboard/ aggregator of my various calendars and my Todoist list

I also use old fashioned pen and paper. It's faster and things really stick to your mind better.

You ready? Alright -- let's get started .

1. Get Clear

Get “IN” to Zero Process

I have a tray on my desk where I set papers as the week goes on -- mostly mail but sometimes other notes that pass my way. I take care most of it during the week but on Saturday I sort through what is left and finish taking care of it. Some I put back -- maybe it's a document that needs physically returned so I add to to my Todo list. But other items like tax forms are filed in my cabinet, dog vaccinations are scanned and saved, etc. etc. By the end of this Zero Process, the Inbox Tray is dealt with, if not ideally empty.

I also go through my notebook to make sure I didn't leave anything there that isn't in Todoist (for something I need ) or Obsidian ( for a note I want to remember). I'm usually good at getting those out of my notebook towards the end of the day that I write it, but I want to make sure.

Then I go through my quick notes in Obsidian. I have a series of Daily Notes that I used during the week. I quickly share a link or sometimes just a thought about something into a daily note. During my review, I do a quick review of each note and figure out if I really need it or not. If I do, I put it in it's own note, or merge with another and make a Todo so I get back at it later. Or I decide that I don't care after all and simply delete it. Once I finish dealing with the contents of the note I immediately delete the note itself . I move on and deal with the next day's note. If I spend more than five minute on an entry, it's way too long.

Empty head

In a notebook, I put in new ideas, new projects and "This is what I need" or "This is what I am waiting for". Why a notebook and not in Obsidian ? Because it's faster to use short-hand to jot the note down quickly and then move on.

Ironically, this doesn't generally take me too long to do. As stated above, I generally have a notepad by me when I work for quick notes or I keep track of ideas in Obsidian and Todoist as the week goes by. But sometimes I don't have things down like I should have -- this is my last ditch effort.

2. Get Current

I have a different notebook specifically for this part - totally separate from the notebook I used during the week and used above . I like having the reviews as a series of pages, one after another for reviewing them.

Review Action Lists

I use an IFTTT Recipe that logs each Todoist task that I complete into a Google Sheet. When I look at the spreadsheet, I make note of the Todoist project or tag to give me the context that it was completed in and I put a tally mark behind it in my notebook. The definition of "context" is squishy -- sometimes I remember the why of it and one item gets tally marks by two contexts, because that made sense to me. I also don't count routine items like "Take out the Trash" because they aren't necessarily part of a larger goal.

This next part is perhaps the most important thing in this whole review -- I look at my tally marks and compare it with my goals from the review prior. Did my completed tasks match what my goals were? If not, why not? Do I need to adjust something from this week to the next?

Review Previous Calendar Data

This doesn't take me too long because I have notes already from previous meetings but sometimes I remember things I missed. It's usually in my notes that I need to flesh them out. Anyway if I find I have other things todo, I make Todoist items on them and schedule them.

Review Upcoming Calendar Review

I have a love/hate relationship with calenders. Besides the work calendar, which things can creep in, our family shares a Google calendar for all our appointments. So things can sneak in there too, or there are events that don't really effect me. Sometimes I usually just need to know something is happening. I used to put each upcoming Calendar item in it's own, separate Todist project but Friday.app has stopped this -- now it's in my daily dashboard so I will see it. Friday also solves a problem that my review couldn't solve -- it also reports new calendar events that gets put on after this.

This is also the time I use Fantastical to look at my week and try to even things out. Fantastical really is the best way to look at Todoist items for a week at a time. If I see some days are heavily-loaded, I used drop and drag them around to even things out. In case you are wondering, yes, I do usually make a Todo item and throw a due date on it that sounds good at the time. This is when I adjust those items to fit into deadlines, to avoid being overburdened, and just general sanity's sake.

Review Project (and Larger Outcome) Lists

This is when I look at each project and evaluate if things are schedule by priority, what is schedule on the right days or even if anything is still valid. Also this is when I set goals Wisdom tells me that I should make a goal or target for every project every week, but I have found that to have disappointing results. I tend to pick out two or no more than three goals and put those in big letters in my notebook. And then make sure (again) that my priority list matches them, and adjust accordingly.

3. Get Creative

This is probably the step I'm the worst at -- some of it because I do this through the week (a la my daily notes) and also by now my brain is tired by all this reflection. But this is where I am supposed to brainstorm and put down wild ideas. I do look at at this list as the week (or weeks) go by and see if there are any patterns that emerge. Those patterns are the goals I turn into projects.

Review Someday Maybe List

This is where I review projects that, yeah, maybe I want to schedule a few things for, some I decide are no longer decided are no long relevant.

Be Creative and Courageous

This is supposed to be "Any new, wonderful, hare-brained, creative, thought-provoking, risk-taking ideas" to added at a high level. Some of this was done as the week went on and reviewed in my notes, others are ridiculous. That said, I will take some down.

It's pretty simple to figure out that Cortex is my favorite podcast. In fact, it's really the only podcast I make sure to listen to when the new episodes come out. Every year they review the apps they are using on their phones (and Macs) and call it State of the Apps. After they published this year's I thought about making my own. If you listen to them very much you will realize that I have been very much influenced by them this year.

Todoist

This isn't a new item but it's a central item of my life. Really Todoist runs my life... there are very few non-routine tasks that aren't in Todoist. And really most of my routine items are there too. With my own weird version of GTD, Todoist lets me rapidly put in my task with a due date (sometimes it arbitrary ) and then I don't worry about it until later. I really talked about Todoist here and I basically run it the same way -- that filter that I use has really worked out well.

In the next apps I talk about next, you will see a common theme -- I use some sort of Todoist integration in all of them. That is how important Todoist has become to me.

Toggl Timer

This is a Cortex-influenced decision and I really don't know what took me to long to try it. Basically I tell Toggl a task and a project and it starts the timer. And when I'm done, I stop the timer. And then I can do reports, etc from this data. And make my timesheet for work from this. But this works so very well and it's cross-platform. Literally -- I can start the timer on my Mac via the web and stop it on my Android phone via the Android app.

Also, Toggl integrates with... well, just about everything. If you install the Chrome plugin, then you can tell it what webapps you want to integrate with Toggl. So if you use Asana and Todoist integration, then you have a Toggl button on tasks there. Click it, set the project on the Toggl dialogue in the app and the timer is started. Clean, easy efficient.

Obsidian

Obsidian is the notes app that I have been waiting for. Basically I can have the same notes on all my devices and edit them and they are sync'ed. Furthermore they are all Markdown files -- so just plain text! And the files reside on my system, not in the Obsidian cloud! And search works! And I can link between files! It's really marvelous.

I'll throw out that this isn't perfect... I would rather have Emacs everywhere and write in OrgMode . Honestly I do think that Orgmode is a better format for this than Markdown but (at least on my Mac) I can use Emacs on my Obsidian files and they work just fine. The d

Obsidian has their own sync service but you don't have to use it... I don't because... well partly because I'm cheap and partly because I have to do things complicated. Between my Mac and iPad, iCloud Drive works nicely. To my Android phone, it's more work but I finally ended up with Syncthing between my Mac and phone. So, basically my Mac is the "server" for both of my mobile devices.

Obsidian by itself is good but really what makes is great is all the plugins. You can really customize it the way you want to work (so really that makes it like Emacs) and your plugins and configuration syncs over! So I install the plugin on my Mac and that plugin is sync'ed to my other devices. Once in a while I've had to enable it on my devices but the configuration is there when I do. I can live with that.

I'm not going to get in deep to my Zettelkasten ideas here... but Obsidian really makes it work.

Plugins that I like:

• DataView -- which lets you create queries on your nows and displays tables on it. I use it for Dashboards to get at things easily.
• Daily Note -- Generally I sometimes I find a link that I'm reading and think "I want to read this later" so Daily Note gives me a quick place to dump it into for later. Sometime later I will go back, read it, take some notes and put that into it's own note.
• Todoist plugin -- You can display Todoist items in Obsidian but I don't use it for that. Instead I use a shortcut that makes a Todo from a note. I use this for projects where I make notes of what I need to do and then make a Todo out of it -- my resulting Todoist item links to my Obsidian note so I have all my context... and after I finish, I still have access to what I was thinking or doing, in case I want to go back to it.

I use many more but the three above are really the essential to my workflow.

Spark

I'm a really bad email user.. I tend to look at a subject and perhaps a preview on an email and leave it alone -- sometimes I take action on it (without opening it up) and sometimes I know it's something I don't need. In either case, I don't even open it up. This is what causes my Gmail to have 30K unread messages (no that is not a joke). This worked... fine? Okayish? for a long time but then lately I realized that I missed some email that I really needed to look at. So I needed to change my bad email ways.

I knew a few people that used Spark and Myke on Coretex has been recommending it and so I decided to give it a shot. And again... what took me so long? I love out is auto-categorizes my messages so I know what I need look at and Notifications and Newsletters that perhaps I can leave for later (or just delete).

Lots of people like the "Snooze" feature of Spark but honestly.. I'll just make a todo out of it's Todoist integration. Once setup (all you need is your API key), you click a button, put in your Todist info in the pop-up and your todo has a link to the email in Spark. I wish the Spark modal did natural language or, at the very least wasn't so clunky putting in the right project or a label.

Tabnine

Tabnine is different than the rest.. it's not an app, per-se, but it's a code-completion helper. Basically it's a little server that runs and connects to plugins to your text editor and/or your IDE and it supercharges your completions. Like just not auto completing variables but in context, lines of code. I've had it complete full streams calls in Java code with no problem. And it works just as well in Intellij as it does in Emacs. This is one tool that works in the background and I really kinda forget about it, yet I use it all the time.

Vivaldi

I had used Firefox for a long time and I liked it sync on my desktop, Android phone, and iPad but I had problems with a few important sites... well, some of it was with the security settings I had more than the browser than the browser itself. That said, the world is getting more and more embedded with Chrome. So I started looking for an alternate. I had read things about Vivaldi but hadn't tried it in a long while. And really it's fantastic. It does lots of things with tabs and my current favorite tab feature is to tile tabs in the windows which is nicer than you may think. I also love Quick Commands, which is like a Spotlight search within the browser. Bring up a tab, bookmark, command, etc with ease.

A little while ago I ran into someone and we ended up chatting about this blog. He asked an interesting question – "Why do you write the blog?". I didn't have an answer then and I'm not sure that I have one now, but I'm writing about it anyway.

A day or two after that conversation I read Katie Baker's article about the death of Google Reader and the real Internet and it reminded me of the "Why blog?" question. They seem related and really that's the real reason I am writing this.

Let's start with this – I've always fancied myself a writer. Even in my grade school days, I did well on my writing assignments and, dare I say, I was good at it. At times I fancied that I was going to write novels someday. Well that feeling is gone but it's still something I enjoy and something perhaps I will always do, in one way or another. Sönke Ahrens wrote in his book How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking – for Students, Academics and Nonfiction Book Writers:

If there is one thing the experts agree on, then it is this: You have
to externalize your ideas, you have to write. Richard Feynman stresses
it as much as Benjamin Franklin. If we write, it is more likely that we
understand what we read, remember what we learn and that our thoughts
make sense.

so, in a huge sense, I write mostly for me. To remember things, to process informartion, to sort things out in my brain. But do I write blogs posts for me? Well… sometimes? But that is something different, which I'll probably get to in a different post.

In another sense, I started my blog simply because I like the other blogs that were out there, especially help in more technical issues but also hearing about things from different people's perspectives. I think that was why blog readers were essential. You found a blog that you were interested in and you followed it. Then you only had one place to go to. It wasn't unusual to have 100 blogs to follow. Of course now I still follow blogs. Very few are updated daily, some (I guess like mine) aren't even updated monthly. But that's fine. Perhaps the most important reason that I write is to be a part of the larger web conversation, even if there really isn't one and I'm typing this to the ether.

I remember thinking once that I didn't need Google Reader anymore – I had Twitter and I didn't need to follow anyone else but I could see what the current conversation was. How naive! The early days of Twitter were fun (anyone else remember TweetUps and Tweet Lunches?). But that didn't last long… soon there was so much noise and very little content. I rarely use Twitter anymore. But how am I going to learn and be exposed to new ideas if there is so much noise in the "noise" in the public forum. Now I'm back to using as RSS reader a lot and even though blogs aren't as active as they once was, there is some great stuff out there.

I don't blog because I want to be a professional blogger. Honestly, I don't care about traffic…I don't even keep track. Maybe no one will read it. I'm fine with that. Maybe this is more online journal for me than "wow someone will find is useful." But I do want someone to find it useful. Maybe not you but perhaps someone still stumble on to this and think "wow, that helped me more than StackOverflow" or "I never thought of it that way."

Basically – I blog for me and hope that other people find it useful as well.