Can we fix the default colorscheme ?

[neutaaaaan]

This post is motivated by this now closed pull request from chrisbra.
I figured that if one of vim's core devs had such different expectations from vim's colorschemes than what they actually are, and have been for 2 decades, issues such as the one highlighted here are likely to have gone unnoticed.

What we currently call "the default colorscheme" is the byproduct of several things :

• the various broken implementations of seemingly unrelated colorschemes that share the same name
• vim failing to autodetect the user's terminal background color over ssh and defaulting to the light version
• distro maintainers doing the usual distro maintainer shuffle to work around perceived issues with vim's default config

As I myself get to use "the default colorscheme" pretty much all day long at work, through various environments, versions of vim, and distributions, I've grown to absolutely abhor it.
Not because it's bad per se, but because the user experience is utterly inconsistent.

There's very little that could be done about the 24bit version without effectively breaking it. It's trash, and I don't even care.
The terminal versions, especially the 256c one, could use a lot of tweaking, to fix design decisions that only make sense within the context of the default xterm palette, or to restore functionalities that break over ssh.

The goal of this post is to figure out if a third colorscheme could be implemented within vim as a fallback when it fails to autodetect the background color. This colorscheme would be based upon the light version, which is the actual default colorscheme.

I believe it is viable, because this version of the colorscheme does so many things wrong that they right themselves.

I'll use screenshots from an xterm analogue, quiet (itself essentially a more legible xterm analogue), a gruvbox analogue, and because I think it's fair to acknowledge that there is no silver bullet, a solarized analogue.

All the makeshift modifications displayed in this post will use similar design rules to what I've used in quiet or sorbet :

• I don't mind using different palettes for elements that I don't expect to stay on screen very long, or that are contextually different enough from the buffer being displayed.
• There is no code highlighting within diffs, selections or searches, as this only creates issues

I use rxvt-unicode and tmux, both configured proprely to use the extended 256c palette.
Here are the aliases used to get vim into the same state it'd be over ssh without a config file:
vim -u NONE +"set nocp" +"syntax on" +"set background=light"
vimdiff -u NONE +"set nocp" +"syntax on" +"set background=light"

Most of the main highlighting palette uses ansi colors, except for spotfixes in places where I believe the default xterm ansi colors really didn't work out, such as the Statement group, and diffs.

Here's what the light version of default looks like, peering at a spot in a random file that contains a bunch of statements:
xterm256c
quiet256c
gruvbox256c
solarized256c

The fact that bad colors were picked 30 years ago for one terminal emulator's ansi palette shouldn't hinder vim users in 2023.
Remarkably, whoever made this tweak, effectively did what most colorscheme/theme authors seem to do : use orange as a proxy for yellow.
Here's what this looks like with Statement pointing to ansi3:
xterm256c_statement
quiet256c_statement
gruvbox256c_statement
solarized256c_statement

Diffs are plain broken.
They rely on the extended 256c palette but still use the ansi palette for DiffText, which ends up being completely illegible.
They also try to preserve syntax highlighting inside diffed elements, which simply cannot work in this context.
xterm256c_diff
quiet256c_diff
gruvbox256c_diff
solarized256c_diff

There's little point trying to move away from the 256c when it comes to diff. Having its ozn palette is fine, it's the behaviour that's busted and ought to be fixed:
xterm256c_diff_coherent
quiet256c_diff_coherent
gruvbox256c_diff_coherent
solarized256c_diff_coherent

Visual selections are broken. We can't rely on fg/bg being different, or even different enough from ansi7 to be legible.
This happens to me on a daily basis:
xterm256c_visual
quiet256c_visual
gruvbox256c_visual
solarized256c_visual

Fixing this requires bolting selections down to a single color:
xterm256c_visual_coherent
quiet256c_visual_coherent
gruvbox256c_visual_coherent
solarized256c_visual_coherent

Search results are kinda crap for similar reasons:
xterm256c_search
quiet256c_search
gruvbox256c_search
solarized256c_search

And the situation can be remedied somewhat by bolting the color down.
xterm256c_search_coherent
quiet256c_search_coherent
gruvbox256c_search_coherent
solarized256c_search_coherent

This is far from being complete and is just meant to get the ball rolling.

[romainl]

Just to be clear, when you say "gruvbox256c", you mean

• the terminal emulator's palette is gruvbox,
• $TERM is set to xterm-256color,
• and you are implicitly using :colorscheme default,

right?

[neutaaaaan]

@romainl

For all intents and purposes, yes.
The only strange thing I'm doing is forcing -u NONE to make sure I don't sideload a distrowide config on my own machine, and force background=light because vim is able to tell I'm using a dark or light background locally.
I've tested the same elements over ssh, without having to do any of that, and had similar results.

[romainl]

Some of those issues have, I think, been demonstrably broken for a while. As a matter of fact, I've had various iterations of the following fixes in my portable minimal vimrcs practically since I started using Vim:

hi Search cterm=NONE ctermbg=yellow ctermfg=black
hi StatusLineNC cterm=bold ctermbg=darkgrey
hi Visual cterm=NONE ctermbg=white ctermfg=darkblue

I am not sure when, exactly, Visual became unusable with syntax on, but one would think it would have been noticed by someone… and fixed, by now. Oh well.

The relevant section of highlight.c is relatively readable so let's discuss which color combo makes the most sense for each of the problematic highlight group and make a PR.

[habamax]

I think we should try to fix it in highlight.c bite by bite.

Visual indeed annoys me the most -- combined with quite unreliable dark/light detection, it is plain unreadable:

@chrisbra what do you think?

PS, even with set bg=dark it is quite bad:

[chrisbra]

Well, what exactly would you suggest to fix here? I had a similar issues with the Visual highlighting before: vim/vim#8247 But nobody saw this as an actual issue 🤷

[habamax]

Well, what exactly would you suggest to fix here?

Make Visual non-transparent with bg and fg setup instead of using term=reverse (at least for TUI), we can try different variants of bg too.

[romainl]

@chrisbra the direction we took with the remakes and the new colorschemes is to explicitly specify Visual's foreground color/style.

The upside is simply that the visual selection is readable in any context, with or without syntax highlighting.

The downside is of course that, the visually selected area loses its syntax highlighting, which some users might find discomforting in some cases.

From our point of view, that of people who dread using visual mode in default because it is so obviously broken, the upside dwarfs the downside by a few orders of magnitude.

FWIW, here are two screenshots of default, the first in 8c and the second in 16c, with the default xterm palette:

Visual is an abomination in 8c and plain unreadable for some of the foreground colors used in 16c. That is how Vim looks by default. No wonder there are literally thousands of third-party colorschemes available. Yes, you sort of see the syntax highlighting but come on.

Changing the style of Visual to some kind of black on white makes it a lot more usable:

Note that I haven't put a lot of effort into this so some other combination might work better.

For reference, here are some of the remakes, some of the new ones, and lastly one of mine, all with readable, working, Visual:

Delek:

Ron:

Zaibatsu:

Habamax:

Malotru:

[chrisbra]

Okay, I guess I am too used to the current broken highlighting, I actually quite like, that the syntax highlighting is still visible in visual mode, but I can see why this causes problems. I am even used to Vim using light background with a default putty colorscheme (which means it is using black background).

So the following patch should remove the reverse stuff and set explicit background colors.

diff --git a/src/highlight.c b/src/highlight.c
index 31c3280e8..a29726865 100644
--- a/src/highlight.c
+++ b/src/highlight.c
@@ -220,8 +220,8 @@ static char *(highlight_init_light[]) = {
     CENT("SignColumn term=standout ctermbg=Grey ctermfg=DarkBlue",
         "SignColumn term=standout ctermbg=Grey ctermfg=DarkBlue guibg=Grey guifg=DarkBlue"),
 #endif
-    CENT("Visual term=reverse",
-        "Visual term=reverse guibg=LightGrey"),
+    CENT("Visual ctermbg=DarkGrey ctermfg=White",
+        "Visual ctermbg=DarkGrey ctermfg=White guibg=Grey42 guifg=LightGrey"),
 #ifdef FEAT_DIFF
     CENT("DiffAdd term=bold ctermbg=LightBlue",
         "DiffAdd term=bold ctermbg=LightBlue guibg=LightBlue"),
@@ -309,8 +309,8 @@ static char *(highlight_init_dark[]) = {
     CENT("SignColumn term=standout ctermbg=DarkGrey ctermfg=Cyan",
         "SignColumn term=standout ctermbg=DarkGrey ctermfg=Cyan guibg=Grey guifg=Cyan"),
 #endif
-    CENT("Visual term=reverse",
-        "Visual term=reverse guibg=DarkGrey"),
+    CENT("Visual ctermbg=235 ctermfg=LightGrey",
+        "Visual ctermbg=235 ctermfg=LightGrey guifg=LightGrey guibg=Grey15"),
 #ifdef FEAT_DIFF
     CENT("DiffAdd term=bold ctermbg=DarkBlue",
         "DiffAdd term=bold ctermbg=DarkBlue guibg=DarkBlue"),
@@ -437,14 +437,8 @@ init_highlight(
     // With 8 colors brown is equal to yellow, need to use black for Search fg
     // to avoid Statement highlighted text disappears.
     // Clear the attributes, needed when changing the t_Co value.
-    if (t_colors > 8)
-       do_highlight((char_u *)(*p_bg == 'l'
-                   ? "Visual cterm=NONE ctermbg=LightGrey"
-                   : "Visual cterm=NONE ctermbg=DarkGrey"), FALSE, TRUE);
-    else
+    if (t_colors <= 8)
     {
-       do_highlight((char_u *)"Visual cterm=reverse ctermbg=NONE",
-                                                                FALSE, TRUE);
        if (*p_bg == 'l')
            do_highlight((char_u *)"Search ctermfg=black", FALSE, TRUE);
     }

It works okay with a dark and light putty terminal, only in 8 color mode, one cannot see the default background color, since it is black as the Normal one. Not sure if 8 color terminals still matter nowadays.

And then I am not good with visualizing, so don't know 🤷

[habamax]

Thx, @chrisbra

I think it would be easier to create a PR and check results there.

We should target all kind of things (no color mode, 8 color mode, etc)

[chrisbra]

vim/vim#13663

[neutaaaaan]

I'm sorry I haven't been able to contribute much lately, work's been pretty hectic.

I think it's fairly obvious from my example screenshots that if I were to fix all versions of the light default colorscheme, I'd basically rebuild them on top of quiet. I do believe trying to preserve syntax highlighting within selections and diffs is not only unnecessary, but counterproductive as far as semantics go given the fairly limited color palette available.

As for 8c terminals, yes, they are relevant. Not only do most people not know anything about terminfo, anytime you use tmux or screen without a config file, they default to the 8c version of their terminfo. This is how most people use multiplexers, and this is not about to change.

[neutaaaaan]

There are 3 intertwined topics here:

• things that are plain broken in all variants and need fixing, ie: visual selections.
• implementation issues in the dark or light variants that are only relevant to their intended use case.
• the default (light) colorscheme breaking down in contexts in which vim cannot figure out the user's background color.

If on paper the best course of action would be to tackle those issues in the order above, I'm afraid there is too much interplay between them to be able to do meaningful work on one item without impacting the others. We'd also introduce the possibility of having to reconsider or further tweak breaking changes after they've been pushed to vim proper, which I believe we'd all rather like to avoid.
For this reason, they all have to be worked on simultaneously.

Similarly, this is work that ought to be out by vim 9.1.0, to avoid having to bother distribution maintainers with implementing specific patches manually.
IF we find a way to work around the issue of vim defaulting to the light colorscheme or implement an agnostic colorscheme based on the current light default one, we will also have to warn distribution maintainers, as I do believe a lot of systemwide .vimrc that default to the dark background only do so to avoid issues such as the broken visual selections.

I should be able to work on a patch this week, so we can at least get the ball rolling.

[neutaaaaan]

I've been putting some work into this branch

This only addresses the issues mentionned in my original post, somewhat less intrusively overall.
The goal of my patch is to be background agnostic. I believe this is the best we can hope for, short of completely rehauling how vim handles its initial colorscheme selection.

The dark colorscheme has similar issues that we're also going to have to fix.