Summary

I demonstrate an imaginary linter with Codex. This predicts an error message for a line of awk code. However, it also infers when the code is correct, so it’s useful.

The accuracy is not really good enough to make a reliable imaginary linter, but should work better with more popular languages such as Python.

In any case, I will build the tooling around prompts to perform linting.

Demo

Correctly identifies syntax error 7 times out of 20

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
6:   }
5:         ^ syntax error
6:     }
5:         ^ syntax error
5:         ^ syntax error
5:         ^
6:     }
5:         ^ syntax error
6:     }
6:   }
6:   }
6:   }
6:     }
6:   }
6:   }
5:         ^ syntax error
6:   }
5:         ^ syntax error
6:   }
5: ^ syntax error

Correctly identifies correct code 19 times out of 20

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
6:   }
6:   }
6:   }
6: }
6:   }
6:   }
6:   }
6:   }
6: }
6:   }
6:   }
6:   }
6:   }
6: }
6:   }
6:     }
6:   }
6:   }
6:   }
5:         ^ syntax error

Prompt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
task: "Imagine an awk linter"
doc: "Given a line of awk code, check it for errors"
issues:
- This doesn't work very well at all.
prompt-version: 1
prompt: |+
    awk -f syntax-test.awk
    awk: syntax-test.awk:1:   x = f(
    awk: syntax-test.awk:1:         ^ unexpected newline or end of string
    awk: syntax-test.awk:2:   for (i=1; i<window_size; i++) {
    awk: syntax-test.awk:3:     s = s memory[i] RS
    awk: syntax-test.awk:4:   }
    awk: syntax-test.awk:5:   <line of code>
    awk: syntax-test.awk:    
engine: "OpenAI Codex"
temperature: 0.0
max-tokens: 60
n-completions: 10
no-uniq-results: on
top-p: 1.0
cache: on
stop-sequences:
- "\n"
vars:
- "line of code"
examples:
- "close cmd, \"to\""
preprocessors:
- "cat"
postprocessor: "grep -v -P \"^6\" | sed \"s/^5: //\""
filter: yes
completion: off
insertion: off